Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.12 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2438796 2438762 2026-07-20T12:22:40Z Stashbot 7414 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-1.tools.eqiad1.wikimedia.cloud 2438796 wikitext text/x-wiki nsuhzsvddwn3cd8zzt285chyfrk07lf 2438798 2438796 2026-07-20T12:24:26Z Stashbot 7414 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-1.tools.eqiad1.wikimedia.cloud 2438798 wikitext text/x-wiki 5li0msjah8lr5eritvliaepoaimk8a5 2438799 2438798 2026-07-20T12:24:30Z Stashbot 7414 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-2.tools.eqiad1.wikimedia.cloud 2438799 wikitext text/x-wiki bxzdxy4zddcdw8iibw8qs35q60roclt 2438800 2438799 2026-07-20T12:26:12Z Stashbot 7414 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-2.tools.eqiad1.wikimedia.cloud 2438800 wikitext text/x-wiki k88ez2dnvg9fflxfy5g91m5g4m47lrk 2438801 2438800 2026-07-20T12:26:49Z Stashbot 7414 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-3.tools.eqiad1.wikimedia.cloud 2438801 wikitext text/x-wiki lyudmk372c34qx6vuofgqe4ivj9ocdm 2438802 2438801 2026-07-20T12:28:36Z Stashbot 7414 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-3.tools.eqiad1.wikimedia.cloud 2438802 wikitext text/x-wiki iy2g4l9o66cmxgj6hk5qclrzvor2cbj 2439382 2438802 2026-07-21T13:40:29Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2439382 wikitext text/x-wiki apaaw5ciu6c4n6pzjvknzj0mc4lda6u 2439467 2439382 2026-07-21T15:20:04Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2439467 wikitext text/x-wiki 1cqg6zu79itm60d2m17c1iyacbih2tg 2439486 2439467 2026-07-21T15:33:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2439486 wikitext text/x-wiki 1ejrhigzmf8x70lev504io27uarmdqg 2439643 2439486 2026-07-22T01:12:35Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config 2439643 wikitext text/x-wiki 6phb4co18pive1zxi6twooo0xnvdmbt 2439644 2439643 2026-07-22T01:21:22Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config 2439644 wikitext text/x-wiki dmfvgeduyyryi6fl77h64s3nbkta9wm 2440282 2439644 2026-07-23T11:47:30Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2440282 wikitext text/x-wiki 2sjilb32e90xn23wwxod2kckesblgo7 0 3788 2439007 2432998 2026-07-20T17:10:33Z BCornwall-WMF 31521 Add syntax template 2439007 wikitext text/x-wiki etw74mhyndf2eies03b1ccx77ihtt8b 2439013 2439007 2026-07-20T17:16:25Z BCornwall-WMF 31521 /* TTL */ Remove historical TTL info, that's already in [[CDN/History]]; put in a See template for the history page. 2439013 wikitext text/x-wiki fi7u1hrip9q4vvxbufarquhuar7g9uu 0 4039 2439744 2430835 2026-07-22T09:17:13Z WBrown (WMF) 37696 /* Security mitigations */ + 2439744 wikitext text/x-wiki 2p67pcm0s67x10489vspfsa9a7uzggv 2439745 2439744 2026-07-22T09:17:38Z WBrown (WMF) 37696 /* Security mitigations */ + 2439745 wikitext text/x-wiki 5i3ptpt2epr5izsn8d4sl5az4myuugd 0 4108 2438874 2438779 2026-07-20T13:53:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1312423]] to Monday, July 20 UTC late backport window 2438874 wikitext text/x-wiki 0ra9c421bi6nheoxvrk3tudfkqxgf1s 2438947 2438874 2026-07-20T15:52:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1312547]] to Monday, July 20 UTC late backport window 2438947 wikitext text/x-wiki ef6kjkmw5enzuhs9u52xm2myw3f526t 2438983 2438947 2026-07-20T16:23:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1312556]] to Monday, July 20 UTC late backport window 2438983 wikitext text/x-wiki joia50ledwlueb8eldq1zwuhpd67szg 2438984 2438983 2026-07-20T16:24:55Z ABreault (WMF) 37749 /* {{Deployment_day|date=2026-07-20}} */ 2438984 wikitext text/x-wiki l9fq8h7iqismrm6y7px71srdukjto96 2439169 2438984 2026-07-20T23:32:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1312614]] to Tuesday, July 21 UTC morning backport window 2439169 wikitext text/x-wiki tnxr6zxhgx9otek7cjokyotjy3clxvi 2439351 2439169 2026-07-21T13:09:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1313092]] to Tuesday, July 21 UTC afternoon backport window 2439351 wikitext text/x-wiki bj22mni13fezube1zjucwyh54ot9d0k 2439352 2439351 2026-07-21T13:09:13Z ScheduleDeploymentBot 37566 Add [[gerrit:1313093]] to Tuesday, July 21 UTC afternoon backport window 2439352 wikitext text/x-wiki idcdaknr66s4f6ah9gk41sj77jghmwy 2439435 2439352 2026-07-21T14:29:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1313200]] to Tuesday, July 21 UTC late backport window 2439435 wikitext text/x-wiki llz2lz76rxscvu0gw1jybzvhraxw2eo 2439436 2439435 2026-07-21T14:30:13Z ScheduleDeploymentBot 37566 Add [[gerrit:1311489]] to Tuesday, July 21 UTC late backport window 2439436 wikitext text/x-wiki m4ryzf8fixdaokmkefylcp56bh389vd 2439582 2439436 2026-07-21T19:56:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1313323]] to Tuesday, July 21 UTC late backport window 2439582 wikitext text/x-wiki g1l0xyxke0z9zurw9i38svnasb3v4g7 2439583 2439582 2026-07-21T19:57:50Z Mutante 20 /* {{Deployment_day|date=2026-07-22}} */ adding window for gerrit restart and new SSH host key 2439583 wikitext text/x-wiki bfgyvzdwnwgvm5kmaea3biwm0kh9yok 2439596 2439583 2026-07-21T20:20:51Z Krinkle 44 /* {{Deployment_day|date=2026-07-21}} */ +me late entry 2439596 wikitext text/x-wiki 2gsz3e2efmn48n0g7wyczvxrzi9h74a 2439597 2439596 2026-07-21T20:22:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1313235]] to Tuesday, July 21 UTC late backport window 2439597 wikitext text/x-wiki mtayw8mv0ew10i4hrqsdrioes3ayqd8 2439599 2439597 2026-07-21T20:22:27Z Krinkle 44 2439599 wikitext text/x-wiki 3tov10sle7gadm25u2msr5idvkf278j 2439650 2439599 2026-07-22T04:42:00Z ScheduleDeploymentBot 37566 Add [[gerrit:1313553]] to Wednesday, July 22 UTC morning backport window 2439650 wikitext text/x-wiki d6mg3af9pew17iubp42l4jivzrmanr1 2439651 2439650 2026-07-22T04:44:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1313554]] to Wednesday, July 22 UTC morning backport window 2439651 wikitext text/x-wiki k1p7hs6aa89d9tu3qi976fq6njv5qhg 2439755 2439651 2026-07-22T09:49:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1313845]] to Wednesday, July 22 UTC afternoon backport window 2439755 wikitext text/x-wiki cjqq6gozfixu0tfl68oscxmzs92s8bb 2439756 2439755 2026-07-22T09:49:53Z ScheduleDeploymentBot 37566 Add [[gerrit:1313842]] to Wednesday, July 22 UTC afternoon backport window 2439756 wikitext text/x-wiki esceyfmu77f2fp8qm2371w5tqyl19vi 2439853 2439756 2026-07-22T12:22:02Z ScheduleDeploymentBot 37566 Add [[gerrit:1313932]] to Wednesday, July 22 UTC afternoon backport window 2439853 wikitext text/x-wiki i3pamlcf8uuru7pcd39gyw9iljx8w5i 2439881 2439853 2026-07-22T12:58:25Z ScheduleDeploymentBot 37566 Add [[gerrit:1313939]] to Wednesday, July 22 UTC afternoon backport window 2439881 wikitext text/x-wiki tl5cwwyzr973id891cr1c63sfq4vx8d 2439898 2439881 2026-07-22T13:20:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1313949]] to Wednesday, July 22 UTC afternoon backport window 2439898 wikitext text/x-wiki fpchcurarnjftpzn4grcyyz18rbzzdc 2440044 2439898 2026-07-22T17:01:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1314046]] to Wednesday, July 22 UTC late backport window 2440044 wikitext text/x-wiki atptc3istbd20ma8wk3eh2oni7k50n7 2440090 2440044 2026-07-22T19:49:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1314106]] to Wednesday, July 22 UTC late backport window 2440090 wikitext text/x-wiki bcqp5homp75y1eixtv523grzoa8gqpm 0 4172 2439506 127129 2026-07-21T16:38:46Z BCornwall-WMF 31521 /* Traffic Flow */ add outdated template 2439506 wikitext text/x-wiki 1np2cl7pkxblvsdxb4a8z5fhruwv0mj 498 6726 2439456 2438753 2026-07-21T15:06:44Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2439456 wikitext text/x-wiki gv135am3ieav9vj16u8rvv77efvzjpt 2439464 2439456 2026-07-21T15:18:14Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2439464 wikitext text/x-wiki 693124ckq24kb66831wneciq4bob5sb 2439641 2439464 2026-07-22T00:54:56Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config 2439641 wikitext text/x-wiki rhtq9t7yjsxy5j6fx8zrg1wet4iw6a3 2439642 2439641 2026-07-22T01:03:00Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config 2439642 wikitext text/x-wiki qy1r7fjpdgu10use80h78d3z88w8xxm 2440274 2439642 2026-07-23T11:35:32Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2440274 wikitext text/x-wiki lmhbntfswcdys4ujeffnp8vnf2j9fkk 2440278 2440274 2026-07-23T11:46:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2440278 wikitext text/x-wiki 0v6iitu51fl69dj1xwwvzdsr72f1vy9 0 7919 2438787 2438777 2026-07-20T12:09:15Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet 2438787 wikitext text/x-wiki p31xkvcmhm5ywd8ou8v3uckaijyitx4 2438788 2438787 2026-07-20T12:10:30Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1075.eqiad.wmnet with reason: host reimage 2438788 wikitext text/x-wiki 5qrycovg4xfgc6anjrkrzka4z5jdqs3 2438789 2438788 2026-07-20T12:12:52Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2070.codfw.wmnet with reason: host reimage 2438789 wikitext text/x-wiki nkdqz9pkmw8usrx6nvwa0z4kfkan7k9 2438791 2438789 2026-07-20T12:14:36Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1075.eqiad.wmnet with reason: host reimage 2438791 wikitext text/x-wiki nj1fkbmcokk5zmsjzodrx4idy8xtof3 2438792 2438791 2026-07-20T12:15:07Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin2001.codfw.wmnet 2438792 wikitext text/x-wiki 5kv91ac6pc5zha1zn32ks3hcw3fzjbu 2438793 2438792 2026-07-20T12:18:33Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2070.codfw.wmnet with reason: host reimage 2438793 wikitext text/x-wiki 7skrabzllhrxbsj2g7x87gr9x4y88pl 2438794 2438793 2026-07-20T12:19:05Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet 2438794 wikitext text/x-wiki tkjerbibb6yvwjehefin6hxtqa5ustc 2438797 2438794 2026-07-20T12:23:00Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin1001.eqiad.wmnet 2438797 wikitext text/x-wiki 5ywm5a79luoerimxdcyyesiob2b6l4w 2438803 2438797 2026-07-20T12:31:56Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 10 hosts 2438803 wikitext text/x-wiki 5yt4lhng24pt4sfjvf9rmc4b9c94syt 2438804 2438803 2026-07-20T12:32:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 10 hosts 2438804 wikitext text/x-wiki 9sjfexjjxpm6fzov6932ilerh9ec6xf 2438805 2438804 2026-07-20T12:35:39Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1075.eqiad.wmnet with OS trixie 2438805 wikitext text/x-wiki 25v8j348now7oy52ffn7zvam804r1y0 2438806 2438805 2026-07-20T12:36:48Z Stashbot 7414 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2438806 wikitext text/x-wiki kuqkvda55lfg6fd0sakrlyn99la2cfi 2438807 2438806 2026-07-20T12:40:58Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2070.codfw.wmnet with OS trixie 2438807 wikitext text/x-wiki 7fkwgvjstmb976ygcstdw1ikzbktpq9 2438808 2438807 2026-07-20T12:42:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 3 hosts 2438808 wikitext text/x-wiki p3n4uj4nkq1a7ebz31scpw3zo959snf 2438809 2438808 2026-07-20T12:42:33Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 3 hosts 2438809 wikitext text/x-wiki hzpash55uhtfs43ry68omj26h5p8qcl 2438810 2438809 2026-07-20T12:42:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 7 hosts 2438810 wikitext text/x-wiki 2nilirarrdgcqhfnl9k9z54biimvmwf 2438812 2438810 2026-07-20T12:43:14Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 7 hosts 2438812 wikitext text/x-wiki rnzuwa1r3e5qwqlrdctodogwwdxg25m 2438813 2438812 2026-07-20T12:45:25Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2438813 wikitext text/x-wiki 6e2cqccyn3v7vbtf3x65tkyltj701bw 2438814 2438813 2026-07-20T12:45:31Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2438814 wikitext text/x-wiki k0q2vfcc8bsvxg5z1pwol2281sxdsej 2438815 2438814 2026-07-20T12:45:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2438815 wikitext text/x-wiki scu11t4jakk9f9ody9dj8a0h8mi4d7g 2438816 2438815 2026-07-20T12:45:52Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2438816 wikitext text/x-wiki f1dnxgaeuey7jbvm5vr4jjcoztel16j 2438817 2438816 2026-07-20T12:46:03Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2438817 wikitext text/x-wiki s40otcmy31m6qcszajxas52k5am72v7 2438818 2438817 2026-07-20T12:46:13Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2438818 wikitext text/x-wiki l1szhuy4q3pb3wez0zjmjygb2q1s948 2438820 2438818 2026-07-20T12:52:59Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2438820 wikitext text/x-wiki 18x1kb2g7us7ri3c0fb336eodpfm6dn 2438821 2438820 2026-07-20T12:53:03Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2438821 wikitext text/x-wiki 95ozagbaey84pd2drohgn5udgq57dj8 2438822 2438821 2026-07-20T12:54:30Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2438822 wikitext text/x-wiki sd1v11t9xkunvyukklp5qbs926puoim 2438823 2438822 2026-07-20T12:54:34Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2438823 wikitext text/x-wiki o5me2d367jfj7ppit22a0qm4b46trl2 2438824 2438823 2026-07-20T12:55:07Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2438824 wikitext text/x-wiki o45et65nhfeq905vxr06e8ii6h9p7gn 2438825 2438824 2026-07-20T12:55:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2438825 wikitext text/x-wiki mstonv062k1awuzk4ph96jct9opqymj 2438826 2438825 2026-07-20T13:03:02Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2071.codfw.wmnet with OS trixie 2438826 wikitext text/x-wiki 0q25ryto2eez26lknfjjzsqml44ttgv 2438827 2438826 2026-07-20T13:03:11Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1076.eqiad.wmnet with OS trixie 2438827 wikitext text/x-wiki nhn294qphpvkeapstz06fu2995jkiwu 2438829 2438827 2026-07-20T13:12:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 10 hosts 2438829 wikitext text/x-wiki bkm9mkvo0pxechuf47s8fy9a25xvuds 2438830 2438829 2026-07-20T13:13:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 10 hosts 2438830 wikitext text/x-wiki sgey8evpnwssmymt0xehvtdi5byrtax 2438831 2438830 2026-07-20T13:13:45Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 3 hosts 2438831 wikitext text/x-wiki h8q314y2oqfy6bf60nihkzcu0hj8qee 2438832 2438831 2026-07-20T13:14:05Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 3 hosts 2438832 wikitext text/x-wiki 6vexiwcp91sjazkcmkdpkdaseeeu6cp 2438834 2438832 2026-07-20T13:17:20Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet 2438834 wikitext text/x-wiki ahlwolk1owqds05gg8zhd05kwrciqql 2438835 2438834 2026-07-20T13:21:17Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet 2438835 wikitext text/x-wiki m2dibpk3nlatvxogympte06mt63uu17 2438836 2438835 2026-07-20T13:21:29Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet 2438836 wikitext text/x-wiki s15gzs57sj4qz5d1nnhb6zxr4oluzjb 2438837 2438836 2026-07-20T13:22:27Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1076.eqiad.wmnet with reason: host reimage 2438837 wikitext text/x-wiki k6fyy2q4vvolngurnhk1i4qhdumqaym 2438838 2438837 2026-07-20T13:23:45Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2071.codfw.wmnet with reason: host reimage 2438838 wikitext text/x-wiki odb1r1cvqwn5fjtw2i0sp8ixxguvyye 2438839 2438838 2026-07-20T13:25:27Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet 2438839 wikitext text/x-wiki 9ingzfi1y4ptmh5hjjaxdpw22pnuirr 2438840 2438839 2026-07-20T13:26:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet 2438840 wikitext text/x-wiki n7fsmoru9hifdlj8x9z5vq99hicwjuw 2438841 2438840 2026-07-20T13:28:51Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw 2438841 wikitext text/x-wiki m2m6hqpy7uq35wyas80cs9dn25bsd41 2438842 2438841 2026-07-20T13:29:01Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw 2438842 wikitext text/x-wiki ht52j0baizpuggpkfpad2hkludngjfc 2438843 2438842 2026-07-20T13:29:17Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1076.eqiad.wmnet with reason: host reimage 2438843 wikitext text/x-wiki a3kjyw88vzsgac94yngbsrp3ekzoyv3 2438844 2438843 2026-07-20T13:30:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2027.codfw.wmnet with OS bookworm 2438844 wikitext text/x-wiki ecjpjwinjf7esxzrgeo82nucprqlzhp 2438845 2438844 2026-07-20T13:30:32Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2027 2438845 wikitext text/x-wiki joitvbtjyshybsmzkaum2rh108ri4td 2438846 2438845 2026-07-20T13:30:33Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2027 2438846 wikitext text/x-wiki gcb7axza4l1g6ummm3etdqe69biuxa3 2438847 2438846 2026-07-20T13:30:45Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2019.codfw.wmnet with OS bookworm 2438847 wikitext text/x-wiki avj79pnz657d3vg61p38nl6o4uxeys4 2438848 2438847 2026-07-20T13:31:02Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet 2438848 wikitext text/x-wiki f4eh08ubrndbqxhc0ahr9srfakyujqc 2438849 2438848 2026-07-20T13:31:16Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet 2438849 wikitext text/x-wiki d5n51q8nokrhmxen8xfa8rwpjuwc96r 2438850 2438849 2026-07-20T13:31:19Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2019 2438850 wikitext text/x-wiki 4cr2r7j5d0ugfvy3gktu0m7orvwz81e 2438851 2438850 2026-07-20T13:31:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2438851 wikitext text/x-wiki 6243dh0akin6fb6a8v4nsqy7coukae7 2438852 2438851 2026-07-20T13:32:08Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2071.codfw.wmnet with reason: host reimage 2438852 wikitext text/x-wiki jds7gds80s0i64ymzen2q3zd72xhfdy 2438854 2438852 2026-07-20T13:35:46Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet 2438854 wikitext text/x-wiki 0vt505e1j48d844d140fb4sxnle663z 2438855 2438854 2026-07-20T13:36:24Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2019 - bking@cumin2003" 2438855 wikitext text/x-wiki jshugzrjd07h3wms7njw96t2oypn6lz 2438856 2438855 2026-07-20T13:36:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2019 - bking@cumin2003" 2438856 wikitext text/x-wiki elzujizi48amikr8w7spp0y3vrb6sy6 2438857 2438856 2026-07-20T13:36:30Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438857 wikitext text/x-wiki e0lz2cxyqibwic9h9nnxz9rphazurze 2438858 2438857 2026-07-20T13:36:32Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2019.codfw.wmnet 156.32.192.10.in-addr.arpa 6.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438858 wikitext text/x-wiki 8itaoaj4m0r6a7cricswjjrznn3nqvw 2438859 2438858 2026-07-20T13:36:34Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2019.codfw.wmnet 156.32.192.10.in-addr.arpa 6.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438859 wikitext text/x-wiki a7ixye03buc71khlbbnvdnew7yxgsuw 2438860 2438859 2026-07-20T13:36:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2019 2438860 wikitext text/x-wiki 1ij3vgz4uf31d2m7llq6of534fhz84r 2438861 2438860 2026-07-20T13:37:41Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet 2438861 wikitext text/x-wiki q7pay13x0zhqv0evufrgteghwlmfz3b 2438862 2438861 2026-07-20T13:37:56Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet 2438862 wikitext text/x-wiki cju7b7x52336sat0ohaht7fuit6cd5m 2438863 2438862 2026-07-20T13:38:26Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host lvs1013.eqiad.wmnet 2438863 wikitext text/x-wiki pgmicfdhimkthpoyhcducjgw4mq0kyl 2438864 2438863 2026-07-20T13:39:34Z Stashbot 7414 kevinbazira@deploy2003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2438864 wikitext text/x-wiki mps86ielpplrhjfyrsrouoo7n84738c 2438865 2438864 2026-07-20T13:44:03Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1013.eqiad.wmnet 2438865 wikitext text/x-wiki q863az25quyfn6mp0ld3dp35h29qtrk 2438866 2438865 2026-07-20T13:45:29Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host lvs1014.eqiad.wmnet 2438866 wikitext text/x-wiki 9d82vq9ze50va820izqwwgodkn5yo4m 2438867 2438866 2026-07-20T13:50:15Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2027.codfw.wmnet with reason: host reimage 2438867 wikitext text/x-wiki h2ba3zn2x98ql9yxfsqhrlzw558f9r5 2438868 2438867 2026-07-20T13:50:27Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1076.eqiad.wmnet with OS trixie 2438868 wikitext text/x-wiki o3s3ce2lmhmbwlhcgat5wq6o7t3lbnc 2438869 2438868 2026-07-20T13:50:55Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1014.eqiad.wmnet 2438869 wikitext text/x-wiki q4iswvgdrxdzbwc00yl0v4rrwibcbla 2438870 2438869 2026-07-20T13:51:02Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host lvs1015.eqiad.wmnet 2438870 wikitext text/x-wiki h6t4v79gq89g7v8rqmymf5h06sfqows 2438871 2438870 2026-07-20T13:51:26Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting A:liberica and P{lvs7003*} and A:liberica 2438871 wikitext text/x-wiki dfmislfryfzziqd32y7vgqpd392ze34 2438872 2438871 2026-07-20T13:51:30Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.loadbalancer.admin (exit_code=97) rebooting A:liberica and P{lvs7003*} and A:liberica 2438872 wikitext text/x-wiki j8juy3rxvgc3sg2d0vt2ppfikiezz7u 2438873 2438872 2026-07-20T13:51:34Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting A:liberica and not P{lvs7003*} and A:liberica 2438873 wikitext text/x-wiki 1f9a8jf8jvd0wchwici4acfiyvne1md 2438875 2438873 2026-07-20T13:54:22Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2071.codfw.wmnet with OS trixie 2438875 wikitext text/x-wiki eau4zckbkkh1d1logbt1xnubfpckkpl 2438876 2438875 2026-07-20T13:54:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2027.codfw.wmnet with reason: host reimage 2438876 wikitext text/x-wiki bgrrq7xgwf1yxvpj7t3jtoqkxpaxzrj 2438877 2438876 2026-07-20T13:56:39Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1015.eqiad.wmnet 2438877 wikitext text/x-wiki 343u8wgjvtcd5bkzpylf8m89a6uwlas 2438878 2438877 2026-07-20T14:00:22Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2019 2438878 wikitext text/x-wiki aujow7vc86ya4qw4sselumk7nuyx3bc 2438879 2438878 2026-07-20T14:00:25Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2019 2438879 wikitext text/x-wiki qouhlzgfpbs1s4d31a22pc1royu9le3 2438880 2438879 2026-07-20T14:05:49Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2438880 wikitext text/x-wiki 7rxyw13qawfz0omkhm3ioi8f1v6jdfx 2438881 2438880 2026-07-20T14:06:22Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply security updates - bking@cumin2003 - T431826 2438881 wikitext text/x-wiki bu7swt5s74qrjgp3m6zf9rfbeqf95k0 2438882 2438881 2026-07-20T14:06:31Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2438882 wikitext text/x-wiki kwup1gm3sbbxdn4h23ccjl2n1lqxmio 2438883 2438882 2026-07-20T14:06:33Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2438883 wikitext text/x-wiki myud0ou1njktk83pvkrudhz3mih4e0z 2438884 2438883 2026-07-20T14:07:13Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2438884 wikitext text/x-wiki 2do98gwmfe64unqt9kjwbulwbwj2417 2438885 2438884 2026-07-20T14:08:27Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2438885 wikitext text/x-wiki 7nvu33nw4lb8evfki1s0jc0pyfdx7gt 2438886 2438885 2026-07-20T14:08:31Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2438886 wikitext text/x-wiki kgosp0oqokwwqasxuu1kmka0ncml6jp 2438887 2438886 2026-07-20T14:08:32Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2438887 wikitext text/x-wiki kcrzoid64507ym0bwitohdi0q05nboc 2438888 2438887 2026-07-20T14:08:35Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2438888 wikitext text/x-wiki 68u9bb09v25s3kc6nj8l2ik0eht9hjc 2438890 2438888 2026-07-20T14:16:37Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet 2438890 wikitext text/x-wiki cahqfktnlktkvvx4zh7hwz86sxhn5lu 2438891 2438890 2026-07-20T14:16:39Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet 2438891 wikitext text/x-wiki stn37pnxe6fwad4muwpe84bdy2eob8z 2438893 2438891 2026-07-20T14:19:48Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2027.codfw.wmnet with OS bookworm 2438893 wikitext text/x-wiki gf6qv93xoesd451fecey62jp5i9yl91 2438894 2438893 2026-07-20T14:19:52Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2019.codfw.wmnet with reason: host reimage 2438894 wikitext text/x-wiki cbkeurzgha4gbaq5fh2bhvhagzfxhba 2438895 2438894 2026-07-20T14:22:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2017.codfw.wmnet -> wdqs2027.codfw.wmnet, repooling source-only afterwards 2438895 wikitext text/x-wiki 9mdb6fv00wjmr2wwszmn6z68xqce69n 2438896 2438895 2026-07-20T14:23:50Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2019.codfw.wmnet with reason: host reimage 2438896 wikitext text/x-wiki qiakkq2k9f24og5n8b3vjhno1ay7ifn 2438897 2438896 2026-07-20T14:24:07Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2438897 wikitext text/x-wiki m3217bc0do714r0q97mqbczy8h9f85v 2438898 2438897 2026-07-20T14:24:11Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2438898 wikitext text/x-wiki 8vmveynsp0by5towyoi44lasp99u1xq 2438899 2438898 2026-07-20T14:24:12Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2438899 wikitext text/x-wiki 12h1hr6ftp4gk0y82y0rx5n2l7id8ae 2438900 2438899 2026-07-20T14:24:14Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2438900 wikitext text/x-wiki 63m7zes17iek04mag8rod6o7u71gkza 2438901 2438900 2026-07-20T14:31:47Z Stashbot 7414 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1312471|postEdit experiment: enroll control users by same criteria]], [[gerrit:1312423|Modify user groups rights in English Wikiquote (T432557)]] 2438901 wikitext text/x-wiki 3cuts8p2ebp4qqjyw2l4vdttqbdyrhe 2438902 2438901 2026-07-20T14:33:45Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: no reason specified, no task ID specified] 2438902 wikitext text/x-wiki ehqxulya9jj7yzhvobfz1i24kccwtbz 2438903 2438902 2026-07-20T14:33:47Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: no reason specified, no task ID specified] 2438903 wikitext text/x-wiki c03blrawh4qym84bo2c0anylq26h7jt 2438904 2438903 2026-07-20T14:39:08Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica 2438904 wikitext text/x-wiki h2wz3gjycpy6zgijpqy648pesasp7dd 2438905 2438904 2026-07-20T14:39:22Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica 2438905 wikitext text/x-wiki t2g74w6277flxda4cwf8ku5hvd3mdqy 2438906 2438905 2026-07-20T14:41:03Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica 2438906 wikitext text/x-wiki f53qd8mdqq4l69nfqfjcda131usx4ef 2438907 2438906 2026-07-20T14:41:16Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica 2438907 wikitext text/x-wiki e4ql0exbigl086me6x10b4bxxc563ls 2438908 2438907 2026-07-20T14:41:39Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica 2438908 wikitext text/x-wiki 39ccg29g707epmnr8x9k62encs6h6cg 2438909 2438908 2026-07-20T14:41:41Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica 2438909 wikitext text/x-wiki dydlq7xt9ihox6v6vj2mpl2v7dtjnyk 2438910 2438909 2026-07-20T14:44:27Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: BGP issues in lvs7003 resolved after liberica restart, no task ID specified] 2438910 wikitext text/x-wiki px1pieb7l4x2nnldfrk58djxzdnfv4j 2438911 2438910 2026-07-20T14:44:36Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: BGP issues in lvs7003 resolved after liberica restart, no task ID specified] 2438911 wikitext text/x-wiki etpzh7u9mbkdyxsudz7f9wloe6nhmj3 2438913 2438911 2026-07-20T14:47:22Z Stashbot 7414 urbanecm@deploy2003: vadymts1, migr, urbanecm: Backport for [[gerrit:1312471|postEdit experiment: enroll control users by same criteria]], [[gerrit:1312423|Modify user groups rights in English Wikiquote (T432557)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438913 wikitext text/x-wiki bda8tgmhunsa2em5e3ktaln6q1jq5ep 2438914 2438913 2026-07-20T14:49:58Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica and not P{lvs7003*} and A:liberica 2438914 wikitext text/x-wiki 4l8zkpq1dpyq8xl49eld4mjvx6saj62 2438916 2438914 2026-07-20T14:51:10Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2019.codfw.wmnet with OS bookworm 2438916 wikitext text/x-wiki slqy7i7jkawgn9549032nidtj6s1azb 2438917 2438916 2026-07-20T14:55:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet 2438917 wikitext text/x-wiki 53roj83vuvccbvwrwl75sysz48ymhk8 2438918 2438917 2026-07-20T14:57:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet 2438918 wikitext text/x-wiki e4xe0uxey9rce45k6carbqwbwgebqev 2438919 2438918 2026-07-20T14:57:15Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling source-only afterwards 2438919 wikitext text/x-wiki lxfq1vva087td4s5cyrcc9hlhmuk5mq 2438920 2438919 2026-07-20T14:58:14Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2438920 wikitext text/x-wiki bg4qgo8suk3tvhecpp4jqg2arczpot6 2438921 2438920 2026-07-20T14:58:21Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 13s) 2438921 wikitext text/x-wiki ovcb77cn492yk166uzqo97gc6q2v1h6 2438922 2438921 2026-07-20T14:58:32Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2438922 wikitext text/x-wiki b4xk0ghe1eou3qfsh45n4qq9ad2z9v9 2438923 2438922 2026-07-20T14:58:38Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 07s) 2438923 wikitext text/x-wiki th8ub8zxt8unpkl5s0tyhlh1h2cy6r6 2438924 2438923 2026-07-20T14:59:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply security updates - bking@cumin2003 - T431826 2438924 wikitext text/x-wiki pul268xzz6g6l9fwudl5st65p7zvl2s 2438925 2438924 2026-07-20T15:00:31Z Stashbot 7414 urbanecm@deploy2003: vadymts1, migr, urbanecm: Continuing with deployment 2438925 wikitext text/x-wiki sjawq4dwpy0nvvzojbwwvl2ezxwqc79 2438927 2438925 2026-07-20T15:06:00Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2017.codfw.wmnet -> wdqs2027.codfw.wmnet, repooling source-only afterwards 2438927 wikitext text/x-wiki jg5fwfbxe86r7vr5c2v5ambwrpff50i 2438928 2438927 2026-07-20T15:07:23Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance 2438928 wikitext text/x-wiki 4zhndo8k7n1n4wu1ss3qr6lpsxldmy2 2438929 2438928 2026-07-20T15:07:32Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2162 (T431660)', diff saved to https://phabricator.wikimedia.org/P94902 and previous config saved to /var/cache/conftool/dbconfig/20260720-150729-cwilliams.json 2438929 wikitext text/x-wiki kvttkdwy6izmvjtbrygxwbere6k9v9h 2438930 2438929 2026-07-20T15:08:31Z Stashbot 7414 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2438930 wikitext text/x-wiki c793r8bn1wkc8mkf4lbxml4a1dghkvn 2438932 2438930 2026-07-20T15:13:03Z Stashbot 7414 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1312471|postEdit experiment: enroll control users by same criteria]], [[gerrit:1312423|Modify user groups rights in English Wikiquote (T432557)]] (duration: 41m 16s) 2438932 wikitext text/x-wiki 809k6g3vg3glhl7782wlqyi4rf8gd5l 2438933 2438932 2026-07-20T15:14:11Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T431660)', diff saved to https://phabricator.wikimedia.org/P94905 and previous config saved to /var/cache/conftool/dbconfig/20260720-151407-cwilliams.json 2438933 wikitext text/x-wiki g04ahmuq41csra4o91n6qebryuo6dwi 2438934 2438933 2026-07-20T15:14:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1023.eqiad.wmnet with OS bookworm 2438934 wikitext text/x-wiki cum8cojf24pej95n9lsx8eqbn6ezk5i 2438935 2438934 2026-07-20T15:14:49Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1023 2438935 wikitext text/x-wiki nm8lzknywvese66uehoph0n012iak9t 2438936 2438935 2026-07-20T15:14:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1023 2438936 wikitext text/x-wiki kxojzvwjkwv9t4mslna46wnx1cxs68y 2438937 2438936 2026-07-20T15:24:19Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P94906 and previous config saved to /var/cache/conftool/dbconfig/20260720-152418-cwilliams.json 2438937 wikitext text/x-wiki lom9wu15e8y69wsx88som03sfp2flfi 2438939 2438937 2026-07-20T15:28:05Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2438939 wikitext text/x-wiki phs0z6jsj0uje363bunwq23eai27kcb 2438941 2438939 2026-07-20T15:33:52Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet 2438941 wikitext text/x-wiki 7hq5oc1h75cjxumyzk8pzgv3jld5z2d 2438942 2438941 2026-07-20T15:34:29Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P94907 and previous config saved to /var/cache/conftool/dbconfig/20260720-153425-cwilliams.json 2438942 wikitext text/x-wiki e66wyt4g4nxt21572qtged6itxrtdql 2438943 2438942 2026-07-20T15:35:21Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet 2438943 wikitext text/x-wiki 90654t04l2rzbfpk5z2v5ez95mctd4t 2438946 2438943 2026-07-20T15:44:35Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T431660)', diff saved to https://phabricator.wikimedia.org/P94908 and previous config saved to /var/cache/conftool/dbconfig/20260720-154433-cwilliams.json 2438946 wikitext text/x-wiki rf0xfj0rs79evm6cyvp5ng5rygcpmiz 2438948 2438946 2026-07-20T15:53:47Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2242.codfw.wmnet with reason: Maintenance 2438948 wikitext text/x-wiki mz3rjgfc5lw5a5769sri86wiojv35rf 2438949 2438948 2026-07-20T15:53:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2242 (T431660)', diff saved to https://phabricator.wikimedia.org/P94909 and previous config saved to /var/cache/conftool/dbconfig/20260720-155353-cwilliams.json 2438949 wikitext text/x-wiki dskho0lk7ncfo5b4l18uv8ka17u554x 2438950 2438949 2026-07-20T15:58:32Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling source-only afterwards 2438950 wikitext text/x-wiki p8svqz0tuztrus8ennuf7ljaezdck7w 2438951 2438950 2026-07-20T16:00:49Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2242 (T431660)', diff saved to https://phabricator.wikimedia.org/P94911 and previous config saved to /var/cache/conftool/dbconfig/20260720-160047-cwilliams.json 2438951 wikitext text/x-wiki 1hdoiyhyeht4zws2kk69opegnlm0cie 2438956 2438951 2026-07-20T16:05:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2020.codfw.wmnet with OS bookworm 2438956 wikitext text/x-wiki r10empdrsd2npfxdkwsm7v3q70ze9lx 2438957 2438956 2026-07-20T16:05:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2020 2438957 wikitext text/x-wiki ihts5xvt2apsxbedr1pl691mru0d33y 2438958 2438957 2026-07-20T16:06:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2438958 wikitext text/x-wiki 0589pcrcoil2xb80frypc53e6itfm8q 2438959 2438958 2026-07-20T16:06:12Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2438959 wikitext text/x-wiki imd2k1skslvtd9witypoe7qptsehyto 2438960 2438959 2026-07-20T16:06:42Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2438960 wikitext text/x-wiki 7khutgzm4ctb244y3rbd8ei7eu9agqp 2438961 2438960 2026-07-20T16:08:33Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2438961 wikitext text/x-wiki lp0p1hcazqslz87un5nx45n0c2liw36 2438962 2438961 2026-07-20T16:09:50Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2438962 wikitext text/x-wiki 63tjq8rh6xvovbekazrfmwd59qd94mv 2438963 2438962 2026-07-20T16:10:57Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2242', diff saved to https://phabricator.wikimedia.org/P94912 and previous config saved to /var/cache/conftool/dbconfig/20260720-161055-cwilliams.json 2438963 wikitext text/x-wiki tgis6f5map3eysh0igt9ed6q812kzb6 2438964 2438963 2026-07-20T16:11:52Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2438964 wikitext text/x-wiki esfnjet1qdpcvh6f75ohz45b9m2egpf 2438965 2438964 2026-07-20T16:12:58Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet 2438965 wikitext text/x-wiki 3nsfpa7eetwh2600znxmbouh2l2228g 2438966 2438965 2026-07-20T16:14:04Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet 2438966 wikitext text/x-wiki fa3oujuw7rjtnairm4k64b3d8ecs742 2438968 2438966 2026-07-20T16:17:32Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating for netbox accounting errors - jhancock@cumin2002" 2438968 wikitext text/x-wiki 9alece3pnoy1lf1uj7xihnxe75byvkf 2438969 2438968 2026-07-20T16:17:37Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating for netbox accounting errors - jhancock@cumin2002" 2438969 wikitext text/x-wiki r9qfwyrsrxi426lggqo7k51umpmd3fe 2438970 2438969 2026-07-20T16:17:39Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438970 wikitext text/x-wiki idadjal6ckhx1ad3y3pt7y45wt000d6 2438971 2438970 2026-07-20T16:18:23Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2438971 wikitext text/x-wiki lfxjd0h8swwzg6hkssxo0b4fj6re6tk 2438972 2438971 2026-07-20T16:18:30Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2438972 wikitext text/x-wiki bjsb52q7fam3jhfk580ai63x94o0r66 2438973 2438972 2026-07-20T16:18:52Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2438973 wikitext text/x-wiki i8i8lfh5qep6v0wiv57nxyu9m5b9vu0 2438974 2438973 2026-07-20T16:19:17Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2438974 wikitext text/x-wiki g5aggqp5l16crpmbjvn7kfh2v2t0g3q 2438976 2438974 2026-07-20T16:21:04Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2242', diff saved to https://phabricator.wikimedia.org/P94913 and previous config saved to /var/cache/conftool/dbconfig/20260720-162103-cwilliams.json 2438976 wikitext text/x-wiki sl93kmqkkm1v6xn8f6u97nh62l88vjq 2438977 2438976 2026-07-20T16:21:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438977 wikitext text/x-wiki j807queupth7fnf8jz6etchhtpa716h 2438978 2438977 2026-07-20T16:21:36Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2020.codfw.wmnet 85.0.192.10.in-addr.arpa 5.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438978 wikitext text/x-wiki ajtils3wynxz3382jbvuoes0tv2gkr0 2438979 2438978 2026-07-20T16:21:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2020.codfw.wmnet 85.0.192.10.in-addr.arpa 5.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438979 wikitext text/x-wiki cdd54k9idqi1q88cfesan1v6q708yy8 2438980 2438979 2026-07-20T16:21:40Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2020 2438980 wikitext text/x-wiki josx92cfy78949sn9worvokjbtuha23 2438981 2438980 2026-07-20T16:23:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2020 2438981 wikitext text/x-wiki dfqdt0xn2kjd1qc5yml6wktsepiu6j3 2438982 2438981 2026-07-20T16:23:09Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2020 2438982 wikitext text/x-wiki 82pcjjx76jjhby0j3d11oahvqajs9ty 2438985 2438982 2026-07-20T16:27:04Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2438985 wikitext text/x-wiki b5gr7zv03x5kcntkdxx71z6yibqgfy9 2438986 2438985 2026-07-20T16:27:19Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2438986 wikitext text/x-wiki lgznafguy7gqsvf2ho01dy3p33b5fjs 2438987 2438986 2026-07-20T16:31:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2242 (T431660)', diff saved to https://phabricator.wikimedia.org/P94914 and previous config saved to /var/cache/conftool/dbconfig/20260720-163111-cwilliams.json 2438987 wikitext text/x-wiki l043t4vy5s8vzcls3oolr9qig103kc0 2438988 2438987 2026-07-20T16:31:33Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2243.codfw.wmnet with reason: Maintenance 2438988 wikitext text/x-wiki 60by8z2zit1lfva3o4jg5hogqpaha3t 2438989 2438988 2026-07-20T16:31:41Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2243 (T431660)', diff saved to https://phabricator.wikimedia.org/P94915 and previous config saved to /var/cache/conftool/dbconfig/20260720-163140-cwilliams.json 2438989 wikitext text/x-wiki nr99yu91a27yxn8zy9w62qh3mgklod2 2438990 2438989 2026-07-20T16:35:26Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1023.eqiad.wmnet with OS bookworm 2438990 wikitext text/x-wiki c9sla3wgv1oc7lxod9p47j6crzs5eqi 2438991 2438990 2026-07-20T16:39:20Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2243 (T431660)', diff saved to https://phabricator.wikimedia.org/P94916 and previous config saved to /var/cache/conftool/dbconfig/20260720-163918-cwilliams.json 2438991 wikitext text/x-wiki is5hlnalpey0andyvweum41d845596h 2438992 2438991 2026-07-20T16:49:27Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2243', diff saved to https://phabricator.wikimedia.org/P94917 and previous config saved to /var/cache/conftool/dbconfig/20260720-164926-cwilliams.json 2438992 wikitext text/x-wiki hpvevrja6tjftsvf4nk932ea7vfuzd9 2438993 2438992 2026-07-20T16:51:40Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet 2438993 wikitext text/x-wiki 5z9mdtwe1tqtqwz4xaqcz2v5izqxd8j 2438994 2438993 2026-07-20T16:52:41Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet 2438994 wikitext text/x-wiki b9l0pzngsyma5ylfp3lqzq0fiatc3xw 2438995 2438994 2026-07-20T16:58:11Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2438995 wikitext text/x-wiki cvjtwpfor84mj3x63n9e9n18f5ry5k4 2438996 2438995 2026-07-20T16:58:16Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2438996 wikitext text/x-wiki 5eqkdt72dhs2qq4e59gmi0z5sz2mv8q 2438997 2438996 2026-07-20T16:59:35Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2243', diff saved to https://phabricator.wikimedia.org/P94918 and previous config saved to /var/cache/conftool/dbconfig/20260720-165933-cwilliams.json 2438997 wikitext text/x-wiki mxasfwk0bqwgxyyca2y3v49rkajbvw1 2439004 2438997 2026-07-20T17:09:43Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2243 (T431660)', diff saved to https://phabricator.wikimedia.org/P94919 and previous config saved to /var/cache/conftool/dbconfig/20260720-170941-cwilliams.json 2439004 wikitext text/x-wiki 3nlwf49gsdio92mcbpzf5xemxb9p4fc 2439005 2439004 2026-07-20T17:10:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2244.codfw.wmnet with reason: Maintenance 2439005 wikitext text/x-wiki ic4e7hxiu23rbif0og0a9c1nr8lrfmu 2439006 2439005 2026-07-20T17:10:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2244 (T431660)', diff saved to https://phabricator.wikimedia.org/P94920 and previous config saved to /var/cache/conftool/dbconfig/20260720-171012-cwilliams.json 2439006 wikitext text/x-wiki 2ya5gjnnahvkpr0cxqywqemb7aea4hd 2439008 2439006 2026-07-20T17:11:09Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 13m 03s) 2439008 wikitext text/x-wiki 6cpla60zyvj61fecx2l936qvf81bg9h 2439009 2439008 2026-07-20T17:11:22Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2439009 wikitext text/x-wiki 3bh5hoe3jy5woinxgvjkhpohn8d2m82 2439014 2439009 2026-07-20T17:16:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2244 (T431660)', diff saved to https://phabricator.wikimedia.org/P94921 and previous config saved to /var/cache/conftool/dbconfig/20260720-171653-cwilliams.json 2439014 wikitext text/x-wiki i8g5mzh86l08lqsrya11c9w5yc7f5n0 2439020 2439014 2026-07-20T17:27:03Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2244', diff saved to https://phabricator.wikimedia.org/P94922 and previous config saved to /var/cache/conftool/dbconfig/20260720-172701-cwilliams.json 2439020 wikitext text/x-wiki 14ftjjj88tkgnm10xqrpmkym8bc0cns 2439021 2439020 2026-07-20T17:27:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1011.eqiad.wmnet with OS bookworm 2439021 wikitext text/x-wiki a3hq0smkb9x6f24v2cpx0o8z0pn2dbv 2439022 2439021 2026-07-20T17:27:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1011 2439022 wikitext text/x-wiki rqizjbo9vbylhzxpr3nc4ph29idcd7v 2439023 2439022 2026-07-20T17:27:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1011 2439023 wikitext text/x-wiki 4cnlrhy9o8c8w8upe8gosk4ve98jk79 2439024 2439023 2026-07-20T17:30:23Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet 2439024 wikitext text/x-wiki fpqm636vrnfq0jtr5jaxvi9gb0ktyl1 2439025 2439024 2026-07-20T17:31:11Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet 2439025 wikitext text/x-wiki 6sy6r1a29s2ed2k7ojm7v72inj9r61a 2439026 2439025 2026-07-20T17:31:53Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 20m 40s) 2439026 wikitext text/x-wiki nlt20jydyxtg5yiu42nrdcsu7mislza 2439029 2439026 2026-07-20T17:35:29Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2439029 wikitext text/x-wiki 9igbhpfpubs7frhlaiej4ofhxetm9ya 2439030 2439029 2026-07-20T17:37:11Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2244', diff saved to https://phabricator.wikimedia.org/P94923 and previous config saved to /var/cache/conftool/dbconfig/20260720-173709-cwilliams.json 2439030 wikitext text/x-wiki 2becax44frxiqjzhd51iq33ooxcxwcc 2439031 2439030 2026-07-20T17:37:52Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2439031 wikitext text/x-wiki 0yc9a0rla4a89a0xmfpqpybihq8syl7 2439032 2439031 2026-07-20T17:38:29Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2439032 wikitext text/x-wiki m6j3vxrnbi7wj3rxo3t9ocz33t9s124 2439033 2439032 2026-07-20T17:38:40Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439033 wikitext text/x-wiki ky46786q0v1nh7hzkcoi68luw3hjags 2439034 2439033 2026-07-20T17:39:33Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2439034 wikitext text/x-wiki 19w7c04fnyr7955w168gbxgk9lauco4 2439035 2439034 2026-07-20T17:39:36Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet 2439035 wikitext text/x-wiki euk9u4u599u2n35wu90zlz6vjre7l76 2439036 2439035 2026-07-20T17:43:19Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet 2439036 wikitext text/x-wiki ngcu0peduyo8s2lk3n6epi4inf6447e 2439037 2439036 2026-07-20T17:43:34Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet 2439037 wikitext text/x-wiki klsmmznwaphmrbq8c3xol6gqt42c3ho 2439038 2439037 2026-07-20T17:43:51Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2020.codfw.wmnet with OS bookworm 2439038 wikitext text/x-wiki lxdxwyixbd6v5z1jn7x0bbo5j3hgt02 2439039 2439038 2026-07-20T17:45:04Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1011.eqiad.wmnet with reason: host reimage 2439039 wikitext text/x-wiki mcoz5id3a3t4nhd060bi27rna53gvkg 2439040 2439039 2026-07-20T17:46:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2020'] 2439040 wikitext text/x-wiki kir9fdhirvmx7xefwojurz4sfl5e24k 2439041 2439040 2026-07-20T17:47:20Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2244 (T431660)', diff saved to https://phabricator.wikimedia.org/P94924 and previous config saved to /var/cache/conftool/dbconfig/20260720-174717-cwilliams.json 2439041 wikitext text/x-wiki b2nb20dg0kjgtlrmqxit3k39aedsjxv 2439042 2439041 2026-07-20T17:47:21Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet 2439042 wikitext text/x-wiki q8s4chyd57zgx9yi8d5w1yxw4zdau4v 2439043 2439042 2026-07-20T17:48:33Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host acmechief1002.eqiad.wmnet 2439043 wikitext text/x-wiki 6rness813iylzpm8f1ln8d7er4phtrl 2439044 2439043 2026-07-20T17:49:34Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1011.eqiad.wmnet with reason: host reimage 2439044 wikitext text/x-wiki j92mtekjsq44mofuhn3fgtcpf9c1v6e 2439045 2439044 2026-07-20T17:50:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439045 wikitext text/x-wiki acvrkh1k6u2dcch31i9hsfy0yyojsb1 2439046 2439045 2026-07-20T17:52:21Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1002.eqiad.wmnet 2439046 wikitext text/x-wiki 5tx6f56z1n1sq24qn2kkqygr3lzid57 2439047 2439046 2026-07-20T17:56:11Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs2020'] 2439047 wikitext text/x-wiki pr4po24gicea0o1lf3qdyadva8dsceh 2439049 2439047 2026-07-20T17:57:56Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host acmechief2002.codfw.wmnet 2439049 wikitext text/x-wiki p1txur1c09fivpl1fdbbcoh60d3gndt 2439050 2439049 2026-07-20T18:01:44Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2002.codfw.wmnet 2439050 wikitext text/x-wiki m3dc0oiuwom1y0n9007d4ylsdd0eu8u 2439051 2439050 2026-07-20T18:02:32Z Stashbot 7414 sukhe: armed keyholder on acmechief1002.eqiad.wmnet and acmechief2002.codfw.wmnet (active host) 2439051 wikitext text/x-wiki fk1q2ca80cuo3avpo8vz5a25hcbvsqw 2439052 2439051 2026-07-20T18:04:25Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance 2439052 wikitext text/x-wiki l7b3i5kef65m4o7lv41ffy188lop7uk 2439053 2439052 2026-07-20T18:04:45Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2439053 wikitext text/x-wiki aqi4tw9diln5gw6gq29ilujoqmybuop 2439054 2439053 2026-07-20T18:04:54Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1211 (T431660)', diff saved to https://phabricator.wikimedia.org/P94925 and previous config saved to /var/cache/conftool/dbconfig/20260720-180452-cwilliams.json 2439054 wikitext text/x-wiki a3z9pzlj2wvnlhezhul9vnrcla3fkkd 2439056 2439054 2026-07-20T18:08:55Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet 2439056 wikitext text/x-wiki r5zxpff427dzzj9p7zvtwxn32fl9607 2439057 2439056 2026-07-20T18:08:56Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw 2439057 wikitext text/x-wiki ebnw4761gxmb46ua58luvsiif58v2aa 2439058 2439057 2026-07-20T18:09:48Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet 2439058 wikitext text/x-wiki g2jjlljl3rdi8yimxws0nsm23s96fyo 2439059 2439058 2026-07-20T18:09:49Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw 2439059 wikitext text/x-wiki jwzxm5k9tjl8yhhjtulohai2prs3vbg 2439060 2439059 2026-07-20T18:11:31Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T431660)', diff saved to https://phabricator.wikimedia.org/P94926 and previous config saved to /var/cache/conftool/dbconfig/20260720-181129-cwilliams.json 2439060 wikitext text/x-wiki 4l3vm8macrmc32h2jq6g98kt1cjb7tl 2439061 2439060 2026-07-20T18:21:39Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P94927 and previous config saved to /var/cache/conftool/dbconfig/20260720-182137-cwilliams.json 2439061 wikitext text/x-wiki ixay70fms9cp16gewio8xwn21t90yn0 2439062 2439061 2026-07-20T18:31:46Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P94928 and previous config saved to /var/cache/conftool/dbconfig/20260720-183145-cwilliams.json 2439062 wikitext text/x-wiki q3th95q7hpm2ea98j0dakshti308r9q 2439063 2439062 2026-07-20T18:37:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2020'] 2439063 wikitext text/x-wiki fg76dfrlc5ycoqsowf87f82d5ui8dxt 2439064 2439063 2026-07-20T18:37:12Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2020'] 2439064 wikitext text/x-wiki 4lm7wudl4kltu99xymwm3c8w0k5yq6p 2439065 2439064 2026-07-20T18:37:22Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2020'] 2439065 wikitext text/x-wiki n3dhgettg266ddy146zqd12atpyyw4m 2439066 2439065 2026-07-20T18:37:30Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2020'] 2439066 wikitext text/x-wiki n4t5lkqnlicjq64w6nzag8cs01z80io 2439067 2439066 2026-07-20T18:37:39Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2020'] 2439067 wikitext text/x-wiki 5mokkjnids4g26x8zivzyikoelxuczw 2439068 2439067 2026-07-20T18:38:50Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 59m 26s) 2439068 wikitext text/x-wiki 9ovhuxmfxvcrg2s7fq12zoqsbmemb99 2439069 2439068 2026-07-20T18:39:07Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2439069 wikitext text/x-wiki m0zugk96ulecpyir5d9i2yxpm4tw3fh 2439070 2439069 2026-07-20T18:39:11Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 16s) 2439070 wikitext text/x-wiki e9vvre5r4if6z7lo1jsw40v23epz2gw 2439071 2439070 2026-07-20T18:39:22Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2020'] 2439071 wikitext text/x-wiki fih6wtd1ay8wddmqecebjk4v9fjt8b1 2439072 2439071 2026-07-20T18:41:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T431660)', diff saved to https://phabricator.wikimedia.org/P94929 and previous config saved to /var/cache/conftool/dbconfig/20260720-184153-cwilliams.json 2439072 wikitext text/x-wiki 7jkevqpkjl0mg114bod8v1g6ebtyfkm 2439073 2439072 2026-07-20T18:42:17Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1255.eqiad.wmnet with reason: Maintenance 2439073 wikitext text/x-wiki ixv6qudw9el3i3862w2wpdpgw2bzg38 2439074 2439073 2026-07-20T18:42:25Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1255 (T431660)', diff saved to https://phabricator.wikimedia.org/P94930 and previous config saved to /var/cache/conftool/dbconfig/20260720-184224-cwilliams.json 2439074 wikitext text/x-wiki 7ykvxcgdnb9ku8qws8uupwbm1n3h707 2439076 2439074 2026-07-20T18:48:17Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1255 (T431660)', diff saved to https://phabricator.wikimedia.org/P94931 and previous config saved to /var/cache/conftool/dbconfig/20260720-184816-cwilliams.json 2439076 wikitext text/x-wiki 1f2ls23tr3rhx5ll5kttjtre6rg42bi 2439077 2439076 2026-07-20T18:50:21Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439077 wikitext text/x-wiki aiqrnxtj142edn639iulj6ma5vg0pzn 2439078 2439077 2026-07-20T18:58:25Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1255', diff saved to https://phabricator.wikimedia.org/P94932 and previous config saved to /var/cache/conftool/dbconfig/20260720-185824-cwilliams.json 2439078 wikitext text/x-wiki 3tv7tniq8pv0d6u8o2kd1nwdji8ikco 2439079 2439078 2026-07-20T19:08:33Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1255', diff saved to https://phabricator.wikimedia.org/P94933 and previous config saved to /var/cache/conftool/dbconfig/20260720-190831-cwilliams.json 2439079 wikitext text/x-wiki lo146pmdd3yfx2woz4ncf0t2ju155cu 2439080 2439079 2026-07-20T19:18:41Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1255 (T431660)', diff saved to https://phabricator.wikimedia.org/P94934 and previous config saved to /var/cache/conftool/dbconfig/20260720-191839-cwilliams.json 2439080 wikitext text/x-wiki lf3ps8msv6ouzzjk7iolbvy8nopj46h 2439081 2439080 2026-07-20T19:18:49Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1256.eqiad.wmnet with reason: Maintenance 2439081 wikitext text/x-wiki izvvffkuf9xur0cqu6f7ekos61yp487 2439082 2439081 2026-07-20T19:18:58Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1256 (T431660)', diff saved to https://phabricator.wikimedia.org/P94935 and previous config saved to /var/cache/conftool/dbconfig/20260720-191856-cwilliams.json 2439082 wikitext text/x-wiki 5devwzcvs8b4czr6cyrruql1328m58n 2439084 2439082 2026-07-20T19:25:44Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1256 (T431660)', diff saved to https://phabricator.wikimedia.org/P94936 and previous config saved to /var/cache/conftool/dbconfig/20260720-192542-cwilliams.json 2439084 wikitext text/x-wiki tncebo8adngzh0d1p21wek1rpjhug8s 2439085 2439084 2026-07-20T19:35:51Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1256', diff saved to https://phabricator.wikimedia.org/P94937 and previous config saved to /var/cache/conftool/dbconfig/20260720-193550-cwilliams.json 2439085 wikitext text/x-wiki kvym3jmbj0l6ibijpd7qqehyahkfq3w 2439087 2439085 2026-07-20T19:41:20Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1011.eqiad.wmnet with OS bookworm 2439087 wikitext text/x-wiki p5h1lk7cfipyvzoa2yd2kw2pgsvxh12 2439088 2439087 2026-07-20T19:42:57Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wikidata from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling source-only afterwards 2439088 wikitext text/x-wiki btbpe3p0wty1kqnkq6mn5d760beeghr 2439089 2439088 2026-07-20T19:43:00Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wikidata from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling source-only afterwards 2439089 wikitext text/x-wiki mw8ig38u341uvnmwbm2bcq0obt5dmgs 2439090 2439089 2026-07-20T19:43:46Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wikidata from wdqs1012.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards 2439090 wikitext text/x-wiki 83v347ucjmhms073snww9t8vmnwnowq 2439091 2439090 2026-07-20T19:43:49Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wikidata from wdqs1012.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards 2439091 wikitext text/x-wiki cwopvm5uxsuwxtgip0ppqr3x0rct8hf 2439092 2439091 2026-07-20T19:44:15Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wikidata from wdqs1012.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards 2439092 wikitext text/x-wiki 1yd2vk9a9phshs9mwxe952860od4g8t 2439093 2439092 2026-07-20T19:45:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1256', diff saved to https://phabricator.wikimedia.org/P94938 and previous config saved to /var/cache/conftool/dbconfig/20260720-194558-cwilliams.json 2439093 wikitext text/x-wiki 8p6tlg32macd3rmde1qr882yzws69ax 2439094 2439093 2026-07-20T19:50:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2020'] 2439094 wikitext text/x-wiki mgwu0h5hg841gcs46rudad7x4iwq3kf 2439095 2439094 2026-07-20T19:51:25Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2020'] 2439095 wikitext text/x-wiki lr3709afi9qpy2jpdslgf7nlsaja80w 2439096 2439095 2026-07-20T19:56:07Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1256 (T431660)', diff saved to https://phabricator.wikimedia.org/P94939 and previous config saved to /var/cache/conftool/dbconfig/20260720-195605-cwilliams.json 2439096 wikitext text/x-wiki iwfm760z6zptkyluz3ps31ynl67eqrt 2439097 2439096 2026-07-20T19:56:26Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1257.eqiad.wmnet with reason: Maintenance 2439097 wikitext text/x-wiki ea3eezd60b2a5z4imlv6k80ebjckyms 2439098 2439097 2026-07-20T19:56:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1257 (T431660)', diff saved to https://phabricator.wikimedia.org/P94940 and previous config saved to /var/cache/conftool/dbconfig/20260720-195633-cwilliams.json 2439098 wikitext text/x-wiki mfqw542y1m621levkpbcy32c6yn1z5v 2439099 2439098 2026-07-20T20:03:11Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1257 (T431660)', diff saved to https://phabricator.wikimedia.org/P94941 and previous config saved to /var/cache/conftool/dbconfig/20260720-200310-cwilliams.json 2439099 wikitext text/x-wiki 7409yn0lsd6rfhq8fnugsnry3srk6d9 2439100 2439099 2026-07-20T20:04:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2007.codfw.wmnet with OS bookworm 2439100 wikitext text/x-wiki j2qrohako1e8ja6nzpfrtv3vp2us31u 2439101 2439100 2026-07-20T20:05:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2007 2439101 wikitext text/x-wiki s99u437sb9zgvt9n8yefj5b3s01g0du 2439102 2439101 2026-07-20T20:10:49Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wikidata from wdqs1012.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards 2439102 wikitext text/x-wiki lij23qdk6pgmcxoneaoxzch0i5nhze5 2439103 2439102 2026-07-20T20:13:13Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1012.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards 2439103 wikitext text/x-wiki c6g00c6klezbn72iww2lo1zbkvjk2ic 2439104 2439103 2026-07-20T20:13:19Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1257', diff saved to https://phabricator.wikimedia.org/P94942 and previous config saved to /var/cache/conftool/dbconfig/20260720-201318-cwilliams.json 2439104 wikitext text/x-wiki bzdx9eknnaznuacwy8ey783in39y185 2439105 2439104 2026-07-20T20:13:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2439105 wikitext text/x-wiki h4xpi9gbhsvmn0ficbievnw6jgkame8 2439106 2439105 2026-07-20T20:17:28Z Stashbot 7414 arlolra@deploy2003: Started scap sync-world: Backport for [[gerrit:1312547|Enable PRV on enwiki talk namespace, template namespace]], [[gerrit:1312554|Bump wikimedia/parsoid to 0.24.0-a15 (T221028 T226289 T353697 T430854 T431771)]], [[gerrit:1312556|Bump wikimedia/parsoid to 0.24.0-a15 (T431771)]] 2439106 wikitext text/x-wiki 298kqjy1z2ubl3ego6fs6h24qkzy9qc 2439107 2439106 2026-07-20T20:21:10Z Stashbot 7414 arlolra@deploy2003: arlolra, cscott: Backport for [[gerrit:1312547|Enable PRV on enwiki talk namespace, template namespace]], [[gerrit:1312554|Bump wikimedia/parsoid to 0.24.0-a15 (T221028 T226289 T353697 T430854 T431771)]], [[gerrit:1312556|Bump wikimedia/parsoid to 0.24.0-a15 (T431771)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439107 wikitext text/x-wiki 4xtpou93zylu4g7emd0sau9qt31umao 2439109 2439107 2026-07-20T20:23:27Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1257', diff saved to https://phabricator.wikimedia.org/P94943 and previous config saved to /var/cache/conftool/dbconfig/20260720-202325-cwilliams.json 2439109 wikitext text/x-wiki tn750ggo4mhhzmu1d6r1of8cqarkunb 2439110 2439109 2026-07-20T20:25:55Z Stashbot 7414 arlolra@deploy2003: arlolra, cscott: Continuing with deployment 2439110 wikitext text/x-wiki j34ehthh963sos3w3a3ppn1j789hbes 2439111 2439110 2026-07-20T20:29:14Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=no; selector: name=wdqs1011.eqiad.wmnet 2439111 wikitext text/x-wiki ayftbseuyopb58bv6gg24hjemqnecwe 2439112 2439111 2026-07-20T20:29:15Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=no; selector: name=wdqs1023.eqiad.wmnet 2439112 wikitext text/x-wiki 7tlopr6dc9bez597dv82ig1oslqhvzm 2439113 2439112 2026-07-20T20:29:17Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=no; selector: name=wdqs2020.codfw.wmnet 2439113 wikitext text/x-wiki 4ykipl2gl6zx2ukjh3819u7ptpojryt 2439114 2439113 2026-07-20T20:30:20Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs1011.eqiad.wmnet 2439114 wikitext text/x-wiki bb8yhfhiyhcnb020s3ipzlo3gc0t11p 2439115 2439114 2026-07-20T20:30:21Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs1023.eqiad.wmnet 2439115 wikitext text/x-wiki omffx7csoy0wcu6s396n3ywgiw0qs05 2439116 2439115 2026-07-20T20:30:23Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2020.codfw.wmnet 2439116 wikitext text/x-wiki 2h3r5mmdhc1h11afco0wkk1wa5yf8w0 2439117 2439116 2026-07-20T20:32:34Z Stashbot 7414 arlolra@deploy2003: Finished scap sync-world: Backport for [[gerrit:1312547|Enable PRV on enwiki talk namespace, template namespace]], [[gerrit:1312554|Bump wikimedia/parsoid to 0.24.0-a15 (T221028 T226289 T353697 T430854 T431771)]], [[gerrit:1312556|Bump wikimedia/parsoid to 0.24.0-a15 (T431771)]] (duration: 15m 07s) 2439117 wikitext text/x-wiki j0ovo34nzfq1sl8ygiqztmq5lrqsdk8 2439118 2439117 2026-07-20T20:33:35Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1257 (T431660)', diff saved to https://phabricator.wikimedia.org/P94944 and previous config saved to /var/cache/conftool/dbconfig/20260720-203333-cwilliams.json 2439118 wikitext text/x-wiki 6yk5r04cqs6o8615kyyl6uuy9y21h1r 2439121 2439118 2026-07-20T20:41:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2007 - bking@cumin2003" 2439121 wikitext text/x-wiki icftdcujaa58n6mdkrc7edmqiazr90u 2439122 2439121 2026-07-20T20:42:52Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2007 - bking@cumin2003" 2439122 wikitext text/x-wiki t52p6jdhse5wng2y861qo9w0dmwcikz 2439123 2439122 2026-07-20T20:42:57Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439123 wikitext text/x-wiki ol7eog2gofjl04kufw9zfrsnvq5iljo 2439124 2439123 2026-07-20T20:43:00Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2007.codfw.wmnet 156.16.192.10.in-addr.arpa 6.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2439124 wikitext text/x-wiki ih7d1xsjarvzb8oofbymusvjbwpqyw8 2439125 2439124 2026-07-20T20:43:03Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2007.codfw.wmnet 156.16.192.10.in-addr.arpa 6.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2439125 wikitext text/x-wiki 33htr42fy9xyu4ok6zh2x70exubpagh 2439126 2439125 2026-07-20T20:43:04Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2007 2439126 wikitext text/x-wiki 9sgyco56dqr1dvuxlmqzu7xv3lokltq 2439127 2439126 2026-07-20T20:44:37Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2007 2439127 wikitext text/x-wiki 32rg33wu3cnc43gi7uwz5m0a5ftq4d7 2439128 2439127 2026-07-20T20:44:38Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2007 2439128 wikitext text/x-wiki lya1attu3540jwdw9kv9uwhgsvysity 2439129 2439128 2026-07-20T20:44:43Z Stashbot 7414 ebernhardson@deploy2003: Started deploy [search/mjolnir/deploy@d4dc3b8]: Update for opensearch 2.x compat 2439129 wikitext text/x-wiki gvzx5o7vz0zp6hgn86qehiodofeg7uy 2439130 2439129 2026-07-20T20:45:18Z Stashbot 7414 ebernhardson@deploy2003: Finished deploy [search/mjolnir/deploy@d4dc3b8]: Update for opensearch 2.x compat (duration: 00m 34s) 2439130 wikitext text/x-wiki 1nml2wdldm98iy161dc2v8zt6ctyyyk 2439131 2439130 2026-07-20T20:46:23Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=wdqs-internal-main,name=codfw 2439131 wikitext text/x-wiki 8ty4t7g9hcrjoyz0gzqafl2qjbq6k5c 2439134 2439131 2026-07-20T20:58:00Z Stashbot 7414 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service 2439134 wikitext text/x-wiki clnk4tzdfxmujenugd2cesdiklh8cx9 2439135 2439134 2026-07-20T20:58:12Z Stashbot 7414 sukhe: pybal restart for IP changes around wdqs-main hosts 2439135 wikitext text/x-wiki 7bnuipszlhxncvwkmlziephakiv0etb 2439136 2439135 2026-07-20T20:59:44Z Stashbot 7414 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service 2439136 wikitext text/x-wiki f219ofx7xh7mjiee7sqvg0f497l51rg 2439137 2439136 2026-07-20T21:06:32Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2007.codfw.wmnet with reason: host reimage 2439137 wikitext text/x-wiki 3ixajkmsou8hjnjlsvkf83yhqbi0zsz 2439138 2439137 2026-07-20T21:08:03Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal-main,name=codfw 2439138 wikitext text/x-wiki irhx9kpktuyx9l5vlmm2uledubp14qh 2439140 2439138 2026-07-20T21:13:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2007.codfw.wmnet with reason: host reimage 2439140 wikitext text/x-wiki 14ty6pe34x3jo6z7zi10dx7a480fj0i 2439148 2439140 2026-07-20T21:27:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1012.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards 2439148 wikitext text/x-wiki hj0cd68foqctkyrrcwnwx0phf0rgzjz 2439149 2439148 2026-07-20T21:32:01Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2439149 wikitext text/x-wiki mejaj9b3qe8w0zvbsgp1myt3cfr2sgq 2439150 2439149 2026-07-20T21:32:08Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 17s) 2439150 wikitext text/x-wiki 03ikrh1xemq89lq5d91q3io6fwgolin 2439151 2439150 2026-07-20T21:33:50Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=no; selector: name=wdqs1011.eqiad.wmnet 2439151 wikitext text/x-wiki pukjturvxy72o2hbahr1k6owoonh8sa 2439152 2439151 2026-07-20T21:33:51Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=no; selector: name=wdqs1023.eqiad.wmnet 2439152 wikitext text/x-wiki cb4l6yse1eetj6alatdgijz6tnmu7cf 2439153 2439152 2026-07-20T21:33:53Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=no; selector: name=wdqs2020.codfw.wmnet 2439153 wikitext text/x-wiki q65f6e580d4cbjkio1dy7tijvik8oli 2439154 2439153 2026-07-20T21:34:55Z Stashbot 7414 sbassett: Deployed security fix for T432424 2439154 wikitext text/x-wiki pisub4wg7vyf26y7am0dv4r42g7qpma 2439156 2439154 2026-07-20T21:37:46Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1023.eqiad.wmnet with OS bookworm 2439156 wikitext text/x-wiki 6wrky5kx6p5fb6ib3b21alqyez9xnwc 2439157 2439156 2026-07-20T21:37:49Z Stashbot 7414 ryankemper@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1023.eqiad.wmnet with OS bookworm 2439157 wikitext text/x-wiki 98l5aa2smj8k05ovnwodwl1r4z69zne 2439159 2439157 2026-07-20T21:42:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling source-only afterwards 2439159 wikitext text/x-wiki rpjga4v6stq4oii51a2e0dweaax5x57 2439160 2439159 2026-07-20T21:42:50Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1023.eqiad.wmnet with OS bookworm 2439160 wikitext text/x-wiki p19oy3kg7h4930ji41md3r5khrkhcns 2439161 2439160 2026-07-20T22:39:05Z Stashbot 7414 maryum: Deployed security fixes for several security bugs 2439161 wikitext text/x-wiki c7no3e3e60boarxsyk2py1ljd9irhgk 2439162 2439161 2026-07-20T22:46:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling source-only afterwards 2439162 wikitext text/x-wiki dbwnbdb3gk5csnm20u95zwngn8l7e0k 2439163 2439162 2026-07-20T22:53:24Z Stashbot 7414 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1312302|Disable MJPEG, enable MPEG-4 Part II (T358266)]] 2439163 wikitext text/x-wiki 4e2rk4jzdloydori1lr58fbmfail3os 2439164 2439163 2026-07-20T23:00:19Z Stashbot 7414 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1312302|Disable MJPEG, enable MPEG-4 Part II (T358266)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439164 wikitext text/x-wiki 9ltofiz78uw817et86r5p9x63xjnicq 2439165 2439164 2026-07-20T23:01:05Z Stashbot 7414 ladsgroup@deploy2003: ladsgroup: Continuing with deployment 2439165 wikitext text/x-wiki p6rqalsqwq0k9ehoz8rfeqt2bz5a286 2439167 2439165 2026-07-20T23:07:33Z Stashbot 7414 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1312302|Disable MJPEG, enable MPEG-4 Part II (T358266)]] (duration: 14m 16s) 2439167 wikitext text/x-wiki 07dqdrzc1rk7ker5ptt02cbuk6dwtsn 2439168 2439167 2026-07-20T23:07:48Z Stashbot 7414 Amir1: deleting echo notifications from 2015 on group1 wikis 2439168 wikitext text/x-wiki 7hn2t7vidd2ryehyt95b51pqz0zh9f3 2439170 2439168 2026-07-20T23:38:06Z Stashbot 7414 ryankemper@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1023.eqiad.wmnet with OS bookworm 2439170 wikitext text/x-wiki l20jc85ihbre59e1k1z2qpkvevejgv5 2439172 2439170 2026-07-21T00:56:04Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1023.eqiad.wmnet with OS bookworm 2439172 wikitext text/x-wiki 2pb0av3z7ri3ipsn44ylgs9n0shi5ia 2439175 2439172 2026-07-21T02:00:28Z Stashbot 7414 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image 2439175 wikitext text/x-wiki 6kq9vl2rxy1gpf0nuruhapxwscs46qh 2439176 2439175 2026-07-21T02:07:15Z Stashbot 7414 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) 2439176 wikitext text/x-wiki fvll6htypbainun6sxy1ln91hl0axu6 2439177 2439176 2026-07-21T02:16:10Z Stashbot 7414 ryankemper@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1023.eqiad.wmnet with OS bookworm 2439177 wikitext text/x-wiki qota9u1246q8bux5oytmizq7ttvyber 2439179 2439177 2026-07-21T02:35:56Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2439179 wikitext text/x-wiki rabh9ckgoj5hki729ay45ln3w98721k 2439180 2439179 2026-07-21T02:36:27Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2439180 wikitext text/x-wiki 3iir4dyhqpmv9dn655bdkl0gbdfht5h 2439181 2439180 2026-07-21T02:36:28Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2439181 wikitext text/x-wiki r1w6vbvzv0bne79zn25jrxfjeemh26t 2439183 2439181 2026-07-21T02:36:53Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2439183 wikitext text/x-wiki 595s0fvkuzsfbbwtb3epf0pl9t8jdlp 2439184 2439183 2026-07-21T02:41:06Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2439184 wikitext text/x-wiki l1mv0cqdqu4ugkpajk8ayqyseghsymp 2439185 2439184 2026-07-21T02:41:10Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2439185 wikitext text/x-wiki 8tzk50qi3z65xieap8f1lc14e1ggmfr 2439186 2439185 2026-07-21T02:41:11Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2439186 wikitext text/x-wiki 9u7sxbtb9dyldkcdwcfk7xzri42591w 2439187 2439186 2026-07-21T02:41:13Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2439187 wikitext text/x-wiki n6hs86l28k3tfni2rpjawpvlrp13c68 2439188 2439187 2026-07-21T02:43:47Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2439188 wikitext text/x-wiki nle2ujukpq3d8euydlnu1fo93qi3o8j 2439189 2439188 2026-07-21T02:43:51Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2439189 wikitext text/x-wiki 9irialk7j503wv86kp4utnbveijxc62 2439190 2439189 2026-07-21T02:43:52Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2439190 wikitext text/x-wiki lk8sv1hiy75cqpxzdhnvc0vq30296hi 2439191 2439190 2026-07-21T02:43:55Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2439191 wikitext text/x-wiki 9bcx7a2wmask21uzij6f0lnqf5qsiem 2439192 2439191 2026-07-21T02:44:15Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2439192 wikitext text/x-wiki ctmyjo3sul3elccwxbcp0drexx8zew3 2439193 2439192 2026-07-21T02:44:18Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2439193 wikitext text/x-wiki 7nhxsl6gnibpcy75sb84c92bgvvj6l0 2439194 2439193 2026-07-21T02:44:20Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2439194 wikitext text/x-wiki 20aaor89o8rza6cxntnwoq2gkgi74m3 2439195 2439194 2026-07-21T02:44:22Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2439195 wikitext text/x-wiki 57blbd6htpvj9zqfosnhu0xmupbou6f 2439196 2439195 2026-07-21T02:45:28Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2439196 wikitext text/x-wiki r63adgtkfpw2dhhqjfk4czi3d2ymo55 2439197 2439196 2026-07-21T02:45:31Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2439197 wikitext text/x-wiki d5k2anyieznvv3lbrky1oo4y9c0rx52 2439198 2439197 2026-07-21T02:45:33Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2439198 wikitext text/x-wiki naj8omy6os7yz5oyx22gvu8ueuz95ai 2439199 2439198 2026-07-21T02:45:35Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2439199 wikitext text/x-wiki o4iymjdsw4tmbigsqz96wdlp3390la0 2439200 2439199 2026-07-21T03:00:54Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2439200 wikitext text/x-wiki 7xp6fjyjv9emwch8xk5vmugsahelxs0 2439201 2439200 2026-07-21T03:01:00Z Stashbot 7414 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2439201 wikitext text/x-wiki n9mpyzn1km41b7pr7my20p2vulxdos5 2439202 2439201 2026-07-21T03:01:01Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2439202 wikitext text/x-wiki dne4yscaithmv2lcwn9dma7oby1c9h3 2439203 2439202 2026-07-21T03:01:06Z Stashbot 7414 ssastry@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2439203 wikitext text/x-wiki ibt22rg8m1gv38w1r81gss1gskuzykc 2439204 2439203 2026-07-21T03:05:08Z Stashbot 7414 mwpresync@deploy2003: Started scap sync-world: testwikis to 1.47.0-wmf.12 refs T430831 2439204 wikitext text/x-wiki kp03llllzhk1vor5gl7lr1flif29kl3 2439205 2439204 2026-07-21T03:41:39Z Stashbot 7414 mwpresync@deploy2003: Finished scap sync-world: testwikis to 1.47.0-wmf.12 refs T430831 (duration: 36m 30s) 2439205 wikitext text/x-wiki cljkjbjrg22eaietyn0tn5c4h5aupy8 2439206 2439205 2026-07-21T04:01:10Z Stashbot 7414 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.9 (duration: 01m 08s) 2439206 wikitext text/x-wiki 0o1mqzkyd37zrualchbgqa37kwcewyn 2439209 2439206 2026-07-21T04:56:08Z Stashbot 7414 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2439209 wikitext text/x-wiki 7ua23kgd9naprn27gn102v1p96e2y1j 2439210 2439209 2026-07-21T05:00:09Z Stashbot 7414 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2439210 wikitext text/x-wiki dnngc5ew5l72u880fcccm68pxfwxcwb 2439211 2439210 2026-07-21T05:04:52Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet 2439211 wikitext text/x-wiki 9zp7pf2t5p9sktq37ktgj7kw4liylza 2439212 2439211 2026-07-21T05:11:49Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet 2439212 wikitext text/x-wiki i4snewfa44rk52jv5wqvupcmidujg1t 2439213 2439212 2026-07-21T05:30:19Z Stashbot 7414 kevinbazira@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2439213 wikitext text/x-wiki 0zla0urs8864iai6u6r3pcplzy6aqjg 2439214 2439213 2026-07-21T05:32:54Z Stashbot 7414 kevinbazira@deploy2003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2439214 wikitext text/x-wiki 1ceqyjkcv0euqbi3xxito3buxbn2p47 2439215 2439214 2026-07-21T05:33:35Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1029.eqiad.wmnet,service=s5 2439215 wikitext text/x-wiki l0ljs2al3s6umkqkqu2ndyfbrqdsfgz 2439216 2439215 2026-07-21T05:33:40Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1029.eqiad.wmnet,service=s8 2439216 wikitext text/x-wiki b6pqnjjmyhyztiqk2luaveskvohk9qk 2439217 2439216 2026-07-21T05:37:39Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s1 2439217 wikitext text/x-wiki oped3kzqkbpsa78igo1ha5fvetw9gq9 2439218 2439217 2026-07-21T05:38:11Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1017.eqiad.wmnet with reason: Cloning 2439218 wikitext text/x-wiki r5k2vl0kmcki7v1948jdh308f11jy1m 2439219 2439218 2026-07-21T05:42:33Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Haproxy-only mode support - oblivian@cumin1003" 2439219 wikitext text/x-wiki t1e5kfgguabsc1no6jykiv72v0cpxqk 2439220 2439219 2026-07-21T05:42:36Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy-only mode support - oblivian@cumin1003 2439220 wikitext text/x-wiki bqcrsfkfqwxk4t9jxomfpv44ogye094 2439221 2439220 2026-07-21T05:43:27Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy-only mode support - oblivian@cumin1003 2439221 wikitext text/x-wiki mays4n88iicxb279zmf0dnqubmzp47j 2439222 2439221 2026-07-21T05:43:28Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Haproxy-only mode support - oblivian@cumin1003" 2439222 wikitext text/x-wiki 62i186wllctyudcqpzycv42jo5i8ou7 2439223 2439222 2026-07-21T06:24:47Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1023.eqiad.wmnet with OS bookworm 2439223 wikitext text/x-wiki sndzhpy6j8l3peltumpa0rpowsszodo 2439224 2439223 2026-07-21T06:46:43Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1023.eqiad.wmnet with reason: host reimage 2439224 wikitext text/x-wiki qha6hyafvmv52xp01r4z41p47mbn55q 2439225 2439224 2026-07-21T06:54:22Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1023.eqiad.wmnet with reason: host reimage 2439225 wikitext text/x-wiki slaq0s4c5naal1ku26chhjl3lkyg7bi 2439227 2439225 2026-07-21T07:08:41Z Stashbot 7414 elukey@deploy2003: helmfile [staging] START helmfile.d/services/proton: sync 2439227 wikitext text/x-wiki ibidplc3aowhs2z2tzd9iei79nufryf 2439228 2439227 2026-07-21T07:09:30Z Stashbot 7414 elukey@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: sync 2439228 wikitext text/x-wiki ibdihx969oje4tijbdkeky1r1ocbspb 2439229 2439228 2026-07-21T07:10:14Z Stashbot 7414 elukey@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: sync 2439229 wikitext text/x-wiki 4542t9nd4j4qyqg5ffjmlf6awglkmvv 2439230 2439229 2026-07-21T07:11:12Z Stashbot 7414 elukey@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: sync 2439230 wikitext text/x-wiki fovzcq4f5qfeqvpx1zvstifqwv6f0aw 2439231 2439230 2026-07-21T07:13:37Z Stashbot 7414 elukey@deploy2003: helmfile [codfw] START helmfile.d/services/proton: sync 2439231 wikitext text/x-wiki em1q1utrtybjgt0p4qcj5t9zqn2up5z 2439232 2439231 2026-07-21T07:14:43Z Stashbot 7414 elukey@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: sync 2439232 wikitext text/x-wiki g5voygyg4hgq5p0zesyngxh6ki79wo7 2439233 2439232 2026-07-21T07:22:49Z Stashbot 7414 elukey: upgrade libtiff6 packages on Wikikube trixie workers for security upgrades 2439233 wikitext text/x-wiki 3m7pe5nn0uxueibqo9dr5nrvqtn9n1g 2439234 2439233 2026-07-21T07:23:32Z Stashbot 7414 elukey: upgrade libtiff6 packages on zuul* trixie hosts for security upgrades 2439234 wikitext text/x-wiki c1g6pcaa3v5pb3u7541x5posotqw556 2439235 2439234 2026-07-21T07:42:17Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host idm2001.wikimedia.org 2439235 wikitext text/x-wiki atm10mw5ujzotg91v4refsyana1s5xa 2439236 2439235 2026-07-21T07:43:54Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2439236 wikitext text/x-wiki qz7fb6uax7lf5jr1mvl8697rup1tbph 2439237 2439236 2026-07-21T07:45:33Z Stashbot 7414 reedy@deploy2003: Started scap sync-world: Backport for [[gerrit:1312614|Enable UserPageEditProtection on jawiki (T392754 T410655)]] 2439237 wikitext text/x-wiki s8lnnektvefu7kpo9pimk63wgdwtsfz 2439238 2439237 2026-07-21T07:46:17Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm2001.wikimedia.org 2439238 wikitext text/x-wiki kz4nsqf6wszn0qwvqc9mu1h1p7tc0af 2439239 2439238 2026-07-21T07:48:08Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1093 hosts 2439239 wikitext text/x-wiki dspluir8107fblxd0u295qn9wao6d6k 2439240 2439239 2026-07-21T07:51:22Z Stashbot 7414 reedy@deploy2003: reedy, neriah: Backport for [[gerrit:1312614|Enable UserPageEditProtection on jawiki (T392754 T410655)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439240 wikitext text/x-wiki 1c2pkutdmo5527csr9xlct68iljxsh2 2439241 2439240 2026-07-21T07:51:31Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2439241 wikitext text/x-wiki asvce7s19rc9ovfekb7rsx8hc78tf5x 2439242 2439241 2026-07-21T07:51:56Z Stashbot 7414 reedy@deploy2003: reedy, neriah: Continuing with deployment 2439242 wikitext text/x-wiki lebvzb43bz9bf3lnyf0dv3rgcngn9wp 2439243 2439242 2026-07-21T07:58:29Z Stashbot 7414 reedy@deploy2003: Finished scap sync-world: Backport for [[gerrit:1312614|Enable UserPageEditProtection on jawiki (T392754 T410655)]] (duration: 12m 55s) 2439243 wikitext text/x-wiki aqn8j6yoyj2f6y7de8e3z96g06z5gus 2439244 2439243 2026-07-21T08:06:14Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Add subrated circuit rate to interface descriptions - CR1312476 - ayounsi@cumin1003 2439244 wikitext text/x-wiki ig4rleg2luz7jhn6grks6yjsqhql9de 2439245 2439244 2026-07-21T08:08:40Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Add subrated circuit rate to interface descriptions - CR1312476 - ayounsi@cumin1003 2439245 wikitext text/x-wiki kzqbpxwo2nadzi8y0szgdhxqpmrwxdv 2439246 2439245 2026-07-21T08:32:39Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet,service=s1 2439246 wikitext text/x-wiki awu0aebgrxructlp82usdnd08ql24a1 2439251 2439246 2026-07-21T08:45:39Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s6 T430964 2439251 wikitext text/x-wiki marrn2b1fcysqpgc5pfrui7g2zqihgw 2439252 2439251 2026-07-21T08:46:16Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2214 with weight 0 T430964', diff saved to https://phabricator.wikimedia.org/P94946 and previous config saved to /var/cache/conftool/dbconfig/20260721-084613-cwilliams.json 2439252 wikitext text/x-wiki 03p90xah8qg0sgjgphjjhesdjss4c11 2439254 2439252 2026-07-21T08:53:37Z Stashbot 7414 cezmunsta: Starting s6 codfw failover from db2229 to db2214 - T430964 2439254 wikitext text/x-wiki l66d3mla3a0m9xpwan1x2nd056nyg3t 2439255 2439254 2026-07-21T08:54:45Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2214 to s6 primary T430964', diff saved to https://phabricator.wikimedia.org/P94947 and previous config saved to /var/cache/conftool/dbconfig/20260721-085442-cwilliams.json 2439255 wikitext text/x-wiki cviiphkrx0i89ofdygqokr8xauw4nxl 2439256 2439255 2026-07-21T08:57:27Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2229 T430964', diff saved to https://phabricator.wikimedia.org/P94948 and previous config saved to /var/cache/conftool/dbconfig/20260721-085724-cwilliams.json 2439256 wikitext text/x-wiki ewzxp7nma7aqdo5ynf5t60ea50ud4x1 2439258 2439256 2026-07-21T09:23:05Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db2229.codfw.wmnet 2439258 wikitext text/x-wiki ot6dwhe3c4f7k1imzzkjmwvq786x2uk 2439259 2439258 2026-07-21T09:23:15Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db2229.codfw.wmnet 2439259 wikitext text/x-wiki mspy224219ztr78xyuuic1mlsl4ri6b 2439261 2439259 2026-07-21T09:42:25Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2229.codfw.wmnet 2439261 wikitext text/x-wiki bxzig81nn8ae0br4giv6am3za0v7b4p 2439262 2439261 2026-07-21T09:42:27Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2229.codfw.wmnet 2439262 wikitext text/x-wiki bhn88sc5mz7jpu2q88cxsarkoxyhmww 2439263 2439262 2026-07-21T09:43:46Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2229: Upgraded kernel 2439263 wikitext text/x-wiki pwabs98izx9qb2t5yremhyyi5h0nlza 2439267 2439263 2026-07-21T10:18:42Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance 2439267 wikitext text/x-wiki amawf81li2gq0gewid5q7f0gw1yrocm 2439268 2439267 2026-07-21T10:18:50Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2158 (T431660)', diff saved to https://phabricator.wikimedia.org/P94954 and previous config saved to /var/cache/conftool/dbconfig/20260721-101848-cwilliams.json 2439268 wikitext text/x-wiki sv1krxj07qz9k683ks67ful1p5v4ult 2439270 2439268 2026-07-21T10:25:41Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T431660)', diff saved to https://phabricator.wikimedia.org/P94955 and previous config saved to /var/cache/conftool/dbconfig/20260721-102539-cwilliams.json 2439270 wikitext text/x-wiki l4sb7v0j4xysmcql02yde5l7hvchck4 2439271 2439270 2026-07-21T10:27:30Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance 2439271 wikitext text/x-wiki dlzsrr6gqmflwskl022bc5brntlhm9d 2439272 2439271 2026-07-21T10:27:51Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025,1028].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2439272 wikitext text/x-wiki 8zyr500upmew3gl6pl182nhtz8jadgp 2439273 2439272 2026-07-21T10:27:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1165 (T431660)', diff saved to https://phabricator.wikimedia.org/P94956 and previous config saved to /var/cache/conftool/dbconfig/20260721-102757-cwilliams.json 2439273 wikitext text/x-wiki cdv5n3gatulegms6n0yzttzzrymc0et 2439275 2439273 2026-07-21T10:29:15Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2229: Upgraded kernel 2439275 wikitext text/x-wiki m0ngxytwcogwohjzyxeyskhdnsybhf1 2439280 2439275 2026-07-21T10:34:58Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T431660)', diff saved to https://phabricator.wikimedia.org/P94958 and previous config saved to /var/cache/conftool/dbconfig/20260721-103456-cwilliams.json 2439280 wikitext text/x-wiki e8zkjmtzz7aiddb3xu2s2agg4uspdli 2439281 2439280 2026-07-21T10:35:48Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P94959 and previous config saved to /var/cache/conftool/dbconfig/20260721-103547-cwilliams.json 2439281 wikitext text/x-wiki tmftgb72qbswbc1dm8pn3evncaiprzi 2439283 2439281 2026-07-21T10:45:06Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P94960 and previous config saved to /var/cache/conftool/dbconfig/20260721-104504-cwilliams.json 2439283 wikitext text/x-wiki eritn7dwyhv6k3eer7ooihgj6pvd0w7 2439284 2439283 2026-07-21T10:45:56Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P94961 and previous config saved to /var/cache/conftool/dbconfig/20260721-104555-cwilliams.json 2439284 wikitext text/x-wiki aw37zqt07nkl818aw6b28tny4do7mrw 2439285 2439284 2026-07-21T10:55:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P94962 and previous config saved to /var/cache/conftool/dbconfig/20260721-105512-cwilliams.json 2439285 wikitext text/x-wiki owivexrt4y80sng6p5c2hts4f4lc03c 2439286 2439285 2026-07-21T10:56:04Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T431660)', diff saved to https://phabricator.wikimedia.org/P94963 and previous config saved to /var/cache/conftool/dbconfig/20260721-105603-cwilliams.json 2439286 wikitext text/x-wiki 2qf4zlpyq03jyh6s273csvwzmd4rtwk 2439287 2439286 2026-07-21T10:56:25Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance 2439287 wikitext text/x-wiki s2g38o81cvfvlquvlrzoih4ebj0id75 2439288 2439287 2026-07-21T10:56:33Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2169 (T431660)', diff saved to https://phabricator.wikimedia.org/P94964 and previous config saved to /var/cache/conftool/dbconfig/20260721-105632-cwilliams.json 2439288 wikitext text/x-wiki c23oqznvxzqwebg28jmoauzjkd84rks 2439289 2439288 2026-07-21T11:04:40Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T431660)', diff saved to https://phabricator.wikimedia.org/P94966 and previous config saved to /var/cache/conftool/dbconfig/20260721-110439-cwilliams.json 2439289 wikitext text/x-wiki 42pq4e8tjmxkz7uzhvjkjxqcqm24t9v 2439290 2439289 2026-07-21T11:05:22Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T431660)', diff saved to https://phabricator.wikimedia.org/P94967 and previous config saved to /var/cache/conftool/dbconfig/20260721-110520-cwilliams.json 2439290 wikitext text/x-wiki gi38jqc67q3o3fgnbrdbjrflaaqc3d2 2439291 2439290 2026-07-21T11:05:41Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance 2439291 wikitext text/x-wiki hcjxze5x8kbhcg5szyno5memr6rmrs2 2439292 2439291 2026-07-21T11:05:50Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1168 (T431660)', diff saved to https://phabricator.wikimedia.org/P94968 and previous config saved to /var/cache/conftool/dbconfig/20260721-110548-cwilliams.json 2439292 wikitext text/x-wiki 9kc8wjyzhrj4dbn5prrhg1k9v03tas7 2439293 2439292 2026-07-21T11:07:21Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1093 hosts 2439293 wikitext text/x-wiki 8cnv0p559cyf185pbbrunajdoaq2qqe 2439294 2439293 2026-07-21T11:10:41Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2439294 wikitext text/x-wiki nhfjzu903ti561gtopo3bz00xwpckqb 2439295 2439294 2026-07-21T11:10:45Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2439295 wikitext text/x-wiki hox3lv2kr4y7ptu4opeg3n4n4z3s803 2439297 2439295 2026-07-21T11:12:43Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T431660)', diff saved to https://phabricator.wikimedia.org/P94969 and previous config saved to /var/cache/conftool/dbconfig/20260721-111242-cwilliams.json 2439297 wikitext text/x-wiki hjwkrk91gsl3io48dsydg9o6xdj2dc4 2439298 2439297 2026-07-21T11:14:48Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P94970 and previous config saved to /var/cache/conftool/dbconfig/20260721-111446-cwilliams.json 2439298 wikitext text/x-wiki 82y3h0f0jd5vhq7m5yp5clvoy7tyec1 2439299 2439298 2026-07-21T11:21:32Z Stashbot 7414 XioNoX: put eqiad-drmrs Arelion link in service 2439299 wikitext text/x-wiki c2pzttjqrysr773ggto8g61zh1igmcu 2439300 2439299 2026-07-21T11:22:51Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P94971 and previous config saved to /var/cache/conftool/dbconfig/20260721-112250-cwilliams.json 2439300 wikitext text/x-wiki cn2uc4nxqb9okxe8ck4u0gwkea5buu1 2439301 2439300 2026-07-21T11:24:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P94972 and previous config saved to /var/cache/conftool/dbconfig/20260721-112453-cwilliams.json 2439301 wikitext text/x-wiki 8zwzbbh0cy5e1re486f1of7lvb3yhyf 2439302 2439301 2026-07-21T11:32:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P94973 and previous config saved to /var/cache/conftool/dbconfig/20260721-113258-cwilliams.json 2439302 wikitext text/x-wiki qk6arj5c0byc2boe8m4bwzyad61z0xf 2439303 2439302 2026-07-21T11:35:03Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T431660)', diff saved to https://phabricator.wikimedia.org/P94974 and previous config saved to /var/cache/conftool/dbconfig/20260721-113501-cwilliams.json 2439303 wikitext text/x-wiki hydkfx7n5eqm4r7f9wr00kjs05piwrq 2439304 2439303 2026-07-21T11:35:24Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance 2439304 wikitext text/x-wiki 544w8x5twuh80q2vprx9jsjwu61d3d8 2439305 2439304 2026-07-21T11:35:32Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2180 (T431660)', diff saved to https://phabricator.wikimedia.org/P94975 and previous config saved to /var/cache/conftool/dbconfig/20260721-113530-cwilliams.json 2439305 wikitext text/x-wiki iyuvuu9fa4peihlwmq244sxukfqd2kt 2439306 2439305 2026-07-21T11:42:17Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T431660)', diff saved to https://phabricator.wikimedia.org/P94976 and previous config saved to /var/cache/conftool/dbconfig/20260721-114215-cwilliams.json 2439306 wikitext text/x-wiki crebk6ljgtr1f0dc9kj4tr4nogtnpwm 2439307 2439306 2026-07-21T11:43:07Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T431660)', diff saved to https://phabricator.wikimedia.org/P94977 and previous config saved to /var/cache/conftool/dbconfig/20260721-114305-cwilliams.json 2439307 wikitext text/x-wiki 3s9hjiy145bw58hhhhbc4uer29njoyu 2439308 2439307 2026-07-21T11:43:26Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance 2439308 wikitext text/x-wiki 7ov0twg1moxjhc1dr1oc1bjfdqtit49 2439309 2439308 2026-07-21T11:43:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1180 (T431660)', diff saved to https://phabricator.wikimedia.org/P94978 and previous config saved to /var/cache/conftool/dbconfig/20260721-114333-cwilliams.json 2439309 wikitext text/x-wiki cyrbzb2fge98waxi70sjj7dmvmhrb1i 2439310 2439309 2026-07-21T11:52:25Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P94979 and previous config saved to /var/cache/conftool/dbconfig/20260721-115223-cwilliams.json 2439310 wikitext text/x-wiki f1028v5chshbt85imarab75geecvc73 2439311 2439310 2026-07-21T12:02:32Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P94980 and previous config saved to /var/cache/conftool/dbconfig/20260721-120231-cwilliams.json 2439311 wikitext text/x-wiki iobgmz41ltm6oheteseqjppukttnbbb 2439313 2439311 2026-07-21T12:12:40Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T431660)', diff saved to https://phabricator.wikimedia.org/P94982 and previous config saved to /var/cache/conftool/dbconfig/20260721-121239-cwilliams.json 2439313 wikitext text/x-wiki 60hpkdcs5j2y784viiwrizlw1gatxe9 2439314 2439313 2026-07-21T12:12:51Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance 2439314 wikitext text/x-wiki ilk2qvaspvzw7dq2nxlraspsqau3j39 2439315 2439314 2026-07-21T12:12:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2193 (T431660)', diff saved to https://phabricator.wikimedia.org/P94983 and previous config saved to /var/cache/conftool/dbconfig/20260721-121257-cwilliams.json 2439315 wikitext text/x-wiki rf5ggl7c9jn98ygfritmrw1z56fpvdy 2439316 2439315 2026-07-21T12:17:35Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2007.codfw.wmnet with OS bookworm 2439316 wikitext text/x-wiki plxqvt5urnv1jih5y931n7s1yprtthq 2439318 2439316 2026-07-21T12:19:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T431660)', diff saved to https://phabricator.wikimedia.org/P94985 and previous config saved to /var/cache/conftool/dbconfig/20260721-121953-cwilliams.json 2439318 wikitext text/x-wiki s1h66hlkplamq764bph4c9byxcgr89o 2439320 2439318 2026-07-21T12:30:03Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P94986 and previous config saved to /var/cache/conftool/dbconfig/20260721-123001-cwilliams.json 2439320 wikitext text/x-wiki co20zl9givzpo067w34v71dvfwk286w 2439321 2439320 2026-07-21T12:31:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2439321 wikitext text/x-wiki r60zf7cze3dqmp58vg4bs2zxf7c9dhs 2439322 2439321 2026-07-21T12:31:19Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2439322 wikitext text/x-wiki bvqqcpojurqn0yf5u8ad3bceiiv9z4u 2439323 2439322 2026-07-21T12:31:37Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2439323 wikitext text/x-wiki nrtj8xvdc3nqn3nx4e2hsf0hcr65cle 2439324 2439323 2026-07-21T12:31:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2439324 wikitext text/x-wiki quw9wplz7fcbdltslwdrhn2wsplx1pq 2439325 2439324 2026-07-21T12:32:55Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 2 hosts 2439325 wikitext text/x-wiki d099shbc2gev1x1y0c6ayr53qvvfffp 2439326 2439325 2026-07-21T12:33:07Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 2 hosts 2439326 wikitext text/x-wiki 3at0i5bxi4u1pdtw91moc46tz1p8ewm 2439327 2439326 2026-07-21T12:33:28Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 3 hosts 2439327 wikitext text/x-wiki eenuuu63oy6p96850vtu0j15yf9dbx2 2439328 2439327 2026-07-21T12:34:20Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 3 hosts 2439328 wikitext text/x-wiki 5ptuducj74422br7th56z8tazgdzdrb 2439329 2439328 2026-07-21T12:34:33Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2439329 wikitext text/x-wiki 164on0ox0utf7anid34r6mrtx60q95t 2439330 2439329 2026-07-21T12:34:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2439330 wikitext text/x-wiki oj3b43n7d0dm872o1p3af7xmrqubfd5 2439331 2439330 2026-07-21T12:34:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2439331 wikitext text/x-wiki kru91ixyseeckptbxzjotfhw9udhoh3 2439332 2439331 2026-07-21T12:35:01Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2439332 wikitext text/x-wiki 7j4avxi8g9atsmgsaqa3m3yy4ph0288 2439334 2439332 2026-07-21T12:40:11Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P94987 and previous config saved to /var/cache/conftool/dbconfig/20260721-124009-cwilliams.json 2439334 wikitext text/x-wiki 4lqhjmat7wmjjnl0zq51iqnrbxzc0mz 2439335 2439334 2026-07-21T12:43:50Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T431660)', diff saved to https://phabricator.wikimedia.org/P94988 and previous config saved to /var/cache/conftool/dbconfig/20260721-124348-cwilliams.json 2439335 wikitext text/x-wiki s93k01gtbquophxg9rpqnx4ggiw4i9q 2439336 2439335 2026-07-21T12:48:28Z Stashbot 7414 elukey: bmc cold reboot for lvs1013 and lvs1015 - T426180 2439336 wikitext text/x-wiki 8lwr2va0tx44d9qdhsjhz13lgzca5cj 2439337 2439336 2026-07-21T12:50:19Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T431660)', diff saved to https://phabricator.wikimedia.org/P94989 and previous config saved to /var/cache/conftool/dbconfig/20260721-125017-cwilliams.json 2439337 wikitext text/x-wiki 0m2smwdzuhyrsvrdwz1eutojo81jwu3 2439338 2439337 2026-07-21T12:50:43Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance 2439338 wikitext text/x-wiki ei4onb1vsdprgehtjygnhyaihh7kpjr 2439339 2439338 2026-07-21T12:50:50Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2217 (T431660)', diff saved to https://phabricator.wikimedia.org/P94990 and previous config saved to /var/cache/conftool/dbconfig/20260721-125049-cwilliams.json 2439339 wikitext text/x-wiki 4b7se1t8hkyscw4m1u7rb6aoexi5904 2439341 2439339 2026-07-21T12:53:58Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P94991 and previous config saved to /var/cache/conftool/dbconfig/20260721-125356-cwilliams.json 2439341 wikitext text/x-wiki 60w8yy7gqzox983u3xvesf5j96ztm8x 2439343 2439341 2026-07-21T12:58:03Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T431660)', diff saved to https://phabricator.wikimedia.org/P94992 and previous config saved to /var/cache/conftool/dbconfig/20260721-125801-cwilliams.json 2439343 wikitext text/x-wiki jknc602lidv3w896zmhrbr3h8o1fwd2 2439344 2439343 2026-07-21T13:02:37Z Stashbot 7414 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2439344 wikitext text/x-wiki 2cg0wbcw0clhmfb7emajoupoiulhxtn 2439345 2439344 2026-07-21T13:02:58Z Stashbot 7414 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2439345 wikitext text/x-wiki n9au3trgb45dw3v5rzngkm9mspelqp4 2439346 2439345 2026-07-21T13:03:06Z Stashbot 7414 blake@deploy2003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2439346 wikitext text/x-wiki r0tyxpk4l2tauvhjmjfpj5ubem34j5l 2439347 2439346 2026-07-21T13:03:18Z Stashbot 7414 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2439347 wikitext text/x-wiki k0vivq9gj9npvwzoe347yhpyagk43vd 2439348 2439347 2026-07-21T13:04:05Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P94993 and previous config saved to /var/cache/conftool/dbconfig/20260721-130404-cwilliams.json 2439348 wikitext text/x-wiki kb7lg6nikpn1wzi6lz94xf0xzl8e07l 2439349 2439348 2026-07-21T13:07:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439349 wikitext text/x-wiki gyfbv75n2fyp9c536kawo7328v68bh3 2439350 2439349 2026-07-21T13:08:10Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P94994 and previous config saved to /var/cache/conftool/dbconfig/20260721-130809-cwilliams.json 2439350 wikitext text/x-wiki b5f6u6yb2w4t2xfdptkxy0hfw2vxeg2 2439354 2439350 2026-07-21T13:13:15Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs1023.eqiad.wmnet, repooling source-only afterwards 2439354 wikitext text/x-wiki a7dk1f8ccpyz07tykp9eeypgfhyt5ld 2439355 2439354 2026-07-21T13:14:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T431660)', diff saved to https://phabricator.wikimedia.org/P94995 and previous config saved to /var/cache/conftool/dbconfig/20260721-131411-cwilliams.json 2439355 wikitext text/x-wiki tdonptake86pw7sp7i4xblvvtu2typy 2439356 2439355 2026-07-21T13:15:27Z Stashbot 7414 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1313092|diff: Mark UnifiedDiffFormatter as stable to extend (T432457)]], [[gerrit:1313093|Add RevisionSnippetGenerator service (T432457)]] 2439356 wikitext text/x-wiki 5e86ul5672ns6qx90swl8b8euwldfv4 2439357 2439356 2026-07-21T13:15:29Z Stashbot 7414 brouberol@dns1004: START - running authdns-update 2439357 wikitext text/x-wiki k03z2i3dyodkne2gn9qnh7bm2wpeceo 2439358 2439357 2026-07-21T13:17:26Z Stashbot 7414 kharlan@deploy2003: kharlan: Backport for [[gerrit:1313092|diff: Mark UnifiedDiffFormatter as stable to extend (T432457)]], [[gerrit:1313093|Add RevisionSnippetGenerator service (T432457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439358 wikitext text/x-wiki slbrkhzdao2ay4kxpqjt78ozcudieps 2439359 2439358 2026-07-21T13:17:31Z Stashbot 7414 brouberol@dns1004: END - running authdns-update 2439359 wikitext text/x-wiki gp10txhnf56n5gekj183408jtxbplwa 2439360 2439359 2026-07-21T13:18:19Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P94996 and previous config saved to /var/cache/conftool/dbconfig/20260721-131817-cwilliams.json 2439360 wikitext text/x-wiki j4ike7joedkdq2hh64scg2bpm19w9sa 2439361 2439360 2026-07-21T13:18:56Z Stashbot 7414 kharlan@deploy2003: kharlan: Continuing with deployment 2439361 wikitext text/x-wiki o2gav71cn10hsj244mw4d3q5ugjpll1 2439362 2439361 2026-07-21T13:20:37Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply 2439362 wikitext text/x-wiki l5arv102ep8y8ngj3atqrsq5wrd0r1i 2439363 2439362 2026-07-21T13:23:17Z Stashbot 7414 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313092|diff: Mark UnifiedDiffFormatter as stable to extend (T432457)]], [[gerrit:1313093|Add RevisionSnippetGenerator service (T432457)]] (duration: 07m 50s) 2439363 wikitext text/x-wiki ptrdv9ro1q26qs0qnurxltuezcyhhxz 2439364 2439363 2026-07-21T13:28:16Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply 2439364 wikitext text/x-wiki 4a54cfqflonq98w93ge5vwn7b00spyr 2439365 2439364 2026-07-21T13:28:28Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T431660)', diff saved to https://phabricator.wikimedia.org/P94998 and previous config saved to /var/cache/conftool/dbconfig/20260721-132826-cwilliams.json 2439365 wikitext text/x-wiki r6bpddoxa81ozqcpo3ru73g6cgqqvpp 2439366 2439365 2026-07-21T13:28:49Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance 2439366 wikitext text/x-wiki neats90trdbtc3fq9rqcldookkubul2 2439367 2439366 2026-07-21T13:28:57Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2224 (T431660)', diff saved to https://phabricator.wikimedia.org/P94999 and previous config saved to /var/cache/conftool/dbconfig/20260721-132855-cwilliams.json 2439367 wikitext text/x-wiki d7tf0hycx0qzdbn2iu235bfvfchbqg4 2439368 2439367 2026-07-21T13:30:57Z Stashbot 7414 blake@deploy2003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2439368 wikitext text/x-wiki q6m52se7982pgen7rriw9n93g6mz942 2439369 2439368 2026-07-21T13:30:59Z Stashbot 7414 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2439369 wikitext text/x-wiki 1s9its0oufbehps4iwozfdyhs8qqca3 2439370 2439369 2026-07-21T13:33:13Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1012.eqiad.wmnet with OS bookworm 2439370 wikitext text/x-wiki 0cb10b87d66jodgzvlyeiu83ow6mqbi 2439371 2439370 2026-07-21T13:33:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1012 2439371 wikitext text/x-wiki bfr62ndjw1jjvmd3fplvxr2oawivvwh 2439372 2439371 2026-07-21T13:33:38Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1012 2439372 wikitext text/x-wiki oyrp6rtlrv7pqffryeqfq65oel3am6x 2439373 2439372 2026-07-21T13:35:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T431660)', diff saved to https://phabricator.wikimedia.org/P95000 and previous config saved to /var/cache/conftool/dbconfig/20260721-133557-cwilliams.json 2439373 wikitext text/x-wiki a90i88bv4zyu6cipvyni99evlvu9isc 2439374 2439373 2026-07-21T13:37:22Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2032.codfw.wmnet 2439374 wikitext text/x-wiki iwyeuqv8xvhohlwns8j70bkkz8ds9zn 2439375 2439374 2026-07-21T13:38:54Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2439375 wikitext text/x-wiki go3yskqq9q5bksxk5y3pmiunzp8ni9d 2439376 2439375 2026-07-21T13:38:56Z Stashbot 7414 jiji@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2439376 wikitext text/x-wiki fd67xeq084w9p26mw7rx2wkzd4hqbv8 2439377 2439376 2026-07-21T13:38:58Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet 2439377 wikitext text/x-wiki 4h2l6tlf0tuyflqof8cc54yizcqxwaq 2439378 2439377 2026-07-21T13:39:02Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2439378 wikitext text/x-wiki 4trdhqmngs9cz1hr5mefvs1panahr6x 2439379 2439378 2026-07-21T13:39:04Z Stashbot 7414 jiji@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2439379 wikitext text/x-wiki lg06acxuhy86c9b8miytp24hl0fxeo8 2439380 2439379 2026-07-21T13:40:22Z Stashbot 7414 jiji@deploy2003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2439380 wikitext text/x-wiki a4w70pfaubih9a70jv3o4ps1yfw87zw 2439381 2439380 2026-07-21T13:40:25Z Stashbot 7414 jiji@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2439381 wikitext text/x-wiki 3l1z6wc2dyeo5qkdj6sn2k204v77978 2439383 2439381 2026-07-21T13:43:46Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2439383 wikitext text/x-wiki pek8haczyteibo8ogsnyv92ui30zxzy 2439384 2439383 2026-07-21T13:45:20Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet 2439384 wikitext text/x-wiki o0mys9t2xjgf41khswtf9080a79jnlv 2439385 2439384 2026-07-21T13:45:22Z Stashbot 7414 cmooney@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2033.codfw.wmnet 2439385 wikitext text/x-wiki gweevngps9xx5srakht8cgrsguasudy 2439386 2439385 2026-07-21T13:45:49Z Stashbot 7414 elukey: move the Docker Registry's /v2/wikimedia/machinelearning.* prefix to the ml S3 backend - T428022 2439386 wikitext text/x-wiki rph4925il44faxog0v8vl6pfiazldg0 2439387 2439386 2026-07-21T13:46:00Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet 2439387 wikitext text/x-wiki cx2h3ovprtsw941jy1h969qu54g3lk6 2439388 2439387 2026-07-21T13:46:02Z Stashbot 7414 cmooney@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2033.codfw.wmnet 2439388 wikitext text/x-wiki m4ev2o55jxdklet38npm8cuwqklnot5 2439389 2439388 2026-07-21T13:46:06Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P95001 and previous config saved to /var/cache/conftool/dbconfig/20260721-134605-cwilliams.json 2439389 wikitext text/x-wiki sod5t0pm4qeog7opb78smmqf4qtj99c 2439390 2439389 2026-07-21T13:46:10Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2439390 wikitext text/x-wiki 14zvuvrmnjtn7vf1ivu40zpz6lib6xd 2439391 2439390 2026-07-21T13:48:23Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2439391 wikitext text/x-wiki 8stypdiibpcfq6qk5s7y79m2ni91fix 2439393 2439391 2026-07-21T13:48:57Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2439393 wikitext text/x-wiki gzeyn8qbr5gblsqctwwjudegzg7dtex 2439394 2439393 2026-07-21T13:51:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1012.eqiad.wmnet with reason: host reimage 2439394 wikitext text/x-wiki 9nfi7g9zmcd8k0d2fqqv7tujdh63055 2439395 2439394 2026-07-21T13:53:48Z Stashbot 7414 cmooney@cumin1003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on 30 hosts with reason: lsw1-b7-codfw JunOS upgrade 2439395 wikitext text/x-wiki sr8m05znl3yweqiwka4aikug6cjjkpk 2439396 2439395 2026-07-21T13:54:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1012.eqiad.wmnet with reason: host reimage 2439396 wikitext text/x-wiki sk5kie4pxv03ud981pc9f0ehae3uff2 2439397 2439396 2026-07-21T13:56:14Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P95003 and previous config saved to /var/cache/conftool/dbconfig/20260721-135613-cwilliams.json 2439397 wikitext text/x-wiki 4dvwvryifa5np14quipct1jene8rx4x 2439398 2439397 2026-07-21T13:57:39Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: lsw1-b7-codfw JunOS upgrade 2439398 wikitext text/x-wiki 7z7zn4t99nrh2lh9l1k21ix1ou63naz 2439399 2439398 2026-07-21T13:57:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs1023.eqiad.wmnet, repooling source-only afterwards 2439399 wikitext text/x-wiki ffhtvudsw7db5fo12qcflrsuk2kszgw 2439400 2439399 2026-07-21T13:59:49Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b7-codfw,lsw1-b7-codfw IPv6,lsw1-b7-codfw.mgmt,ssw1-a[1,8]-codfw.mgmt with reason: lsw1-b7-codfw JunOS upgrade 2439400 wikitext text/x-wiki gf72bj2jq1qriinumwxagodealzlkag 2439401 2439400 2026-07-21T14:00:26Z Stashbot 7414 Dreamy_Jazz: Created cusi_case, cusi_signal, and cusi_user on svwiki, dewiki, jawiki, eswiki 2439401 wikitext text/x-wiki gyms2k3yq77x69hq29nnttza56k8bwg 2439402 2439401 2026-07-21T14:00:41Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2139-2140,2157,2284-2285].codfw.wmnet 2439402 wikitext text/x-wiki ajth5s0gr5ptpchugkrx8kz5tj4m6ud 2439403 2439402 2026-07-21T14:02:27Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2049.codfw.wmnet 2439403 wikitext text/x-wiki 686cfudnidm6j2qv3u863xr93q28207 2439404 2439403 2026-07-21T14:03:29Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet 2439404 wikitext text/x-wiki 6ui9e8ny5wyud63nd88k9ffi1lfrkl5 2439405 2439404 2026-07-21T14:03:30Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet 2439405 wikitext text/x-wiki lhudgq9gtfp9w0rchyerx42to4u3lru 2439406 2439405 2026-07-21T14:03:32Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2139-2140,2157,2284-2285].codfw.wmnet 2439406 wikitext text/x-wiki ed75m99gyph5kidg51gmaxfzbkj4i61 2439407 2439406 2026-07-21T14:04:01Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply 2439407 wikitext text/x-wiki 21ma7q9a3z53vyvy5cwezredunb2r0u 2439408 2439407 2026-07-21T14:04:06Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply 2439408 wikitext text/x-wiki bdbc48jusrk6vvo4340daz71v5fsstd 2439409 2439408 2026-07-21T14:04:21Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2009.codfw.wmnet 2439409 wikitext text/x-wiki pk2m26pcfow20pd0pe8z7zu5vbws2ap 2439410 2439409 2026-07-21T14:05:20Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=eqiad 2439410 wikitext text/x-wiki lkfd0upz9rtruzpuby3qvp68lozwu7z 2439411 2439410 2026-07-21T14:05:34Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2049.codfw.wmnet 2439411 wikitext text/x-wiki hffxzmnw45pl7h6sx3ar455bikk07kq 2439412 2439411 2026-07-21T14:06:22Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T431660)', diff saved to https://phabricator.wikimedia.org/P95005 and previous config saved to /var/cache/conftool/dbconfig/20260721-140620-cwilliams.json 2439412 wikitext text/x-wiki 9rvra70iazpxrfraoxcigezbfpbgkim 2439413 2439412 2026-07-21T14:07:06Z Stashbot 7414 cmooney@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on aux-k8s-etcd2004.codfw.wmnet,ml-etcd2001.codfw.wmnet with reason: lsw1-b7-codfw JunOS upgrade 2439413 wikitext text/x-wiki 9b9t9n4gux4ym92ctevsiev1nj0woia 2439414 2439413 2026-07-21T14:08:37Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet 2439414 wikitext text/x-wiki ajzkbsobn2ohabmvi3983iplfh66dww 2439415 2439414 2026-07-21T14:12:20Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool pc2017: codfw rack B7 depool for maintenance 2439415 wikitext text/x-wiki ng3h2qzl7awgufxfb5heyen70brc6pv 2439416 2439415 2026-07-21T14:12:21Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.parsercache 2439416 wikitext text/x-wiki h89do4ywj3w4on4r6teyxlr37xci9sl 2439417 2439416 2026-07-21T14:12:24Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439417 wikitext text/x-wiki b32uqwwnw8r5j9nv89nkk4u9fz5n7im 2439418 2439417 2026-07-21T14:12:25Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2017: codfw rack B7 depool for maintenance 2439418 wikitext text/x-wiki 4lj38rddhwv6twq9398zmo99ntebsvn 2439419 2439418 2026-07-21T14:12:29Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool db2229: codfw rack B7 depool for maintenance 2439419 wikitext text/x-wiki 9nw5prmp7qxs4w044nbjnc8x66avh6s 2439420 2439419 2026-07-21T14:12:42Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2229: codfw rack B7 depool for maintenance 2439420 wikitext text/x-wiki 4iad278ns7mdb32eke80eg1i4trqcek 2439421 2439420 2026-07-21T14:12:46Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool db2228: codfw rack B7 depool for maintenance 2439421 wikitext text/x-wiki ayjli4x45aeqssnynbn0oswqjfppbkc 2439422 2439421 2026-07-21T14:12:59Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: codfw rack B7 depool for maintenance 2439422 wikitext text/x-wiki 5j8insxpv3kfz3v0eyf458w7gxmbvzb 2439423 2439422 2026-07-21T14:13:04Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool db2242: codfw rack B7 depool for maintenance 2439423 wikitext text/x-wiki sf1xwgps8mj3edqhx6tqne11e4nnrx3 2439424 2439423 2026-07-21T14:13:17Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2242: codfw rack B7 depool for maintenance 2439424 wikitext text/x-wiki oo4d6oso98ops4pd5aif7ucu63ucrx8 2439425 2439424 2026-07-21T14:14:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1180: Security update 2439425 wikitext text/x-wiki 93j0u03fxnrglp9z96sgzp1gbp16xuk 2439426 2439425 2026-07-21T14:14:32Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2009.codfw.wmnet 2439426 wikitext text/x-wiki 2o1t83mdwxf6shbdg4rx4otks7ghfgp 2439427 2439426 2026-07-21T14:18:41Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet 2439427 wikitext text/x-wiki dem8g51h95ya1lo07z9mxh7wmb7g52c 2439428 2439427 2026-07-21T14:23:34Z Stashbot 7414 topranks: reboot lsw1-b7-codfw to upgrade JunOS (affects all hosts in rack) T430928 2439428 wikitext text/x-wiki j42h5qed6oku4woshekp9ztooffeosj 2439430 2439428 2026-07-21T14:24:32Z Stashbot 7414 dreamyjazz@deploy2003: Started scap sync-world: Backport for [[gerrit:1313194|CheckUser Suggested Investigations: Enable for 4 more wikis]] 2439430 wikitext text/x-wiki mp8wd0dz0g5eee27plvcajb2luu1znv 2439431 2439430 2026-07-21T14:26:35Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Backport for [[gerrit:1313194|CheckUser Suggested Investigations: Enable for 4 more wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439431 wikitext text/x-wiki 48ekyd73zlhte7c81bw9v07f56u1fnz 2439432 2439431 2026-07-21T14:28:01Z Stashbot 7414 elukey@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2439432 wikitext text/x-wiki tvudwdh7cyo36n50ysjkjltni1lvofw 2439433 2439432 2026-07-21T14:28:11Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment 2439433 wikitext text/x-wiki 2r5qxeslw14a23t47n6zlx23t13bkxs 2439434 2439433 2026-07-21T14:29:05Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ssw1-a[1,8]-codfw with reason: lsw1-b7-codfw JunOS upgrade 2439434 wikitext text/x-wiki 7bua1a8li3ij1k10stva28kqp1puf7l 2439437 2439434 2026-07-21T14:32:28Z Stashbot 7414 dreamyjazz@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313194|CheckUser Suggested Investigations: Enable for 4 more wikis]] (duration: 07m 56s) 2439437 wikitext text/x-wiki t95cr7ewkgmi9thc4lr4ja6fyzqbo6h 2439438 2439437 2026-07-21T14:34:33Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2439438 wikitext text/x-wiki jccce3lhx7g1s8k1zs8e5bu2e6odj5v 2439439 2439438 2026-07-21T14:35:06Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2439439 wikitext text/x-wiki ehxn46948k2y9764dzq7sudx4bif1y3 2439441 2439439 2026-07-21T14:40:36Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.hosts.remove-downtime for 29 hosts 2439441 wikitext text/x-wiki ho0xv0bax3drf28j4d6uhpvjo9awhwk 2439442 2439441 2026-07-21T14:40:53Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 29 hosts 2439442 wikitext text/x-wiki b7hgrnppdxhevbk89qnevqov2lt4nsr 2439443 2439442 2026-07-21T14:41:48Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2139-2140,2157,2284-2285].codfw.wmnet 2439443 wikitext text/x-wiki ptg0byctq0obu2fdmx9skbwxswhrlmd 2439444 2439443 2026-07-21T14:41:52Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2139-2140,2157,2284-2285].codfw.wmnet 2439444 wikitext text/x-wiki 4706jjznrmh6c1rg7zemmdypp5ue46j 2439445 2439444 2026-07-21T14:42:43Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2009.codfw.wmnet 2439445 wikitext text/x-wiki 2dyed3rk0kqrsce8f5b1jk8u5s7onwi 2439446 2439445 2026-07-21T14:42:45Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2009.codfw.wmnet 2439446 wikitext text/x-wiki l0pjbxm7ubvmh4bzxv6tq7k464lolgq 2439447 2439446 2026-07-21T14:43:00Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet 2439447 wikitext text/x-wiki 0irgdaba9wqmc1vz740geo06bf4uqr5 2439448 2439447 2026-07-21T14:43:02Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet 2439448 wikitext text/x-wiki r0rgu1b1u34dljnawvd0ious0ultil7 2439449 2439448 2026-07-21T14:43:54Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool pc2017: codfw rack B7 depool for maintenance 2439449 wikitext text/x-wiki ovjugcffywefj9ekj6m4ktp1oo4vmgy 2439450 2439449 2026-07-21T14:43:55Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.parsercache 2439450 wikitext text/x-wiki pzrhrdf154uhashj96eebqrdziywa5k 2439451 2439450 2026-07-21T14:44:06Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439451 wikitext text/x-wiki mwa1ja3q6ptbkz0y6rfug0d64q24jjk 2439452 2439451 2026-07-21T14:44:08Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2017: codfw rack B7 depool for maintenance 2439452 wikitext text/x-wiki mx79fx8v0kvefs9mtixwru8d9p7x60e 2439453 2439452 2026-07-21T14:44:11Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool db2229: codfw rack B7 depool for maintenance 2439453 wikitext text/x-wiki 3z2tskt03p4mqkexownwxcxefy8lo8g 2439454 2439453 2026-07-21T14:48:12Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d1-eqiad 2439454 wikitext text/x-wiki 0lho5s3ye2mb2mae558cidi18piy1ao 2439455 2439454 2026-07-21T14:50:30Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d1-eqiad 2439455 wikitext text/x-wiki arvtk5sth0ekcmazu4v12uhqqrf6aup 2439461 2439455 2026-07-21T15:13:17Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply 2439461 wikitext text/x-wiki pc5i44rvm2w7juxepxu96nurth0i57e 2439462 2439461 2026-07-21T15:14:20Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1180: Security update 2439462 wikitext text/x-wiki 0746g2cvc5n1rvqzklnqkrfi80off1u 2439463 2439462 2026-07-21T15:14:58Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply 2439463 wikitext text/x-wiki lg68maead3gz7bv28dubfacgzwghs93 2439465 2439463 2026-07-21T15:19:32Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2439465 wikitext text/x-wiki k7kewopxvxxmgo5j5qd30qg9hnyems5 2439466 2439465 2026-07-21T15:19:40Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 24s) 2439466 wikitext text/x-wiki 50jngoxxl3ryngsft6ihhkl960cog4z 2439470 2439466 2026-07-21T15:21:19Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply 2439470 wikitext text/x-wiki 0z6egt80zdfuk6de60ea7ayi7wlttha 2439472 2439470 2026-07-21T15:21:56Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply 2439472 wikitext text/x-wiki at22kckroaxe8x17xld5ily93021jcw 2439476 2439472 2026-07-21T15:25:31Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1180: Security update 2439476 wikitext text/x-wiki p6nzfv5y4rknlrbj4a1ts8pzq53odo5 2439477 2439476 2026-07-21T15:25:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1180: Security update 2439477 wikitext text/x-wiki 16s47zuco9lcojtzzfyhgkfsv6vcjn4 2439479 2439477 2026-07-21T15:27:17Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439479 wikitext text/x-wiki a748ocl8s2qhxexxtfq6dekra73c27l 2439483 2439479 2026-07-21T15:29:23Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2229: codfw rack B7 depool for maintenance 2439483 wikitext text/x-wiki pqh4i3icmtp2mld3qhek6p74pcu7s9p 2439484 2439483 2026-07-21T15:29:28Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool db2228: codfw rack B7 depool for maintenance 2439484 wikitext text/x-wiki rots7qv3rergrm5dyulyv0ubpud36aq 2439485 2439484 2026-07-21T15:29:58Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439485 wikitext text/x-wiki pff763gdjtmurepgf3b0i9x71n61ueg 2439488 2439485 2026-07-21T15:34:42Z Stashbot 7414 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311536|Set $wgMathInternalRestbaseURL explicitly (take 2) (T349582)]] 2439488 wikitext text/x-wiki bj081kaiz0za7qy93ntr2k9htyse0qg 2439489 2439488 2026-07-21T15:39:18Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2439489 wikitext text/x-wiki pmowih9k426ivletw1vtb6q7pddfavo 2439490 2439489 2026-07-21T15:39:26Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s) 2439490 wikitext text/x-wiki 65uzymhxgdst2u9t7y7bux6wllneatl 2439491 2439490 2026-07-21T15:39:27Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp3073.* 2439491 wikitext text/x-wiki f4hngnaytx9ztfr82rx9nbxta8qt6wk 2439492 2439491 2026-07-21T15:48:08Z Stashbot 7414 inflatador: bking@apt1002 `sudo reprepro copy bookworm-wikimedia bullseye-wikimedia jvmquake` T430880 2439492 wikitext text/x-wiki 1u1nxcaq582mobiu4tnb2njxhy1q639 2439495 2439492 2026-07-21T16:03:15Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1012.eqiad.wmnet with OS bookworm 2439495 wikitext text/x-wiki kvooubnmi82cnzhx3ylw2bs3c9n7cnd 2439496 2439495 2026-07-21T16:03:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling source-only afterwards 2439496 wikitext text/x-wiki ihnqrn87bwyg5sle4uh2vwg13g6fmkw 2439497 2439496 2026-07-21T16:08:08Z Stashbot 7414 kamila@deploy2003: Locking from deployment [MediaWiki]: deploy1003 reimage 2439497 wikitext text/x-wiki 3l1zzvvjfgvammbinxnjmh02xnmxtqf 2439499 2439497 2026-07-21T16:08:33Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host deploy1003.eqiad.wmnet with OS bookworm 2439499 wikitext text/x-wiki qathrg0npmu9gv126o28lfvlf1oc7qe 2439500 2439499 2026-07-21T16:13:07Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3073.esams.wmnet with OS trixie 2439500 wikitext text/x-wiki 0o84s7v58rtj9f48cb4bj0cv8fibwgm 2439501 2439500 2026-07-21T16:13:35Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1023.eqiad.wmnet with OS bookworm 2439501 wikitext text/x-wiki 6o6lndghne98310nxccv36067r88yfk 2439502 2439501 2026-07-21T16:14:41Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: codfw rack B7 depool for maintenance 2439502 wikitext text/x-wiki 019guwk9eosh63hmvw4n6begrrc60sm 2439503 2439502 2026-07-21T16:14:45Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool db2242: codfw rack B7 depool for maintenance 2439503 wikitext text/x-wiki c7xrdf83zhme9u20vzpt9o6ycrhi2c9 2439504 2439503 2026-07-21T16:27:07Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on deploy1003.eqiad.wmnet with reason: host reimage 2439504 wikitext text/x-wiki 7eqm0g31ecrglv90jr50bypdgihg4v0 2439505 2439504 2026-07-21T16:32:32Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on deploy1003.eqiad.wmnet with reason: host reimage 2439505 wikitext text/x-wiki c1tbp9qn12sec50vwyh781449x7g1h1 2439507 2439505 2026-07-21T16:39:15Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3073.esams.wmnet with reason: host reimage 2439507 wikitext text/x-wiki 41pbzz0dx8b6351q1sql66degtljntp 2439508 2439507 2026-07-21T16:42:45Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439508 wikitext text/x-wiki bkp84fa8k2an2gq5lj8icswigdot1as 2439509 2439508 2026-07-21T16:43:41Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3073.esams.wmnet with reason: host reimage 2439509 wikitext text/x-wiki 9awqthorc8mmtqxbsm30lgapa78y8gz 2439510 2439509 2026-07-21T16:59:21Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling source-only afterwards 2439510 wikitext text/x-wiki 53al8zpitxi5r17l5t6omjy75dsr6lf 2439511 2439510 2026-07-21T16:59:59Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2242: codfw rack B7 depool for maintenance 2439511 wikitext text/x-wiki j73yr6e68edvabft66sb34e3q2ku3gt 2439512 2439511 2026-07-21T17:00:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling source-only afterwards 2439512 wikitext text/x-wiki olfuw39jlkeygjkfumeuh1txyvk06rv 2439513 2439512 2026-07-21T17:03:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1024.eqiad.wmnet with OS bookworm 2439513 wikitext text/x-wiki 9afnl4a54xhtrl3asxrtfmdrxu2lpjc 2439514 2439513 2026-07-21T17:04:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1024 2439514 wikitext text/x-wiki jlrc4fv6nijmvkpqn2q3c5azhwuewh9 2439515 2439514 2026-07-21T17:04:17Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1024 2439515 wikitext text/x-wiki m4v0o5jgs0h1wrkasu65ozg7zgputg4 2439516 2439515 2026-07-21T17:11:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439516 wikitext text/x-wiki f8nrjflhgytv38evrjy9kv4is0p6g7r 2439517 2439516 2026-07-21T17:11:21Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3073.esams.wmnet with OS trixie 2439517 wikitext text/x-wiki 2ur5qxbgqit3abv2we7u4lh7amllk03 2439518 2439517 2026-07-21T17:13:52Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wcqs1003.eqiad.wmnet 2439518 wikitext text/x-wiki 92ugfjwyohe75xpwzjdx1bxtfpgcerc 2439519 2439518 2026-07-21T17:15:47Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp3073.* 2439519 wikitext text/x-wiki sd5l6zoqss1vi2viq1a664101p6c2e6 2439520 2439519 2026-07-21T17:16:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1180.eqiad.wmnet 2439520 wikitext text/x-wiki 3sjsbyagdshkk5rqjpz26099r8vzxfz 2439521 2439520 2026-07-21T17:16:08Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1180.eqiad.wmnet 2439521 wikitext text/x-wiki il6tg35ekg9o29ig6bbqej98bgwnr7n 2439522 2439521 2026-07-21T17:17:31Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Security updates 2439522 wikitext text/x-wiki 1fugj15djhi93wamc2b7dy00lksnwgh 2439523 2439522 2026-07-21T17:20:38Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wcqs1003.eqiad.wmnet 2439523 wikitext text/x-wiki 0bhx0hqthmmdogg23431y7of7ctdi4a 2439525 2439523 2026-07-21T17:35:28Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet 2439525 wikitext text/x-wiki 40janiwx0jkheb1dinv9h3464hkal7h 2439526 2439525 2026-07-21T17:35:29Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet 2439526 wikitext text/x-wiki brtdqqgybzgq5zxiblddliuccge95in 2439527 2439526 2026-07-21T17:35:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1165.eqiad.wmnet 2439527 wikitext text/x-wiki 3viy96y2mlao3hxx77djd11tiwi39yo 2439528 2439527 2026-07-21T17:36:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1165.eqiad.wmnet 2439528 wikitext text/x-wiki izt03hd3hjbnqsp0r4dua99ctyz5lyh 2439529 2439528 2026-07-21T17:36:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2169.codfw.wmnet 2439529 wikitext text/x-wiki tmihdxwpzdwmcf09o6inpu6z1bqgtjy 2439530 2439529 2026-07-21T17:36:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2169.codfw.wmnet 2439530 wikitext text/x-wiki m9j7u7gbplvwnyyzosw96bwhd7qfgy4 2439531 2439530 2026-07-21T17:36:15Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1168.eqiad.wmnet 2439531 wikitext text/x-wiki ra5p64uo0ealymw2ktdnbszuyyrfnk3 2439532 2439531 2026-07-21T17:36:17Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1168.eqiad.wmnet 2439532 wikitext text/x-wiki 1kpik33ff3aghyzt4x74yv6qglhg1el 2439533 2439532 2026-07-21T17:38:14Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2180.codfw.wmnet 2439533 wikitext text/x-wiki 5bius68gynklvn5vp0pb7yozo6b28q4 2439534 2439533 2026-07-21T17:38:16Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2180.codfw.wmnet 2439534 wikitext text/x-wiki kjpsf1ul67sf0sou45in9fudl76a35c 2439535 2439534 2026-07-21T17:38:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2193.codfw.wmnet 2439535 wikitext text/x-wiki gjhfcgtq38zvudfhy6qnk02foqfh1j3 2439536 2439535 2026-07-21T17:38:18Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2193.codfw.wmnet 2439536 wikitext text/x-wiki rwavhf8bhpabr997689aajblwp3op63 2439537 2439536 2026-07-21T17:38:20Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2217.codfw.wmnet 2439537 wikitext text/x-wiki aupn7la431hvnabohtkr3t7nmo8zeim 2439538 2439537 2026-07-21T17:38:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2217.codfw.wmnet 2439538 wikitext text/x-wiki jn0sr5zbmsykyh7fkgkxt26ko3gsv48 2439539 2439538 2026-07-21T17:38:23Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2224.codfw.wmnet 2439539 wikitext text/x-wiki ghfv4jhnuvgzkuuis0fxl4izwde2brt 2439540 2439539 2026-07-21T17:38:25Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2224.codfw.wmnet 2439540 wikitext text/x-wiki dyjnbsvkhnnrbtiwwgg093sxft70hnj 2439541 2439540 2026-07-21T17:38:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1155.eqiad.wmnet 2439541 wikitext text/x-wiki o6d0cgpr4w87rc89udwsz0ud189wonk 2439542 2439541 2026-07-21T17:38:28Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1155.eqiad.wmnet 2439542 wikitext text/x-wiki fhp0ndypc6tnrqpb95saqxbdo4gpijn 2439543 2439542 2026-07-21T17:48:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wcqs2003.codfw.wmnet 2439543 wikitext text/x-wiki tjzyz54ucfe7d7r948bcri33fj7rkh4 2439544 2439543 2026-07-21T17:55:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wcqs2003.codfw.wmnet 2439544 wikitext text/x-wiki 5sluupdeljetwxq9td1qs6a9lznixh4 2439545 2439544 2026-07-21T18:02:59Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Security updates 2439545 wikitext text/x-wiki aa1h8xl5hro0vwulhpypwuf8qzje4dg 2439546 2439545 2026-07-21T18:03:34Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling source-only afterwards 2439546 wikitext text/x-wiki 149h2xqbsdfspk43lvvyhdc5q3sausi 2439547 2439546 2026-07-21T18:09:24Z Stashbot 7414 kamila@deploy2003: Unlocked for deployment [MediaWiki]: deploy1003 reimage (duration: 121m 16s) 2439547 wikitext text/x-wiki kh70zwzxy4kpic38y4kuu4rdyy8oyg4 2439548 2439547 2026-07-21T18:20:15Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host deploy1003.eqiad.wmnet with OS bookworm 2439548 wikitext text/x-wiki peeqhl504dgze8husy86yplro6ox2ei 2439549 2439548 2026-07-21T18:24:53Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1024.eqiad.wmnet with OS bookworm 2439549 wikitext text/x-wiki 73qwqgsvtzgopxsxu5nosp1cbam672d 2439550 2439549 2026-07-21T18:27:01Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.netbox 2439550 wikitext text/x-wiki q20bxf5n9e2q0jg9qb9r1xz34rcjn4a 2439551 2439550 2026-07-21T18:30:16Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2439551 wikitext text/x-wiki hyi033538wm2pw0mogerul30fk9d09k 2439552 2439551 2026-07-21T18:32:01Z Stashbot 7414 dancy@deploy2003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.12 refs T430831 2439552 wikitext text/x-wiki 00g626pdoi05ld40v5trvfrrowk7vgc 2439553 2439552 2026-07-21T18:32:11Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.netbox 2439553 wikitext text/x-wiki 37d25fcfteczf3dteemni0oo2uuiw94 2439554 2439553 2026-07-21T18:37:18Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding urldownloader service IPs - sukhe@cumin1003" 2439554 wikitext text/x-wiki l9s0h4qjficdr5pvsw36a4en34jo2y4 2439555 2439554 2026-07-21T18:37:22Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding urldownloader service IPs - sukhe@cumin1003" 2439555 wikitext text/x-wiki 6tgmckrb019izyzlc6ndpg2eufkxtpf 2439556 2439555 2026-07-21T18:37:23Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439556 wikitext text/x-wiki qs3ehl733fxvrxcjd1gx5713gvc2856 2439557 2439556 2026-07-21T18:57:56Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.remove-downtime for deploy1003.eqiad.wmnet 2439557 wikitext text/x-wiki 6q1hq6fsgxi34dcfxz33llf08826wk4 2439558 2439557 2026-07-21T18:57:58Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for deploy1003.eqiad.wmnet 2439558 wikitext text/x-wiki 1efluvkobuwsctdo6xqzb242wbx877j 2439560 2439558 2026-07-21T19:08:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439560 wikitext text/x-wiki lyvhfy19fk9fknwdok8i2pwp80oafsa 2439561 2439560 2026-07-21T19:10:22Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs1012.eqiad.wmnet 2439561 wikitext text/x-wiki 5rj5yvzqxflshjnpc2eyirv0q5fqlqz 2439562 2439561 2026-07-21T19:10:47Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=eqiad 2439562 wikitext text/x-wiki 18v8owii9zu5c63i33fj2d9j3fr6b4u 2439564 2439562 2026-07-21T19:11:35Z Stashbot 7414 ryankemper: [wdqs] T430880 Repooled wdqs1012.eqiad.wmnet after successful Bookworm reimage, data transfer, service checks, readiness probe, and cross-graph federation query validation 2439564 wikitext text/x-wiki 57iuroxiii1a1gghcy40xkhjq0514uq 2439566 2439564 2026-07-21T19:12:06Z Stashbot 7414 ryankemper: [wdqs] T430880 Repooled `wdqs-scholarly` discovery in `eqiad` after validating `wdqs1023` end-to-end; `wdqs1024` remains disabled pending reimage recovery 2439566 wikitext text/x-wiki qxvghymbpymerxtrda6kg9bzuwmt2ms 2439568 2439566 2026-07-21T19:24:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs1024'] 2439568 wikitext text/x-wiki cakfyywi5xq6aand122hfba0e9kuf0z 2439569 2439568 2026-07-21T19:26:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439569 wikitext text/x-wiki tug4cp07htlmpwwutvxlvz1fq89yjsw 2439570 2439569 2026-07-21T19:26:55Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs1024'] 2439570 wikitext text/x-wiki hmq9305qde6d0rrmdm7qow6sukcq78t 2439571 2439570 2026-07-21T19:27:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439571 wikitext text/x-wiki 5yatoi439skmpi6dbpqde8st7ygflnh 2439572 2439571 2026-07-21T19:32:45Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['wdqs1024'] 2439572 wikitext text/x-wiki exhrua8pmj4szj284c6gnbcydqrh4fo 2439573 2439572 2026-07-21T19:35:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1024.eqiad.wmnet with OS bookworm 2439573 wikitext text/x-wiki dkj500zkb00mzau1y6dm9lhax5cfp2v 2439574 2439573 2026-07-21T19:45:21Z Stashbot 7414 zabe@deploy2003: Started scap sync-world: Backport for [[gerrit:1312603|Prepare Wikipedia Bole (T429921)]] 2439574 wikitext text/x-wiki ods8w21ysxz3n5i05jq9rdex5hdtg7s 2439575 2439574 2026-07-21T19:47:22Z Stashbot 7414 zabe@deploy2003: zabe: Backport for [[gerrit:1312603|Prepare Wikipedia Bole (T429921)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439575 wikitext text/x-wiki o0qve3qwpedi0ef2it51q82006ygxlu 2439576 2439575 2026-07-21T19:48:19Z Stashbot 7414 zabe@deploy2003: zabe: Continuing with deployment 2439576 wikitext text/x-wiki 97pcr35brvwz5p07dkbr7lavsah2ts5 2439579 2439576 2026-07-21T19:52:47Z Stashbot 7414 zabe@deploy2003: Finished scap sync-world: Backport for [[gerrit:1312603|Prepare Wikipedia Bole (T429921)]] (duration: 07m 25s) 2439579 wikitext text/x-wiki g38fp32wom13j26y4hn9d2a6xhywh6n 2439581 2439579 2026-07-21T19:56:32Z Stashbot 7414 zabe@deploy2003: Started scap sync-world: Backport for [[gerrit:1312604|Activate bolwiki (T429921)]] 2439581 wikitext text/x-wiki ep8mm00l66s7vghm1p9jqtjizscwxku 2439584 2439581 2026-07-21T19:58:29Z Stashbot 7414 zabe@deploy2003: zabe: Backport for [[gerrit:1312604|Activate bolwiki (T429921)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439584 wikitext text/x-wiki bf0z1ofey9qf55f0wngo5gigh6e95yl 2439586 2439584 2026-07-21T19:59:08Z Stashbot 7414 zabe@deploy2003: zabe: Continuing with deployment 2439586 wikitext text/x-wiki a8iya4ohp92axj491gr8ket4cbsnzcb 2439587 2439586 2026-07-21T20:01:08Z Stashbot 7414 mutante: Gerrit - tomorrow a new SSH host key will appear - it will be ed25519 and has already been added to wmf-laptop. you can verify it here: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418 (T240266) 2439587 wikitext text/x-wiki q1uth1ef7qykfdv2f2nb0ozy93x48ba 2439588 2439587 2026-07-21T20:03:36Z Stashbot 7414 zabe@deploy2003: Finished scap sync-world: Backport for [[gerrit:1312604|Activate bolwiki (T429921)]] (duration: 07m 04s) 2439588 wikitext text/x-wiki 7qm4ytbwyzz4twykuqz71c303ay8zl8 2439589 2439588 2026-07-21T20:05:48Z Stashbot 7414 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1313200|Restore the server-side SPARQL endpoint config for Wikidata (T421250)]], [[gerrit:1311489|Article Guidance: migrate wikidata config (T421250)]] 2439589 wikitext text/x-wiki 30h9zzwj1lszg4pe65a6y6yr7fqq2ou 2439590 2439589 2026-07-21T20:07:42Z Stashbot 7414 sbisson@deploy2003: sbisson: Backport for [[gerrit:1313200|Restore the server-side SPARQL endpoint config for Wikidata (T421250)]], [[gerrit:1311489|Article Guidance: migrate wikidata config (T421250)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439590 wikitext text/x-wiki pupq3j5fl5fujzsepxn323lrm1qonk2 2439591 2439590 2026-07-21T20:10:28Z Stashbot 7414 sbisson@deploy2003: sbisson: Continuing with deployment 2439591 wikitext text/x-wiki 04bhm0cnjgnqebed8wv0monaow8t0gh 2439592 2439591 2026-07-21T20:11:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2020.codfw.wmnet with OS bookworm 2439592 wikitext text/x-wiki dr256s1ves5zjwzefox05nzmowwwx68 2439593 2439592 2026-07-21T20:14:48Z Stashbot 7414 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313200|Restore the server-side SPARQL endpoint config for Wikidata (T421250)]], [[gerrit:1311489|Article Guidance: migrate wikidata config (T421250)]] (duration: 09m 01s) 2439593 wikitext text/x-wiki qnaokk2bnkqtcaknd4kok8piss1ndq3 2439594 2439593 2026-07-21T20:19:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet 2439594 wikitext text/x-wiki f75kvv8ijym1qt2ui7nbkewm6scx341 2439595 2439594 2026-07-21T20:19:40Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet 2439595 wikitext text/x-wiki 0ta16nr828ghnemqivkzvijtvdfotiw 2439598 2439595 2026-07-21T20:22:22Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439598 wikitext text/x-wiki 2eu3ryyof3ngydrm0a44ugfslkhn7e6 2439600 2439598 2026-07-21T20:22:44Z Stashbot 7414 dani@deploy2003: Started scap sync-world: Backport for [[gerrit:1313323|Pre-Deploy Referring Experiences survey on enwiki (T432289)]] 2439600 wikitext text/x-wiki 18wk733pa0wyyjt7wb2c9qh15fq7o6d 2439601 2439600 2026-07-21T20:23:17Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet 2439601 wikitext text/x-wiki 520nu3m8nrldkmkplkdt9ovl44g2f2r 2439602 2439601 2026-07-21T20:23:33Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet 2439602 wikitext text/x-wiki 199fqsw2t9qend5kfw5xu31hcxdzw3m 2439603 2439602 2026-07-21T20:24:45Z Stashbot 7414 dani@deploy2003: dani: Backport for [[gerrit:1313323|Pre-Deploy Referring Experiences survey on enwiki (T432289)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439603 wikitext text/x-wiki 5691xoh97u4zx6pioxsnqd46wt7xao7 2439604 2439603 2026-07-21T20:26:42Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet 2439604 wikitext text/x-wiki b59w2zr7ssrdnmnog07zvgrphhtgb29 2439605 2439604 2026-07-21T20:26:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet 2439605 wikitext text/x-wiki 02lw2y7gg3twwspdf5moguzumqy4n1l 2439606 2439605 2026-07-21T20:29:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2020.codfw.wmnet with reason: host reimage 2439606 wikitext text/x-wiki 893ao0x690dd9chnyn4a2p4xb1skr52 2439607 2439606 2026-07-21T20:29:44Z Stashbot 7414 dani@deploy2003: dani: Continuing with deployment 2439607 wikitext text/x-wiki kxeh7q34hbgn0e1wany1cy33ypj8r00 2439608 2439607 2026-07-21T20:30:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439608 wikitext text/x-wiki iw6myvb9h2tav1wbhyuecj0uri8mfyt 2439609 2439608 2026-07-21T20:30:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet 2439609 wikitext text/x-wiki kaokiz1uyqhvp6oew3vlfla7tha6bgv 2439610 2439609 2026-07-21T20:30:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet 2439610 wikitext text/x-wiki 2ulmox6iiamj4absu0m3biq2excnaqr 2439611 2439610 2026-07-21T20:33:19Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2020.codfw.wmnet with reason: host reimage 2439611 wikitext text/x-wiki qvzhyhcmirz8hpiq62hhr67x9g85m13 2439612 2439611 2026-07-21T20:33:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet 2439612 wikitext text/x-wiki 2e3om3vybhxv8n1ntqo3djxg7t7u76j 2439613 2439612 2026-07-21T20:33:33Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet 2439613 wikitext text/x-wiki 0otzwyywuleljj0gm5ail4k5137umra 2439614 2439613 2026-07-21T20:34:00Z Stashbot 7414 dani@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313323|Pre-Deploy Referring Experiences survey on enwiki (T432289)]] (duration: 11m 15s) 2439614 wikitext text/x-wiki 9jlzrijtbzh7rc9mdy5z1tjmf82v6go 2439615 2439614 2026-07-21T20:37:14Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet 2439615 wikitext text/x-wiki p903dubkraysyx72tg00x8p6rnqh5ek 2439616 2439615 2026-07-21T20:37:34Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet 2439616 wikitext text/x-wiki q3zsnwm22vbukcz4b58m7xvqi4udekk 2439617 2439616 2026-07-21T20:39:37Z Stashbot 7414 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1313235|Set $wgMathInternalRestbaseURL explicitly (take 3) (T349582)]] 2439617 wikitext text/x-wiki p00nw0pdam5gaa4zrc0sleh5z9jxw5l 2439618 2439617 2026-07-21T20:41:31Z Stashbot 7414 krinkle@deploy2003: krinkle: Backport for [[gerrit:1313235|Set $wgMathInternalRestbaseURL explicitly (take 3) (T349582)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439618 wikitext text/x-wiki p92yyc3l5746a1hy7iuhfkkltm5iy45 2439619 2439618 2026-07-21T20:44:36Z Stashbot 7414 krinkle@deploy2003: krinkle: Rolling back deployment 2439619 wikitext text/x-wiki pe8ncutexvxz1t65fi3gsdu441t6g2p 2439620 2439619 2026-07-21T20:45:18Z Stashbot 7414 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313235|Set $wgMathInternalRestbaseURL explicitly (take 3) (T349582)]] (duration: 05m 42s) 2439620 wikitext text/x-wiki ft6orgw2l25i5p07uih5ud3i43ovkh6 2439621 2439620 2026-07-21T20:47:58Z Stashbot 7414 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1313340|Fixup closure setting $wgMathInternalRestbaseURL]] 2439621 wikitext text/x-wiki fmnxbx1qav6bby2bzb1mpho6kzyjh3d 2439622 2439621 2026-07-21T20:49:56Z Stashbot 7414 krinkle@deploy2003: krinkle: Backport for [[gerrit:1313340|Fixup closure setting $wgMathInternalRestbaseURL]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439622 wikitext text/x-wiki t3qn9s9y5l2s8hxijqfews7xmm8oj9h 2439623 2439622 2026-07-21T20:52:50Z Stashbot 7414 krinkle@deploy2003: krinkle: Continuing with deployment 2439623 wikitext text/x-wiki gtu8jylyi1rpwmkw8qsjttihqtspesm 2439624 2439623 2026-07-21T20:54:22Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2020.codfw.wmnet with OS bookworm 2439624 wikitext text/x-wiki trio9g5l0v11g6utbwt0yoqorxwphjb 2439625 2439624 2026-07-21T20:55:11Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1024.eqiad.wmnet with OS bookworm 2439625 wikitext text/x-wiki adn41bfyq3fgnwxlaxilz3i0xkp3ls8 2439627 2439625 2026-07-21T20:57:09Z Stashbot 7414 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313340|Fixup closure setting $wgMathInternalRestbaseURL]] (duration: 09m 10s) 2439627 wikitext text/x-wiki h410ovb9q01kfxu62v2oq82m3l2ngeq 2439628 2439627 2026-07-21T21:49:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2019.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling source-only afterwards 2439628 wikitext text/x-wiki 98igjske6b5nyhdlpm2wm97xw1o6dbn 2439629 2439628 2026-07-21T21:53:39Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439629 wikitext text/x-wiki 2u8m3b998ph17aazrmyur244ge3cvq8 2439630 2439629 2026-07-21T21:53:50Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs1024'] 2439630 wikitext text/x-wiki ggp45ikyavuu19lqvz5h2prchkqwjgw 2439631 2439630 2026-07-21T21:54:01Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439631 wikitext text/x-wiki 8aertwnhki0m15ixvnh0icmjzvjl1ud 2439632 2439631 2026-07-21T21:54:12Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs1024'] 2439632 wikitext text/x-wiki tsgdf1q5ix56t1lg1zvdjc6fbncw4fh 2439633 2439632 2026-07-21T22:11:26Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439633 wikitext text/x-wiki egdbkqzhr0xxaqu8mbr4u7rjzb92mxp 2439634 2439633 2026-07-21T22:22:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs1024'] 2439634 wikitext text/x-wiki cyj8qsx2tvyilqtr4tgi69coyb5rki4 2439635 2439634 2026-07-21T22:23:21Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439635 wikitext text/x-wiki 1o7qedyi0swwjo60li91vazrxmzjne7 2439636 2439635 2026-07-21T22:24:03Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2439636 wikitext text/x-wiki ejfuwmt092dlciw4u8bar1m3o3uxezs 2439637 2439636 2026-07-21T22:24:11Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 01m 26s) 2439637 wikitext text/x-wiki lwgav0x1vyfewc3bk3tkucjhr9vi3ss 2439638 2439637 2026-07-21T22:47:37Z Stashbot 7414 cwhite: force reboot arclamp2001 - appears to have run out of memory and gone unresponsive 2439638 wikitext text/x-wiki dnzyqc2ythembgs0zf0ng295m8sey1q 2439639 2439638 2026-07-21T22:50:27Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2019.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling source-only afterwards 2439639 wikitext text/x-wiki afvxnhzgr6dvmyqp385fh3o0zooims9 2439646 2439639 2026-07-22T02:00:29Z Stashbot 7414 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image 2439646 wikitext text/x-wiki is6zara2toemvxr148hf7nol9ad1a49 2439647 2439646 2026-07-22T02:07:11Z Stashbot 7414 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) 2439647 wikitext text/x-wiki n84zzz5vn8wpjs7a4ofj4wjw83rse2h 2439652 2439647 2026-07-22T05:25:38Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis bolwiki in section s5 2439652 wikitext text/x-wiki fs3yh8susq2fl6bwkmx2lli9li9s3ds 2439653 2439652 2026-07-22T05:32:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis bolwiki in section s5 2439653 wikitext text/x-wiki aiczcr9u3pu0ohmtowsyifokcdz3603 2439655 2439653 2026-07-22T06:42:15Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2020.codfw.wmnet 2439655 wikitext text/x-wiki bp8jk0r67k3vi80iiiha3yp9m859l21 2439656 2439655 2026-07-22T06:42:42Z Stashbot 7414 ryankemper: [WDQS] T430880 Repooled wdqs2020 after successful Bookworm reimage, data transfer, and postflight validation 2439656 wikitext text/x-wiki fpbpuqopc6wdmlakz0fyomg4bd3eat7 2439658 2439656 2026-07-22T07:23:50Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Turnilo import support - oblivian@cumin1003" 2439658 wikitext text/x-wiki 0bkofjv2yj5u03f3xxtn17wjqcjsq6r 2439659 2439658 2026-07-22T07:23:52Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Turnilo import support - oblivian@cumin1003 2439659 wikitext text/x-wiki aq7ay53wxyt0nckbzc10ko0w903rgbo 2439660 2439659 2026-07-22T07:24:44Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Turnilo import support - oblivian@cumin1003 2439660 wikitext text/x-wiki 04m1f58vnugd38kbbq63hn84jf4cgua 2439661 2439660 2026-07-22T07:24:46Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Turnilo import support - oblivian@cumin1003" 2439661 wikitext text/x-wiki e765j1snyit1akpl9181bxmtyrf0va4 2439663 2439661 2026-07-22T07:29:51Z Stashbot 7414 phuedx@deploy2003: Started scap sync-world: Backport for [[gerrit:1313554|Bump VEFU schema and remove hCaptcha AB test (T432737)]], [[gerrit:1313553|Bump VEFU schema and remove hCaptcha AB test (T432737)]] 2439663 wikitext text/x-wiki 0x1fo24fqsbr3stjhv9juqr8hlp70yp 2439665 2439663 2026-07-22T07:31:52Z Stashbot 7414 phuedx@deploy2003: phuedx: Backport for [[gerrit:1313554|Bump VEFU schema and remove hCaptcha AB test (T432737)]], [[gerrit:1313553|Bump VEFU schema and remove hCaptcha AB test (T432737)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439665 wikitext text/x-wiki i47gcz9akun2gkpm89lipgcqcrc1zul 2439668 2439665 2026-07-22T07:39:14Z Stashbot 7414 phuedx@deploy2003: phuedx: Continuing with deployment 2439668 wikitext text/x-wiki ql44ms80lxyofzm688fanslgrit06hn 2439669 2439668 2026-07-22T07:43:31Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2439669 wikitext text/x-wiki 1bxb0cgla578paw8vlfq5ojy0tv4olg 2439670 2439669 2026-07-22T07:43:35Z Stashbot 7414 phuedx@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313554|Bump VEFU schema and remove hCaptcha AB test (T432737)]], [[gerrit:1313553|Bump VEFU schema and remove hCaptcha AB test (T432737)]] (duration: 13m 44s) 2439670 wikitext text/x-wiki 1abc609f0295v2gltoh9bpef2u7enr3 2439671 2439670 2026-07-22T07:43:48Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2439671 wikitext text/x-wiki o5qklfeshx9070gst6ssznju2gqcy32 2439672 2439671 2026-07-22T07:47:14Z Stashbot 7414 phuedx: End of UTC morning backport window 2439672 wikitext text/x-wiki smhhezf4vezf25setszvjpnezovb8yz 2439673 2439672 2026-07-22T07:47:45Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2439673 wikitext text/x-wiki lco0o43xyr4mvrgo1plszbhlmzpfmjm 2439674 2439673 2026-07-22T07:48:01Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2439674 wikitext text/x-wiki gg1iiywhd1ngcpbh8zxcpxwaco0yrrb 2439675 2439674 2026-07-22T07:48:10Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2439675 wikitext text/x-wiki 9l4fbzhctyiujm8zz36ojmlo86vri0a 2439676 2439675 2026-07-22T07:48:32Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2439676 wikitext text/x-wiki 0kx00b65hw1nbyqrnw2joytqvg2htli 2439677 2439676 2026-07-22T07:49:00Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2439677 wikitext text/x-wiki 99teucmie1s3j7f57b48qmi56xjt1wz 2439678 2439677 2026-07-22T07:49:16Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2439678 wikitext text/x-wiki kvoxm0c010msq8h960n2geew0vwd4rl 2439679 2439678 2026-07-22T07:49:24Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2439679 wikitext text/x-wiki kd9oj9mrzbksi1b7zxofd8a0n99cb88 2439680 2439679 2026-07-22T07:49:42Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2439680 wikitext text/x-wiki 36ee6740cnths92fnu8m50gybw7se5b 2439681 2439680 2026-07-22T07:50:08Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply 2439681 wikitext text/x-wiki 9lwwm0isye5aflou4nao5v2ffgdlvci 2439682 2439681 2026-07-22T07:50:23Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply 2439682 wikitext text/x-wiki 2v06gevy13v31ao1u27e41lqz32d962 2439683 2439682 2026-07-22T07:50:30Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply 2439683 wikitext text/x-wiki i3in1a4y8sgbvgijypz99t81kexi3e9 2439684 2439683 2026-07-22T07:50:46Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply 2439684 wikitext text/x-wiki ps8s09lidnkonnzvonsyk9rmn4d7g37 2439685 2439684 2026-07-22T07:51:49Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2439685 wikitext text/x-wiki cy5m7k32k8k1foo8fq3hc5x058bleb6 2439686 2439685 2026-07-22T07:52:07Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2439686 wikitext text/x-wiki 6h956qiwbbaobi5dgnoo0znh3h97kgi 2439687 2439686 2026-07-22T07:52:26Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2439687 wikitext text/x-wiki sz18p249sy1ydg5uw0v5cayk8tvi7pl 2439688 2439687 2026-07-22T07:52:42Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2439688 wikitext text/x-wiki ah4tsirj8uiixc5dyvoyxm0dhuxtj7j 2439689 2439688 2026-07-22T07:52:49Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2439689 wikitext text/x-wiki sngyqwm9wqme0aed8amnyu6t17zjhgs 2439690 2439689 2026-07-22T07:53:05Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2439690 wikitext text/x-wiki hvwaofs24d3ait2whde7g2rmq53yc28 2439691 2439690 2026-07-22T07:53:12Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply 2439691 wikitext text/x-wiki 0efsspmux4pmrkxzrx3j9urgisjc6mb 2439692 2439691 2026-07-22T07:53:30Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply 2439692 wikitext text/x-wiki lybsszpm2h6wuc0tj62t4195zsz19a3 2439693 2439692 2026-07-22T08:04:03Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1030.eqiad.wmnet,service=s1 2439693 wikitext text/x-wiki hcvhsvil83siarjvv2471on65pjxfw2 2439694 2439693 2026-07-22T08:04:17Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1030.eqiad.wmnet,service=s1 2439694 wikitext text/x-wiki 28tor80fn0bl6vxc44hwbyuz6uyt1sd 2439698 2439694 2026-07-22T08:11:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1152: OS upgrade 2439698 wikitext text/x-wiki i23vnfvsfxuu95pmxc8k1v03sjdm9ry 2439699 2439698 2026-07-22T08:11:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: OS upgrade 2439699 wikitext text/x-wiki 3egujlcnrmugih7iwishrh6aqbccz3a 2439700 2439699 2026-07-22T08:11:55Z Stashbot 7414 Dreamy_Jazz: Created cusi_signal, cusi_case, and cusi_user on ukwiki and enwikivoyage in extension1 2439700 wikitext text/x-wiki pn9j0ar4q1xs9ki4bif5du7oxxu9zwz 2439701 2439700 2026-07-22T08:13:48Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1152: OS upgrade 2439701 wikitext text/x-wiki ayikul9uq9rwis3g3xqgb0dfcdro1fk 2439702 2439701 2026-07-22T08:13:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: OS upgrade 2439702 wikitext text/x-wiki hwhomaznfeioc2n61czfawqfwgqzg1h 2439703 2439702 2026-07-22T08:15:44Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2251.codfw.wmnet,db1152.eqiad.wmnet with reason: OS upgrade 2439703 wikitext text/x-wiki 3kg7d67pzgzhs7gwmwbmegh764hn35h 2439704 2439703 2026-07-22T08:16:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1152: OS upgrade 2439704 wikitext text/x-wiki lu7zyxgdwgdgrywc4771wzockis8z2c 2439705 2439704 2026-07-22T08:16:28Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.parsercache 2439705 wikitext text/x-wiki 4egickrc083nr42qjslypkzjivgw60n 2439706 2439705 2026-07-22T08:16:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439706 wikitext text/x-wiki r75wt23kynpe63z8ybkt47zi47g14ac 2439707 2439706 2026-07-22T08:16:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: OS upgrade 2439707 wikitext text/x-wiki 5ck29rk4tnqjpw2sca22pl5ar30wpca 2439708 2439707 2026-07-22T08:17:11Z Stashbot 7414 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1313823|[Growth] Set revise tone threshold to 0.79 (T432790)]] 2439708 wikitext text/x-wiki mptquffeqg4b9dh11ewakfyyao6keab 2439709 2439708 2026-07-22T08:19:07Z Stashbot 7414 urbanecm@deploy2003: urbanecm: Backport for [[gerrit:1313823|[Growth] Set revise tone threshold to 0.79 (T432790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439709 wikitext text/x-wiki 63eaqcxft8ya3t8g8tkw4jwaaype1cz 2439710 2439709 2026-07-22T08:20:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2439710 wikitext text/x-wiki 7237l8t129l9mjmj3isjfw4antzfef4 2439711 2439710 2026-07-22T08:20:30Z Stashbot 7414 cwilliams@cumin1003: dbmaint on ms1@codfw T432321 2439711 wikitext text/x-wiki rftvg71mqylor2158sce3ktc84v5ks2 2439712 2439711 2026-07-22T08:20:54Z Stashbot 7414 urbanecm@deploy2003: urbanecm: Continuing with deployment 2439712 wikitext text/x-wiki s9kdw6m2630uyfpsc43coyxsxxb4gzt 2439713 2439712 2026-07-22T08:22:19Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2251: Upgrading db2251.codfw.wmnet 2439713 wikitext text/x-wiki nrc19eu22wfu2pf4mbdsd4jseqh4z02 2439714 2439713 2026-07-22T08:22:21Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.parsercache 2439714 wikitext text/x-wiki b3iuriptga6quj2lyso78u621vr3swo 2439715 2439714 2026-07-22T08:22:24Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439715 wikitext text/x-wiki 107c4mq3enefh5yjppyfzr75rxrr3pf 2439716 2439715 2026-07-22T08:22:25Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2251: Upgrading db2251.codfw.wmnet 2439716 wikitext text/x-wiki mpj16gofzl9rktrnv0lhlqcoo9pzpf8 2439717 2439716 2026-07-22T08:25:10Z Stashbot 7414 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313823|[Growth] Set revise tone threshold to 0.79 (T432790)]] (duration: 07m 59s) 2439717 wikitext text/x-wiki d6wawhuvrjcjmk1rw6b91z62rydrt1b 2439720 2439717 2026-07-22T08:28:37Z Stashbot 7414 dreamyjazz@deploy2003: Started scap sync-world: Backport for [[gerrit:1313832|CheckUser SI: Enable on ukwiki and enwikivoyage without UI]] 2439720 wikitext text/x-wiki cx9konam1irak5o37qq0the3qxj5se4 2439721 2439720 2026-07-22T08:30:27Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Backport for [[gerrit:1313832|CheckUser SI: Enable on ukwiki and enwikivoyage without UI]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439721 wikitext text/x-wiki 40m0si3g9svp20ylyf60lckxftmrz4n 2439724 2439721 2026-07-22T08:31:38Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2251.codfw.wmnet with OS trixie 2439724 wikitext text/x-wiki a8vl8ytaioxrr6ebblisasxnz3jxpmv 2439725 2439724 2026-07-22T08:36:33Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment 2439725 wikitext text/x-wiki j62scv9qpsmflkh9dj5p4gtto0c20yt 2439726 2439725 2026-07-22T08:38:52Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1030.eqiad.wmnet,service=s1 2439726 wikitext text/x-wiki hoi077wm6k3r1nklgryodyb8lb39lw5 2439727 2439726 2026-07-22T08:40:55Z Stashbot 7414 dreamyjazz@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313832|CheckUser SI: Enable on ukwiki and enwikivoyage without UI]] (duration: 12m 26s) 2439727 wikitext text/x-wiki 4jvg7ss4amhg42oxi1i4izlyvrq7sen 2439728 2439727 2026-07-22T08:45:22Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2251.codfw.wmnet with reason: host reimage 2439728 wikitext text/x-wiki cbxyqiqiyqwshvbs3r52a8l4hjmjrkx 2439729 2439728 2026-07-22T08:50:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2251.codfw.wmnet with reason: host reimage 2439729 wikitext text/x-wiki 4eo0smzmkh0opss36cibd7j15nugdmf 2439736 2439729 2026-07-22T09:04:31Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2251.codfw.wmnet with OS trixie 2439736 wikitext text/x-wiki aly8sdu7rb271sau8wtlnikl9t82bnj 2439737 2439736 2026-07-22T09:05:14Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2439737 wikitext text/x-wiki fjclhaj9d5nn20va7cdqo49z7objxlr 2439738 2439737 2026-07-22T09:09:06Z Stashbot 7414 Dreamy_Jazz: Deployed patch for T432453 and T432454 2439738 wikitext text/x-wiki lao7kdstmf4u41cxv1hiv06ivmkeb2s 2439739 2439738 2026-07-22T09:10:38Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1018.eqiad.wmnet with reason: Cloning 2439739 wikitext text/x-wiki k9vvxahl1j0oz8bhpznp5303815q88o 2439740 2439739 2026-07-22T09:11:34Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates 2439740 wikitext text/x-wiki 93pt3cs67dw989vgjsrzziufswtpxab 2439741 2439740 2026-07-22T09:11:35Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2439741 wikitext text/x-wiki azbwij3h4lsj0k1hhb9mk9xntk2k1hm 2439742 2439741 2026-07-22T09:11:40Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439742 wikitext text/x-wiki mv1b4mgofs5cdbglkct2u7o4bk115ik 2439743 2439742 2026-07-22T09:11:42Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates 2439743 wikitext text/x-wiki t8eakfdl3lipdd0puj71psocuy697th 2439747 2439743 2026-07-22T09:26:11Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates 2439747 wikitext text/x-wiki bqbczq8j5ajpf2vjffy4288zh4r1dog 2439748 2439747 2026-07-22T09:26:13Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2439748 wikitext text/x-wiki 5yfp0w1skv2ms6iav3m6ob8cprvmxuj 2439749 2439748 2026-07-22T09:26:26Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439749 wikitext text/x-wiki a0pd2s13bco4l3ilie9eqb2tfvnd20y 2439750 2439749 2026-07-22T09:26:28Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates 2439750 wikitext text/x-wiki 5bxtyw3qfbuo80n7q9blo3wvj1kku2m 2439751 2439750 2026-07-22T09:35:49Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Turnilo import: UI improvement; support url shortener - oblivian@cumin1003" 2439751 wikitext text/x-wiki d59z1fv8t8o71rh00isvwm8x4f1olax 2439752 2439751 2026-07-22T09:35:52Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Turnilo import: UI improvement; support url shortener - oblivian@cumin1003 2439752 wikitext text/x-wiki aodwrb5yp6j2cfte3bmj1qjyved2lvg 2439753 2439752 2026-07-22T09:36:41Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Turnilo import: UI improvement; support url shortener - oblivian@cumin1003 2439753 wikitext text/x-wiki dtds7eaq8rd8vtpk7ynm0lxsqt40u6p 2439754 2439753 2026-07-22T09:36:43Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Turnilo import: UI improvement; support url shortener - oblivian@cumin1003" 2439754 wikitext text/x-wiki 5bt3e34qphrjnybtouyay17ztfyhfoy 2439760 2439754 2026-07-22T09:56:30Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2439760 wikitext text/x-wiki 8m5g4b4kt6t2zcivp6yj8q8pm7cm2mr 2439761 2439760 2026-07-22T09:56:33Z Stashbot 7414 cwilliams@cumin1003: dbmaint on ms2@codfw T432321 2439761 wikitext text/x-wiki cbu0p18q3vlh8fetyo14pnydljc86ml 2439762 2439761 2026-07-22T09:56:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2253: Upgrading db2253.codfw.wmnet 2439762 wikitext text/x-wiki q108gzdu2xhbq52t6b6qmllurqt4pr7 2439763 2439762 2026-07-22T09:57:00Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.parsercache 2439763 wikitext text/x-wiki k9r9oqkanf0mu7sm0o4nfgz6g1cj2os 2439764 2439763 2026-07-22T09:57:08Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439764 wikitext text/x-wiki icbsc2z0i6j845wf79biqgzmcuisamm 2439765 2439764 2026-07-22T09:57:10Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2253: Upgrading db2253.codfw.wmnet 2439765 wikitext text/x-wiki 5d75lr7ru2na7dw14pf34qz3rbnzutp 2439767 2439765 2026-07-22T09:58:49Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1151.eqiad.wmnet with reason: Security updates 2439767 wikitext text/x-wiki mqc9x844hzjwhmvj0zt3ln4fioma054 2439768 2439767 2026-07-22T10:00:19Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2253.codfw.wmnet with OS trixie 2439768 wikitext text/x-wiki 1wld3183yhr2arr31y2p2e83fwtpuqk 2439770 2439768 2026-07-22T10:13:49Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2253.codfw.wmnet with reason: host reimage 2439770 wikitext text/x-wiki ngz11fjam2gvce78yw670eeo9e0yvzm 2439771 2439770 2026-07-22T10:18:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2253.codfw.wmnet with reason: host reimage 2439771 wikitext text/x-wiki l1kx106g4axtmzchyjoukcudq3syfm9 2439773 2439771 2026-07-22T10:26:10Z Stashbot 7414 dreamyjazz@deploy2003: Started scap sync-world: Backport for [[gerrit:1313885|Display SI in svwiki and enwikivoyage]] 2439773 wikitext text/x-wiki 6p0ilt2czw3id1lmby9t1dltb1xupr3 2439775 2439773 2026-07-22T10:28:05Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Backport for [[gerrit:1313885|Display SI in svwiki and enwikivoyage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439775 wikitext text/x-wiki ctx8s6v9x67e9okp94b4rgq65hlmg54 2439776 2439775 2026-07-22T10:33:27Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2253.codfw.wmnet with OS trixie 2439776 wikitext text/x-wiki iijlaods3c2ugmbdwxhfc6xrmg1mpju 2439777 2439776 2026-07-22T10:34:10Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2439777 wikitext text/x-wiki a4dhjzglvnedhm44xx2z8kab1vjwson 2439778 2439777 2026-07-22T10:34:36Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment 2439778 wikitext text/x-wiki iek6rjm6zuqxjw6ho01bpj1j56qqm95 2439781 2439778 2026-07-22T10:38:57Z Stashbot 7414 dreamyjazz@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313885|Display SI in svwiki and enwikivoyage]] (duration: 12m 47s) 2439781 wikitext text/x-wiki mjli40l2fdwz5omemz7quxh4uox6whb 2439782 2439781 2026-07-22T10:42:27Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates 2439782 wikitext text/x-wiki hjh2655amgfoevt75sx4nsdvy63qc1t 2439783 2439782 2026-07-22T10:42:29Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2439783 wikitext text/x-wiki 6a29dclgpgidzym3apci232h10n7dsn 2439784 2439783 2026-07-22T10:42:34Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439784 wikitext text/x-wiki tt5z67k5zzya8r22s28dqbuvd6dhgjn 2439785 2439784 2026-07-22T10:42:35Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates 2439785 wikitext text/x-wiki m1g24hbtc2afqedy90hsac37lyzm339 2439786 2439785 2026-07-22T10:45:59Z Stashbot 7414 zabe@deploy2003: Started scap sync-world: Backport for [[gerrit:1313339|Update interwiki cache (T429921)]] 2439786 wikitext text/x-wiki 5ntmqcrsaejtx2dcl7agsk7tlk0jfor 2439787 2439786 2026-07-22T10:47:58Z Stashbot 7414 zabe@deploy2003: zabe: Backport for [[gerrit:1313339|Update interwiki cache (T429921)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439787 wikitext text/x-wiki oliququ4c4efokhcs3g9zdef75m5jfc 2439788 2439787 2026-07-22T10:50:19Z Stashbot 7414 zabe@deploy2003: zabe: Continuing with deployment 2439788 wikitext text/x-wiki 8jsqr7nnay1afqntb9g941uie6w73zc 2439790 2439788 2026-07-22T10:52:56Z Stashbot 7414 fnegri@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply 2439790 wikitext text/x-wiki ksqry1mkxcsdy4soulbvlnq7e23no31 2439791 2439790 2026-07-22T10:53:48Z Stashbot 7414 fnegri@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply 2439791 wikitext text/x-wiki o17ju3wyi3osefpaxw7otv30ba9w3xp 2439792 2439791 2026-07-22T10:53:59Z Stashbot 7414 fnegri@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply 2439792 wikitext text/x-wiki k3053sn2x85zdlisdb93wyka2cecx18 2439793 2439792 2026-07-22T10:54:38Z Stashbot 7414 zabe@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313339|Update interwiki cache (T429921)]] (duration: 08m 38s) 2439793 wikitext text/x-wiki s864yjxl7rzmcy55kdiyhpdn59p2xwr 2439794 2439793 2026-07-22T10:55:15Z Stashbot 7414 fnegri@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply 2439794 wikitext text/x-wiki tolxq0gdxxoehpwxtp3byfciykfsw58 2439795 2439794 2026-07-22T10:57:01Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates 2439795 wikitext text/x-wiki f4bu5odinpbaoaoqculevijna51295e 2439796 2439795 2026-07-22T10:57:03Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2439796 wikitext text/x-wiki 0ab0sz9g835dwmstn5idilxypeeex8y 2439797 2439796 2026-07-22T10:57:14Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439797 wikitext text/x-wiki d16pwa9bhpan5agdfxewerutpjc8nj7 2439798 2439797 2026-07-22T10:57:16Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates 2439798 wikitext text/x-wiki k8xoy9zeyo9udzksqm6czsx793bcyh4 2439799 2439798 2026-07-22T10:58:52Z Stashbot 7414 fnegri@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply 2439799 wikitext text/x-wiki 3q19v61t1pz13zqzqj882zn3sk8a8z3 2439801 2439799 2026-07-22T11:00:17Z Stashbot 7414 fnegri@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply 2439801 wikitext text/x-wiki bk9fomi4uzv33ldd8krqec5v35z7nct 2439802 2439801 2026-07-22T11:01:18Z Stashbot 7414 dreamyjazz@deploy2003: Started scap sync-world: Backport for [[gerrit:1307751|WikimediaAntiAbuse: Enable everywhere (T431023)]], [[gerrit:1313187|WikimediaAntiAbuse: Enable personal info tagging on testwiki (T431292)]] 2439802 wikitext text/x-wiki hwcjff5me0lsokp0ebxm4fozvpr2l45 2439803 2439802 2026-07-22T11:01:25Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1153.eqiad.wmnet with reason: Security updates 2439803 wikitext text/x-wiki legfb9u72psiu1w3ftk7hgk3frgw4lt 2439804 2439803 2026-07-22T11:01:58Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2439804 wikitext text/x-wiki j6181dlpjb6m35zrmxd1iy8lrib2enn 2439805 2439804 2026-07-22T11:02:01Z Stashbot 7414 cwilliams@cumin1003: dbmaint on ms3@codfw T432321 2439805 wikitext text/x-wiki nirvxhi0p7hfall0bf6xcmso3mv834e 2439806 2439805 2026-07-22T11:02:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2252: Upgrading db2252.codfw.wmnet 2439806 wikitext text/x-wiki r74a0fvx8fdk6ahgq77udyttkhp7rgx 2439807 2439806 2026-07-22T11:02:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.parsercache 2439807 wikitext text/x-wiki 3jfn9dvzchehkgfop49gbps95c18b3h 2439808 2439807 2026-07-22T11:02:37Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439808 wikitext text/x-wiki hn32t15s4f78zirp8wzuhc4sfaakz48 2439809 2439808 2026-07-22T11:02:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2252: Upgrading db2252.codfw.wmnet 2439809 wikitext text/x-wiki 5j8c5ioovtiwq0rn2ck3av1as9g36oa 2439810 2439809 2026-07-22T11:03:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2252.codfw.wmnet with OS trixie 2439810 wikitext text/x-wiki p05imx6og78xxyd13gh2gyjqhqtadwy 2439811 2439810 2026-07-22T11:03:40Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz, kharlan: Backport for [[gerrit:1307751|WikimediaAntiAbuse: Enable everywhere (T431023)]], [[gerrit:1313187|WikimediaAntiAbuse: Enable personal info tagging on testwiki (T431292)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439811 wikitext text/x-wiki dqo83l1kz27w1obod7vq2is4md7gszb 2439812 2439811 2026-07-22T11:08:00Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz, kharlan: Continuing with deployment 2439812 wikitext text/x-wiki mabv5r93bej58sm86ze6d4i0dg2e6qb 2439813 2439812 2026-07-22T11:10:40Z Stashbot 7414 mvolz@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply 2439813 wikitext text/x-wiki j95lg27o6l6eh8jl4lsnd1heulmb525 2439814 2439813 2026-07-22T11:11:12Z Stashbot 7414 mvolz@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2439814 wikitext text/x-wiki ilzoau64cm04orbxk8d6ibzxq2ea5di 2439815 2439814 2026-07-22T11:12:23Z Stashbot 7414 dreamyjazz@deploy2003: Finished scap sync-world: Backport for [[gerrit:1307751|WikimediaAntiAbuse: Enable everywhere (T431023)]], [[gerrit:1313187|WikimediaAntiAbuse: Enable personal info tagging on testwiki (T431292)]] (duration: 11m 05s) 2439815 wikitext text/x-wiki lzl1u6i7o228jh3t2kplhcb1bbq6w7u 2439816 2439815 2026-07-22T11:13:04Z Stashbot 7414 mvolz@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply 2439816 wikitext text/x-wiki 3m3d8hcthf0ujluoj7toj7rlwm3p4zu 2439817 2439816 2026-07-22T11:13:31Z Stashbot 7414 mvolz@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply 2439817 wikitext text/x-wiki 3k97wqb0s9yez0epnr2i0clb307ehxo 2439818 2439817 2026-07-22T11:14:54Z Stashbot 7414 mvolz@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply 2439818 wikitext text/x-wiki 644lpy2ka715u4ldke7fdg1rsq4y4of 2439819 2439818 2026-07-22T11:15:02Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1068.eqiad.wmnet 2439819 wikitext text/x-wiki cvzsldo5pa3ewd5hn6yyba2pfzvbzwr 2439820 2439819 2026-07-22T11:15:05Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet 2439820 wikitext text/x-wiki 9h8k3jqj02fm7gegcqwnm9ipd6rx76w 2439821 2439820 2026-07-22T11:15:26Z Stashbot 7414 mvolz@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply 2439821 wikitext text/x-wiki 0335ew3wbsztz8050gqumk7mpvtbas7 2439822 2439821 2026-07-22T11:15:40Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet 2439822 wikitext text/x-wiki jxf7gdtiy5pms1h3ofcbp50jbahntda 2439823 2439822 2026-07-22T11:17:03Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2252.codfw.wmnet with reason: host reimage 2439823 wikitext text/x-wiki ei3sn9r7nrgtjdrk60ockwesy7ghagx 2439824 2439823 2026-07-22T11:17:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1068.eqiad.wmnet with OS trixie 2439824 wikitext text/x-wiki ijilwe9f0ycm84pd78pacea412wtg7u 2439825 2439824 2026-07-22T11:17:27Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1068 2439825 wikitext text/x-wiki kaxewa8kc6up8y5n2cgwmqgceygyyjx 2439826 2439825 2026-07-22T11:17:34Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2439826 wikitext text/x-wiki 5k1mnt9xryix9ll42x00oznc77vv9eu 2439828 2439826 2026-07-22T11:22:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2252.codfw.wmnet with reason: host reimage 2439828 wikitext text/x-wiki dyjvqvey5jq2qd5guaa6aetg5i6rr6d 2439829 2439828 2026-07-22T11:26:12Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1068 - jiji@cumin1003" 2439829 wikitext text/x-wiki 0nehpqgjyi8dtfbu1efv1teq1bihae7 2439830 2439829 2026-07-22T11:26:16Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1068 - jiji@cumin1003" 2439830 wikitext text/x-wiki tb3btgxjv988pob2icvfnclca3p3sga 2439831 2439830 2026-07-22T11:26:18Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439831 wikitext text/x-wiki jyhg021d075ra9amfnukg9lve36uhtr 2439832 2439831 2026-07-22T11:26:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1068.eqiad.wmnet 46.48.64.10.in-addr.arpa 6.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2439832 wikitext text/x-wiki 2ihvk7jl2yp4db7ltc2yuz0s9mflc7x 2439833 2439832 2026-07-22T11:26:21Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1068.eqiad.wmnet 46.48.64.10.in-addr.arpa 6.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2439833 wikitext text/x-wiki nd4nzoyz5fo7jxwb3kst1fzup6amlr2 2439834 2439833 2026-07-22T11:26:23Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1068 2439834 wikitext text/x-wiki mglzg11j9k6gs6tl3cq6mm5rpdjo2jb 2439836 2439834 2026-07-22T11:34:33Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1068 2439836 wikitext text/x-wiki gvbm8nco2681u08jqn7fwut8n0mpvdb 2439837 2439836 2026-07-22T11:34:35Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1068 2439837 wikitext text/x-wiki ajg53cpx4n8ja2lcqfcplc75pljs1vf 2439838 2439837 2026-07-22T11:37:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2252.codfw.wmnet with OS trixie 2439838 wikitext text/x-wiki kl07t2e7n21m5mccbovzaq1w5sn4vzb 2439839 2439838 2026-07-22T11:38:20Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2439839 wikitext text/x-wiki 4cp2trwbrhiaphglna98wvljf48oyu8 2439840 2439839 2026-07-22T11:40:43Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply 2439840 wikitext text/x-wiki b8oiobzp1spb0gquau635mr71l13tnu 2439841 2439840 2026-07-22T11:42:45Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply 2439841 wikitext text/x-wiki 3mhghoijxlfnflgbo3vtrkxwd7xikrb 2439842 2439841 2026-07-22T11:42:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2252: Security updates 2439842 wikitext text/x-wiki q6g04vgclzque0j3vfz9ndtcrzrc27j 2439843 2439842 2026-07-22T11:43:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.parsercache 2439843 wikitext text/x-wiki 9mummc7lsbj09e8qf5jfj39o6lgu7ul 2439844 2439843 2026-07-22T11:43:13Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2439844 wikitext text/x-wiki 23467qkg9b2yh7di2bb86mqjjntc9qe 2439845 2439844 2026-07-22T11:43:14Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2252: Security updates 2439845 wikitext text/x-wiki b1tluvwtqq0s78akf87hi9s41iehd4v 2439846 2439845 2026-07-22T11:47:48Z Stashbot 7414 dreamyjazz@deploy2003: Started scap sync-world: Backport for [[gerrit:1313918|Disable SI special page on enwikivoyage]] 2439846 wikitext text/x-wiki lhmvl8ouc3fgwypg5u1nwxah7m9z775 2439847 2439846 2026-07-22T11:49:45Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Backport for [[gerrit:1313918|Disable SI special page on enwikivoyage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439847 wikitext text/x-wiki ror06ha5xapgjs9ozjg0177nhxwe82h 2439848 2439847 2026-07-22T11:51:24Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment 2439848 wikitext text/x-wiki 9q2etmgkcm9jlp2eq2v8qwqoknq6qv9 2439849 2439848 2026-07-22T11:52:29Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1068.eqiad.wmnet with reason: host reimage 2439849 wikitext text/x-wiki r5s0nrj689c8w5p65etj5zbriqcug6a 2439850 2439849 2026-07-22T11:58:55Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1068.eqiad.wmnet with reason: host reimage 2439850 wikitext text/x-wiki b340ctzqte8lrqnat391t88cl6hxf7e 2439851 2439850 2026-07-22T12:01:07Z Stashbot 7414 dreamyjazz@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313918|Disable SI special page on enwikivoyage]] (duration: 13m 19s) 2439851 wikitext text/x-wiki inqw0cbwwbf4u2d0x6a1dwpo3cigo9w 2439852 2439851 2026-07-22T12:20:15Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1068.eqiad.wmnet with OS trixie 2439852 wikitext text/x-wiki 3e4vohg0rn7rrmda5om9ffjcsv6x6nw 2439854 2439852 2026-07-22T12:27:46Z Stashbot 7414 brouberol@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'. 2439854 wikitext text/x-wiki 5a5czeay8t4xcqxsph4k6ed2g7k3792 2439855 2439854 2026-07-22T12:28:20Z Stashbot 7414 brouberol@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2439855 wikitext text/x-wiki cejcw06ixhdypyoaa72bj496n8txssn 2439856 2439855 2026-07-22T12:30:04Z Stashbot 7414 brouberol@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2439856 wikitext text/x-wiki g6q36sxdqj0ygl2ee6idjb2acs479vx 2439857 2439856 2026-07-22T12:30:26Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudvirt1048.eqiad.wmnet 2439857 wikitext text/x-wiki h14hmq0j6rpjah9dcylhtussny9dsmt 2439858 2439857 2026-07-22T12:31:43Z Stashbot 7414 brouberol@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2439858 wikitext text/x-wiki dexbka4rnggjt1o26t7i18hdr6hd7y6 2439859 2439858 2026-07-22T12:34:10Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2439859 wikitext text/x-wiki gfigrshdebx8hzk3uv6os7h988xh8pf 2439860 2439859 2026-07-22T12:35:09Z Stashbot 7414 brouberol@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2439860 wikitext text/x-wiki 83pmyp4xlnh2qn17etmlwko8lkqjfmo 2439861 2439860 2026-07-22T12:35:15Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2439861 wikitext text/x-wiki g2wct4alkxhoai4i4ln69wei5uilvpu 2439862 2439861 2026-07-22T12:36:02Z Stashbot 7414 elukey@cumin1003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for crm2001.codfw.wmnet: Renew puppet certificate - elukey@cumin1003 2439862 wikitext text/x-wiki r7iar0zyrmxbp6nevr4nw80n4omch40 2439863 2439862 2026-07-22T12:36:13Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2439863 wikitext text/x-wiki j2nwffgnkdz6s9tyifavrpjoyzd5g7m 2439864 2439863 2026-07-22T12:36:29Z Stashbot 7414 brouberol@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2439864 wikitext text/x-wiki eyjv9znewahm83smich6akznh75z2df 2439865 2439864 2026-07-22T12:36:57Z Stashbot 7414 brouberol@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2439865 wikitext text/x-wiki ootqae3fcba5mwqx83x05j4gxxhuqxo 2439866 2439865 2026-07-22T12:37:44Z Stashbot 7414 brouberol@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2439866 wikitext text/x-wiki 6dwerto02m5rydfynvzqmwxl3ranprq 2439867 2439866 2026-07-22T12:37:58Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2439867 wikitext text/x-wiki 6ufc7c9subagmgzqvxi7zv0xs9scxga 2439868 2439867 2026-07-22T12:38:43Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2439868 wikitext text/x-wiki a7znhbsktcxii50a3r04nlxnq33vsct 2439869 2439868 2026-07-22T12:39:51Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" 2439869 wikitext text/x-wiki 8g4oz37y8x4gha0sulp8641bzy83bu6 2439870 2439869 2026-07-22T12:40:11Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" 2439870 wikitext text/x-wiki 4ez69nrnle3l4qfqm9dptfqavckprft 2439871 2439870 2026-07-22T12:40:13Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439871 wikitext text/x-wiki 08kq8arrq7ihbsr5rsex90usg7vcktq 2439872 2439871 2026-07-22T12:40:14Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1048.eqiad.wmnet 2439872 wikitext text/x-wiki lozhjk5xn67f6yb07ezqj8vkbmfsp68 2439873 2439872 2026-07-22T12:40:29Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2439873 wikitext text/x-wiki kkvjrgcqyxpmtlyvs9xp4t8iy6uonci 2439874 2439873 2026-07-22T12:41:04Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2439874 wikitext text/x-wiki h53bn4t7cmaxkwl0my8fouz5as6ft2p 2439875 2439874 2026-07-22T12:43:20Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1036 hosts 2439875 wikitext text/x-wiki jvp5uyjjqm3at6nfzykryjhev1yuymq 2439878 2439875 2026-07-22T12:54:47Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Test 2439878 wikitext text/x-wiki 9mxp68mepczgcwny49sjbrygldvbgcc 2439879 2439878 2026-07-22T12:55:07Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Test 2439879 wikitext text/x-wiki 0pxj9jssgzd2k9aqzodem6ce3z2ogen 2439880 2439879 2026-07-22T12:55:18Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2051: Test 2439880 wikitext text/x-wiki as8xtvsl3i03iwm10zhkir29uv46lbw 2439882 2439880 2026-07-22T13:08:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439882 wikitext text/x-wiki 9034l179s4rf0rk6xrbdoztpk7cf0ds 2439883 2439882 2026-07-22T13:11:05Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw 2439883 wikitext text/x-wiki 08oqtvt2bwhpm4c3o0m0bkq2a5ttpot 2439884 2439883 2026-07-22T13:11:06Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw 2439884 wikitext text/x-wiki k4gy5hzda0vmfyhu8e4yl09bz18u935 2439889 2439884 2026-07-22T13:13:43Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw 2439889 wikitext text/x-wiki lx75bc54rjrxnvu3kfsr0af2dz1j7rx 2439890 2439889 2026-07-22T13:13:44Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw 2439890 wikitext text/x-wiki o8ae2f9sbrixmzbzlor6aas3214a8f9 2439891 2439890 2026-07-22T13:14:04Z Stashbot 7414 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1313845|Send an exposure event in the IRS instrument (T432718)]], [[gerrit:1313842|Send an exposure event in the IRS instrument (T432718)]] 2439891 wikitext text/x-wiki 91fnsa1kvwgbgxvmxvlgr2orn43ts8a 2439892 2439891 2026-07-22T13:16:01Z Stashbot 7414 stran@deploy2003: stran: Backport for [[gerrit:1313845|Send an exposure event in the IRS instrument (T432718)]], [[gerrit:1313842|Send an exposure event in the IRS instrument (T432718)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439892 wikitext text/x-wiki 9vae99wlbrqeqyrmgqb08dd2oddcf86 2439893 2439892 2026-07-22T13:18:04Z Stashbot 7414 stran@deploy2003: stran: Continuing with deployment 2439893 wikitext text/x-wiki n2jlxd8tik5kce95ns488c6otbtg3r3 2439894 2439893 2026-07-22T13:19:19Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2439894 wikitext text/x-wiki m711vm4x8wqym68pse5jrhhvmzleuwn 2439895 2439894 2026-07-22T13:19:43Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2439895 wikitext text/x-wiki bm16yht3ohhv210vl410uws486jxi1w 2439896 2439895 2026-07-22T13:19:49Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2439896 wikitext text/x-wiki pqzar5r7ledeu9rhwbectta9fbz30g8 2439897 2439896 2026-07-22T13:20:02Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2439897 wikitext text/x-wiki 6vl03w38eq9k0064pm0byi7w8ag56cr 2439899 2439897 2026-07-22T13:21:22Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439899 wikitext text/x-wiki lxflf0ahydn0e7yp88qdeglot45kssl 2439900 2439899 2026-07-22T13:22:34Z Stashbot 7414 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313845|Send an exposure event in the IRS instrument (T432718)]], [[gerrit:1313842|Send an exposure event in the IRS instrument (T432718)]] (duration: 08m 20s) 2439900 wikitext text/x-wiki g4g8pu75s7xkim4oqvz33khabjmpp7d 2439902 2439900 2026-07-22T13:24:43Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2439902 wikitext text/x-wiki im91i2ykd69vsr7ayd24xpl91dw9gnu 2439903 2439902 2026-07-22T13:25:42Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2439903 wikitext text/x-wiki mzzfxd62jkubv73h1d7u89iac2i6mjo 2439904 2439903 2026-07-22T13:25:48Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2439904 wikitext text/x-wiki 9nxrs7rzr8335s1pcuhgy7h6bw5drrr 2439905 2439904 2026-07-22T13:26:48Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2439905 wikitext text/x-wiki akqxzczvq9y3jfsvqbe1jchg251eeh1 2439906 2439905 2026-07-22T13:28:25Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['wdqs1024'] 2439906 wikitext text/x-wiki gksrp1y9kb9gmz3njz0b5l5ku4mo6cg 2439907 2439906 2026-07-22T13:30:10Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1024'] 2439907 wikitext text/x-wiki 2s26avt68vgrgq0l0dg0gsbj1yq961f 2439908 2439907 2026-07-22T13:35:06Z Stashbot 7414 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1313932|jawiki: remove redundant permission for confirmed group (T410655 T432850)]] 2439908 wikitext text/x-wiki t93b2mfl8enqsow6uccxwf1delzaesi 2439909 2439908 2026-07-22T13:35:46Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439909 wikitext text/x-wiki 10tcha9rmrq2xifb7rjq5mzmvn0de9v 2439910 2439909 2026-07-22T13:35:54Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['wdqs1024'] 2439910 wikitext text/x-wiki b72js5q6dol12vv2e0hj5c6zrp1mqus 2439911 2439910 2026-07-22T13:35:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.decommission for hosts datahubsearch[1002-1003].eqiad.wmnet 2439911 wikitext text/x-wiki 7doyllzd2vusi1iizxl6v8e3771sqi0 2439912 2439911 2026-07-22T13:37:01Z Stashbot 7414 stran@deploy2003: dragoniez, stran: Backport for [[gerrit:1313932|jawiki: remove redundant permission for confirmed group (T410655 T432850)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439912 wikitext text/x-wiki or9xx8fqxdbco7yr41mq8ja8r1yfoez 2439913 2439912 2026-07-22T13:38:16Z Stashbot 7414 stran@deploy2003: dragoniez, stran: Continuing with deployment 2439913 wikitext text/x-wiki 8ymejjp79cojuflcwwywdfkku180ouj 2439914 2439913 2026-07-22T13:40:49Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: Test 2439914 wikitext text/x-wiki k2aoq6l4w6kxi506zcgzyw7i1o05jiw 2439915 2439914 2026-07-22T13:42:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2439915 wikitext text/x-wiki 7xe4v7wvps91zrzdxx24zxj487dncul 2439916 2439915 2026-07-22T13:42:36Z Stashbot 7414 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313932|jawiki: remove redundant permission for confirmed group (T410655 T432850)]] (duration: 07m 30s) 2439916 wikitext text/x-wiki dc8vo9vhk9zep40d970bq81soeotlis 2439917 2439916 2026-07-22T13:51:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1024.eqiad.wmnet with OS bookworm 2439917 wikitext text/x-wiki 1bf27mcm6q5tqdpiv4t99vdzgz6u46t 2439918 2439917 2026-07-22T13:52:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: datahubsearch[1002-1003].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2003" 2439918 wikitext text/x-wiki ftzcf3q68j2592v5a30nop9qwvjtywb 2439919 2439918 2026-07-22T13:53:40Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: datahubsearch[1002-1003].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2003" 2439919 wikitext text/x-wiki ph6yxpzv9ugdbk4kchn7c5fn0uhs7wz 2439920 2439919 2026-07-22T13:53:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439920 wikitext text/x-wiki l503qijvcvwchywx86gc8ygn6a03fn5 2439921 2439920 2026-07-22T13:53:43Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts datahubsearch[1002-1003].eqiad.wmnet 2439921 wikitext text/x-wiki 9e4p4vrrl5qvq145211m50rd6u8vrn9 2439922 2439921 2026-07-22T13:55:26Z Stashbot 7414 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1313939|Allow users with suppressrevision to see mw-private-personal-info (T431292)]], [[gerrit:1313949|hCaptcha: Stop looking up the viewer's block on every page view (T432518)]] 2439922 wikitext text/x-wiki krnyopyrofkf1x011sk8ctx3ppz8494 2439923 2439922 2026-07-22T13:57:20Z Stashbot 7414 kharlan@deploy2003: dreamyjazz, kharlan: Backport for [[gerrit:1313939|Allow users with suppressrevision to see mw-private-personal-info (T431292)]], [[gerrit:1313949|hCaptcha: Stop looking up the viewer's block on every page view (T432518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2439923 wikitext text/x-wiki ksymtfgcyp9bookjq47som56vpcpv8m 2439924 2439923 2026-07-22T13:58:49Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.decommission for hosts datahubsearch1001.eqiad.wmnet 2439924 wikitext text/x-wiki c753ho2w035fkh17wrr0ijh3j313jse 2439925 2439924 2026-07-22T13:59:52Z Stashbot 7414 kharlan@deploy2003: dreamyjazz, kharlan: Continuing with deployment 2439925 wikitext text/x-wiki ja81qk1hl7epg1nihbhim8unatbnnym 2439926 2439925 2026-07-22T14:04:55Z Stashbot 7414 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1313939|Allow users with suppressrevision to see mw-private-personal-info (T431292)]], [[gerrit:1313949|hCaptcha: Stop looking up the viewer's block on every page view (T432518)]] (duration: 09m 28s) 2439926 wikitext text/x-wiki k81or7ie12h0z3f5l270o40t4lpw0h9 2439927 2439926 2026-07-22T14:17:58Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2439927 wikitext text/x-wiki 8w9fnepfxok9hx68pdrmc4jh3oza7h0 2439928 2439927 2026-07-22T14:19:43Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for 1036 hosts 2439928 wikitext text/x-wiki o61yh2b4nmixawcw8z6j4mywo2kwjrw 2439929 2439928 2026-07-22T14:24:08Z Stashbot 7414 sukhe: restart pybal on lvs1020 2439929 wikitext text/x-wiki jkb51lpq7b2jny0od34vljrsnjb1m6t 2439930 2439929 2026-07-22T14:24:47Z Stashbot 7414 sukhe: restart pybal on lvs1019 2439930 wikitext text/x-wiki sbnk51nst7603a0kdenpy3iqk2hfg0i 2439931 2439930 2026-07-22T14:26:22Z Stashbot 7414 sukhe: sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal' 2439931 wikitext text/x-wiki 2d9jaxgtw07sn9bwy05etmz10ulvykx 2439932 2439931 2026-07-22T14:26:27Z Stashbot 7414 sukhe: sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal': lvs2014 2439932 wikitext text/x-wiki d5n9i9vqy6mynkpb6qgbam0fq1tdncx 2439933 2439932 2026-07-22T14:28:20Z Stashbot 7414 sukhe: sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal': lvs2013 2439933 wikitext text/x-wiki ioodfh99lhb9c8ucwwy0bbrh1jpesmg 2439935 2439933 2026-07-22T14:31:09Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw 2439935 wikitext text/x-wiki 9f5b1xoeacpwjkmjz5hq9ivkvlfx6nq 2439936 2439935 2026-07-22T14:31:17Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: datahubsearch1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2003" 2439936 wikitext text/x-wiki kliwv3iyi1kuxyz2ygaiq0pw2pasb1k 2439937 2439936 2026-07-22T14:32:13Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: datahubsearch1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2003" 2439937 wikitext text/x-wiki ks5j5oibqp5f1o8x5265vpb0jcq0wfr 2439938 2439937 2026-07-22T14:32:14Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439938 wikitext text/x-wiki mito9f1oio2yqgvx36dysyfmvxj38x0 2439939 2439938 2026-07-22T14:32:16Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts datahubsearch1001.eqiad.wmnet 2439939 wikitext text/x-wiki h7hky1oyu8zi4uyiut8smydy8smzadg 2439940 2439939 2026-07-22T14:34:42Z Stashbot 7414 ecarg@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2439940 wikitext text/x-wiki 17l86aktcqsphfa28xo36zu9n6qul25 2439941 2439940 2026-07-22T14:36:10Z Stashbot 7414 ecarg@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2439941 wikitext text/x-wiki dtea4tcl1aa2b1qf42cekswg625rf54 2439942 2439941 2026-07-22T14:37:10Z Stashbot 7414 ecarg@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2439942 wikitext text/x-wiki d38wj2468h0otvcbnohwhxt3kpne4ce 2439943 2439942 2026-07-22T14:37:38Z Stashbot 7414 ecarg@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2439943 wikitext text/x-wiki b3s30zvrgvdda8tzsu6cyx7j506y62u 2439944 2439943 2026-07-22T14:38:10Z Stashbot 7414 ecarg@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2439944 wikitext text/x-wiki 4e7oi00qq15ki2kz6wn0xqin1cr95dg 2439945 2439944 2026-07-22T14:38:34Z Stashbot 7414 ecarg@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2439945 wikitext text/x-wiki b3vetha4rzq7an2hcso73eb0n9tzz5h 2439946 2439945 2026-07-22T14:39:33Z Stashbot 7414 sukhe: ipvsadm --delete-service --tcp-service 10.2.2.55:8087: T432445 2439946 wikitext text/x-wiki o9uh9qns318y3ci84wfb2kupm7whstz 2439947 2439946 2026-07-22T14:39:36Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439947 wikitext text/x-wiki co2o5uootmpwu7jya4c1wfkvdxckuh5 2439950 2439947 2026-07-22T14:41:06Z Stashbot 7414 sukhe: ipvsadm --delete-service --tcp-service 10.2.1.55:8087: lvs2014 and lvs2013 2439950 wikitext text/x-wiki 9p08ullsmab5ds1yd27p3dvs8raw9lm 2439951 2439950 2026-07-22T14:43:09Z Stashbot 7414 ecarg@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2439951 wikitext text/x-wiki kdvbn6k4d5zvve2kqtufoc8slf779z9 2439952 2439951 2026-07-22T14:43:34Z Stashbot 7414 ecarg@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2439952 wikitext text/x-wiki k9t57ye2nld75t5q5gh0xrdvld799ok 2439953 2439952 2026-07-22T14:44:17Z Stashbot 7414 ecarg@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2439953 wikitext text/x-wiki ahd8jcvr0ox3yvk5hpdsyi1mwyt0nqf 2439954 2439953 2026-07-22T14:44:52Z Stashbot 7414 ecarg@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2439954 wikitext text/x-wiki gbd4l1fpfl0wygvm209gi0t26uiyy2t 2439955 2439954 2026-07-22T14:45:07Z Stashbot 7414 ecarg@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2439955 wikitext text/x-wiki 9u0u6idvba57gezgapuxbo9qslbvro1 2439956 2439955 2026-07-22T14:45:53Z Stashbot 7414 ecarg@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2439956 wikitext text/x-wiki hiomihgtlz8px78yy9nibgjb0fthstv 2439959 2439956 2026-07-22T14:48:55Z Stashbot 7414 jiji@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2439959 wikitext text/x-wiki ramdfownt8b1807chom5bxgdjcroloi 2439960 2439959 2026-07-22T14:49:02Z Stashbot 7414 jiji@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2439960 wikitext text/x-wiki qeoyvqvqjz5iewdbiodhpeji9k4lcxo 2439961 2439960 2026-07-22T14:49:10Z Stashbot 7414 jiji@deploy2003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2439961 wikitext text/x-wiki awhtv08yp1u4cbcbxs6hkkv7pl32qaz 2439962 2439961 2026-07-22T14:49:16Z Stashbot 7414 jiji@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2439962 wikitext text/x-wiki e6hjwdv3wr8utnhlfworffset9zhgop 2439963 2439962 2026-07-22T14:50:41Z Stashbot 7414 dancy@deploy2003: Started deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit2002 - T240266 2439963 wikitext text/x-wiki l5mbnjrit5iwtnlx40qjaew81p62b75 2439964 2439963 2026-07-22T14:50:49Z Stashbot 7414 dancy@deploy2003: Finished deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit2002 - T240266 (duration: 00m 09s) 2439964 wikitext text/x-wiki 04v0id9o7qdalanukm7c30qgdqec1dt 2439965 2439964 2026-07-22T14:52:40Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1024.eqiad.wmnet with OS bookworm 2439965 wikitext text/x-wiki gv7c32a0bw8zkhategpvddl4y1andyy 2439966 2439965 2026-07-22T14:55:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1024.eqiad.wmnet with OS bookworm 2439966 wikitext text/x-wiki 01art370ifqyjo8b436st5mlv0u5r90 2439967 2439966 2026-07-22T15:11:23Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439967 wikitext text/x-wiki kkj8x9klet2hxaxedprqfpqbvu3cn1v 2439968 2439967 2026-07-22T15:17:01Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1024.eqiad.wmnet with reason: host reimage 2439968 wikitext text/x-wiki hkiwu7459qse23rpcjvt041e9fqu1ao 2439969 2439968 2026-07-22T15:20:09Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2439969 wikitext text/x-wiki rl2ee3gxxdmags64xy42ixbqs4c1awj 2439970 2439969 2026-07-22T15:20:21Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet 2439970 wikitext text/x-wiki l54d131uhe13enid1519bugkqhjocjl 2439971 2439970 2026-07-22T15:20:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet 2439971 wikitext text/x-wiki 21omvt9uhv28yrwieujicxry0jgxxg5 2439972 2439971 2026-07-22T15:20:24Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1068.eqiad.wmnet 2439972 wikitext text/x-wiki 55ucqq99ok38saukrgla72pwnuwzj24 2439973 2439972 2026-07-22T15:23:16Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439973 wikitext text/x-wiki 04i2vor5q2xfwk2ug1ln273freonyuh 2439974 2439973 2026-07-22T15:23:30Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc-gp2006 2439974 wikitext text/x-wiki fudksehl6wpd5zeywt78h07zphtks6i 2439975 2439974 2026-07-22T15:23:40Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-gp2006 2439975 wikitext text/x-wiki s4lw92ecdix5wpucx45rzhufjyip8fa 2439976 2439975 2026-07-22T15:24:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1024.eqiad.wmnet with reason: host reimage 2439976 wikitext text/x-wiki k7qq83i28avuj2flz0sh852fps7hwvg 2439977 2439976 2026-07-22T15:27:28Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc-gp2006 2439977 wikitext text/x-wiki ewerh8ye1l61rngmsketih85g0xgm5n 2439978 2439977 2026-07-22T15:27:38Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-gp2006 2439978 wikitext text/x-wiki 5l5vcd1qcf0700lrlfhoqfwb3z5oj92 2439979 2439978 2026-07-22T15:28:25Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2439979 wikitext text/x-wiki 5ryh9gn8nkg2daqdptq2s00fce0otg6 2439980 2439979 2026-07-22T15:31:37Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439980 wikitext text/x-wiki jyzpu79e0z6x6t5ldfhmuse3lb9rxu3 2439981 2439980 2026-07-22T15:33:43Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc-gp2006 2439981 wikitext text/x-wiki e25nq5sikigepy48enj8izqz0aufal8 2439982 2439981 2026-07-22T15:33:51Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2439982 wikitext text/x-wiki s2l7br2u9sdy11dg9icmg0jngw2zmhv 2439983 2439982 2026-07-22T15:33:54Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-gp2006 2439983 wikitext text/x-wiki 0umpq49sm1vu5yqw4eo479iq9e2ojgc 2439984 2439983 2026-07-22T15:33:56Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2439984 wikitext text/x-wiki fxzinlir8xr7uyg1s854bu2fw0rs9rr 2439986 2439984 2026-07-22T15:36:06Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc-gp2006 2439986 wikitext text/x-wiki g7u04rnj0meu4lnnl4pf3f1rk94hvy0 2439987 2439986 2026-07-22T15:36:15Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-gp2006 2439987 wikitext text/x-wiki rmmzu8o6rqowftmzpjowccj62ajirdv 2439988 2439987 2026-07-22T15:36:57Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc-gp2006 2439988 wikitext text/x-wiki 9pvclon6vd1tidn8zjuzse103fn8sl4 2439989 2439988 2026-07-22T15:37:11Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-gp2006 2439989 wikitext text/x-wiki sr7qr1dgjsiqjuw7njg8nn73a5ji7ae 2439990 2439989 2026-07-22T15:37:42Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2439990 wikitext text/x-wiki oq7yh67ezq5g2172sum5n8z5ysgk1wq 2439991 2439990 2026-07-22T15:40:54Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2439991 wikitext text/x-wiki n13iql7z25ns19utw03kegu6857oekr 2439992 2439991 2026-07-22T15:42:47Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc-gp2006 2439992 wikitext text/x-wiki 8vit4vkxynh294xmg04nh3omnq656iy 2439993 2439992 2026-07-22T15:42:57Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-gp2006 2439993 wikitext text/x-wiki dgc631b2w15l68gfvxj4c7ew39yw87g 2439994 2439993 2026-07-22T15:46:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2439994 wikitext text/x-wiki c94g6uxmkx4wfjmjzmbpjh1mn89hg4a 2439995 2439994 2026-07-22T15:47:14Z Stashbot 7414 dancy@deploy2003: Started deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit2002 - T240266 2439995 wikitext text/x-wiki 5b6cnw2b4ep53tz7k785j4uhj8mh70k 2439996 2439995 2026-07-22T15:47:25Z Stashbot 7414 dancy@deploy2003: Finished deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit2002 - T240266 (duration: 00m 10s) 2439996 wikitext text/x-wiki 19axljzlmq8x8ozwg25wsh8zfxg9rzg 2439997 2439996 2026-07-22T15:48:11Z Stashbot 7414 dancy@deploy2003: Started deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit1003 - T240266 2439997 wikitext text/x-wiki fjs1q66qszthe6ml33uvecyaf9gn1rb 2439998 2439997 2026-07-22T15:48:26Z Stashbot 7414 dancy@deploy2003: Finished deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit1003 - T240266 (duration: 00m 15s) 2439998 wikitext text/x-wiki mh33vl8a6tcjxt4is8ib6duxqkni3yy 2439999 2439998 2026-07-22T15:49:43Z Stashbot 7414 dancy@deploy2003: Started deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit1003 - T240266 2439999 wikitext text/x-wiki m8b3fqsbs7rij9ezf9a6xzokiu5jjvm 2440000 2439999 2026-07-22T15:49:54Z Stashbot 7414 dancy@deploy2003: Finished deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit1003 - T240266 (duration: 00m 10s) 2440000 wikitext text/x-wiki schupk4vcy2ozsvepfmi0ljxayv2920 2440001 2440000 2026-07-22T15:50:20Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1024.eqiad.wmnet with OS bookworm 2440001 wikitext text/x-wiki 6eqo0nv0ph03f6hrt7fpt9hw6xzq27p 2440003 2440001 2026-07-22T15:58:08Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw 2440003 wikitext text/x-wiki rcgq2azeo19rycb59xj2zanwgep9rh3 2440004 2440003 2026-07-22T15:58:10Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw 2440004 wikitext text/x-wiki o5fpx7xv3a8yq4k61hu90g5d2lj7c92 2440005 2440004 2026-07-22T15:58:11Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw 2440005 wikitext text/x-wiki 442tqk74607sykzfy8bnb9h5v2fccot 2440007 2440005 2026-07-22T16:01:03Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin2003 - T431826 2440007 wikitext text/x-wiki nc11riyzf3atmny1cy8z6a8t15fmykm 2440008 2440007 2026-07-22T16:04:24Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw 2440008 wikitext text/x-wiki 1rj52b91mf2ln86ihi98h1hjbk0hvtn 2440009 2440008 2026-07-22T16:04:26Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw 2440009 wikitext text/x-wiki q800zosoxg23spx7ixxhepf02wkjai8 2440010 2440009 2026-07-22T16:04:27Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw 2440010 wikitext text/x-wiki kvxcd80svfea6voe9sdp0w13w70vo6l 2440011 2440010 2026-07-22T16:07:51Z Stashbot 7414 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2440011 wikitext text/x-wiki 1xo3gx5874vntx9dx0vswxjuvzm4u32 2440012 2440011 2026-07-22T16:08:11Z Stashbot 7414 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2440012 wikitext text/x-wiki ifds2iludrcgxt2w36dolwsn3dxdkse 2440013 2440012 2026-07-22T16:18:18Z Stashbot 7414 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply 2440013 wikitext text/x-wiki fd2hzi4a5t9y2nzhyqv5r047gqndxj6 2440014 2440013 2026-07-22T16:18:32Z Stashbot 7414 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply 2440014 wikitext text/x-wiki 3cwc0me11alrm71zl82cm80i591ylom 2440015 2440014 2026-07-22T16:21:22Z Stashbot 7414 kevinbazira@deploy2003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2440015 wikitext text/x-wiki sxd1nqur8g1z9l96def4sc2yuwc8qw0 2440016 2440015 2026-07-22T16:24:47Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=eqiad 2440016 wikitext text/x-wiki 0lpoblah6k6g7lcbn2e6zh6htrmavyw 2440017 2440016 2026-07-22T16:25:33Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2440017 wikitext text/x-wiki 2cemeto40shmhx0bcd4ypll52uim74l 2440018 2440017 2026-07-22T16:25:41Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 29s) 2440018 wikitext text/x-wiki 8d5b9hqzhfeirb6p89jhf7ctlgv24as 2440019 2440018 2026-07-22T16:26:23Z Stashbot 7414 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply 2440019 wikitext text/x-wiki 00xz8cejyqqpb5d47u7zc4s1mciamr0 2440020 2440019 2026-07-22T16:26:52Z Stashbot 7414 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply 2440020 wikitext text/x-wiki la120jvk72dvb43atm0k4usy4ul0ipl 2440021 2440020 2026-07-22T16:26:57Z Stashbot 7414 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply 2440021 wikitext text/x-wiki hhm9inzngd30xq65yblqkx3qrbuztva 2440022 2440021 2026-07-22T16:27:27Z Stashbot 7414 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply 2440022 wikitext text/x-wiki tndghljc3z2zshpnnese34djtnhzlzr 2440023 2440022 2026-07-22T16:27:46Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=eqiad 2440023 wikitext text/x-wiki m89w1jo0scpnxnytg051629cr0pbvb9 2440024 2440023 2026-07-22T16:28:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1024.eqiad.wmnet, repooling source-only afterwards 2440024 wikitext text/x-wiki 1f5ifa4f3xxcjo7f9pkes7ifg885jhe 2440025 2440024 2026-07-22T16:34:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1013.eqiad.wmnet with OS bookworm 2440025 wikitext text/x-wiki mrlhm03ifb957lyu08mm837kzz5hl12 2440026 2440025 2026-07-22T16:34:39Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1013 2440026 wikitext text/x-wiki 96y4rbk077rl4ee3mi51d70g7kwsfwc 2440027 2440026 2026-07-22T16:34:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2440027 wikitext text/x-wiki 1t8tkv92c7qp8eb4k1ld9byq4pjnd28 2440028 2440027 2026-07-22T16:39:07Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1013 - bking@cumin2003" 2440028 wikitext text/x-wiki l27ql674gen57ajt7pifr3igin0v0es 2440029 2440028 2026-07-22T16:39:37Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1013 - bking@cumin2003" 2440029 wikitext text/x-wiki qz7slw9xobutith7z7w4s5amz4ttogf 2440030 2440029 2026-07-22T16:39:38Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440030 wikitext text/x-wiki ik7ae5ge5t22soikhrl1j3czfafp5zz 2440031 2440030 2026-07-22T16:39:40Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1013.eqiad.wmnet 105.32.64.10.in-addr.arpa 5.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440031 wikitext text/x-wiki m9mmc18soz44ppph7ly3ns6grc4ofd6 2440032 2440031 2026-07-22T16:39:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1013.eqiad.wmnet 105.32.64.10.in-addr.arpa 5.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440032 wikitext text/x-wiki dcca0v1k6p40ycelng0ft8qecn3npea 2440033 2440032 2026-07-22T16:39:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1013 2440033 wikitext text/x-wiki cb9f3l90lifnm9xep8z83bqcepj6n7h 2440034 2440033 2026-07-22T16:40:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1013 2440034 wikitext text/x-wiki 8rx020mvk9zevpy3f3kbqjfar3nvvhh 2440035 2440034 2026-07-22T16:40:32Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1013 2440035 wikitext text/x-wiki gtab8utz1at9jw3g3qrfi3mudhr8lc9 2440036 2440035 2026-07-22T16:54:52Z Stashbot 7414 dancy@deploy2003: Started deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit2003 - T240266 2440036 wikitext text/x-wiki c4vjellc3iflzkdbvmsqxft2f9q2z7u 2440037 2440036 2026-07-22T16:55:03Z Stashbot 7414 dancy@deploy2003: Finished deploy [gerrit/gerrit@2fa4c79]: Applying updated motd plugin on gerrit2003 - T240266 (duration: 00m 11s) 2440037 wikitext text/x-wiki d4bhc4vgvzpdvsdvm2lqpr44vz3rvjx 2440038 2440037 2026-07-22T16:55:24Z Stashbot 7414 kamila@deploy2003: Locking from deployment [MediaWiki]: deployment server switchover - T423714 2440038 wikitext text/x-wiki 4bmvq62yymcsjxgta19hso9yy8aah5n 2440039 2440038 2026-07-22T16:57:57Z Stashbot 7414 kamila@deploy2003: Unlocked for deployment [MediaWiki]: deployment server switchover - T423714 (duration: 02m 33s) 2440039 wikitext text/x-wiki 0lnci1bg0rx9npkpds8489vmzxn1q1y 2440040 2440039 2026-07-22T16:58:06Z Stashbot 7414 kamila@deploy2003: Locking from deployment [MediaWiki]: deployment server switchover - T423714 2440040 wikitext text/x-wiki oowsgsjcnmrwpda6hqmdmb8iqqz0v2v 2440042 2440040 2026-07-22T17:00:19Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1013.eqiad.wmnet with reason: host reimage 2440042 wikitext text/x-wiki oesb1gqivp9n1ojh3nsn2kmrvcdl152 2440043 2440042 2026-07-22T17:01:18Z Stashbot 7414 kamila@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet,releases1003.eqiad.wmnet with reason: Deployment server switchover 2440043 wikitext text/x-wiki 422s1ez5ms8aln57goioskm5zfie9h1 2440045 2440043 2026-07-22T17:02:38Z Stashbot 7414 kamila@dns7001: START - running authdns-update 2440045 wikitext text/x-wiki 14zp1yfghfxmsokiyk68d7yl6scyi5h 2440046 2440045 2026-07-22T17:03:55Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1013.eqiad.wmnet with reason: host reimage 2440046 wikitext text/x-wiki 7mm1x2d1ybd65871x2ikvc69s9qzft2 2440047 2440046 2026-07-22T17:04:39Z Stashbot 7414 kamila@dns7001: END - running authdns-update 2440047 wikitext text/x-wiki csi9ea0hxln1udrpcayd9ip15vbjf2g 2440048 2440047 2026-07-22T17:04:56Z Stashbot 7414 Raine: point deployment.eqiad.wmnet to deploy1003 2440048 wikitext text/x-wiki eqspd7rvpxsjjurd7b312w38tbl2gk5 2440049 2440048 2026-07-22T17:12:30Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1024.eqiad.wmnet, repooling source-only afterwards 2440049 wikitext text/x-wiki 04on9tlxrlmp0v9tr1dwcfjqulmhvez 2440050 2440049 2026-07-22T17:20:57Z Stashbot 7414 kamila@deploy2003: Unlocked for deployment [MediaWiki]: deployment server switchover - T423714 (duration: 22m 50s) 2440050 wikitext text/x-wiki j19ai12nzin9nzz46prpv13fe3chdw3 2440052 2440050 2026-07-22T17:27:17Z Stashbot 7414 kamila@deploy1003: Started scap sync-world: Test deployment to validate deployment server switchover - T423714 2440052 wikitext text/x-wiki pk9usinnova93qf42qy3318gp68f60z 2440054 2440052 2026-07-22T17:29:20Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1013.eqiad.wmnet with OS bookworm 2440054 wikitext text/x-wiki 0zmy0mjuvgdzhpfyzeic4qzo2k0qdg3 2440055 2440054 2026-07-22T17:54:36Z Stashbot 7414 mutante: restarting gerrit for maintenance 2440055 wikitext text/x-wiki 4yw5afq0krkmyzl53kmib3hbysgpv2o 2440056 2440055 2026-07-22T17:55:42Z Stashbot 7414 kamila@deploy1003: Finished scap sync-world: Test deployment to validate deployment server switchover - T423714 (duration: 28m 24s) 2440056 wikitext text/x-wiki am3y6lkcinhc0cb9jzo9lj8y9u5tzfz 2440057 2440056 2026-07-22T17:56:44Z Stashbot 7414 Raine: deployment server switchover => deploy1003 is primary now 2440057 wikitext text/x-wiki ewjf2l27yw75600ppv8dw3fy6ovacdr 2440063 2440057 2026-07-22T18:20:10Z Stashbot 7414 amastilovic@deploy1003: Started deploy [analytics/refinery@2a25417] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2a25417d] 2440063 wikitext text/x-wiki 5xgg168wnylvdavyltkeqwsb8eydul3 2440064 2440063 2026-07-22T18:22:09Z Stashbot 7414 amastilovic@deploy1003: Finished deploy [analytics/refinery@2a25417] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2a25417d] (duration: 01m 59s) 2440064 wikitext text/x-wiki 8qvrukwphdf97lbwfq0wmb9g2uffzby 2440065 2440064 2026-07-22T18:23:44Z Stashbot 7414 amastilovic@deploy1003: Started deploy [analytics/refinery@2a25417]: Regular analytics weekly train [analytics/refinery@2a25417d] 2440065 wikitext text/x-wiki ts0ijs6fri6yj4jjeze44yeguxp0zd4 2440067 2440065 2026-07-22T18:27:47Z Stashbot 7414 dduvall: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/1314025 (4 jobs updated) 2440067 wikitext text/x-wiki q40cohe8jt9i9n7di3acvaya6781n7p 2440068 2440067 2026-07-22T18:28:15Z Stashbot 7414 amastilovic@deploy1003: Finished deploy [analytics/refinery@2a25417]: Regular analytics weekly train [analytics/refinery@2a25417d] (duration: 04m 31s) 2440068 wikitext text/x-wiki h879k5met78xvb8462ieskpveo1q0fl 2440069 2440068 2026-07-22T18:28:23Z Stashbot 7414 amastilovic@deploy1003: Started deploy [analytics/refinery@2a25417] (thin): Regular analytics weekly train THIN [analytics/refinery@2a25417d] 2440069 wikitext text/x-wiki swyuxody3allbaeu4p4ue2fx0jtxv4t 2440070 2440069 2026-07-22T18:30:33Z Stashbot 7414 amastilovic@deploy1003: Finished deploy [analytics/refinery@2a25417] (thin): Regular analytics weekly train THIN [analytics/refinery@2a25417d] (duration: 02m 09s) 2440070 wikitext text/x-wiki gwvlam8cjisftfoy7aohhem6siopw5k 2440071 2440070 2026-07-22T18:36:25Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=eqiad 2440071 wikitext text/x-wiki 0e4woyb5xdrzltvt6u4q6peym3cchnd 2440072 2440071 2026-07-22T18:39:29Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2440072 wikitext text/x-wiki t6pkk1qf088wiwobfp32izp57a6z6w3 2440073 2440072 2026-07-22T18:39:37Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 11s) 2440073 wikitext text/x-wiki i3to9c22ph1tjlfs06fln4sgz2bpidb 2440075 2440073 2026-07-22T18:51:38Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2016.codfw.wmnet with OS bookworm 2440075 wikitext text/x-wiki 9p4m7bwr6bbc169yd9q8zdeft162h4i 2440076 2440075 2026-07-22T18:51:42Z Stashbot 7414 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.12 refs T430831 2440076 wikitext text/x-wiki 0ztllvjw2uout1oq3hr4ilfms9w3dbi 2440077 2440076 2026-07-22T18:51:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2016 2440077 wikitext text/x-wiki cjk789syteeolwgwvrtzwalwflw20x7 2440078 2440077 2026-07-22T18:52:00Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2016 2440078 wikitext text/x-wiki sr8dte80wop0ra03siu6mw1nyhjw9zg 2440079 2440078 2026-07-22T18:54:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling source-only afterwards 2440079 wikitext text/x-wiki soexyer5if0blf341bkmts9ovq2xm5n 2440080 2440079 2026-07-22T19:11:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2016.codfw.wmnet with reason: host reimage 2440080 wikitext text/x-wiki r11e7xk8vpvgzvwes0f0xrgu1tcqjbv 2440081 2440080 2026-07-22T19:14:45Z Stashbot 7414 dancy@deploy1003: Started deploy [zuul/deploy@d92e238]: Freshening Zuul installation 2440081 wikitext text/x-wiki 0hsh6h1ks8tr79qkm5k1hbo0xpy9llq 2440082 2440081 2026-07-22T19:15:01Z Stashbot 7414 dancy@deploy1003: Finished deploy [zuul/deploy@d92e238]: Freshening Zuul installation (duration: 00m 15s) 2440082 wikitext text/x-wiki e4mnq835o1fbton8aewnm3n138umojw 2440083 2440082 2026-07-22T19:19:14Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2016.codfw.wmnet with reason: host reimage 2440083 wikitext text/x-wiki j3f7nrlz6mp0iaaq83ff3fl3mv0ms0e 2440085 2440083 2026-07-22T19:27:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2244: Testing 2440085 wikitext text/x-wiki 08xw48xd5e5tw0vzr8zm4oi6cysuzqc 2440086 2440085 2026-07-22T19:27:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2244: Testing 2440086 wikitext text/x-wiki mtns8crnwvz2ttohb6cxillcxod2600 2440087 2440086 2026-07-22T19:29:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2244: Testing 2440087 wikitext text/x-wiki pjmfo2cj0wmxn06fyk1wwark0k237y6 2440088 2440087 2026-07-22T19:40:07Z Stashbot 7414 mutante: gerrit - one more service restart is needed - restarting 2440088 wikitext text/x-wiki pbtjb9ae4jbutoioo8ycy5em6603xqs 2440089 2440088 2026-07-22T19:43:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2016.codfw.wmnet with OS bookworm 2440089 wikitext text/x-wiki 0n123gdvldwrt2374leu9duclirv4ff 2440094 2440089 2026-07-22T19:56:58Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling source-only afterwards 2440094 wikitext text/x-wiki gbwok97bnsxm1orhyfq15cu0v2gfv89 2440095 2440094 2026-07-22T20:06:09Z Stashbot 7414 dani@deploy1003: Started scap sync-world: Backport for [[gerrit:1314046|Deploy Referring Experiences survey on enwiki (T432289)]], [[gerrit:1314106|Set $wgAutoConfirmCount to 10 for enwikiquote (T432895)]] 2440095 wikitext text/x-wiki rw1mn3219lzv1pk53mmegdaex23qn74 2440097 2440095 2026-07-22T20:14:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2244: Testing 2440097 wikitext text/x-wiki nkqdbamd582b7b0olcf2d89krpmjis2 2440099 2440097 2026-07-22T20:24:52Z Stashbot 7414 dani@deploy1003: dani, vadymts1: Backport for [[gerrit:1314046|Deploy Referring Experiences survey on enwiki (T432289)]], [[gerrit:1314106|Set $wgAutoConfirmCount to 10 for enwikiquote (T432895)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2440099 wikitext text/x-wiki 2j0fmgn3u3rl67ba2kzzy0er7uvk4pb 2440100 2440099 2026-07-22T20:26:19Z Stashbot 7414 dani@deploy1003: dani, vadymts1: Continuing with deployment 2440100 wikitext text/x-wiki fmnmqsi7233798j2ywneaojya2ntnz4 2440105 2440100 2026-07-22T20:38:29Z Stashbot 7414 ryankemper: [WDQS] Pruned obsolete requestctl action+pattern `wdqs_20260715_p2003_ring_ja3n` (actor rotated JA3Ns; rule inert) 2440105 wikitext text/x-wiki bqx7l5x2o8acli22hrgzyu31jxmic89 2440106 2440105 2026-07-22T20:38:59Z Stashbot 7414 dani@deploy1003: Finished scap sync-world: Backport for [[gerrit:1314046|Deploy Referring Experiences survey on enwiki (T432289)]], [[gerrit:1314106|Set $wgAutoConfirmCount to 10 for enwikiquote (T432895)]] (duration: 32m 51s) 2440106 wikitext text/x-wiki io4rrd0xp4ib0zryf9o5wxmbs3x3qzo 2440115 2440106 2026-07-22T21:39:43Z Stashbot 7414 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1314116|OATHManage: Don't create recovery codes when viewing the page (T432915 T432916)]] 2440115 wikitext text/x-wiki 3e0333sn9jpug0dp4edtdjuic04lxiq 2440116 2440115 2026-07-22T21:43:45Z Stashbot 7414 sbassett@deploy1003: sbassett: Backport for [[gerrit:1314116|OATHManage: Don't create recovery codes when viewing the page (T432915 T432916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2440116 wikitext text/x-wiki h8o8o2c8mtvsctqcdxm180qbg83m1x6 2440117 2440116 2026-07-22T21:44:14Z Stashbot 7414 sbassett@deploy1003: sbassett: Continuing with deployment 2440117 wikitext text/x-wiki lniy9c0ki5xamqgdx7gtwror6ub3ko5 2440118 2440117 2026-07-22T21:51:36Z Stashbot 7414 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1314116|OATHManage: Don't create recovery codes when viewing the page (T432915 T432916)]] (duration: 11m 52s) 2440118 wikitext text/x-wiki 3t8rst0dxsyinvravv4fa6ww4n3j77g 2440119 2440118 2026-07-22T22:06:17Z Stashbot 7414 ryankemper: [WDQS] Added requestctl per-IP ratelimit `wdqs_heavy_sparql_bots_jul_2026_ratelimit` (chronic heavy-query bot tier driving deadlock-remediation restarts); pruned superseded `wdqs_2026_05_11_worobot` 2440119 wikitext text/x-wiki msm7q64euxgnx9flo08a59h297yqjw5 2440122 2440119 2026-07-22T23:14:25Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: Apply upgrade to JVM17 - eevans@cumin1003 2440122 wikitext text/x-wiki 4p23w247ccv4safvidxf694gz13r8yc 2440123 2440122 2026-07-22T23:34:07Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: Apply upgrade to JVM17 - eevans@cumin1003 2440123 wikitext text/x-wiki hmwr3buyhmp3luobqlfwvh2a44dmsyp 2440127 2440123 2026-07-23T05:54:36Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore scholarly data after Bookworm reimage) xfer scholarly_articles from wdqs2024.codfw.wmnet -> wdqs2016.codfw.wmnet, repooling source-only afterwards 2440127 wikitext text/x-wiki m9zsrrxaxp0ga6ue4q9nn1xxfuxmjt2 2440128 2440127 2026-07-23T05:58:44Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs1013.eqiad.wmnet 2440128 wikitext text/x-wiki dsvfyy3kku4mcx5z7zu335p23u3x1v2 2440129 2440128 2026-07-23T05:58:45Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2007.codfw.wmnet 2440129 wikitext text/x-wiki 25149cr5l8jxi8tpwpidcnoznvv6aeb 2440130 2440129 2026-07-23T06:16:30Z Stashbot 7414 ryankemper: [WDQS] T430880 Repooled wdqs2007 after successful Bookworm reimage, data transfer, and postflight validation; wdqs1013 also passed postflights and is enabled in conftool, but remains out of IPVS pending a rolling pybal restart to clear its stale pre-VLAN-move address 2440130 wikitext text/x-wiki mbgibivy5b384a8groikgu6nlp29jl9 2440131 2440130 2026-07-23T07:13:22Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440131 wikitext text/x-wiki 93bfrm046x2iqak1mlt0xk7mec6gdwb 2440132 2440131 2026-07-23T07:14:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 45 hosts 2440132 wikitext text/x-wiki k241llho9v9f7ccjio0fa5gn7zsqm7i 2440133 2440132 2026-07-23T07:17:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1069.eqiad.wmnet 2440133 wikitext text/x-wiki p9q4vgtbfjjjjtfhspwm9f1qu7wp9up 2440134 2440133 2026-07-23T07:17:57Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet 2440134 wikitext text/x-wiki jhclgxvcgqcca47a8vv10f7bc01lxfc 2440135 2440134 2026-07-23T07:18:54Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 45 hosts 2440135 wikitext text/x-wiki ssxokj0ve951zglhkz234hf2qv40xcm 2440136 2440135 2026-07-23T07:19:04Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet 2440136 wikitext text/x-wiki b3itg72i2mcvr2ynyafn6moms9evygm 2440137 2440136 2026-07-23T07:19:23Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1069.eqiad.wmnet with OS trixie 2440137 wikitext text/x-wiki pgm5yvh808rhb9gc94rzawpwcpchamm 2440138 2440137 2026-07-23T07:19:51Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1069 2440138 wikitext text/x-wiki cyr2kj30h065odjboia2xrix5xe7gm7 2440139 2440138 2026-07-23T07:19:59Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2440139 wikitext text/x-wiki ji5hoy01ip0vezng728vsb0yhsjcbhj 2440140 2440139 2026-07-23T07:20:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 2 hosts 2440140 wikitext text/x-wiki n9cei39d0ux889jclw4p1ikuifh3v0d 2440141 2440140 2026-07-23T07:20:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 2 hosts 2440141 wikitext text/x-wiki nqerhrkxd6nlke8sl8rg1n66igtpplu 2440142 2440141 2026-07-23T07:20:43Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1031.eqiad.wmnet,service=s2 2440142 wikitext text/x-wiki qew1phvrjb17h5qvibhmrlhtz3wqyuy 2440143 2440142 2026-07-23T07:20:46Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1031.eqiad.wmnet,service=s7 2440143 wikitext text/x-wiki ea4lqj4i66jxgmhmx8oycighc06hpwm 2440144 2440143 2026-07-23T07:20:58Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1031.eqiad.wmnet,service=s2 2440144 wikitext text/x-wiki 7tm9efy1ms8opfr0zrs4464p65mxh07 2440145 2440144 2026-07-23T07:21:02Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1031.eqiad.wmnet,service=s7 2440145 wikitext text/x-wiki bxqzmfgo6dzcf4m91rr98zxx5z4fwut 2440146 2440145 2026-07-23T07:21:25Z Stashbot 7414 jiji@deploy1003: Started scap sync-world: 1314018 mediawiki: bump mcrouter and mesh modules 2440146 wikitext text/x-wiki cg8y4z97dnj0k582og6z4kr59pg2tgg 2440147 2440146 2026-07-23T07:22:35Z Stashbot 7414 jiji@deploy1003: jiji: 1314018 mediawiki: bump mcrouter and mesh modules synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2440147 wikitext text/x-wiki hxze8624iaqjc3vlqk22jkl7xw2ur43 2440148 2440147 2026-07-23T07:24:48Z Stashbot 7414 jiji@deploy1003: jiji: Continuing with deployment 2440148 wikitext text/x-wiki 56cggg2fk14ox7qyplch497kckej53u 2440149 2440148 2026-07-23T07:25:27Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply 2440149 wikitext text/x-wiki mm11fwq08ctrr35ozn7hp0y5ql8hadd 2440150 2440149 2026-07-23T07:25:34Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply 2440150 wikitext text/x-wiki 0o373h5irh4xeq4wh4sewbhx3ra1l24 2440151 2440150 2026-07-23T07:25:36Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1069 - jiji@cumin1003" 2440151 wikitext text/x-wiki dn16yzfwsrpjbyk60dod0rwijtp38ai 2440152 2440151 2026-07-23T07:25:40Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1069 - jiji@cumin1003" 2440152 wikitext text/x-wiki fwc4arjr06jxdbp9aka4trtplza9in0 2440153 2440152 2026-07-23T07:25:41Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440153 wikitext text/x-wiki 67mf778s20gae0pmvcyto1jcjhr5t5w 2440154 2440153 2026-07-23T07:25:43Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1069.eqiad.wmnet 164.48.64.10.in-addr.arpa 4.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440154 wikitext text/x-wiki kf71xwul5xl9xlybu22a5592m7i3mvs 2440155 2440154 2026-07-23T07:25:44Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1069.eqiad.wmnet 164.48.64.10.in-addr.arpa 4.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440155 wikitext text/x-wiki 308corygc4pkvdvxqjqa4heql5irpjr 2440156 2440155 2026-07-23T07:25:46Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1069 2440156 wikitext text/x-wiki 9r6i6nrptpmcvrdzlh8aaok3puyp2t2 2440157 2440156 2026-07-23T07:26:37Z Stashbot 7414 jiji@deploy1003: Finished scap sync-world: 1314018 mediawiki: bump mcrouter and mesh modules (duration: 06m 01s) 2440157 wikitext text/x-wiki rbpkydxcez5hpzzzzlubp8vwffwa4ks 2440158 2440157 2026-07-23T07:29:12Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1069 2440158 wikitext text/x-wiki 03z03gnfq65ihr2kevl0ukjcq6w5tje 2440159 2440158 2026-07-23T07:29:14Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1069 2440159 wikitext text/x-wiki qypbktkiolhy5jxhkcywun18fg9f0fd 2440160 2440159 2026-07-23T07:32:47Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2440160 wikitext text/x-wiki tteb8390ebpfdqfyxnq5b9p4fl2z0fb 2440161 2440160 2026-07-23T07:33:00Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2440161 wikitext text/x-wiki gj9c02hn7tm2vg3z7icul4n19i66jim 2440162 2440161 2026-07-23T07:33:10Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2440162 wikitext text/x-wiki 28trb1kpafwyxta847dmwrb23fynn52 2440163 2440162 2026-07-23T07:33:26Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2440163 wikitext text/x-wiki c57f1nuls0ws3lyfqpmowyfervw9h20 2440164 2440163 2026-07-23T07:33:35Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2440164 wikitext text/x-wiki a62fsxk4vk5b1aznhjouohzeonecug3 2440165 2440164 2026-07-23T07:33:39Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2440165 wikitext text/x-wiki k1tt02rynf5yjkjbcru8isudzg343hv 2440166 2440165 2026-07-23T07:34:16Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1035 hosts 2440166 wikitext text/x-wiki 1u5wsfl7l6cifkpjpzilngb48uzaw3v 2440167 2440166 2026-07-23T07:45:45Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1069.eqiad.wmnet with reason: host reimage 2440167 wikitext text/x-wiki 98715w5q88t7p00oeff2ja59sk0k0hs 2440168 2440167 2026-07-23T07:49:03Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1069.eqiad.wmnet with reason: host reimage 2440168 wikitext text/x-wiki 99fx5tn11rw1vr05ml3fu8f3gd7ahp8 2440169 2440168 2026-07-23T08:10:34Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1069.eqiad.wmnet with OS trixie 2440169 wikitext text/x-wiki s70dbm189hd4hlkel8tqlye2fita1xg 2440170 2440169 2026-07-23T08:35:13Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440170 wikitext text/x-wiki rx067277akaw3mmlywt50otscm3jgp0 2440171 2440170 2026-07-23T08:37:02Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2440171 wikitext text/x-wiki olpo2ijz0d6jwxd5vb0932lk9pnhahx 2440172 2440171 2026-07-23T08:38:43Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2440172 wikitext text/x-wiki kyyblfe311n02ri1lcqslh5w980mt1u 2440173 2440172 2026-07-23T08:38:51Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2440173 wikitext text/x-wiki f45hpcsh0ae4j2nojuwu8lwgd8b1i7h 2440174 2440173 2026-07-23T08:39:08Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2440174 wikitext text/x-wiki o6h4wlw54ut763funmbqo4vs5tr2xx5 2440175 2440174 2026-07-23T08:46:40Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet 2440175 wikitext text/x-wiki ps470h6jm5once6yox6suuhjr7wyyal 2440176 2440175 2026-07-23T08:46:42Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet 2440176 wikitext text/x-wiki ma9htw894z0yhh7hpkodfbqy8a6l124 2440177 2440176 2026-07-23T08:46:44Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1069.eqiad.wmnet 2440177 wikitext text/x-wiki ltnfy7aarcyj5gec8o8icve0jlaxmd4 2440178 2440177 2026-07-23T08:49:17Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2440178 wikitext text/x-wiki t2es7tvqw07t9q1ahwc3lk05uoqqvcg 2440179 2440178 2026-07-23T08:49:45Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2440179 wikitext text/x-wiki 66n1t6xh3355g9y9l2tdi0jvhikw85h 2440180 2440179 2026-07-23T08:49:51Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2440180 wikitext text/x-wiki surx7jpx1n5lutglr79vuik9h98h7u1 2440181 2440180 2026-07-23T08:50:10Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2440181 wikitext text/x-wiki m56i8fxmm6dm13ixcufnbr4087hlet6 2440182 2440181 2026-07-23T08:50:16Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2440182 wikitext text/x-wiki cyf6y8z2c4hhaiu9zkhbdprx17mjqm0 2440183 2440182 2026-07-23T08:50:38Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2440183 wikitext text/x-wiki ttasl5qcls3xglloxm7wdiqrxqbp0xy 2440184 2440183 2026-07-23T09:09:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1055: testing 2440184 wikitext text/x-wiki 81bftc739df1155fnemibh3qn8c1o72 2440185 2440184 2026-07-23T09:09:34Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1055: testing 2440185 wikitext text/x-wiki h1t90uqq3b08ioq2entb5v5m2mxjm5v 2440186 2440185 2026-07-23T09:09:44Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440186 wikitext text/x-wiki 0z2l4ai3g5zqg72x6o6u1ky7mw1tgu7 2440187 2440186 2026-07-23T09:11:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1055: testing 2440187 wikitext text/x-wiki mzrmm2z6rmkhl2otqpre8gs4b2hk9z6 2440188 2440187 2026-07-23T09:12:03Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: testing 2440188 wikitext text/x-wiki abwazo861yrpntepwjesgwehbv3nk63 2440189 2440188 2026-07-23T09:12:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: testing 2440189 wikitext text/x-wiki t6221d58ktzsl922u990fpk3l9z845w 2440190 2440189 2026-07-23T09:12:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: testing 2440190 wikitext text/x-wiki iqu68y77kyr4h5weoie0642hu981zuh 2440192 2440190 2026-07-23T09:27:03Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440192 wikitext text/x-wiki b1t9avifcwk3zuwirgvn7xtx6fmamro 2440194 2440192 2026-07-23T09:41:40Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1070.eqiad.wmnet 2440194 wikitext text/x-wiki 2g711o06oxgg4kac0pd01x1i1mzd5xf 2440195 2440194 2026-07-23T09:41:44Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet 2440195 wikitext text/x-wiki thggta60zyk86r1dd6z3b9j2ulrdoeb 2440196 2440195 2026-07-23T09:42:17Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet 2440196 wikitext text/x-wiki 52t5w81ibcqgr7kp69o54ul0uqyioh5 2440197 2440196 2026-07-23T09:42:43Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1070.eqiad.wmnet with OS trixie 2440197 wikitext text/x-wiki ggakhzkfs2q3fukkfku6an1aas5imrv 2440198 2440197 2026-07-23T09:43:00Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1070 2440198 wikitext text/x-wiki dzi7yd9znbej1i6z20kq0p5zldoc8i1 2440199 2440198 2026-07-23T09:45:40Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2440199 wikitext text/x-wiki bi800rcwztd6udm24s6jkqtf8quacte 2440200 2440199 2026-07-23T09:47:21Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for 1035 hosts 2440200 wikitext text/x-wiki 4hgqx49ba1twa3l0wrfp32mvdkmqxst 2440201 2440200 2026-07-23T09:56:19Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1070 - jiji@cumin1003" 2440201 wikitext text/x-wiki ezp5dzogfucbli3lcjvemu9nczn0kob 2440202 2440201 2026-07-23T09:56:23Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1070 - jiji@cumin1003" 2440202 wikitext text/x-wiki pxcwerj762fuyb872s29x3t7m3sbb64 2440203 2440202 2026-07-23T09:56:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440203 wikitext text/x-wiki 2htkxfk65qzocwkhkps16dfsw01hr8l 2440204 2440203 2026-07-23T09:56:26Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1070.eqiad.wmnet 165.48.64.10.in-addr.arpa 5.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440204 wikitext text/x-wiki 9fx1fspxoijvvzezwg4dfo81dtc1xm9 2440205 2440204 2026-07-23T09:56:27Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1070.eqiad.wmnet 165.48.64.10.in-addr.arpa 5.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440205 wikitext text/x-wiki p3ncu98w4zvxqpcjb3m3y53vo8jax6p 2440206 2440205 2026-07-23T09:56:29Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1070 2440206 wikitext text/x-wiki d6nm56t6zlo7h14gtkfxq7p6wwktmzs 2440207 2440206 2026-07-23T09:56:42Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1055: testing 2440207 wikitext text/x-wiki opsn451eogbne5j58u97njp4csocu2n 2440208 2440207 2026-07-23T09:57:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1054: testing 2440208 wikitext text/x-wiki tdlywm9qp9g4m977wm941l0bmfsslmv 2440209 2440208 2026-07-23T09:57:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1054: testing 2440209 wikitext text/x-wiki m2zz8l6u4cl57qs7hz5r69q1zwql10j 2440210 2440209 2026-07-23T09:57:38Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1070 2440210 wikitext text/x-wiki mzp9xeu07jcgtwpoc4331gkfgclm17o 2440211 2440210 2026-07-23T09:57:39Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1070 2440211 wikitext text/x-wiki 3odjg106tnw2hb1rtyb04topc0gj9z4 2440212 2440211 2026-07-23T09:57:54Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1054: testing 2440212 wikitext text/x-wiki 8pc2h4zkikpcwl2lqrk82phvh7lxkgj 2440213 2440212 2026-07-23T09:58:32Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: testing 2440213 wikitext text/x-wiki jjftmcfz4limg72mtdx705vv1ejzgmx 2440214 2440213 2026-07-23T10:00:26Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2440214 wikitext text/x-wiki lhoua3ibjgeox36jrfeevq9knobfy90 2440215 2440214 2026-07-23T10:02:24Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2440215 wikitext text/x-wiki r6dg6gsof7j5dansv49blc3317kn22i 2440216 2440215 2026-07-23T10:05:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1038: testing 2440216 wikitext text/x-wiki faluc83fxti5v5pxetmg96s5njwjd84 2440217 2440216 2026-07-23T10:05:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1038: testing 2440217 wikitext text/x-wiki 8044w2uusmbcni4xhi8twdilplpgv3g 2440218 2440217 2026-07-23T10:06:55Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1038: testing 2440218 wikitext text/x-wiki ts43ncqc17dkfg3p29wea43ye8jzsvc 2440220 2440218 2026-07-23T10:16:32Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1070.eqiad.wmnet with reason: host reimage 2440220 wikitext text/x-wiki 5no1p72ythiruied9uf42ch63np1f8n 2440221 2440220 2026-07-23T10:20:32Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1070.eqiad.wmnet with reason: host reimage 2440221 wikitext text/x-wiki p3b62tdqsdoslmc9kmxojwa4jua7f7h 2440222 2440221 2026-07-23T10:24:43Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2440222 wikitext text/x-wiki rdjsc2dx9fzk0jsvlda64703tbdl515 2440223 2440222 2026-07-23T10:24:55Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2440223 wikitext text/x-wiki p3aht3458dn181m4veeu3ww54nuyepa 2440224 2440223 2026-07-23T10:30:31Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1037 hosts 2440224 wikitext text/x-wiki tmves1d0w7uuh217wip2lps9tefq93i 2440225 2440224 2026-07-23T10:41:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1070.eqiad.wmnet with OS trixie 2440225 wikitext text/x-wiki t8dtq1dyyx7rd8snmyz9a3gpsjgr30x 2440226 2440225 2026-07-23T10:43:24Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1054: testing 2440226 wikitext text/x-wiki gqwacb4tcsoyw73vmwfu01fitr74mm6 2440227 2440226 2026-07-23T10:49:50Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance 2440227 wikitext text/x-wiki m0mqr11uvma5aj0tfuqbjw4198hsaui 2440228 2440227 2026-07-23T10:49:58Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2186 (T431660)', diff saved to https://phabricator.wikimedia.org/P95072 and previous config saved to /var/cache/conftool/dbconfig/20260723-104956-cwilliams.json 2440228 wikitext text/x-wiki 8wn7yg9t5h4ckhf3g4dyifefsqk8yhy 2440229 2440228 2026-07-23T10:52:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1038: testing 2440229 wikitext text/x-wiki 8ast3go5ccli0qurulh2husxiatnq0a 2440231 2440229 2026-07-23T10:53:06Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1037: testing 2440231 wikitext text/x-wiki crbqyiiju5edq3kll25ae88znjbi8h4 2440232 2440231 2026-07-23T10:53:15Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1037: testing 2440232 wikitext text/x-wiki m934udc0td4tmfei6l557cdvc8iskmn 2440233 2440232 2026-07-23T10:53:58Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1037: testing 2440233 wikitext text/x-wiki byitykhz1udr8tv8mhut5r0ds1mqjhv 2440234 2440233 2026-07-23T10:54:12Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1037: testing 2440234 wikitext text/x-wiki 3xv4w10wjlpcxnist0o7ga158yjxtu7 2440235 2440234 2026-07-23T10:56:15Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2186: Maintenance 2440235 wikitext text/x-wiki dtzyq79ul0ptks9of9vv1u02bij1w0k 2440238 2440235 2026-07-23T11:11:34Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2160,2235].codfw.wmnet with reason: Upgrading 2440238 wikitext text/x-wiki 9fgyo5yp7u10x2bmsafd0xxgyxe2kdo 2440239 2440238 2026-07-23T11:11:53Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440239 wikitext text/x-wiki 3zcmfcvdl023ajwz8actio8nyvg08co 2440240 2440239 2026-07-23T11:12:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2235.codfw.wmnet with OS trixie 2440240 wikitext text/x-wiki lhpsnywy4hsnnjgpccx2vjvxpxjjvqx 2440245 2440240 2026-07-23T11:23:04Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d1-eqiad 2440245 wikitext text/x-wiki mosvlgmhoytsrazzqtz1dv72wo7vkpn 2440246 2440245 2026-07-23T11:23:32Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d1-eqiad 2440246 wikitext text/x-wiki 00l8nk3bpm8rr6isr0f4mzaov653kii 2440247 2440246 2026-07-23T11:23:34Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d8-eqiad 2440247 wikitext text/x-wiki r36kwpvjlosi386rxlbbu6262q97hmj 2440248 2440247 2026-07-23T11:24:17Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d8-eqiad 2440248 wikitext text/x-wiki kfh1kg8rotv09a1k4a2jh66k58fy3ce 2440249 2440248 2026-07-23T11:24:20Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d2-eqiad 2440249 wikitext text/x-wiki 6gpprnd84pw7hssv78s2jjyybt33b70 2440250 2440249 2026-07-23T11:24:31Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d2-eqiad 2440250 wikitext text/x-wiki e6fhcatnl1c5df9m0dnxmen906dtksm 2440251 2440250 2026-07-23T11:24:40Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d3-eqiad 2440251 wikitext text/x-wiki 329fbwuihvqqgexv6uk0n5ipi906zja 2440252 2440251 2026-07-23T11:24:42Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-eqiad 2440252 wikitext text/x-wiki r55tuuwwzjg1kaw9p4exobjtnvt90sx 2440253 2440252 2026-07-23T11:24:43Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d4-eqiad 2440253 wikitext text/x-wiki o3xljl2orriuw9mig1eao3wotv97kzt 2440254 2440253 2026-07-23T11:24:45Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d4-eqiad 2440254 wikitext text/x-wiki 0cbt68wbb0zpgk3t4dxjvjqhbg37fab 2440255 2440254 2026-07-23T11:24:47Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d7-eqiad 2440255 wikitext text/x-wiki 49e5rzpp08r44i24c7ksnnhafq42cf5 2440256 2440255 2026-07-23T11:24:50Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d7-eqiad 2440256 wikitext text/x-wiki 8l5ndg41pqtlxpelt6csu5h9ezptpkr 2440257 2440256 2026-07-23T11:24:53Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c2-eqiad 2440257 wikitext text/x-wiki ec06sogeml1ny5pzvuqpx06ukglcm0b 2440258 2440257 2026-07-23T11:24:59Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c2-eqiad 2440258 wikitext text/x-wiki kts33z5pceu2f79r8li8otuck8t3d6a 2440259 2440258 2026-07-23T11:25:02Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c3-eqiad 2440259 wikitext text/x-wiki 05d2z1c3xx47n06ul2330064zotbgyf 2440260 2440259 2026-07-23T11:25:08Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c3-eqiad 2440260 wikitext text/x-wiki apzqpn6rgnpoivzylsuadxfaognf2p9 2440261 2440260 2026-07-23T11:25:11Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c4-eqiad 2440261 wikitext text/x-wiki 0h2knopeqlcd8gdc3765esidl2zpjhj 2440262 2440261 2026-07-23T11:25:16Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c4-eqiad 2440262 wikitext text/x-wiki tayl1sdkr18t1flhjluh1b52bil87vn 2440263 2440262 2026-07-23T11:25:19Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c5-eqiad 2440263 wikitext text/x-wiki drx40mp201n0nu0zhsgyxbg1ypa8u4u 2440264 2440263 2026-07-23T11:25:24Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c5-eqiad 2440264 wikitext text/x-wiki ksrncz41uua47eaehwv0cddm4ixdfbl 2440265 2440264 2026-07-23T11:25:27Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c6-eqiad 2440265 wikitext text/x-wiki szum1f480gjth3rlnreqd5srndvhop1 2440266 2440265 2026-07-23T11:25:32Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c6-eqiad 2440266 wikitext text/x-wiki 3ndg3zikjbqnsajkt5u7kr8la3f7ggx 2440267 2440266 2026-07-23T11:25:35Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c7-eqiad 2440267 wikitext text/x-wiki sjjohvixbqxm6t5776ych7lxg77crx7 2440268 2440267 2026-07-23T11:25:42Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c7-eqiad 2440268 wikitext text/x-wiki q3kq4yip5nkt07dupjx6jy9vcmt8ef6 2440270 2440268 2026-07-23T11:28:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2235.codfw.wmnet with reason: host reimage 2440270 wikitext text/x-wiki dwyd733g2vu5ilm3p5tu078823hulsd 2440271 2440270 2026-07-23T11:33:54Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2235.codfw.wmnet with reason: host reimage 2440271 wikitext text/x-wiki lxi225g8euval48nalgsvx59h7kvkgq 2440272 2440271 2026-07-23T11:34:10Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 46375 2440272 wikitext text/x-wiki gahqizg6z3fh1d2gsuj0ms3yd77002h 2440273 2440272 2026-07-23T11:35:01Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 46375 2440273 wikitext text/x-wiki cclycn4ttvatf97udgqienqoog4if5n 2440275 2440273 2026-07-23T11:41:45Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2186: Maintenance 2440275 wikitext text/x-wiki jgbsp4yeozo3dx8ssxm0puvebxy4k1r 2440276 2440275 2026-07-23T11:43:01Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2191.codfw.wmnet with reason: Maintenance 2440276 wikitext text/x-wiki 5r92ulxz5uz93wsr8nww1evf9mfndfb 2440277 2440276 2026-07-23T11:43:10Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2191 (T431660)', diff saved to https://phabricator.wikimedia.org/P95080 and previous config saved to /var/cache/conftool/dbconfig/20260723-114308-cwilliams.json 2440277 wikitext text/x-wiki gpptvihl3swrrdp1l9pv931usf8g7kd 2440279 2440277 2026-07-23T11:46:46Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet 2440279 wikitext text/x-wiki accl2jrp86rznvbc719t3ybtcgdpz4a 2440280 2440279 2026-07-23T11:46:48Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet 2440280 wikitext text/x-wiki 00jk7a1adqpr3xqa4l60n4e8z1tgk34 2440281 2440280 2026-07-23T11:46:50Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1070.eqiad.wmnet 2440281 wikitext text/x-wiki rt5e1ditwyhdaszcam7tnxz7ks3bj4w 2440283 2440281 2026-07-23T11:50:33Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2191: Maintenance 2440283 wikitext text/x-wiki rbkwlpfnzyauknzj2eeb9gxjdi763c5 2440284 2440283 2026-07-23T11:56:38Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2235.codfw.wmnet with OS trixie 2440284 wikitext text/x-wiki ezh6icygpsfy3ixestxdkuf56u9l02y 12 8727 2439577 1908069 2026-07-21T19:51:59Z Mutante 20 adding new ed25519 key (https://phabricator.wikimedia.org/T240266) 2439577 wikitext text/x-wiki pad53xyh26xemzu8mmykw4hr5i4attd 2439578 2439577 2026-07-21T19:52:17Z Mutante 20 2439578 wikitext text/x-wiki s5ng4wscbx6pcp60say10psvdra0m2p 2439585 2439578 2026-07-21T19:58:53Z Mutante 20 2439585 wikitext text/x-wiki jzs4yu1ig20auvrnufejza1evzqewlz 498 14256 2438790 2435572 2026-07-20T12:13:38Z Stashbot 7414 wikibugs: Updated gerrit-channels.yaml to: 01b3b4d1f33f616a6233abade8a005081fc09e6dc966d4e50da8451c24191657 2438790 wikitext text/x-wiki 78k1vqoq92pszhemfotm5472vuwuywa 2438795 2438790 2026-07-20T12:21:46Z Stashbot 7414 wikibugs: Updated channels.yaml to: 855ab59e0a3c427c4a0460ba0f1df4b20864f21eb9439a886f6f16b13013cd9e 2438795 wikitext text/x-wiki s977lhfpdusbbdgr14m5uix2zkajuzz 2438811 2438795 2026-07-20T12:43:12Z Stashbot 7414 wikibugs: Updated gerrit-channels.yaml to: f30065c45b80543d21ea36f42f0d441652f7676a3addad9ced53b1cc37abb6c0 2438811 wikitext text/x-wiki ivmnepcw1oiwcvqp00ijaesj8jjqqfz 0 17290 2439086 2438285 2026-07-20T19:36:33Z Stashbot 7414 dancy: Restarting jenkins on contint1003 to clear out hours-long stuck jobs 2439086 wikitext text/x-wiki i0397m94mbzaxln4nq3xboropda8wpf 2439226 2439086 2026-07-21T07:08:23Z Stashbot 7414 hashar: integration: cleaned up Jenkins workspace on integration-agent-docker-1084 2439226 wikitext text/x-wiki 60rrzh8ao2pt30ayf5fpixmxvg44qxv 2439494 2439226 2026-07-21T15:59:25Z Stashbot 7414 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1313220 2439494 wikitext text/x-wiki cjik69ohfld9o5lhuvkehxypxthkyqf 2439759 2439494 2026-07-22T09:54:35Z Stashbot 7414 James_F: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure for 1313321, per Lucas_WMDE. 2439759 wikitext text/x-wiki e71yc8ddip8q1unq6x7xtg1vduysfwa 2440006 2439759 2026-07-22T15:59:32Z Stashbot 7414 dancy: Restarting Gerrit (T240266) 2440006 wikitext text/x-wiki mellemwyb3qjoybcz7wps9g51wku9da 2440041 2440006 2026-07-22T16:58:32Z Stashbot 7414 dancy: Restarting Gerrit (T240266) (again) 2440041 wikitext text/x-wiki t4oxf8hl6mxnumf4cc3gqquuknink44 2440051 2440041 2026-07-22T17:24:29Z Stashbot 7414 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1314014/1 (T432886) 2440051 wikitext text/x-wiki dotmnfav5rt42l7ypr00gdtxox1kssq 2440066 2440051 2026-07-22T18:25:24Z Stashbot 7414 dancy: Zuul is currently broken due to the Gerrit SSH key update. I'm investigating 2440066 wikitext text/x-wiki ptrdp7w5xdc31ikmgg11l5rvnd9vb67 2440074 2440066 2026-07-22T18:41:39Z Stashbot 7414 dancy: Zuul dependencies upgraded and Zuul restarted. 2440074 wikitext text/x-wiki h0y0fjrflch2gzdgnrfufq6ysjzgnkc 2440112 2440074 2026-07-22T21:32:44Z Stashbot 7414 dancy: Moved /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/npm to /srv/castor-debug-T20260722-npm-torn-cacache/ on integration-castor06 2440112 wikitext text/x-wiki k2dpm0ipa8v6dtqwcg7aq8figx7womu 0 23327 2439158 1777924 2026-07-20T21:40:09Z Star walker 40724 internal link 2439158 wikitext text/x-wiki rfgfscongo61vfj475n2apwukgelttv 498 30942 2438784 2438780 2026-07-20T12:01:35Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=0) 2438784 wikitext text/x-wiki aqobc9glctvkitn69hv5oz1peq9v35p 2438785 2438784 2026-07-20T12:07:03Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2438785 wikitext text/x-wiki b9zufjvpwt1v8xt27q9jw4b8c10vr5k 2438786 2438785 2026-07-20T12:07:06Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=99) 2438786 wikitext text/x-wiki 4dzih0fi5quw4vnf7f1e1z9wqy3delk 2439119 2438786 2026-07-20T20:34:47Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439119 wikitext text/x-wiki nhya1j6z0o9ffubkq8vytoqv54jozdy 2439120 2439119 2026-07-20T20:35:33Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439120 wikitext text/x-wiki 2ntdzxf2zuvdygcohrn89mn9whqgyg2 2439132 2439120 2026-07-20T20:51:06Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439132 wikitext text/x-wiki 8x3d1zn0k76hv3i9zp7eelhdmusxa5l 2439133 2439132 2026-07-20T20:51:37Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439133 wikitext text/x-wiki a1o66nrfob719si42b2m820uddru81e 2439139 2439133 2026-07-20T21:12:56Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439139 wikitext text/x-wiki 1gf4c4uzetzfwvryvgegpsjxsi9szph 2439141 2439139 2026-07-20T21:13:26Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439141 wikitext text/x-wiki csj3qe7x3xptn5bzn2s54ujqrc7jf68 2439142 2439141 2026-07-20T21:25:06Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439142 wikitext text/x-wiki gg7068a7urfid4vauiwf9kdqe436hcu 2439143 2439142 2026-07-20T21:25:31Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439143 wikitext text/x-wiki 70jnoyxpvnzv0jdor3uxwfss2z8xqjc 2439144 2439143 2026-07-20T21:26:49Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439144 wikitext text/x-wiki dtgsdx2scjw9j8dqdnvdkduirgbaxaf 2439145 2439144 2026-07-20T21:27:16Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439145 wikitext text/x-wiki epo2nklo7u8ynqz7kkaskruyeu2sxvq 2439146 2439145 2026-07-20T21:27:19Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439146 wikitext text/x-wiki a10c9swugffv6cpvg31063qb9djeokz 2439147 2439146 2026-07-20T21:27:38Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439147 wikitext text/x-wiki 2kt9p78avvxhba1wut0c0sdtyij2tdt 2439440 2439147 2026-07-21T14:38:27Z Stashbot 7414 godog: switch from SystemdUnitDown (eqiad only) to SystemdUnitFailed (codfw/eqiad) alerts -- there might be some codfw noise coming - T428873 2439440 wikitext text/x-wiki 7w3p5fpaasky97dbf64b91mzcl6i5ad 2439457 2439440 2026-07-21T15:07:00Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439457 wikitext text/x-wiki 4uez1lieqcqciwgvkt992jeidoxvr34 2439458 2439457 2026-07-21T15:07:30Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439458 wikitext text/x-wiki ib2wv5wj6o39wz3ll2170jwcp6z5a49 2439459 2439458 2026-07-21T15:10:37Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439459 wikitext text/x-wiki myh1sg90vjrqic30sjp6bf99k5aov5z 2439460 2439459 2026-07-21T15:11:06Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439460 wikitext text/x-wiki k83i9agiywfbru07t8hm1dctmsodooh 2439468 2439460 2026-07-21T15:20:12Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439468 wikitext text/x-wiki 90t1bc59hsqymyq7wqaj4nbr5y4025v 2439469 2439468 2026-07-21T15:20:38Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439469 wikitext text/x-wiki 4y48vs73ceeiay0vyjmqd537ddsmr0n 2439471 2439469 2026-07-21T15:21:48Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439471 wikitext text/x-wiki 7q0m5i88gmj6fpx96h91xxymsi6kxvo 2439473 2439471 2026-07-21T15:22:40Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439473 wikitext text/x-wiki gtbjp9il5242x81aurhbx5iwcq4xzkj 2439474 2439473 2026-07-21T15:22:43Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439474 wikitext text/x-wiki d7bpk6nr6s8iwmhlh2ksmw54wrt4qig 2439475 2439474 2026-07-21T15:24:00Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2439475 wikitext text/x-wiki 382oxk4sqf8n5a7oswyk43ojf2qsv0h 2439478 2439475 2026-07-21T15:27:09Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439478 wikitext text/x-wiki m14q7cfhb96icsk4q29t7h0qa17z99m 2439480 2439478 2026-07-21T15:27:50Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2439480 wikitext text/x-wiki 8p5xpqa89sxhkq21w4i4iicqo9u8fc5 2439481 2439480 2026-07-21T15:27:59Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2439481 wikitext text/x-wiki in05b7jgnp4w2tymnwwpxeomyt97j4y 2439482 2439481 2026-07-21T15:28:26Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2439482 wikitext text/x-wiki qywx9lg4cxxbhrstiwzosn37oa2yjb2 2439648 2439482 2026-07-22T02:24:59Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch 2439648 wikitext text/x-wiki q4fw2y8nvib0zimdrqdg6qxhlxgov1j 2439649 2439648 2026-07-22T02:25:30Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch 2439649 wikitext text/x-wiki nba5ersy0texffpd75pssa7ns6f31ek 2439876 2439649 2026-07-22T12:46:39Z Stashbot 7414 taavi: add security group rules for all projects for new metricsinfra VIPs T401813 2439876 wikitext text/x-wiki ja2xgueibjmwkmbpah2ewccghbvmc0k 2440058 2439876 2026-07-22T18:04:03Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431374) 2440058 wikitext text/x-wiki njo9dvi5l8ig6ol8sw1azs0yv6jdnfr 2440059 2440058 2026-07-22T18:04:07Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431374) 2440059 wikitext text/x-wiki 0zh1shrghx5vbpyftj6biadkjbixoss 2440060 2440059 2026-07-22T18:05:58Z Stashbot 7414 andrewbogott: serveraction powercycle on cloudvirt1071 2440060 wikitext text/x-wiki eoavcqh9stm045j6cy1hme5t4j5pclp 2440061 2440060 2026-07-22T18:11:23Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1071.eqiad.wmnet' (T431374) 2440061 wikitext text/x-wiki 1r9szc644q8dpn68mpsttlagjc3g24c 2440062 2440061 2026-07-22T18:16:19Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' (T431374) 2440062 wikitext text/x-wiki 9wqc5najr07r2sfrgzlilhpy5ofjt67 0 33560 2439048 2365461 2026-07-20T17:57:15Z SSingh (WMF) 13928 add note for acme-chief keyholder 2439048 wikitext text/x-wiki c60t6dbdia4vvuvgsc9lmozro0dja11 0 67018 2439626 2435859 2026-07-21T20:56:39Z Krinkle 44 WMF migrated from php-tideways (back) to php-xhprof last year. Decision: [[phab:T398245]]: Migration parent task: [[phab:T348379]] 2439626 wikitext text/x-wiki 505tqpue9tkdq5yti3fxkcdeqrfnss9 498 117521 2439247 2394560 2026-07-21T08:38:20Z Stashbot 7414 lucaswerkmeister: (re?)add Reedy (should be covered by T249774; SSH key still present in puppet.git root-authorized-keys.erb AFAICT) 2439247 wikitext text/x-wiki g8jlaqliabqrxmpppqlza19cx8c541h 2440236 2439247 2026-07-23T10:57:22Z Stashbot 7414 lucaswerkmeister: granted Reedy some more comprehensive access per https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolforge_roots_and_Toolforge_admins (tools member in horizon, toollabs-trusted on Gerrit) 2440236 wikitext text/x-wiki 01j48hw8varagtq0mmqp1jfy76tdbh6 0 350482 2439108 2434005 2026-07-20T20:23:09Z BCornwall-WMF 31521 /* Packaging */ flip git diff versions 2439108 wikitext text/x-wiki csoa82cxkrgh570bxssektugt7cp6wh 498 442283 2439731 2326909 2026-07-22T08:58:57Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: refreshed the GitHub token in cdnjs-index/tokenfile (now expires 2026-10-20) 2439731 wikitext text/x-wiki qxyuogjttqtzeaifccze7ng7t85nor9 12 442409 2440237 2380008 2026-07-23T10:57:31Z Taavi-WMF 41365 add explicit footnotes section 2440237 wikitext text/x-wiki mhedjpeejqf9gshd974tndg9abtd786 498 443060 2440120 2438626 2026-07-22T22:24:27Z Stashbot 7414 bd808: logging-logstash-04 hard reboot via Horizon (T432254) 2440120 wikitext text/x-wiki mffznbcci2r0c4gzco644soishlm2ic 498 443628 2438828 2438781 2026-07-20T13:07:01Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29744681881 (https://github.com/cluebotng/component-configs/commits/8b5c46ef715a2e57f383d35d7acc37bfbfa4db2d) 2438828 wikitext text/x-wiki 1te2mk6jk1l25rfdh51f101c9c8yc43 2438889 2438828 2026-07-20T14:11:18Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29749326972 (https://github.com/cluebotng/component-configs/commits/83d05d6d10640a88d80bdbc041955ca27603a0fe) 2438889 wikitext text/x-wiki ovgbvubt0ydz5jj8t7e35y1yxre3ywp 2438926 2438889 2026-07-20T15:04:08Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29753268267 (https://github.com/cluebotng/component-configs/commits/3cd242ed54e9146f6a3dfd9305f97d4580c2fc06) 2438926 wikitext text/x-wiki t4qd4m92bkzs8rszzg1kvesu4efnfm0 2440193 2438926 2026-07-23T09:28:10Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29995256262 (https://github.com/cluebotng/component-configs/commits/c08c3e19100546a5f8ed26a2bfa7095cb4633147) 2440193 wikitext text/x-wiki tvjhr7pqo69umhl0fr4i1g5kbn6k75i 2440219 2440193 2026-07-23T10:11:30Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29998195703 (https://github.com/cluebotng/component-configs/commits/e8762796572d00e69633a1b076413d5aa4ee3508) 2440219 wikitext text/x-wiki 94xv8p5hgnmduyakrxoadj6j4xbwyed 2440269 2440219 2026-07-23T11:28:03Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/30003154188 (https://github.com/cluebotng/component-configs/commits/97520d86ff57924e8237a0446010f837e79b9062) 2440269 wikitext text/x-wiki jakbsq5o12jr3fkp6okhf9kg8tgcdkq 0 443895 2439278 2095222 2026-07-21T10:32:06Z LSobanski (WMF) 18783 2439278 wikitext text/x-wiki f5gor4r9hi336w62fw8igt8m6znfx23 0 443904 2440053 2434948 2026-07-22T17:28:27Z KSoučková-WMF 35993 /* Service */ change active deployment server after switchover 2440053 wikitext text/x-wiki g7ofu1ytks6jw2jsqc08w3m60dadz2r 498 443947 2439780 2432975 2026-07-22T10:37:21Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed e33530c9d2 (fix README) 2439780 wikitext text/x-wiki 4mitc8zbb20y6mji2gtffcflzp5lzvx 498 443948 2439779 2363188 2026-07-22T10:35:34Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed fc6fda67f8 (fix README) 2439779 wikitext text/x-wiki tdypd21cmb5p3a0pygpwwtagicrvebj 498 443949 2438952 2438648 2026-07-20T16:03:15Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 2fa79d1 (Fix QLever link in SSE error UI: add SPARQL prefixes) for T425477 2438952 wikitext text/x-wiki tgqlfs61wfllz3i197yfwckka7g5wzn 2438953 2438952 2026-07-20T16:03:17Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 4235204 (Show SPARQL queries in SSE progress messages) for T425477 2438953 wikitext text/x-wiki 6b2ljbb2gfbntiy4f0doahi6ke4hvdg 2438954 2438953 2026-07-20T16:03:19Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy a4029f2 (Mark completed steps with ✅ via DOM mutation) for T425477 2438954 wikitext text/x-wiki 3hccema1onzxw1l06b6c2mhv3kcftyy 2438955 2438954 2026-07-20T16:03:22Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 7dbac6d (Make streaming update the default) for T425477 2438955 wikitext text/x-wiki a3987jhzzm1v3pd55wl3y0ebvz1okuu 2438998 2438955 2026-07-20T17:00:28Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy b3e24a6 (Log all totals and no-grouping SPARQL queries in SSE stream) for T425477 2438998 wikitext text/x-wiki h5blm78dlipywzgujmsjfgln3rod4p0 2438999 2438998 2026-07-20T17:00:29Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy dd7677a (Fix footer layout: use sticky footer instead of fixed position) 2438999 wikitext text/x-wiki rxzhg22t6iueu19zalsd7lp72edqvxf 2439155 2438999 2026-07-20T21:36:52Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 14ca09e (Collapse completed sections in SSE progress view) for T425477 2439155 wikitext text/x-wiki i56vkimou8nei6jftu5x1hy2k6h6gds 2439282 2439155 2026-07-21T10:38:11Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 6b79fa1 (Add noscript fallback for streaming update page) for T425477 2439282 wikitext text/x-wiki 6rvm786x3wnwdm0ptn6f9jarkzl79z4 2439886 2439282 2026-07-22T13:13:37Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 116fe90 (Override format_html_snippet in ReferenceColumn) for T428637 2439886 wikitext text/x-wiki e0liszedmcz0lgtdtworijka2khs0dg 2439887 2439886 2026-07-22T13:13:40Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 47fd820 (Refactor: extract ReferenceCheck strategy from ReferenceColumn) for T428637 2439887 wikitext text/x-wiki ez8yc7z821004f1z4z0jou7l9tlwrz3 2439888 2439887 2026-07-22T13:13:42Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy eaa22b6 (Add P123/S456 syntax for specific reference property) for T428637 2439888 wikitext text/x-wiki byip0zhzpr39n1gwzvor5znjp5lxvpb 0 444622 2439075 2376045 2026-07-20T18:42:39Z AndrewBogott (WMF) 51047 /* Etcd */ we have a newer version now but are still behind the recommended version 2439075 wikitext text/x-wiki lorzxrst3xapf3rxspjldy9q6sni3b2 498 446686 2439948 2434995 2026-07-22T14:40:12Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 8 cores, 16384 ram 2439948 wikitext text/x-wiki 9jdlbywf3nr17mzbw22n1v5es2hm5bb 2439949 2439948 2026-07-22T14:40:19Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 8 cores, 16384 ram 2439949 wikitext text/x-wiki jaqm0ufs6tmslv63pd10hkk6xppfs5o 2439957 2439949 2026-07-22T14:46:05Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-4.metricsinfra.eqiad1.wikimedia.cloud 2439957 wikitext text/x-wiki nufcb4zkxeyr9h3plo64kkb8lgj4wgx 2439958 2439957 2026-07-22T14:47:28Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-prometheus-4.metricsinfra.eqiad1.wikimedia.cloud 2439958 wikitext text/x-wiki 2bbj4wvngwksmg0o3xk46bed7pjundh 2440241 2439958 2026-07-23T11:20:50Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-controller-2 2440241 wikitext text/x-wiki 1v60v3cp4v4rqox9pdel8xtvb593pmm 2440242 2440241 2026-07-23T11:21:35Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-controller-2 2440242 wikitext text/x-wiki 4v7glc16od7cyfaiekt4cqwlhzhf3om 2440243 2440242 2026-07-23T11:22:10Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-prometheus-2 2440243 wikitext text/x-wiki n913416ynz43lkwo84qa8w2nlcov1d6 2440244 2440243 2026-07-23T11:22:56Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-prometheus-2 2440244 wikitext text/x-wiki njhcr3s48ktk667brt64a580inu43hi 3 446735 2439173 2269310 2026-07-21T01:41:06Z Mfield 51590 Mfield moved page [[User talk:Renamed user cdb78c3737e6b7f6ba7e28cedcc6608711202eee]] to [[User talk:IVORK]] over redirect: Automatically moved page while renaming the user "[[Special:CentralAuth/Renamed user cdb78c3737e6b7f6ba7e28cedcc6608711202eee|Renamed user cdb78c3737e6b7f6ba7e28cedcc6608711202eee]]" to "[[Special:CentralAuth/IVORK|IVORK]]" 2269310 wikitext text/x-wiki rtm8p1892o82p9zuhle6cibqxm2rjm8 0 447017 2440126 2436938 2026-07-23T01:08:14Z RLazarus (WMF) 15215 2440126 wikitext text/x-wiki fvd1b36sibcrbfzr3p7icbth1mdf23j 498 447034 2439774 2433158 2026-07-22T10:26:29Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 258b59cd93 (fix README) 2439774 wikitext text/x-wiki bhx4p38hcsywt2xborea78df5sqsy8x 0 447310 2439985 2435471 2026-07-22T15:34:16Z MPopov (WMF) 5891 /* Experimentation templates */ Remove link to archived scorecard 2439985 wikitext text/x-wiki 7u69agl33wse9zmiq7008hejqjrvyz5 0 447718 2439559 2401063 2026-07-21T19:00:31Z ADancy (WMF) 17694 Break up the long first sentence. 2439559 wikitext text/x-wiki cmp85vcki34fu01pcu1lc6tmfqsqsvr 498 448506 2439027 2200470 2026-07-20T17:33:19Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 23 cores, 122880 ram 2439027 wikitext text/x-wiki 7d9edm0oloo3c8tv8qwdbema3eyd9je 2439028 2439027 2026-07-20T17:33:26Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 23 cores, 122880 ram 2439028 wikitext text/x-wiki ez7jfn9c4cmn8oxqm4lxj6d67aib8bp 0 448785 2439166 2383552 2026-07-20T23:01:15Z EMcnaughton (WMF) 6074 Added new processors 2439166 wikitext text/x-wiki okhrvhdfsway14ijvcw4lwcumvagibb 0 449031 2439567 2433474 2026-07-21T19:21:29Z LSandergreen-WMF 46345 2439567 wikitext text/x-wiki 5boxxff5kwlvt28x6m71mwz6pily8y0 0 449079 2440084 2433395 2026-07-22T19:26:39Z DAdedoyin (WMF) 39150 /* Setup and Administration */ add instruction for setting up new proxy. 2440084 wikitext text/x-wiki feuorm8vj7ysi9eox7o78y9db0mmtss 0 449160 2438931 2438706 2026-07-20T15:10:05Z Dexbot 30554 Bot: Updating the report 2438931 wikitext text/x-wiki trfkhv601qkab87tbqvbsm88ib2nsoh 2439055 2438931 2026-07-20T18:07:52Z Dexbot 30554 Bot: Updating the report 2439055 wikitext text/x-wiki ij3b0yx6hx0691w36qtltqo2xac1njz 2439171 2439055 2026-07-21T00:02:52Z Dexbot 30554 Bot: Updating the report 2439171 wikitext text/x-wiki 9lmridspfxnr89dtwmnv0vmjk19k9e3 2439253 2439171 2026-07-21T08:48:59Z Dexbot 30554 Bot: Updating the report 2439253 wikitext text/x-wiki q61vjrpdet6a63gxh1a0jdbxsot3c4u 2439269 2439253 2026-07-21T10:19:11Z Dexbot 30554 Bot: Updating the report 2439269 wikitext text/x-wiki 96edcl5gd69rjfkbdxejid3m7bwbdkh 2439274 2439269 2026-07-21T10:28:18Z Dexbot 30554 Bot: Updating the report 2439274 wikitext text/x-wiki 3nlvg383iyl5k6queq9uw3ytbc0j4pt 2439640 2439274 2026-07-22T00:00:05Z Dexbot 30554 Bot: Updating the report 2439640 wikitext text/x-wiki r1pufxlgyo8jwnzsxwdoub0c4jnhsr8 2440124 2439640 2026-07-23T00:00:13Z Dexbot 30554 Bot: Updating the report 2440124 wikitext text/x-wiki hyccglgqhb0akutgfqc0qled96qo1d4 2440230 2440124 2026-07-23T10:52:33Z Dexbot 30554 Bot: Updating the report 2440230 wikitext text/x-wiki qvhemv78hxd4voof6zq0j33swny4m16 0 449344 2439757 2425609 2026-07-22T09:51:08Z GLavagetto (WMF) 2905 /* Examples */ 2439757 wikitext text/x-wiki 6yiuhasv30cq0ug6a58hesziplw84k7 2439758 2439757 2026-07-22T09:51:30Z GLavagetto (WMF) 2905 /* Expression grammar and inline patterns */ 2439758 wikitext text/x-wiki 9lp7i0ijiq5005sle2pqzz4w6m6ldt3 0 451533 2439207 2432353 2026-07-21T04:07:35Z SSapaty (WMF) 28955 Update team names 2439207 wikitext text/x-wiki blkw6i8sytjsc5ux7zlf7to9xdcjwb9 2439208 2439207 2026-07-21T04:28:43Z SSapaty (WMF) 28955 Add Google Meet link 2439208 wikitext text/x-wiki 3332owrcewxchdn0dokk6efrknlygmf 2439772 2439208 2026-07-22T10:25:48Z Taavi-WMF 41365 expand the documentation a bit 2439772 wikitext text/x-wiki 8zhh9sek9w5wacr429qvy42y0x3z15t 2439789 2439772 2026-07-22T10:52:37Z Taavi-WMF 41365 restore meet link 2439789 wikitext text/x-wiki 45rtxw3s8rqcnwff5k8zecldhpz1bbf 828 452250 2439885 2412376 2026-07-22T13:12:06Z Toolforge Image Bot 35765 Bot: Updating available container images 2439885 json application/json dxxx6gfvz2jg51khigyqygkrv65bu0m 116 453905 2438853 2256135 2026-07-20T13:32:19Z Quiddity 1884 add link to more specific Gitlab subpage on mw (IIUC) 2438853 wikitext text/x-wiki e1mdl41jct88tjqfpqdbpswyx8gby3m 116 453906 2438938 2438712 2026-07-20T15:27:25Z Gitlabaccountapprovalbot 37332 @rodrigoargenton was approved. 2438938 wikitext text/x-wiki owvucsqwh0gj62viwwsgjuhlsecuelc 2439012 2438938 2026-07-20T17:15:13Z Gitlabaccountapprovalbot 37332 leonidlednev was rejected. 2439012 wikitext text/x-wiki dhpkzdqjay95em70bhfpdlrpuzhlr9m 2439260 2439012 2026-07-21T09:24:26Z Gitlabaccountapprovalbot 37332 @xanonymusx was approved. 2439260 wikitext text/x-wiki j125tqrvwjy66sr1b676k0vzfnzckg9 2439264 2439260 2026-07-21T10:09:24Z Gitlabaccountapprovalbot 37332 @francyskus was approved. 2439264 wikitext text/x-wiki s81axih9lym2732w6ly2r07f8h6j2ha 2439266 2439264 2026-07-21T10:15:32Z Gitlabaccountapprovalbot 37332 @yongjiapeng was approved. 2439266 wikitext text/x-wiki gpdnr7ozlpr5kwj1apuoc9ktcky9cpi 2439276 2439266 2026-07-21T10:30:44Z Gitlabaccountapprovalbot 37332 @majicita was approved. 2439276 wikitext text/x-wiki d0pf5gx6kouc1p35ehdsnv4hgddek23 2439277 2439276 2026-07-21T10:30:45Z Gitlabaccountapprovalbot 37332 @johlig was approved. 2439277 wikitext text/x-wiki ioakx31cywo1l6yu3ei794lwex0162u 2439279 2439277 2026-07-21T10:33:25Z Gitlabaccountapprovalbot 37332 @nivopol was approved. 2439279 wikitext text/x-wiki d0nghkl2796pb00jqvyseo93fuvpb84 2439317 2439279 2026-07-21T12:18:31Z Gitlabaccountapprovalbot 37332 @wijltcher was approved. 2439317 wikitext text/x-wiki 67eumcz7oagq187m4gmkuzaif53pifm 2439319 2439317 2026-07-21T12:21:24Z Gitlabaccountapprovalbot 37332 @nadiantara was approved. 2439319 wikitext text/x-wiki t88bh7eeogjl7xs1w0d11jct1zv25my 2439333 2439319 2026-07-21T12:36:27Z Gitlabaccountapprovalbot 37332 @jeannenoiraud was approved. 2439333 wikitext text/x-wiki dladxqf6h1uvb6cr7513g9l4fedew0l 2439342 2439333 2026-07-21T12:54:23Z Gitlabaccountapprovalbot 37332 @ctdo17 was approved. 2439342 wikitext text/x-wiki cnbfueq537uarx3iipcqz376wkr8xu2 2439353 2439342 2026-07-21T13:09:24Z Gitlabaccountapprovalbot 37332 @deevad was approved. 2439353 wikitext text/x-wiki 8ngd85dzatk7mtxscrvmyjymgpqod1h 2439392 2439353 2026-07-21T13:48:26Z Gitlabaccountapprovalbot 37332 @yru was approved. 2439392 wikitext text/x-wiki hlabjvbcs22hws4gwrjug9e13pda0mz 2439487 2439392 2026-07-21T15:33:26Z Gitlabaccountapprovalbot 37332 @luisvilla-personal was approved. 2439487 wikitext text/x-wiki 4svzna3v4q9loqy4pgis9e4cgyk7vfx 2439493 2439487 2026-07-21T15:51:25Z Gitlabaccountapprovalbot 37332 @panieravide was approved. 2439493 wikitext text/x-wiki k99hlnh16u955ti251qde2z61datcid 2439657 2439493 2026-07-22T07:06:14Z Gitlabaccountapprovalbot 37332 ayush8620 was rejected. 2439657 wikitext text/x-wiki limx6ok98s6khzupvmyv1tsvicdi6vc 2439722 2439657 2026-07-22T08:30:28Z Gitlabaccountapprovalbot 37332 @plyd was approved. 2439722 wikitext text/x-wiki dw9t24oa44m1thkmmj8aaze3tgejrir 2439723 2439722 2026-07-22T08:30:29Z Gitlabaccountapprovalbot 37332 @nabbegat was approved. 2439723 wikitext text/x-wiki n8jw6giyqezzzqgpfhhs88vl2kz4c2i 2439877 2439723 2026-07-22T12:48:28Z Gitlabaccountapprovalbot 37332 @cyberpower678 was approved. 2439877 wikitext text/x-wiki 3oyebr19ffbulg6l6ydmgch45e7ti3h 2439901 2439877 2026-07-22T13:24:13Z Gitlabaccountapprovalbot 37332 praveen5638 was rejected. 2439901 wikitext text/x-wiki nt1jf1yf2w7qs154iwpl8y34izt33sp 2440002 2439901 2026-07-22T15:54:24Z Gitlabaccountapprovalbot 37332 @lmedley was approved. 2440002 wikitext text/x-wiki cef295tocl36pv6p5mryw2fahiqt7wb 2440191 2440002 2026-07-23T09:21:23Z Gitlabaccountapprovalbot 37332 @slashme was approved. 2440191 wikitext text/x-wiki djuwuvcqjhk6tz92cigjld4fz7a0di1 0 455132 2439563 2344610 2026-07-21T19:11:25Z BKing (WMF) 29017 2439563 wikitext text/x-wiki 3a78sepdcjah61nrexvu7om55ntib10 2439565 2439563 2026-07-21T19:11:48Z BKing (WMF) 29017 /* Object Storage (Flink Savepoints/Checkpoints) */ 2439565 wikitext text/x-wiki mm9628i5pm8lrhx4ajde1n877dpwe5d 12 455426 2439580 2214984 2026-07-21T19:54:29Z RLazarus (WMF) 15215 update after reimage (T423714) 2439580 wikitext text/x-wiki 3ipkupmlr2sx3xlv1agj5kd93wugmb6 3 457268 2439250 2269949 2026-07-21T08:40:39Z StrikerBot 8475 Welcome to Toolforge! 2439250 wikitext text/x-wiki t399pbjq42g75cl0ha6qnss4n28wj85 0 458576 2439666 2360613 2026-07-22T07:31:54Z BRouberol-WMF 39141 /* DAGs and Airflow instance */ 2439666 wikitext text/x-wiki 500gsvhms19sob1xbi7bu3ogyw4250g 2439667 2439666 2026-07-22T07:32:26Z BRouberol-WMF 39141 /* Pools and parallelism */ 2439667 wikitext text/x-wiki 15jwhs9au1st3dww6nsqcstc4aq3be2 0 458584 2439662 2364229 2026-07-22T07:29:00Z BRouberol-WMF 39141 /* The fetch_wiki_list_from_noc dag is failing */ 2439662 wikitext text/x-wiki 0df3b8f3qp5kkgh9plwyhhwyx6couec 2439664 2439662 2026-07-22T07:30:46Z BRouberol-WMF 39141 /* Making a change to the SQL/XML Dump DAGs */ 2439664 wikitext text/x-wiki 486lhhz1t15oufliuou25mft0tici85 0 459044 2439934 2435481 2026-07-22T14:28:21Z BKing (WMF) 29017 /* Operations */ 2439934 wikitext text/x-wiki 7icdyprofcy4vt3svupbchi7lgamzqf 2440092 2439934 2026-07-22T19:50:55Z BKing (WMF) 29017 /* Alerts/Dashboards */ 2440092 wikitext text/x-wiki lef2i0lezsl5u2vex8c1j9vy2gdaxr6 2440093 2440092 2026-07-22T19:55:50Z BKing (WMF) 29017 2440093 wikitext text/x-wiki 5qw3hxhv3io215ewaqcgn19acoz9wwx 2440096 2440093 2026-07-22T20:11:18Z BKing (WMF) 29017 2440096 wikitext text/x-wiki 1fqehzirp2y4lpb4b77umrcoqxufj36 2440098 2440096 2026-07-22T20:15:58Z BKing (WMF) 29017 /* Alerts */ 2440098 wikitext text/x-wiki o8rtiaxa0yy4st1ufbis9pnl367g451 2440101 2440098 2026-07-22T20:26:23Z BKing (WMF) 29017 /* Alerts */ 2440101 wikitext text/x-wiki s8zt2ciuv7hlqmcpkq3i9vtqn2lgedo 2440102 2440101 2026-07-22T20:26:53Z BKing (WMF) 29017 /* Alerts */ 2440102 wikitext text/x-wiki 43wxy4zmcnxc3ebiqnzumaaqt8svzr7 2440103 2440102 2026-07-22T20:27:39Z BKing (WMF) 29017 /* Alerts/Dashboards */ 2440103 wikitext text/x-wiki 5xopr1thd9ywszui0el23hm9puf6xkr 2440104 2440103 2026-07-22T20:34:19Z BKing (WMF) 29017 /* Alerts/Dashboards */ 2440104 wikitext text/x-wiki 629az3c2xjbop2gfcgeh08cndqb53kn 2440107 2440104 2026-07-22T20:41:18Z BKing (WMF) 29017 /* Alerts */ 2440107 wikitext text/x-wiki o9ooscavi02qyybna5q7rqhx7jt2h7d 2440108 2440107 2026-07-22T20:43:30Z BKing (WMF) 29017 /* Alerts */ 2440108 wikitext text/x-wiki 3pzzedpmnd6aj4i9345258doc1x0tl9 2440109 2440108 2026-07-22T20:44:13Z BKing (WMF) 29017 /* Alerts */ 2440109 wikitext text/x-wiki 4z5h5y5jm1ac6vx68kqjxxf2av7x7nm 2440110 2440109 2026-07-22T20:47:02Z BKing (WMF) 29017 /* Alerts */ 2440110 wikitext text/x-wiki 334umkvhjyfscghqvcsk15kginxb1fw 2440111 2440110 2026-07-22T20:49:03Z BKing (WMF) 29017 /* Alerts */ 2440111 wikitext text/x-wiki h1scte8uxp2s782lo09k99w43yyy0xe 0 459087 2439654 2376497 2026-07-22T05:35:00Z DSantamaria-WMF 39115 Promote the AW SLO out of draft 2439654 wikitext text/x-wiki oaijeki81adjaqk89xsk2ovvkhi3k4o 0 459093 2438833 2435803 2026-07-20T13:15:18Z AYounsi (WMF) 10040 /* Limitations */ 2438833 wikitext text/x-wiki 3hn4dkvyl7x03n7niovcxsb5q13af5h 0 459290 2439019 2438735 2026-07-20T17:24:54Z Ottomata 88 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2439019 wikitext text/x-wiki 61gia4dz2ahm4imvbhn7hzp3nginpil 0 459699 2439645 2438345 2026-07-22T01:36:27Z Stashbot 7414 eileen: civicrm upgraded from 2eba4374 to 5a04e8a9 2439645 wikitext text/x-wiki qhkzjizdfq52vp4hydxmlheb7z8kz3d 2440113 2439645 2026-07-22T21:36:42Z Stashbot 7414 eileen: SmashPig upgraded from 6921d862 to 038018d4 2440113 wikitext text/x-wiki a5kelj8h9kk7t5yscyqf38urjr66rmd 2440114 2440113 2026-07-22T21:38:06Z Stashbot 7414 eileen: SmashPig upgraded from 6921d862 to 038018d4 2440114 wikitext text/x-wiki j0seksrhsigchrbnxr14jgm8u99ovs5 2440121 2440114 2026-07-22T22:35:09Z Stashbot 7414 eileen: civicrm upgraded from 5a04e8a9 to 068b2e3c 2440121 wikitext text/x-wiki 5pitef90fq013k3ed8u848yv69sld5t 2440125 2440121 2026-07-23T00:04:51Z Stashbot 7414 eileen: civicrm upgraded from 068b2e3c to bf93abac 2440125 wikitext text/x-wiki oqhi1si9cl4hhqepgxic34ahay5ignw 0 460018 2440091 2407461 2026-07-22T19:49:57Z GModena (WMF) 20194 2440091 wikitext text/x-wiki i4yvjecev8qewcq81pbej473a51ouja 0 460192 2439732 2435515 2026-07-22T08:59:47Z Effie Mouzeli (WMF) 12880 2439732 wikitext text/x-wiki kqgc0yaq08g4w138ydbnstjw6if99g7 0 460234 2439083 2438076 2026-07-20T19:22:20Z JVanderhoop-WMF 41069 2439083 wikitext text/x-wiki enm8h9yxm3c8bivt99lbc15kkts1nwp 2439429 2439083 2026-07-21T14:24:19Z MPopov (WMF) 5891 /* Step 2: Specify how to enroll and how many */ 2439429 wikitext text/x-wiki nei411wpohgdkrvi7lm6o99t57dydli 0 460277 2439010 2433001 2026-07-20T17:13:19Z BCornwall-WMF 31521 /* 2016 */ ce: use past tense 2439010 wikitext text/x-wiki k3zxkttk62aly2ik2ov1y01irxab2za 2439011 2439010 2026-07-20T17:14:25Z BCornwall-WMF 31521 /* Through the years */ Add 2017 cdn backend ttl 2439011 wikitext text/x-wiki tgn916fieo0p07upipz8s5d4nsljvu9 0 460377 2439340 2437961 2026-07-21T12:52:11Z Quiddity 1884 lang="text" 2439340 wikitext text/x-wiki jnbaf8cbuu7jq9zdhhte1lzg0uiliet 0 460417 2438819 2438757 2026-07-20T12:48:46Z AChou-WMF 15621 /* Deployment Information */ 2438819 wikitext text/x-wiki tfhkut486nythju5s6vbvfw1u696kog 2438892 2438819 2026-07-20T14:17:27Z AChou-WMF 15621 De-V0'd; Add new Events & programs hub 2438892 wikitext text/x-wiki rpq4e8sjp4bgxs4ufxjf95j37je182s 2438912 2438892 2026-07-20T14:47:12Z AChou-WMF 15621 /* Available Models */ Link base model to HF model cards; Update Qwen3-14B to Qwen3-14B-FP8 2438912 wikitext text/x-wiki 29gl9ikc6mq3wyu547oel9f7b35qsgw 2438915 2438912 2026-07-20T14:51:08Z AChou-WMF 15621 /* Deployments (ml-serve-eqiad) */ Update llm-qwen3-14b (now FP8/16K on 24GB partition) 2438915 wikitext text/x-wiki f2p2viz1fc3jqwab7hd3yzlsytbefl2 0 460424 2439733 2436397 2026-07-22T09:00:30Z Effie Mouzeli (WMF) 12880 2439733 wikitext text/x-wiki ciiaadbtyedx7454jxujgl3bfh6vwj1 0 460460 2438940 2026-07-20T15:32:47Z AChou-WMF 15621 Created page with "{{Draft}} ''Part of [[Machine Learning/LiftWing/Large Language Models|LiftWing Large Language Models]] · Wikimania 2026'' LiftWing hosts open-weight LLMs you can use in your Wikimedia projects and tools. This page gets you from zero to a working request in a few minutes — chat with a model in your browser, or call it from your code. For the full platform reference (internal models, deployment, performance), see the Machine Learning/LiftWing/Large Language Models|m..." 2438940 wikitext text/x-wiki ad4o3wbg6yvyb6vxdthzrf4ff5lr78f 2438945 2438940 2026-07-20T15:37:24Z AChou-WMF 15621 /* API usage */ 2438945 wikitext text/x-wiki 85c0xn96uidhaum66oag1lk0tbc4vpi 2 460461 2438944 2026-07-20T15:35:27Z BCornwall-WMF 31521 Created page with "{{Cleanup}}" 2438944 wikitext text/x-wiki 8pxrg5k7u6ys0du18jfm139yvhgt3cp 2438967 2438944 2026-07-20T16:15:15Z BCornwall-WMF 31521 2438967 wikitext text/x-wiki dwojldpsm7v8n1ahchtigdw93jh02ne 2438975 2438967 2026-07-20T16:20:44Z BCornwall-WMF 31521 2438975 wikitext text/x-wiki eaisx7c2tgwib11c9vj9qk17b9c566v 2439000 2438975 2026-07-20T17:00:44Z BCornwall-WMF 31521 2439000 wikitext text/x-wiki qzsxeclhwaqq7o7bfn3kuo35t34az9z 2439018 2439000 2026-07-20T17:21:10Z BCornwall-WMF 31521 Blanked the page 2439018 wikitext text/x-wiki phoiac9h4m842xq45sp7s6u21eteeq1 10 460462 2439001 2026-07-20T17:00:55Z BCornwall-WMF 31521 Created page with "<onlyinclude>{{mbox |type=style |image=[[File:Edit-clear.svg|44px]] |text='''This article or section needs language, wiki syntax or style improvements.'''{{#if:{{{1|}}}<small>Reason: {{1}}</small>}} }} </onlyinclude> ==Usage== This template should be added at the beginning of articles or sections with language, wiki syntax or style issues. Ensure to cite specific concerns with the first argument and possibly also on the flagged article's discussion page. <nowiki>{{St..." 2439001 wikitext text/x-wiki gwukn42zpkr6y2j8xwtz6ma2h532zvs 2439002 2439001 2026-07-20T17:05:26Z BCornwall-WMF 31521 syntax error fixes 2439002 wikitext text/x-wiki d2h9wvruj8g0phvv776j8fmkf02f3pz 2439003 2439002 2026-07-20T17:06:26Z BCornwall-WMF 31521 Add cat for pages 2439003 wikitext text/x-wiki jpkrbqf44zy54asfojr96f8jte2yzn5 14 460464 2439016 2026-07-20T17:19:33Z BCornwall-WMF 31521 Created page with "{{Hidden category}}" 2439016 wikitext text/x-wiki hkte2duffkjywpxxt3lp18bww66fee6 2439017 2439016 2026-07-20T17:20:42Z BCornwall-WMF 31521 Use existing cat 2439017 wikitext text/x-wiki 9ebva6ziys8jjyu1va27a7fprlq61v7 3 460465 2439174 2026-07-21T01:41:06Z Mfield 51590 Mfield moved page [[User talk:Renamed user cdb78c3737e6b7f6ba7e28cedcc6608711202eee]] to [[User talk:IVORK]] over redirect: Automatically moved page while renaming the user "[[Special:CentralAuth/Renamed user cdb78c3737e6b7f6ba7e28cedcc6608711202eee|Renamed user cdb78c3737e6b7f6ba7e28cedcc6608711202eee]]" to "[[Special:CentralAuth/IVORK|IVORK]]" 2439174 wikitext text/x-wiki e13fo5wwb22h0kacdrc3vxcovesc88u 3 460467 2439248 2026-07-21T08:39:09Z StrikerBot 8475 Welcome to Toolforge! 2439248 wikitext text/x-wiki 9l3xey0xiq5kxe1p0tf4zjy9tywpdce 2439257 2439248 2026-07-21T09:02:24Z Dagmawi-M 45860 /* Welcome to Toolforge! */ Reply 2439257 wikitext text/x-wiki ooxhh2axzbykanj7eynrjc3icxvoy4p 3 460468 2439249 2026-07-21T08:39:39Z StrikerBot 8475 Welcome to Toolforge! 2439249 wikitext text/x-wiki shqaqd80fnmgnrpwrzd68thbhplrxw6 3 460469 2439265 2026-07-21T10:10:46Z StrikerBot 8475 Welcome to Toolforge! 2439265 wikitext text/x-wiki ed86ul4v3hiviboq50590f42q68a3q1 0 460470 2439296 2026-07-21T11:11:24Z ESanders (WMF) 953 create stub 2439296 wikitext text/x-wiki 444x8oylmbfy85bvt89l4dcft3p145e 2439734 2439296 2026-07-22T09:01:14Z Effie Mouzeli (WMF) 12880 2439734 wikitext text/x-wiki cve0ziiabsbvttqtghvu7335yfn34un 2439735 2439734 2026-07-22T09:02:16Z Effie Mouzeli (WMF) 12880 2439735 wikitext text/x-wiki sqxy6ixxalac51l5raehlejrn55m0tt 2439746 2439735 2026-07-22T09:20:10Z Effie Mouzeli (WMF) 12880 2439746 wikitext text/x-wiki fh0ifcbweccdueo8vkmamznq39x23er 3 460471 2439312 2026-07-21T12:12:10Z StrikerBot 8475 Welcome to Toolforge! 2439312 wikitext text/x-wiki mifqe31930k5rolw3s8roavrvkgu5cv 3 460472 2439498 2026-07-21T16:08:24Z StrikerBot 8475 Welcome to Toolforge! 2439498 wikitext text/x-wiki 2cessyky3pnoi2ssytkvii7lhbuhupx 3 460473 2439524 2026-07-21T17:28:56Z StrikerBot 8475 Welcome to Toolforge! 2439524 wikitext text/x-wiki rwhjbfgvc71ps94b8d12tgasqkr2ib1 498 460474 2439695 2026-07-22T08:09:20Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: toolforge envvars create TOOL_SECRET_KEY < /proc/sys/kernel/random/uuid 2439695 wikitext text/x-wiki 8ddmmkeic7iomr8ukkt6k4dabcvxikm 2439696 2439695 2026-07-22T08:09:44Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: toolforge envvars create TOOL_OAUTH__CONSUMER_KEY 321c54d7da6e0ce319dfec8f5eb85119 2439696 wikitext text/x-wiki fohr7i5kd8x5fidqym6xflpcsgniwys 2439697 2439696 2026-07-22T08:10:01Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET 2439697 wikitext text/x-wiki 4ue99q2prx6q14cximuumsnyav8yv23 2439718 2439697 2026-07-22T08:25:21Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: toolforge build start --use-latest-versions https://gitlab.wikimedia.org/toolforge-repos/win-the-kiwi && webservice --mount=none --health-check-path=/healthz buildservice start 2439718 wikitext text/x-wiki amx62rz5socvolmbklzsqsfhf7leofm 2439766 2439718 2026-07-22T09:58:12Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: toolforge envvars create OAUTH__CHANGE_TAG "OAuth CID: 18794" 2439766 wikitext text/x-wiki gghg50acas0rhd0z85rs3vgtrfcjspz 2439769 2439766 2026-07-22T10:00:39Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 9bd2cbb20c 2439769 wikitext text/x-wiki kp2vbg2bjp6sc2uxk01w9wb5rj6h3pi 2439800 2439769 2026-07-22T10:59:23Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 02a8568de9 2439800 wikitext text/x-wiki k8z8lbi2j5f6o6sm5n7w8e64m3bpk6w 3 460475 2439719 2026-07-22T08:27:14Z StrikerBot 8475 Welcome to Toolforge! 2439719 wikitext text/x-wiki cntw1ser920v057axbx61b893ngrbao 0 460476 2439730 2026-07-22T08:57:25Z Effie Mouzeli (WMF) 12880 Created page with "[[File:Books_and_Scroll_Ornament_with_Open_Book.png|right|frameless|150px|alt=A scroll]] <h2> 📄 Need a Wikitech page?</h2> A good Wikitech component page is the home for your component's documentation, and it provides most of the information a SCROLL reviewer will need. Stand one up early. <div class="mw-tpl-createincidentreport-box" style="background: #f8f9fa; border: 1px solid #a2a9b1; border-top: 4px solid #4b8063; border-radius: 6px; padding: 16px 20px; margi..." 2439730 wikitext text/x-wiki fmvpwsyx2utbzmh9kzbcbq2ulnjn3y5