Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.15 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 12 1568 2447575 2443310 2026-08-14T09:43:10Z Taavi-WMF 41365 document scratch volume deprecation 2447575 wikitext text/x-wiki i8amfof3j1pa1od462dbaakljamjfvu 498 3086 2447011 2446649 2026-08-13T15:48:07Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers 2447011 wikitext text/x-wiki cdckmaus4sco05zsg0qansj0ci40xve 2447066 2447011 2026-08-13T16:43:27Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli 2447066 wikitext text/x-wiki 9mwgcg291khjx4d7z5wigz1czfv0bn0 2447067 2447066 2026-08-13T16:45:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli 2447067 wikitext text/x-wiki mdweqlz5is0kor8i1gy16ityewm5w2j 2447593 2447067 2026-08-14T10:06:20Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7, tools-k8s-gateway-1 2447593 wikitext text/x-wiki kmqf4ichl4yj4pipei6cuyq9kt8j07z 2447599 2447593 2026-08-14T10:08:03Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7, tools-k8s-gateway-1 2447599 wikitext text/x-wiki 0k6q80vpf5f434469dbkhxk4n85ivc8 2447600 2447599 2026-08-14T10:09:06Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8, tools-k8s-gateway-2 2447600 wikitext text/x-wiki ctsqrx4spow6xe1ni42gatq2wpund06 2447602 2447600 2026-08-14T10:10:43Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8, tools-k8s-gateway-2 2447602 wikitext text/x-wiki 4jdfdiiy3nrfqivvisihsvjg12mec8z 2447610 2447602 2026-08-14T10:12:52Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-gateway-3 2447610 wikitext text/x-wiki ssg06b4jcgnstnhvzc9yr9v5wft67gk 2447611 2447610 2026-08-14T10:14:29Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-gateway-3 2447611 wikitext text/x-wiki kni53kauy2vjkxveuuth5apai0gfm5q 0 4108 2446766 2446539 2026-08-13T13:01:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1325476]] to Thursday, August 13 UTC afternoon backport window 2446766 wikitext text/x-wiki q7wry7zces4turhq09icizgms3sky3x 2446805 2446766 2026-08-13T13:29:58Z ScheduleDeploymentBot 37566 Add [[gerrit:1325480]] to Thursday, August 13 UTC afternoon backport window 2446805 wikitext text/x-wiki 4rq674otrg4vm3ragdiwr2h8z0cbdtn 2447034 2446805 2026-08-13T16:14:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1319906]] to Thursday, August 13 UTC late backport window 2447034 wikitext text/x-wiki o4u6rs0jsmxo9f32lbdtxxl37v6qm4x 2447053 2447034 2026-08-13T16:33:26Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-08-13}} */ UTC-late infra window 2447053 wikitext text/x-wiki 45ci5p8aa4g634idarh6ndjjz2zo3bz 2447212 2447053 2026-08-13T19:08:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1325549]] to Thursday, August 13 UTC late backport window 2447212 wikitext text/x-wiki kzpw89l51gnkvjs7yhs2aw1z1b7ja3s 2447224 2447212 2026-08-13T19:34:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1320163]] to Thursday, August 13 UTC late backport window 2447224 wikitext text/x-wiki ioswopbg8kmmf0h4eve2p3ufalg4c4i 2447225 2447224 2026-08-13T19:38:40Z ScheduleDeploymentBot 37566 Add [[gerrit:1325553]] to Thursday, August 13 UTC late backport window 2447225 wikitext text/x-wiki fr25rsf3cv0dx9atvyn3uafgk4f08vn 498 6726 2447060 2446597 2026-08-13T16:37:32Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli 2447060 wikitext text/x-wiki 9pi4drbjkycac7hznxnm0pc97ivjknr 2447063 2447060 2026-08-13T16:39:14Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli 2447063 wikitext text/x-wiki t0fn9reybel38a67ybh0wfcfpdbjp66 2447108 2447063 2026-08-13T17:38:07Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2447108 wikitext text/x-wiki 5hdqybtnb40uktmw7oqocjxjqlfflp5 2447109 2447108 2026-08-13T17:38:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2447109 wikitext text/x-wiki apc3gboqktb995ddc0eo0nfhxekpowi 0 7919 2446682 2446681 2026-08-13T12:01:07Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2446682 wikitext text/x-wiki ciedm8a1964xkvgzlvuivd64puinjad 2446683 2446682 2026-08-13T12:01:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet 2446683 wikitext text/x-wiki f8ppmkpb7rgmxyaxuyo03ke9u4pba9o 2446684 2446683 2026-08-13T12:01:35Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet 2446684 wikitext text/x-wiki lh6e39rrj05f4s06qsnjdqfqdg3yw0a 2446685 2446684 2026-08-13T12:02:09Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1005.eqiad.wmnet 2446685 wikitext text/x-wiki 6gpqqf405e2miy0ze542ydk98wlnopy 2446686 2446685 2026-08-13T12:03:12Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be1006.eqiad.wmnet 2446686 wikitext text/x-wiki dpogny25jsbg60omthk154tm0yaj51e 2446687 2446686 2026-08-13T12:04:18Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446687 wikitext text/x-wiki idspbeg4082fwr96z2bsijhpqc5wazh 2446688 2446687 2026-08-13T12:04:31Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2446688 wikitext text/x-wiki kb6sm0r2ze9ub7d4yelyr924sb2pzht 2446689 2446688 2026-08-13T12:05:56Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet 2446689 wikitext text/x-wiki snezaj2tpgmi0hztvybmd2rq53coy9n 2446690 2446689 2026-08-13T12:05:58Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446690 wikitext text/x-wiki scno3vnge5p7xd1hxrz7jcph2pi2jzr 2446691 2446690 2026-08-13T12:08:17Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet 2446691 wikitext text/x-wiki nbr1htbjaz1vo6wk3sxwcbzwzp9pybd 2446692 2446691 2026-08-13T12:09:23Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet 2446692 wikitext text/x-wiki mi27uu6synuj6soumgwq5dfiwpok07w 2446693 2446692 2026-08-13T12:10:22Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446693 wikitext text/x-wiki kd26qthnmi9gyg2xnpyny0fukz18c9d 2446694 2446693 2026-08-13T12:10:26Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446694 wikitext text/x-wiki b76mbcv15ztu1hndjopwcye2rhsrcmq 2446695 2446694 2026-08-13T12:10:29Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446695 wikitext text/x-wiki ms40qp4qie5dm4cudzyrw4gtd42oqov 2446696 2446695 2026-08-13T12:10:32Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446696 wikitext text/x-wiki 2xyh3qtkzqf9oeky0ky5t95nepejcde 2446697 2446696 2026-08-13T12:10:34Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446697 wikitext text/x-wiki 9ozk43onwkj1wx6sjxbyzbsd6r79xt8 2446698 2446697 2026-08-13T12:10:36Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1006.eqiad.wmnet 2446698 wikitext text/x-wiki nj26k2ccuutvixx8p7qokz94em7lbqv 2446699 2446698 2026-08-13T12:10:48Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446699 wikitext text/x-wiki c22s6s1zghzhcybagttqa4erc1y9enu 2446700 2446699 2026-08-13T12:10:59Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host moss-be1003.eqiad.wmnet 2446700 wikitext text/x-wiki cyfl8bd533vqdu80v3yw9pr44e9x2gp 2446701 2446700 2026-08-13T12:12:30Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1178.eqiad.wmnet with OS bookworm 2446701 wikitext text/x-wiki 6fzc29ue2e0xfbnm8yyuayteo2n3jf9 2446702 2446701 2026-08-13T12:13:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet 2446702 wikitext text/x-wiki ivf9lrnw8im14qgpudq4tayl921m8vp 2446703 2446702 2026-08-13T12:13:54Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet 2446703 wikitext text/x-wiki 22a5hi8drum7kd6144plqsahfs84n4c 2446704 2446703 2026-08-13T12:15:05Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1211.eqiad.wmnet with OS bookworm 2446704 wikitext text/x-wiki jciiwg4by6qic4a8paaj7itcxigh11l 2446705 2446704 2026-08-13T12:15:09Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1210.eqiad.wmnet with OS bookworm 2446705 wikitext text/x-wiki ko92bm7dkmpz08qvc5u9w676lb0af7a 2446706 2446705 2026-08-13T12:15:12Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1233.eqiad.wmnet with OS bookworm 2446706 wikitext text/x-wiki 5rxg47ib0kjc3hug19oc10ffmqmz55v 2446707 2446706 2026-08-13T12:15:56Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2446707 wikitext text/x-wiki s8l5binffjnhowj08ktab6w2mhg2t6v 2446708 2446707 2026-08-13T12:17:28Z Stashbot 7414 moritzm: installin curl security updates 2446708 wikitext text/x-wiki s0d3omaurdfix4gnw25d14t71jpfylu 2446709 2446708 2026-08-13T12:17:53Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet 2446709 wikitext text/x-wiki 681hg7rzippmjlgwgs83p9vrxk2945i 2446711 2446709 2026-08-13T12:18:14Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1003.eqiad.wmnet 2446711 wikitext text/x-wiki rdri23jgf54lu2ukey6rjjru3nvl7k7 2446712 2446711 2026-08-13T12:19:11Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446712 wikitext text/x-wiki 54lss8yk4o3fvrvdkl79z3egbfkte7w 2446713 2446712 2026-08-13T12:19:17Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet 2446713 wikitext text/x-wiki locfmz8ro8of4tz9eogw56tqjscefva 2446714 2446713 2026-08-13T12:19:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet 2446714 wikitext text/x-wiki s3rrvqtzmvlemy63p34dhaqbr3kug1f 2446715 2446714 2026-08-13T12:19:56Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446715 wikitext text/x-wiki eoexgh9cb9s90dt014mexnxw5cmoq2q 2446716 2446715 2026-08-13T12:19:58Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446716 wikitext text/x-wiki 20sbb2c6so6wtjsbmri9nwse1qyhxp3 2446717 2446716 2026-08-13T12:20:01Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446717 wikitext text/x-wiki s46hb0qxbkyc358715u6jo5wfu7y3j3 2446718 2446717 2026-08-13T12:20:02Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446718 wikitext text/x-wiki 150gasco4o3hvdjb0lplm6yzngkftw6 2446719 2446718 2026-08-13T12:20:06Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox dns entries for expanded public1-603-eqsin subnet - cmooney@cumin1003" 2446719 wikitext text/x-wiki ood9vzpq7qtbmrgk5p9jck7cu0cgsky 2446720 2446719 2026-08-13T12:20:11Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox dns entries for expanded public1-603-eqsin subnet - cmooney@cumin1003" 2446720 wikitext text/x-wiki r01w43ihrux0ya0abd9ss9sphbs53n2 2446721 2446720 2026-08-13T12:20:13Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446721 wikitext text/x-wiki a4b8fftxh804s68jl7dizy8nr63ywf6 2446722 2446721 2026-08-13T12:20:15Z Stashbot 7414 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet 2446722 wikitext text/x-wiki m7weq7l428e8ja8jfxy7ecwz26z6k9j 2446723 2446722 2026-08-13T12:20:28Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet 2446723 wikitext text/x-wiki c6a0ikyer7vwxbz4vcxan1cn2p3zbft 2446724 2446723 2026-08-13T12:21:52Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2446724 wikitext text/x-wiki t82dlisa0zy8fzno3yhwtbp8hwbkps2 2446725 2446724 2026-08-13T12:21:57Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet 2446725 wikitext text/x-wiki o28ksnjg1mr2h142tt8p7kzmh471xgu 2446727 2446725 2026-08-13T12:24:01Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2446727 wikitext text/x-wiki qkr7joitea86etzy8icrt9ou6vdv4zv 2446728 2446727 2026-08-13T12:26:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet 2446728 wikitext text/x-wiki 505vyrjm1hq398nlrpe50420v9x04wp 2446729 2446728 2026-08-13T12:26:34Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow5003.eqsin.wmnet 2446729 wikitext text/x-wiki kcufs880w8ysjjrqdtrcbru612o1stj 2446730 2446729 2026-08-13T12:29:18Z Stashbot 7414 moritzm: installing Wireshark security updates 2446730 wikitext text/x-wiki nlaheld81p5jk3x7r65lahknrjo8ptu 2446731 2446730 2026-08-13T12:29:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1211.eqiad.wmnet with reason: host reimage 2446731 wikitext text/x-wiki aij7ozx9j0jq6megiydu24u7ftz73xt 2446732 2446731 2026-08-13T12:29:46Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1210.eqiad.wmnet with reason: host reimage 2446732 wikitext text/x-wiki 0ycu5bww6n712g97cbfrzkn9err198j 2446733 2446732 2026-08-13T12:31:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet 2446733 wikitext text/x-wiki g7nsn00wtqi5kglul9lo1h1x7rmddl3 2446734 2446733 2026-08-13T12:32:00Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet 2446734 wikitext text/x-wiki dvt0o6w3ya1axhdma86o5cmzch0v85u 2446735 2446734 2026-08-13T12:32:06Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5003.eqsin.wmnet 2446735 wikitext text/x-wiki 5rvl48yj2rlvwcspi9jf7n9hx5ki4az 2446736 2446735 2026-08-13T12:32:09Z Stashbot 7414 marostegui: Drop SecurePoll tables from closed wikis T423128 2446736 wikitext text/x-wiki 9puvutciu0ycf30pnmrvm7wrorpkfug 2446737 2446736 2026-08-13T12:32:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org 2446737 wikitext text/x-wiki l0ipj5yn3vlo8hc8oqdszrpo1vmtxry 2446738 2446737 2026-08-13T12:32:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet 2446738 wikitext text/x-wiki tdmmssx3qdqlzc81vyc9r9y9w2a8u7c 2446739 2446738 2026-08-13T12:33:27Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1233.eqiad.wmnet with reason: host reimage 2446739 wikitext text/x-wiki hyx7vefu4n6of06zugwcp1wj2r3he61 2446741 2446739 2026-08-13T12:34:53Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1211.eqiad.wmnet with reason: host reimage 2446741 wikitext text/x-wiki 43fx1dhpcmv2bbir6gu41uial79r953 2446742 2446741 2026-08-13T12:37:48Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet 2446742 wikitext text/x-wiki mwe4935aes78r4yhuo5ibbc6a3wu9f6 2446743 2446742 2026-08-13T12:38:08Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1233.eqiad.wmnet with reason: host reimage 2446743 wikitext text/x-wiki ncmm38d049ltknf14h3cv63chpnfofg 2446744 2446743 2026-08-13T12:38:35Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org 2446744 wikitext text/x-wiki c5ogm3idqxks4cvk7eukrt7ilq9sudy 2446745 2446744 2026-08-13T12:39:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Added db1274', diff saved to https://phabricator.wikimedia.org/P96062 and previous config saved to /var/cache/conftool/dbconfig/20260813-123907-cwilliams.json 2446745 wikitext text/x-wiki krfpunbf222fgdblbug8j3m6rcjvxeb 2446746 2446745 2026-08-13T12:39:38Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1274: New host 2446746 wikitext text/x-wiki dqukwshit9s8l4ti1yfzcb0gtvbo9g5 2446747 2446746 2026-08-13T12:40:05Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2446747 wikitext text/x-wiki 5n0ptvrp3wzfu6u9lyis0kq3p04wogk 2446748 2446747 2026-08-13T12:41:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Pool db1167.eqiad.wmnet in after cloning 2446748 wikitext text/x-wiki 2nezhp2br2movzyktgrp4degx8rc7vf 2446749 2446748 2026-08-13T12:42:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1210.eqiad.wmnet with reason: host reimage 2446749 wikitext text/x-wiki p81xcj29l8i38y2qam8lo8ea4dtqffu 2446750 2446749 2026-08-13T12:43:23Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet 2446750 wikitext text/x-wiki l1s7sqq4m5f3kusa9vytoqql6ygwn0a 2446751 2446750 2026-08-13T12:43:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet 2446751 wikitext text/x-wiki 9otn1tg8mt3k473xuxr70b06wghkg78 2446752 2446751 2026-08-13T12:44:56Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns record for deleted IP reservations eqsin lvs vlan ints - cmooney@cumin1003" 2446752 wikitext text/x-wiki f8fwirvix44iip1nrwxuwlp3r5xtrz2 2446753 2446752 2026-08-13T12:45:00Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns record for deleted IP reservations eqsin lvs vlan ints - cmooney@cumin1003" 2446753 wikitext text/x-wiki 8tiv1q0t0fudsa5tp9aefuzl4wd206h 2446754 2446753 2026-08-13T12:45:02Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446754 wikitext text/x-wiki 2pg69qew8v2ie21mzcq2t6o9a0rbrpc 2446755 2446754 2026-08-13T12:46:40Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.clone of db1169.eqiad.wmnet onto db1277.eqiad.wmnet 2446755 wikitext text/x-wiki aic1na0cv9d1gs8y0qc5de6fjhku8r3 2446756 2446755 2026-08-13T12:46:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1277.eqiad.wmnet - cwilliams@cumin1003 2446756 wikitext text/x-wiki lrfsz9mcnhrgji2r0juxfq7yfr05cpb 2446757 2446756 2026-08-13T12:49:19Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1277.eqiad.wmnet - cwilliams@cumin1003 2446757 wikitext text/x-wiki 9bc3cs02yc85asvi9vji6amhbjk268v 2446758 2446757 2026-08-13T12:52:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet 2446758 wikitext text/x-wiki lz6wbtib08jaaqb25grx8onc87gnriu 2446759 2446758 2026-08-13T12:52:21Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet 2446759 wikitext text/x-wiki 5mv58dyjsls4b1uj9i15afx6ch0nc17 2446760 2446759 2026-08-13T12:55:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet 2446760 wikitext text/x-wiki hohr1fp20lgztr5ns8s0cbo5dxztqfp 2446761 2446760 2026-08-13T12:55:21Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1211.eqiad.wmnet with OS bookworm 2446761 wikitext text/x-wiki rg5578cksny455dw6telwn5qc9z58rq 2446762 2446761 2026-08-13T12:58:14Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet 2446762 wikitext text/x-wiki ibmu679vtepap1gad94kxtc8ks8097c 2446763 2446762 2026-08-13T13:00:25Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet 2446763 wikitext text/x-wiki 446v2yeyn2qael5uy1amzxe58q94i6i 2446764 2446763 2026-08-13T13:00:31Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet 2446764 wikitext text/x-wiki 3o37x1mwbq2q22m54ui6l6aykz0qp8n 2446765 2446764 2026-08-13T13:01:07Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet 2446765 wikitext text/x-wiki t5u6y6cvwvl2jjafwdk4zmw1min7irl 2446767 2446765 2026-08-13T13:02:20Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1210.eqiad.wmnet with OS bookworm 2446767 wikitext text/x-wiki joeoa2i3atuyptv77gs2zl0zzg6wfwe 2446768 2446767 2026-08-13T13:03:27Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet 2446768 wikitext text/x-wiki c122vpy0lauvwmv8101j39wio79ms1i 2446769 2446768 2026-08-13T13:03:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow3004.esams.wmnet 2446769 wikitext text/x-wiki fgrzzspq9bdpf7oh4qydevc4i12uk8m 2446770 2446769 2026-08-13T13:04:22Z Stashbot 7414 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1324731|prv: Enable parsoid rendering for 5 wikisource wikis]] 2446770 wikitext text/x-wiki 6fgnewu1avjvn07nmw5gm8j1c6gln74 2446771 2446770 2026-08-13T13:06:25Z Stashbot 7414 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1324731|prv: Enable parsoid rendering for 5 wikisource wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446771 wikitext text/x-wiki gh44pfq7fn4wt4pxi3ujdd2eguze868 2446772 2446771 2026-08-13T13:07:00Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1233.eqiad.wmnet with OS bookworm 2446772 wikitext text/x-wiki lmpaidqf0h3qxztcv5oyadujjogsmfj 2446773 2446772 2026-08-13T13:07:39Z Stashbot 7414 jgiannelos@deploy1003: jgiannelos: Continuing with deployment 2446773 wikitext text/x-wiki 73pbymmw4ljpal1hz78rqd5x6w51gdt 2446774 2446773 2026-08-13T13:07:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3004.esams.wmnet 2446774 wikitext text/x-wiki g2olptq5kndm7vxceqc0ybrlrrq9wi9 2446775 2446774 2026-08-13T13:11:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet 2446775 wikitext text/x-wiki tn3v7ewnjfay56a54q8qooq2sqrx0x9 2446776 2446775 2026-08-13T13:11:08Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet 2446776 wikitext text/x-wiki f84h598txeh1xt1rb0k5ohzee3awkn0 2446777 2446776 2026-08-13T13:11:21Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet 2446777 wikitext text/x-wiki suf3adzl3qe1ykus8vh68bzhhvnrp39 2446778 2446777 2026-08-13T13:11:48Z Stashbot 7414 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324731|prv: Enable parsoid rendering for 5 wikisource wikis]] (duration: 07m 26s) 2446778 wikitext text/x-wiki 1uu7tox0ysvv2e3djtoksxehocz8e09 2446779 2446778 2026-08-13T13:17:12Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1325476|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] 2446779 wikitext text/x-wiki tkojwngs4w92mo3uajemo9yft7r01m2 2446780 2446779 2026-08-13T13:18:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1149.eqiad.wmnet with OS bookworm 2446780 wikitext text/x-wiki 6hto663w05euftums7r6zt26l5brjs8 2446781 2446780 2026-08-13T13:18:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1214.eqiad.wmnet with OS bookworm 2446781 wikitext text/x-wiki shjk2xizqq4qirizqv5maw77knrrw8q 2446782 2446781 2026-08-13T13:18:36Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1215.eqiad.wmnet with OS bookworm 2446782 wikitext text/x-wiki j6ug3mc76ribz7018rlqzuuxnch2pnb 2446783 2446782 2026-08-13T13:18:50Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus3004.esams.wmnet 2446783 wikitext text/x-wiki adxwkz24ebkk1odoc0v5xzzjkgllhga 2446785 2446783 2026-08-13T13:19:14Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1325476|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446785 wikitext text/x-wiki l6p13naqygflwr5j0qauc4p8srbb4wh 2446786 2446785 2026-08-13T13:19:18Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough 2446786 wikitext text/x-wiki ax70qejt825hhoez7uyp0t8wtb7vxh0 2446787 2446786 2026-08-13T13:19:45Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet 2446787 wikitext text/x-wiki 1jks36neue3vs2sdqxjoac5b189ai28 2446789 2446787 2026-08-13T13:20:26Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2446789 wikitext text/x-wiki kbxmp0ym6i1wb34ioziptwoa7t4jag5 2446790 2446789 2026-08-13T13:20:29Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.decommission for hosts lvs1013.eqiad.wmnet 2446790 wikitext text/x-wiki kys28w7dfl7iesllkx6t8mjsfwpszq1 2446791 2446790 2026-08-13T13:22:08Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet 2446791 wikitext text/x-wiki 9hpvu96rkzhgbdh8cr3y8h8qe4kzp80 2446792 2446791 2026-08-13T13:23:34Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet 2446792 wikitext text/x-wiki ap3zt6ret53rov8llv9ug3jozlcu9ek 2446793 2446792 2026-08-13T13:24:11Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260803000000" --end-timestamp="20260804000000" --sleep="5" --batch-size="10"` for T434688 2446793 wikitext text/x-wiki jqdjw8tgoq34c66pgtn7xc3ya6i6v44 2446794 2446793 2026-08-13T13:24:19Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260804000000" --end-timestamp="20260805000000" --sleep="5" --batch-size="10"` for T434688 2446794 wikitext text/x-wiki h54inu4vuopncspttz8splewdzvydgr 2446795 2446794 2026-08-13T13:24:32Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325476|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] (duration: 07m 19s) 2446795 wikitext text/x-wiki 6a4srrcs1k9wgmgxxavv8xpi1nsit20 2446796 2446795 2026-08-13T13:24:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: New host 2446796 wikitext text/x-wiki 67w0v3irlwl89l7twpy198mmgkgpv27 2446797 2446796 2026-08-13T13:24:55Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3004.esams.wmnet 2446797 wikitext text/x-wiki 3toihp6ujv0e6ffvkzfhycfkbpbddck 2446798 2446797 2026-08-13T13:24:57Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.netbox 2446798 wikitext text/x-wiki 8c8plsq4sd0dhpcga0tqvx2p5iy6ba0 2446799 2446798 2026-08-13T13:25:14Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260802000000" --end-timestamp="20260803000000" --sleep="5" --batch-size="10"` for T434688 2446799 wikitext text/x-wiki 6xnr9rkgh1v3r4zmbgfrwh2eglqgfz1 2446800 2446799 2026-08-13T13:25:17Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus4003.ulsfo.wmnet 2446800 wikitext text/x-wiki r1tcbr6y3ylwwnqnp1977w62nqfcpel 2446801 2446800 2026-08-13T13:26:33Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Pool db1167.eqiad.wmnet in after cloning 2446801 wikitext text/x-wiki nivvo419egvf66goiqm1lgq9elkhqg1 2446802 2446801 2026-08-13T13:26:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1167.eqiad.wmnet onto db1281.eqiad.wmnet 2446802 wikitext text/x-wiki e9ljyjgdjs7hea4rbayn9m23md059s5 2446803 2446802 2026-08-13T13:27:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet 2446803 wikitext text/x-wiki 65wvcy7hretqk10tlisxhxu6q5dfo5a 2446804 2446803 2026-08-13T13:27:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet 2446804 wikitext text/x-wiki nezuk5739rans47ehvq0mil45mmawjj 2446806 2446804 2026-08-13T13:30:05Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" 2446806 wikitext text/x-wiki rglot2pyvuo42t3k1x5d19a33pzblkp 2446807 2446806 2026-08-13T13:30:33Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" 2446807 wikitext text/x-wiki l3jrvlrcfaczma5xctqqy4h5edfsh7i 2446808 2446807 2026-08-13T13:30:36Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446808 wikitext text/x-wiki 8x9x30mu5hhebc7t6bs9t6jmayoh227 2446809 2446808 2026-08-13T13:30:38Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1013.eqiad.wmnet 2446809 wikitext text/x-wiki ec83pqlgcat4xgypu0vojy7or9w7aw5 2446810 2446809 2026-08-13T13:30:53Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.decommission for hosts lvs1014.eqiad.wmnet 2446810 wikitext text/x-wiki gqkk2cjhsjf76q9ch06flm6qj0xlqs7 2446811 2446810 2026-08-13T13:31:21Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4003.ulsfo.wmnet 2446811 wikitext text/x-wiki 0id7r8homsor2fyp8ude4ty32f3jg22 2446812 2446811 2026-08-13T13:31:34Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2003 2446812 wikitext text/x-wiki lmdfe95j7eyn8vr90fccgc0y6rwj99z 2446813 2446812 2026-08-13T13:31:47Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus5003.eqsin.wmnet 2446813 wikitext text/x-wiki 526c96flwjrrxbi3nks2kt6dduu6jcp 2446814 2446813 2026-08-13T13:33:19Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet 2446814 wikitext text/x-wiki 67gl299nah8mjogkx15k5ky7ugmbpku 2446815 2446814 2026-08-13T13:33:22Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1214.eqiad.wmnet with reason: host reimage 2446815 wikitext text/x-wiki j2ohf2ps9bj5trr46xafki23mcbaa89 2446816 2446815 2026-08-13T13:33:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2004.codfw.wmnet 2446816 wikitext text/x-wiki odnkz0gqi79y2vi6yvjxwkwy0k54uqe 2446817 2446816 2026-08-13T13:33:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1215.eqiad.wmnet with reason: host reimage 2446817 wikitext text/x-wiki 5o60x8wakfx5xewwy158v244p4ajyil 2446818 2446817 2026-08-13T13:35:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1149.eqiad.wmnet with reason: host reimage 2446818 wikitext text/x-wiki 7nuzncp8aub4bhrt06xt1ta40dpr320 2446819 2446818 2026-08-13T13:35:45Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.netbox 2446819 wikitext text/x-wiki 1ftskugt9ihigy6vok4htonveocnpfh 2446820 2446819 2026-08-13T13:38:02Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5003.eqsin.wmnet 2446820 wikitext text/x-wiki 20irk4oe855ujap3yygee2nkjxpv68n 2446821 2446820 2026-08-13T13:38:29Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet 2446821 wikitext text/x-wiki 240dspc6aua7wr9jj7ushm8rd2gwbkg 2446822 2446821 2026-08-13T13:38:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2004.codfw.wmnet 2446822 wikitext text/x-wiki e9gr22q121jaqarbnlj2u1ajpsb8y4x 2446823 2446822 2026-08-13T13:39:24Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1214.eqiad.wmnet with reason: host reimage 2446823 wikitext text/x-wiki 5aef1rdkpyr7ftsfux6x12gren824su 2446824 2446823 2026-08-13T13:40:12Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" 2446824 wikitext text/x-wiki izm9h0rydsxvq6iaw0hql0cgbhqr5l3 2446825 2446824 2026-08-13T13:40:16Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet 2446825 wikitext text/x-wiki qvm5jiy50dri1zzox57dhrmh6osgkvb 2446826 2446825 2026-08-13T13:40:28Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" 2446826 wikitext text/x-wiki oav819me3b6c667npuvpvjaobkh72e4 2446827 2446826 2026-08-13T13:40:30Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446827 wikitext text/x-wiki c63ma03402ao7kgo4yj5nysxg2ki9t9 2446828 2446827 2026-08-13T13:40:32Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1014.eqiad.wmnet 2446828 wikitext text/x-wiki 12x74lgelwm1e7b1lq89ddpb536piah 2446829 2446828 2026-08-13T13:40:58Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.decommission for hosts lvs1015.eqiad.wmnet 2446829 wikitext text/x-wiki kk941g5tdcxmqahhb3ul3r6zhdefgzx 2446830 2446829 2026-08-13T13:41:12Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1003.eqiad.wmnet 2446830 wikitext text/x-wiki lwjh4sbq7dc5b4pq6t15ocxa7yb19xb 2446831 2446830 2026-08-13T13:42:43Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1325480|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] 2446831 wikitext text/x-wiki nhpzq7hmwbho6la2cyfi2e8v9xvczuj 2446832 2446831 2026-08-13T13:43:07Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1149.eqiad.wmnet with reason: host reimage 2446832 wikitext text/x-wiki i8zgdwaxtb3azeglnzcl2ijwkbku7fm 2446833 2446832 2026-08-13T13:43:48Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260805000000" --end-timestamp="20260806000000" --sleep="5" --batch-size="10"` for T434688 2446833 wikitext text/x-wiki 75sanf5om6iqg0ddojt4ofm10ir6wi3 2446834 2446833 2026-08-13T13:44:34Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6002.drmrs.wmnet 2446834 wikitext text/x-wiki 6vmdjm1rmr5in85ogye2z8l12oc3cqb 2446835 2446834 2026-08-13T13:44:45Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1325480|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446835 wikitext text/x-wiki dv6tl8tm17sxu1szcdtkzyb5tbjn4uf 2446836 2446835 2026-08-13T13:44:49Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus7002.magru.wmnet 2446836 wikitext text/x-wiki 3jmmu8470sog6a317dqfqbo9pd789ki 2446837 2446836 2026-08-13T13:45:11Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2446837 wikitext text/x-wiki kr9x3fzxvis9d18qlr7hzs8v7xca4xv 2446838 2446837 2026-08-13T13:45:17Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus1005.eqiad.wmnet 2446838 wikitext text/x-wiki nja6snwi1m6gp5uo64qnhlwln9lwmow 2446839 2446838 2026-08-13T13:45:33Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.netbox 2446839 wikitext text/x-wiki swtahmlg8sdnixckt75qht2a5egxiol 2446840 2446839 2026-08-13T13:45:38Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet 2446840 wikitext text/x-wiki acd3px07otoh1fj6ulqy5n4eibvpssb 2446841 2446840 2026-08-13T13:45:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet 2446841 wikitext text/x-wiki d9hqq2aehmm4cstb1pygb4xujojvmrk 2446842 2446841 2026-08-13T13:45:56Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=no; selector: name=dns5003.* 2446842 wikitext text/x-wiki qotqyog1iqphnhhnbei2u7vndmvnjws 2446843 2446842 2026-08-13T13:46:35Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1003.eqiad.wmnet 2446843 wikitext text/x-wiki amw9wa66ui9cgknzgeoohee2oc9jtg6 2446844 2446843 2026-08-13T13:46:40Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.hosts.reboot-single for host dns5003.wikimedia.org 2446844 wikitext text/x-wiki qltbs7rhdr23nf0rfgis0q0a8ayl9rv 2446845 2446844 2026-08-13T13:47:06Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1215.eqiad.wmnet with reason: host reimage 2446845 wikitext text/x-wiki l3n6rbyns7w4b5a92wndb9w52y7cenw 2446846 2446845 2026-08-13T13:49:22Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325480|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] (duration: 06m 39s) 2446846 wikitext text/x-wiki rcfwbqhk3r8axeze0tf19e9qmxi07ha 2446847 2446846 2026-08-13T13:49:55Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" 2446847 wikitext text/x-wiki 8sslin8ceb6pxgiup1t70lc4uatwo51 2446848 2446847 2026-08-13T13:50:19Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" 2446848 wikitext text/x-wiki bcs5rki7082xlauh1nhu144aeotr8m9 2446849 2446848 2026-08-13T13:50:21Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446849 wikitext text/x-wiki 54vu4nwd6rnn235e9jssnnxw6aw2xd2 2446850 2446849 2026-08-13T13:50:24Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1015.eqiad.wmnet 2446850 wikitext text/x-wiki 0hgojyvvr1zgcvhptxh18zjrzrq40y7 2446851 2446850 2026-08-13T13:50:55Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7002.magru.wmnet 2446851 wikitext text/x-wiki bobdswqol944pril5955rixwvmi31u9 2446852 2446851 2026-08-13T13:51:11Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host apifeatureusage1001.eqiad.wmnet with OS bookworm 2446852 wikitext text/x-wiki 01alsyppftodvk1anbpeq1kksc3en14 2446853 2446852 2026-08-13T13:51:14Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet 2446853 wikitext text/x-wiki t8rlch3hx7zeune12ezbhnvag0uor69 2446854 2446853 2026-08-13T13:51:16Z Stashbot 7414 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw 2446854 wikitext text/x-wiki 59l1bil1ovksc8rpy6mw1qvz5712shf 2446855 2446854 2026-08-13T13:51:36Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet 2446855 wikitext text/x-wiki pz1j2le8wmtw96ew2aa37t0tuusu3ys 2446856 2446855 2026-08-13T13:55:01Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1005.eqiad.wmnet 2446856 wikitext text/x-wiki 5stb3d74988tag8z1mcaibtu3czhviv 2446857 2446856 2026-08-13T13:55:15Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dns5003.wikimedia.org 2446857 wikitext text/x-wiki g7ya01wr89wzhibc2rrjlu0c8d6k49s 2446858 2446857 2026-08-13T13:55:21Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus1007.eqiad.wmnet 2446858 wikitext text/x-wiki 1397zo6g43jr9c0wybi8jmw1217yu6x 2446859 2446858 2026-08-13T13:55:35Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet 2446859 wikitext text/x-wiki b3f02uni3mw2se5aaeh7bobu379n1o8 2446860 2446859 2026-08-13T13:56:26Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=yes; selector: name=dns5003.*,service=authdns-update 2446860 wikitext text/x-wiki 4zl3ae6dgo9nxhaugfki41bkz2hzm2b 2446861 2446860 2026-08-13T13:56:32Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2446861 wikitext text/x-wiki 03egfonp7gr133kep9brh55ant2kbbe 2446862 2446861 2026-08-13T13:56:35Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2446862 wikitext text/x-wiki r0zggo93pa6jumblgalu0mkefk6rgbs 2446863 2446862 2026-08-13T13:57:12Z Stashbot 7414 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw 2446863 wikitext text/x-wiki l43xre3bgu8vlzqb46hyevsd6tnaw3g 2446864 2446863 2026-08-13T13:57:14Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2003 2446864 wikitext text/x-wiki 32r30ddzaaxnblze8ochjmol0bofass 2446865 2446864 2026-08-13T13:57:23Z Stashbot 7414 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad 2446865 wikitext text/x-wiki hojp3foucjmqotthtvuz0yi2yno6j0h 2446866 2446865 2026-08-13T13:57:37Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet 2446866 wikitext text/x-wiki a451pato1dbntv0ivv3ck67f2x2p2lm 2446867 2446866 2026-08-13T13:58:37Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2446867 wikitext text/x-wiki s8wynup79wssct3xbs2q4s8dgv877n2 2446868 2446867 2026-08-13T13:59:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet 2446868 wikitext text/x-wiki 3ga9cklmusp4p8wmiv0471uajgn2i3k 2446869 2446868 2026-08-13T13:59:20Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet 2446869 wikitext text/x-wiki 0mb04o540woix1of166sbnrpov1xmaq 2446870 2446869 2026-08-13T13:59:29Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=yes; selector: name=dns5003.* 2446870 wikitext text/x-wiki gx0hiwitpnyne8pghq0wrm8lvqhz1jg 2446871 2446870 2026-08-13T13:59:49Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1214.eqiad.wmnet with OS bookworm 2446871 wikitext text/x-wiki afl55ui6z3nr50envifyapwb9guvypq 2446872 2446871 2026-08-13T14:00:33Z Stashbot 7414 moritzm: installing libxml2 security updates 2446872 wikitext text/x-wiki in9owohfmcnna8qacxn7zhk9pfi7xnu 2446873 2446872 2026-08-13T14:01:38Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet 2446873 wikitext text/x-wiki s7gmy9rzl6brgq9fpi7g30bbat74kff 2446874 2446873 2026-08-13T14:02:03Z Stashbot 7414 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad 2446874 wikitext text/x-wiki 9242cnjzkymrahldgwmks9t5wsjnlbc 2446875 2446874 2026-08-13T14:02:57Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on apifeatureusage1001.eqiad.wmnet with reason: host reimage 2446875 wikitext text/x-wiki tf3hqry83waqn77jngyzxaykdqi81jl 2446876 2446875 2026-08-13T14:03:05Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet 2446876 wikitext text/x-wiki deppqvg27pxa3w27zctmie06x8ufruj 2446877 2446876 2026-08-13T14:03:13Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1007.eqiad.wmnet 2446877 wikitext text/x-wiki db72k45h027la1mchioqmu4oqgecxzt 2446878 2446877 2026-08-13T14:03:54Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2005.codfw.wmnet 2446878 wikitext text/x-wiki drf5xpjtyiqnpn5up5kp7emwaas4f1g 2446879 2446878 2026-08-13T14:04:00Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus2007.codfw.wmnet 2446879 wikitext text/x-wiki c7fxno92s6zuneaos7ppppawmbubw95 2446880 2446879 2026-08-13T14:04:13Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus1006.eqiad.wmnet 2446880 wikitext text/x-wiki 7hslxcpxgj4oar6lvhxn5x1x3j2tqzc 2446881 2446880 2026-08-13T14:06:09Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet 2446881 wikitext text/x-wiki 91d3ud3ro5293iqf9eleoopnis2zwkb 2446882 2446881 2026-08-13T14:06:53Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2446882 wikitext text/x-wiki fcisprdrdn788wt91oydo2j6vlkurux 2446883 2446882 2026-08-13T14:07:01Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1149.eqiad.wmnet with OS bookworm 2446883 wikitext text/x-wiki csfe5j2afj1ih8wmm6b3pg4xkoav64r 2446884 2446883 2026-08-13T14:08:34Z Stashbot 7414 jayme: updated calico to v3.30.7 on wikikube codfw T427400 2446884 wikitext text/x-wiki q68hqt4082vwwairbkc7npyelmlbyd2 2446885 2446884 2026-08-13T14:08:37Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet 2446885 wikitext text/x-wiki 7krazc2prv8z8uh6y52u97fz1bymxjj 2446886 2446885 2026-08-13T14:08:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1215.eqiad.wmnet with OS bookworm 2446886 wikitext text/x-wiki rh0x9nfitb30m74ipmbg6r1yfym92wy 2446887 2446886 2026-08-13T14:09:01Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on apifeatureusage1001.eqiad.wmnet with reason: host reimage 2446887 wikitext text/x-wiki of4u3b27i6e1jrcpdv8z7oofdlmgftr 2446888 2446887 2026-08-13T14:09:13Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad 2446888 wikitext text/x-wiki 1wqkofio69ye57tmib24np9kl4dh1li 2446889 2446888 2026-08-13T14:09:17Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet 2446889 wikitext text/x-wiki lfnmc4hptcr05ibjidaahjxg8wzqjyl 2446890 2446889 2026-08-13T14:09:19Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet 2446890 wikitext text/x-wiki qdllh9t51fj2xb67mjpre9e7aztw3ay 2446892 2446890 2026-08-13T14:11:36Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet 2446892 wikitext text/x-wiki 65jfjydp0mpgii0yyync95vvrtilkf9 2446893 2446892 2026-08-13T14:11:38Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet 2446893 wikitext text/x-wiki ls22myotb9y7s0fwnqy5a7jb8q8siwv 2446894 2446893 2026-08-13T14:11:41Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2007.codfw.wmnet 2446894 wikitext text/x-wiki 2wb3bvjz28tpxpt1xwkp9qmqm1akg83 2446895 2446894 2026-08-13T14:12:33Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet 2446895 wikitext text/x-wiki bg6yd0o9wqsama39kwfv7yirkdb7leo 2446896 2446895 2026-08-13T14:13:00Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus2006.codfw.wmnet 2446896 wikitext text/x-wiki fptq0f6lnuia8lpfok1g9o6d3et637p 2446897 2446896 2026-08-13T14:14:04Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1006.eqiad.wmnet 2446897 wikitext text/x-wiki scbt5hfu9thlcp4lv27m9drv9x4eu0g 2446898 2446897 2026-08-13T14:15:46Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus1008.eqiad.wmnet 2446898 wikitext text/x-wiki cs7b8ixctfcjcat5t3ede524vwf54d0 2446899 2446898 2026-08-13T14:16:48Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet 2446899 wikitext text/x-wiki gjx847vpr28x5zs0g9a0iztr222e9g8 2446900 2446899 2026-08-13T14:16:50Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet 2446900 wikitext text/x-wiki 7mzovfv7dpebzok5nciyb1elf2lgqe6 2446901 2446900 2026-08-13T14:16:55Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet 2446901 wikitext text/x-wiki c5xmm8mypt23qmk36fig5zuiybd9vky 2446902 2446901 2026-08-13T14:16:58Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet 2446902 wikitext text/x-wiki 1nd3lezdpiefzn9lrlm1uyrwk2jfeo5 2446903 2446902 2026-08-13T14:17:09Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=yes; selector: name=cp5022.* 2446903 wikitext text/x-wiki 3nwh59jevm03x9mpi7cadf8zmj6jd2l 2446904 2446903 2026-08-13T14:18:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet 2446904 wikitext text/x-wiki 6a805kmk50ssajdgqlimu8b4i1yu347 2446905 2446904 2026-08-13T14:19:27Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet 2446905 wikitext text/x-wiki eznskud7t7c60i31ocen8qdnrrwk8mx 2446906 2446905 2026-08-13T14:22:55Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2006.codfw.wmnet 2446906 wikitext text/x-wiki ahkxr75dqz5y1zn7ov4axsh0td7h92g 2446907 2446906 2026-08-13T14:23:15Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet 2446907 wikitext text/x-wiki ayhsqyxe1pyl3apnis4stkw7g9e5yen 2446908 2446907 2026-08-13T14:23:48Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1008.eqiad.wmnet 2446908 wikitext text/x-wiki ex9snb9g87ltzlxnw7bschbjydzxm4l 2446909 2446908 2026-08-13T14:23:57Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus2008.codfw.wmnet 2446909 wikitext text/x-wiki nzwpmkajld26z4ikg58xrwcgel7j3x5 2446910 2446909 2026-08-13T14:24:10Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet 2446910 wikitext text/x-wiki 5c6c86e6a7s0cd3odlc1sx6837gjeok 2446911 2446910 2026-08-13T14:24:12Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet 2446911 wikitext text/x-wiki 3i0138kx94jmdnumflp38stvqyuf1tz 2446912 2446911 2026-08-13T14:24:18Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet 2446912 wikitext text/x-wiki 8c85ywl47avkewhbaedpjm5ci5z9k0u 2446913 2446912 2026-08-13T14:24:21Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet 2446913 wikitext text/x-wiki etdf5fi83tr2pw02q4p0zbifnqtqzd7 2446914 2446913 2026-08-13T14:25:06Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1216.eqiad.wmnet with OS bookworm 2446914 wikitext text/x-wiki 0piz7quephf7ietdo3cvb4g1qvcep9h 2446915 2446914 2026-08-13T14:25:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1217.eqiad.wmnet with OS bookworm 2446915 wikitext text/x-wiki j3qnn85xkkm7naxwmapyq1yeryq8i9r 2446918 2446915 2026-08-13T14:26:13Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough 2446918 wikitext text/x-wiki 43pnumgr2oae4llf4obqy94l7ne0wm8 2446919 2446918 2026-08-13T14:26:23Z Stashbot 7414 moritzm: installing Django security updates 2446919 wikitext text/x-wiki aksdfdkh7x4ss8cm7xr26lvqomf07kt 2446920 2446919 2026-08-13T14:27:13Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet 2446920 wikitext text/x-wiki ez9cp9g5zsvbxecmmtxbb36g344f686 2446921 2446920 2026-08-13T14:29:40Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet 2446921 wikitext text/x-wiki 75shlxdwmu21afwkiggfkmqfgwojl11 2446922 2446921 2026-08-13T14:31:12Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet 2446922 wikitext text/x-wiki jgm66641nd9cmkwg9gtu8zdwi71b4fn 2446923 2446922 2026-08-13T14:31:57Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2008.codfw.wmnet 2446923 wikitext text/x-wiki 1ycpg92arpz7cfr73699wrfk81femir 2446924 2446923 2026-08-13T14:32:04Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet 2446924 wikitext text/x-wiki o23g1pkjwk8e64qydfbl5b1tmptj4ii 2446925 2446924 2026-08-13T14:32:07Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet 2446925 wikitext text/x-wiki 9f5149pdlrab5wdaj6346d7o43cf286 2446926 2446925 2026-08-13T14:32:11Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet 2446926 wikitext text/x-wiki taibc2ep20ebe2dth21sz8q9ei59lxc 2446927 2446926 2026-08-13T14:32:13Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet 2446927 wikitext text/x-wiki bc2rvrrdro1pkuq1o8ire0mt9owhry0 2446928 2446927 2026-08-13T14:39:31Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1216.eqiad.wmnet with reason: host reimage 2446928 wikitext text/x-wiki h5fat9eacubqpgags8qhmolg7qa6ph3 2446929 2446928 2026-08-13T14:39:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1169: Pool db1169.eqiad.wmnet in after cloning 2446929 wikitext text/x-wiki 0nfila7qoumaxos3cubvt1f1btvpolk 2446930 2446929 2026-08-13T14:40:02Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1217.eqiad.wmnet with reason: host reimage 2446930 wikitext text/x-wiki kjhwyx2hdsg7qstsry3l1ejssghgafd 2446931 2446930 2026-08-13T14:44:18Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1216.eqiad.wmnet with reason: host reimage 2446931 wikitext text/x-wiki 3dxah8j7tp4ov8yp5dptcjprfs8f3wc 2446932 2446931 2026-08-13T14:44:42Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet 2446932 wikitext text/x-wiki 7kqkx9ee67jx5ve87lqifi6lveh9ytj 2446933 2446932 2026-08-13T14:44:45Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet 2446933 wikitext text/x-wiki e04uva01ireqxi6w3liahdl2rpiuht8 2446934 2446933 2026-08-13T14:44:47Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad 2446934 wikitext text/x-wiki ntfrf2829bg8f0chh5945h8oauvr21c 2446935 2446934 2026-08-13T14:48:01Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2004.codfw.wmnet 2446935 wikitext text/x-wiki 4l8qfx8xvm48921qupup7n8e9hpvrby 2446936 2446935 2026-08-13T14:48:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2446936 wikitext text/x-wiki 7xrh6asairhc0p1bet9re4ubfpg614t 2446937 2446936 2026-08-13T14:48:09Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1217.eqiad.wmnet with reason: host reimage 2446937 wikitext text/x-wiki m9ug94yklp54vde1n01c2c4s40iqf1v 2446938 2446937 2026-08-13T14:50:15Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1325455|Api: Use correct query when continue prop=categories (T433922)]] 2446938 wikitext text/x-wiki 4ux896wq50zg65yczh0jvjljxw492rd 2446939 2446938 2026-08-13T14:52:21Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1325455|Api: Use correct query when continue prop=categories (T433922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446939 wikitext text/x-wiki n2fghccoqim8vvgtea4z8lmcpebhm1o 2446940 2446939 2026-08-13T14:52:27Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet 2446940 wikitext text/x-wiki qjzpxaqhozrnbl24mvd2rdmq1ibobpl 2446941 2446940 2026-08-13T14:52:29Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet 2446941 wikitext text/x-wiki g712yspvde0s7ty5ogafuv7eqtp2ima 2446942 2446941 2026-08-13T14:52:36Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw 2446942 wikitext text/x-wiki gok223cuq2fl1j0n98wwdimqu3d28q2 2446943 2446942 2026-08-13T14:52:40Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet 2446943 wikitext text/x-wiki 7n9j4wednxr00dyg6vno2bhfctakwzc 2446944 2446943 2026-08-13T14:52:42Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet 2446944 wikitext text/x-wiki ptujp54vxtcb9koig83i7qg5mg9zuz0 2446945 2446944 2026-08-13T14:53:24Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" 2446945 wikitext text/x-wiki qkiyeik5tw5wvfhu5mt6lj1jelc0fin 2446946 2446945 2026-08-13T14:53:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" 2446946 wikitext text/x-wiki lmfrdxuomoogj3zx38xyx9y24y1znlu 2446947 2446946 2026-08-13T14:53:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446947 wikitext text/x-wiki mdm7gl83z659b2pnxptpegaxu8x1i0v 2446948 2446947 2026-08-13T14:53:32Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2004.codfw.wmnet on all recursors 2446948 wikitext text/x-wiki jdb0pzmbib38m3xw738ydzuj13e78p0 2446949 2446948 2026-08-13T14:53:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2004.codfw.wmnet on all recursors 2446949 wikitext text/x-wiki pj2cg9mxoj9ovpmj0pxgc67zxgzvovm 2446950 2446949 2026-08-13T14:53:49Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2446950 wikitext text/x-wiki f6hand3p7dizx935dxzc9t1mp3uq16f 2446951 2446950 2026-08-13T14:55:26Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with deployment 2446951 wikitext text/x-wiki bog6kml04fpwj5ow2zeawcdch33yrah 2446953 2446951 2026-08-13T14:58:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" 2446953 wikitext text/x-wiki a0e5nfxcxej3h5ol0tttqo9maa60drt 2446954 2446953 2026-08-13T14:58:52Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" 2446954 wikitext text/x-wiki 03jd6sfufsuu31ff48qc2rxao3fv8m5 2446955 2446954 2026-08-13T14:58:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446955 wikitext text/x-wiki t3sxjmjyyu7bvf6jpyizg1dcjjanyay 2446956 2446955 2026-08-13T14:58:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2004.codfw.wmnet on all recursors 2446956 wikitext text/x-wiki dh0x8ued8su3kzd13oy7rvwbzw2zobt 2446957 2446956 2026-08-13T14:58:58Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2004.codfw.wmnet on all recursors 2446957 wikitext text/x-wiki 8ka5sh8drhcrpj32jjrdgjxvlsk9zst 2446958 2446957 2026-08-13T14:59:08Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2004.codfw.wmnet 2446958 wikitext text/x-wiki kgqbhpjy9nem1m8hpbcin8os8ssy0x3 2446959 2446958 2026-08-13T15:00:01Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325455|Api: Use correct query when continue prop=categories (T433922)]] (duration: 09m 47s) 2446959 wikitext text/x-wiki ak0ul7xcx9kgg8o7grt3knnxxydoygb 2446960 2446959 2026-08-13T15:01:25Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet 2446960 wikitext text/x-wiki e9t04zlqgn28p56zt2ncy3oy4kk7fh6 2446961 2446960 2026-08-13T15:01:31Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet 2446961 wikitext text/x-wiki gw5meye5n78pj0p3gz4usmksnkobjfc 2446962 2446961 2026-08-13T15:02:09Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet 2446962 wikitext text/x-wiki k8pomcy3bxwrin02l4zwaojixk48h5y 2446963 2446962 2026-08-13T15:02:12Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet 2446963 wikitext text/x-wiki mknxkhoby7jy6jcg6u7kk6wkbfbvrnd 2446964 2446963 2026-08-13T15:02:17Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet 2446964 wikitext text/x-wiki b59jyp8khbl8897wo4292kqqkt70y5b 2446965 2446964 2026-08-13T15:02:20Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet 2446965 wikitext text/x-wiki avwhg742jej66tm4vnhv1yolm23crao 2446966 2446965 2026-08-13T15:04:12Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1216.eqiad.wmnet with OS bookworm 2446966 wikitext text/x-wiki 2nfbadkpx8y7t7f0hgf6y7gpjr5oc6v 2446967 2446966 2026-08-13T15:10:01Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: quash java safepoint logspam - bking@cumin2003 - T434685 2446967 wikitext text/x-wiki 3almqx0szwc0b8ktloe4hn818ggqojy 2446968 2446967 2026-08-13T15:10:12Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet 2446968 wikitext text/x-wiki p4w462gkj113d7cpzfjqfhf6jzkymdg 2446969 2446968 2026-08-13T15:10:15Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet 2446969 wikitext text/x-wiki oyl5l91320hh0x644exy2xbx8lau9j1 2446970 2446969 2026-08-13T15:10:18Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet 2446970 wikitext text/x-wiki c3wnnd772b6azhscqm4xrbjbyengoln 2446971 2446970 2026-08-13T15:10:20Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet 2446971 wikitext text/x-wiki qvasd9uv1j55guo8gmktw7oicc1rknj 2446972 2446971 2026-08-13T15:10:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1217.eqiad.wmnet with OS bookworm 2446972 wikitext text/x-wiki 1ckb9j3o517qs5riroix7wll1o7lndd 2446973 2446972 2026-08-13T15:13:40Z Stashbot 7414 cdobbins@dns1004: START - running authdns-update 2446973 wikitext text/x-wiki em1bgq0kpjn95iamhv1rb5cynmsfskk 2446974 2446973 2026-08-13T15:15:42Z Stashbot 7414 cdobbins@dns1004: END - running authdns-update 2446974 wikitext text/x-wiki ok6q72vx4zefoy3mi7ujlko4v1g31sv 2446975 2446974 2026-08-13T15:17:12Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet 2446975 wikitext text/x-wiki 7tzz2dccndouachpmgalpj5fbsmgx7q 2446976 2446975 2026-08-13T15:17:14Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet 2446976 wikitext text/x-wiki fk2ptrumhyb1dxq650knf29fxquatgk 2446977 2446976 2026-08-13T15:17:19Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet 2446977 wikitext text/x-wiki 28tr9hje9b118w3xcehuw5nf65ecydx 2446978 2446977 2026-08-13T15:17:21Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet 2446978 wikitext text/x-wiki g86a8ugv2xie82nyozpybwuf4udu1hr 2446979 2446978 2026-08-13T15:18:23Z Stashbot 7414 inflatador: bking@ganeti2048 sudo gnt-node failover -f ganeti2046.codfw.wmnet T434681 2446979 wikitext text/x-wiki but9isnnx6t4l1n85g25dqnn1ga26jc 2446980 2446979 2026-08-13T15:23:43Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet 2446980 wikitext text/x-wiki cfe0okq22hkou82sib0iy4tbf4rbe69 2446981 2446980 2026-08-13T15:23:46Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet 2446981 wikitext text/x-wiki omo04d0c64go1924dlbus2ylx0dwznf 2446982 2446981 2026-08-13T15:23:50Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet 2446982 wikitext text/x-wiki kwoirm4q6iamsp5521ketq1s4cg45wh 2446983 2446982 2026-08-13T15:23:52Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet 2446983 wikitext text/x-wiki bkxy2fs948bglhf1g1c2uh1iq4nk3qc 2446984 2446983 2026-08-13T15:25:13Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1169: Pool db1169.eqiad.wmnet in after cloning 2446984 wikitext text/x-wiki i1m9fthtn2t45z1nvvv9icz2hujb4mg 2446985 2446984 2026-08-13T15:25:22Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1169.eqiad.wmnet onto db1277.eqiad.wmnet 2446985 wikitext text/x-wiki mlyn5ep7q08i3y6ncii052feicxkrv9 2446986 2446985 2026-08-13T15:30:12Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet 2446986 wikitext text/x-wiki d2gn14dvy7n1r2awp7u4c4jpc2agebm 2446987 2446986 2026-08-13T15:30:14Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet 2446987 wikitext text/x-wiki f9pyw6os497zc3o7nxi2zak8z6k54l0 2446988 2446987 2026-08-13T15:30:16Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw 2446988 wikitext text/x-wiki kbek9udov60rq5ulqxjaayggkoqunmi 2446989 2446988 2026-08-13T15:32:07Z Stashbot 7414 cgoubert@deploy1003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot 2446989 wikitext text/x-wiki 721gmvvvw6wyoj80v1rk639ik10n39i 2446990 2446989 2026-08-13T15:32:49Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet 2446990 wikitext text/x-wiki biz27but6ybqgmvv4sy67z42xfhcwmg 2446991 2446990 2026-08-13T15:35:34Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2446991 wikitext text/x-wiki s8cx2i598jesoqrc6bpkb0vlzv6568k 2446992 2446991 2026-08-13T15:36:38Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet 2446992 wikitext text/x-wiki 06vxlojeejp4iv6x4scfddb9vdmixht 2446993 2446992 2026-08-13T15:37:31Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet 2446993 wikitext text/x-wiki bdo1p40a3lsdxlq4e7nr4chld957qq4 2446994 2446993 2026-08-13T15:37:45Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s7 T434802 2446994 wikitext text/x-wiki d7vn8wf5mudvi39ccuoijyhjal4idzs 2446995 2446994 2026-08-13T15:37:49Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2446995 wikitext text/x-wiki tgxek5iy4n8zexubol5d6smi8xomga9 2446996 2446995 2026-08-13T15:38:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2159 with weight 0 T434802', diff saved to https://phabricator.wikimedia.org/P96077 and previous config saved to /var/cache/conftool/dbconfig/20260813-153806-cwilliams.json 2446996 wikitext text/x-wiki 15yvnv9om2o40mtsi54zd8vyv7a4gug 2446998 2446996 2026-08-13T15:39:15Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2446998 wikitext text/x-wiki 23vzmxodi82wwnwqoxye23p9shwh5jh 2446999 2446998 2026-08-13T15:39:39Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2446999 wikitext text/x-wiki krgm1yjfx7vb8jgm5ivaijm2c13lc34 2447000 2446999 2026-08-13T15:39:52Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2447000 wikitext text/x-wiki a0bm9ckzn1uhw8w5kq6d2wv90yrmvmh 2447001 2447000 2026-08-13T15:39:56Z Stashbot 7414 inflatador: bking@ganeti2048] ~$ sudo gnt-node failover -f --ignore-consistency ganeti2046.codfw.wmnet T434681 2447001 wikitext text/x-wiki on5larzthfu3t9w2zgbc4w622oulatx 2447002 2447001 2026-08-13T15:41:31Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet 2447002 wikitext text/x-wiki onnmy8dxzxe87q6t6adh9v2obukco6q 2447003 2447002 2026-08-13T15:41:50Z Stashbot 7414 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 42s) 2447003 wikitext text/x-wiki ndqh87tu3u6evzljpljyf8u8nxfmtwt 2447004 2447003 2026-08-13T15:43:25Z Stashbot 7414 cezmunsta: Starting s7 codfw failover from db2220 to db2159 - T434802 2447004 wikitext text/x-wiki hzwri4g1fv3qdzr7gr9dye43osdjjgp 2447005 2447004 2026-08-13T15:44:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2159 to s7 primary T434802', diff saved to https://phabricator.wikimedia.org/P96078 and previous config saved to /var/cache/conftool/dbconfig/20260813-154405-cwilliams.json 2447005 wikitext text/x-wiki 9ph8ar20kpzumlbfz9w4groog93u34j 2447006 2447005 2026-08-13T15:44:40Z Stashbot 7414 cjd91: depooling dns3004 to reimage to trixie 2447006 wikitext text/x-wiki rceug7n2yl30j6rqif9vs9j2wogeu18 2447008 2447006 2026-08-13T15:45:23Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=no; selector: name=dns3004.* 2447008 wikitext text/x-wiki gywwuuo07kejm4e44athputn2mux25u 2447009 2447008 2026-08-13T15:46:28Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2220 T434802', diff saved to https://phabricator.wikimedia.org/P96079 and previous config saved to /var/cache/conftool/dbconfig/20260813-154624-cwilliams.json 2447009 wikitext text/x-wiki occdkrd73a9ss3w4jlln89r698zlmkb 2447010 2447009 2026-08-13T15:46:44Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns3004.wikimedia.org with OS trixie 2447010 wikitext text/x-wiki 6oi8x9n622g2alpzwrg3soc53qoyjr5 2447012 2447010 2026-08-13T15:48:45Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.upgrade for 1 hosts 2447012 wikitext text/x-wiki 8swjnlc6vb3d96syn93reektljierqj 2447013 2447012 2026-08-13T15:48:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2220: Upgrading db2220.codfw.wmnet 2447013 wikitext text/x-wiki 6bx10ooh6sjhghbd5s60p1ue9z293xg 2447014 2447013 2026-08-13T15:49:00Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2220: Upgrading db2220.codfw.wmnet 2447014 wikitext text/x-wiki io8b8c1oi92ubzrdpub1pn8m1i9igag 2447016 2447014 2026-08-13T15:56:36Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Upgrade of db2220.codfw.wmnet completed 2447016 wikitext text/x-wiki dsn91dd4kjyrxidr8plalx4oy4yy3eq 2447017 2447016 2026-08-13T15:59:59Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2447017 wikitext text/x-wiki 07uk86bivksvmrmigbvni03veoc7ldl 2447018 2447017 2026-08-13T16:00:18Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2447018 wikitext text/x-wiki jhxwmu8ant5p4hsiumj3bnceouy0gck 2447023 2447018 2026-08-13T16:04:40Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2447023 wikitext text/x-wiki 7ft7241lkj6tcucz6j9rhccyphqdpay 2447024 2447023 2026-08-13T16:05:01Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2447024 wikitext text/x-wiki 00xuzk4cgiiva479mmql82zutloanrf 2447027 2447024 2026-08-13T16:07:46Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns3004.wikimedia.org with reason: host reimage 2447027 wikitext text/x-wiki b82ao8oxw4jogp59dyssm9hnbhrr4mc 2447028 2447027 2026-08-13T16:07:54Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2447028 wikitext text/x-wiki q3egn8pcz0c3c7ju6ay04bkvssmwfxd 2447029 2447028 2026-08-13T16:08:07Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2447029 wikitext text/x-wiki m4r3p7ztldkrrx6olfun9j11s96h143 2447033 2447029 2026-08-13T16:14:13Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns3004.wikimedia.org with reason: host reimage 2447033 wikitext text/x-wiki 8ifsuyivhoaw9bap5de8jnopb6pgh6z 2447041 2447033 2026-08-13T16:20:16Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: quash java safepoint logspam - bking@cumin2003 - T434685 2447041 wikitext text/x-wiki m4nd8zmk8rrn4lfuv49hk955pnif3vq 2447044 2447041 2026-08-13T16:22:11Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.280.0" for 3 host(s) 2447044 wikitext text/x-wiki 0g5qxn34vk84jt6ei2bvsn5ab44y8oc 2447049 2447044 2026-08-13T16:24:06Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.280.0" completed for 3 hosts 2447049 wikitext text/x-wiki cjhxtiklws5f4ayshwfz9fubwrgw5mc 2447050 2447049 2026-08-13T16:31:41Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1150.eqiad.wmnet with OS bookworm 2447050 wikitext text/x-wiki arn1u26wdbvkf3blwaxihtn9c1lf0w4 2447051 2447050 2026-08-13T16:32:02Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1150 2447051 wikitext text/x-wiki nkn6rg5klo1fjfdjc9y1tylrovwwudy 2447052 2447051 2026-08-13T16:32:05Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1150 2447052 wikitext text/x-wiki qmrwdmtyj0htay54lnyk7odw8jkdlag 2447054 2447052 2026-08-13T16:33:48Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bookworm 2447054 wikitext text/x-wiki 9k8s7b8xq3oyh9nxrotyindokairyc4 2447055 2447054 2026-08-13T16:34:10Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1179 2447055 wikitext text/x-wiki 189x06c1j8y38g7b5kb9rs7srrti6s8 2447056 2447055 2026-08-13T16:34:12Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1179 2447056 wikitext text/x-wiki csffyj1ywvwzmsh75vu9u96bu0rd55f 2447057 2447056 2026-08-13T16:34:31Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1218.eqiad.wmnet with OS bookworm 2447057 wikitext text/x-wiki er7tkvbz9hzbf0z0aq57i7o8h2i2n78 2447058 2447057 2026-08-13T16:34:52Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1218 2447058 wikitext text/x-wiki ib7h0eg4gs80x9zt6dom7m9cvmjorly 2447059 2447058 2026-08-13T16:34:55Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1218 2447059 wikitext text/x-wiki 6tqscma7l3fxxlzbec0rnbx2jkx9brx 2447061 2447059 2026-08-13T16:38:08Z Stashbot 7414 swfrench-wmf: kubectl delete node kubestagemaster2005.codfw.wmnet - T434681 2447061 wikitext text/x-wiki 0s40nxvx5tucwt0sq7zz497dlfii480 2447062 2447061 2026-08-13T16:38:14Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2447062 wikitext text/x-wiki 1gnf4tjkcuveqn1ahux8rt8xc91so9r 2447064 2447062 2026-08-13T16:41:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Upgrade of db2220.codfw.wmnet completed 2447064 wikitext text/x-wiki 9imydb3axogyonjzzdyj3hnjgotnjp1 2447065 2447064 2026-08-13T16:41:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for 1 hosts 2447065 wikitext text/x-wiki by8zsn790xjghdfucw3nyyzkosos1bn 2447068 2447065 2026-08-13T16:47:44Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1150.eqiad.wmnet with reason: host reimage 2447068 wikitext text/x-wiki klcm7ppux39clg9g7x1a6x4qpkjxqlz 2447069 2447068 2026-08-13T16:49:00Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1179.eqiad.wmnet with reason: host reimage 2447069 wikitext text/x-wiki ckca3zg2f5t7lyrdxd6yrm1kwij08g2 2447070 2447069 2026-08-13T16:49:05Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1218.eqiad.wmnet with reason: host reimage 2447070 wikitext text/x-wiki 7v8pytzstp035e44ld2f0qgnotfeak4 2447071 2447070 2026-08-13T16:50:39Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.clone of db1166.eqiad.wmnet onto db1280.eqiad.wmnet 2447071 wikitext text/x-wiki 38msu8jnh4c1tsca10ihxlwi4zkrn2j 2447072 2447071 2026-08-13T16:50:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1166: Depool db1166.eqiad.wmnet to then clone it to db1280.eqiad.wmnet - cwilliams@cumin1003 2447072 wikitext text/x-wiki 5s3qfs4v0hlci3g19mq8ykc4mhe6uk9 2447073 2447072 2026-08-13T16:51:58Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1166: Depool db1166.eqiad.wmnet to then clone it to db1280.eqiad.wmnet - cwilliams@cumin1003 2447073 wikitext text/x-wiki 8jndk0s4kvgqbybzf3xq9brj3lmwhjm 2447074 2447073 2026-08-13T16:53:59Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1324810|deployment-info.php: Report dbname and branch for the requested wiki (T434726)]] 2447074 wikitext text/x-wiki rmu5jp3sdkk8lxi73tn3ek91t8wqulc 2447075 2447074 2026-08-13T16:54:28Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1150.eqiad.wmnet with reason: host reimage 2447075 wikitext text/x-wiki ep6k0kcys9zyertwsia3att8bjdo6wn 2447077 2447075 2026-08-13T16:56:13Z Stashbot 7414 dancy@deploy1003: dancy: Backport for [[gerrit:1324810|deployment-info.php: Report dbname and branch for the requested wiki (T434726)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447077 wikitext text/x-wiki nbrqqdpzs65mrvlx55gfel7lanomoqj 2447078 2447077 2026-08-13T16:56:41Z Stashbot 7414 dancy@deploy1003: dancy: Continuing with deployment 2447078 wikitext text/x-wiki rb7j0gm7kt0bes8s34m9bdncbixgl1l 2447079 2447078 2026-08-13T16:57:37Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1179.eqiad.wmnet with reason: host reimage 2447079 wikitext text/x-wiki t1ys6o8p4ztm1ez107w6b5ui2mnin6l 2447080 2447079 2026-08-13T16:58:03Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2447080 wikitext text/x-wiki f15qzyvfhrjt2eicy6dqwu3foeozcys 2447081 2447080 2026-08-13T17:00:50Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324810|deployment-info.php: Report dbname and branch for the requested wiki (T434726)]] (duration: 06m 52s) 2447081 wikitext text/x-wiki b2mchmuf2ipz6uar8982iomb11gsq4i 2447082 2447081 2026-08-13T17:01:11Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447082 wikitext text/x-wiki nw7zk4vh3burrbpv3go0jhknr2m7604 2447083 2447082 2026-08-13T17:01:35Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447083 wikitext text/x-wiki mmhd7qvkktc93m9a7u521c4k70ofx0w 2447084 2447083 2026-08-13T17:01:43Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1218.eqiad.wmnet with reason: host reimage 2447084 wikitext text/x-wiki 1pgsoq6xjsan4s54k7a1em9owtk25kr 2447085 2447084 2026-08-13T17:01:58Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447085 wikitext text/x-wiki gp5455xeml4l0x9t30rax62n82581fo 2447086 2447085 2026-08-13T17:02:19Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447086 wikitext text/x-wiki m8nsh52cxv7vto94c7lczp8w071bkwo 2447088 2447086 2026-08-13T17:09:13Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment for mediawiki chart bump - T427666 2447088 wikitext text/x-wiki q29zynhdatvrsan8qv0t7h1cv8ntarw 2447089 2447088 2026-08-13T17:12:16Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment for mediawiki chart bump - T427666 (duration: 03m 03s) 2447089 wikitext text/x-wiki 7w7m4zc7j0qlrfs74jfv0bdfaeg31bz 2447090 2447089 2026-08-13T17:12:30Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447090 wikitext text/x-wiki 06ekpb9m5jhk0dkrboiw0vobb4fm49m 2447091 2447090 2026-08-13T17:13:56Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2447091 wikitext text/x-wiki jvfpi8c050v4jlowzj0ammhkem9dlz5 2447092 2447091 2026-08-13T17:14:15Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2447092 wikitext text/x-wiki 42fqlbl629kbhh11irzcjazj1lmme1t 2447093 2447092 2026-08-13T17:15:56Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447093 wikitext text/x-wiki 7jgo3zwz17b7h6jjrdldl4lm3g3nhoy 2447094 2447093 2026-08-13T17:18:16Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns3004.wikimedia.org with OS trixie 2447094 wikitext text/x-wiki gwgpicg7sdctxqd4hvt9m8yfpuafwnl 2447095 2447094 2026-08-13T17:18:35Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1150.eqiad.wmnet with OS bookworm 2447095 wikitext text/x-wiki 3gyrrb19ww46pb8e75uu6wmi7b92qmn 2447096 2447095 2026-08-13T17:19:22Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1179.eqiad.wmnet with OS bookworm 2447096 wikitext text/x-wiki rur6zzadvprsk2b789lj7jxn93nrtxq 2447097 2447096 2026-08-13T17:20:30Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2447097 wikitext text/x-wiki 5p8f56gsefq9ia42zejnognztrm0uyp 2447098 2447097 2026-08-13T17:20:45Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2447098 wikitext text/x-wiki qhgq62ypb35o77ngun4dp1dqn8tkmsh 2447099 2447098 2026-08-13T17:21:38Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1218.eqiad.wmnet with OS bookworm 2447099 wikitext text/x-wiki lel2hkllxrk9xb05q7fcflfvh2u1x5v 2447100 2447099 2026-08-13T17:23:55Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2447100 wikitext text/x-wiki h8ckoxojutuar4b3nbnb9o3k4bg70m2 2447101 2447100 2026-08-13T17:24:07Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2447101 wikitext text/x-wiki lmhiaq0ja6k24szpua6vizz6i1i27pd 2447102 2447101 2026-08-13T17:26:29Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2447102 wikitext text/x-wiki g3sia6tw4uetfaow459r9q4w9dick0d 2447103 2447102 2026-08-13T17:26:40Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2447103 wikitext text/x-wiki bqq4zr878dmzmvjz338f3nts5se9on0 2447104 2447103 2026-08-13T17:28:01Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2447104 wikitext text/x-wiki rcxe4ibhnxsxk902m6yriaaeuh9ne3k 2447105 2447104 2026-08-13T17:32:26Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2447105 wikitext text/x-wiki pmivxekn8504bi2i1f1dci138h9mmsw 2447106 2447105 2026-08-13T17:32:37Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2447106 wikitext text/x-wiki md1kzb6ce5ncns8pzcqrkyueyq8504g 2447107 2447106 2026-08-13T17:36:28Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447107 wikitext text/x-wiki 6elko2h3mevmc02r536u360mt6ghf12 2447111 2447107 2026-08-13T17:38:58Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447111 wikitext text/x-wiki fbzrpuom8sboj8jhsua11vtiectv3v1 2447112 2447111 2026-08-13T17:39:14Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447112 wikitext text/x-wiki sfe78qz1q085ktwud7rmv3ml3pxvhm8 2447113 2447112 2026-08-13T17:40:35Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447113 wikitext text/x-wiki 8dz0jnt4asorq0le1mu2sv8dp4xjuf5 2447114 2447113 2026-08-13T17:40:45Z Stashbot 7414 inflatador: bking@ganeti2048] `sudo gnt-instance remove --force --ignore-failures --shutdown-timeout=0` on non-DRBD VMs T434681 2447114 wikitext text/x-wiki h9hu9qqmfcpcyh3xtzbzqlm6o90trpf 2447115 2447114 2026-08-13T17:40:56Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447115 wikitext text/x-wiki 02cny58yw2tx7abwho93y44gqueof8e 2447116 2447115 2026-08-13T17:41:33Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2001.codfw.wmnet 2447116 wikitext text/x-wiki s24erasrq2xde17w2da18zn50muph4y 2447117 2447116 2026-08-13T17:41:36Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2001.codfw.wmnet 2447117 wikitext text/x-wiki 7x7vlu8icej063bfsztpetijsrsti4a 2447118 2447117 2026-08-13T17:43:51Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003*} and A:liberica 2447118 wikitext text/x-wiki mr62y1txqf5mj793v2epzov405zvnc1 2447119 2447118 2026-08-13T17:45:04Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1220.eqiad.wmnet with OS bookworm 2447119 wikitext text/x-wiki 3ssvgbfygoy6l1vhewwmhw2ruowb1wr 2447120 2447119 2026-08-13T17:45:33Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1220 2447120 wikitext text/x-wiki 5dadxr3utpxj3qibzahydc5opxpsl3z 2447121 2447120 2026-08-13T17:45:39Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1221.eqiad.wmnet with OS bookworm 2447121 wikitext text/x-wiki gs0simc4z47an8gqq2zmtfdq9p2kg2z 2447122 2447121 2026-08-13T17:46:05Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447122 wikitext text/x-wiki luhon3yh0ib5q3e1u3muw34qlah1drx 2447123 2447122 2026-08-13T17:46:20Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1234.eqiad.wmnet with OS bookworm 2447123 wikitext text/x-wiki co9ae9snpgi8g8wj2s2zhpmlhoqnxaq 2447124 2447123 2026-08-13T17:47:26Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003*} and A:liberica 2447124 wikitext text/x-wiki iton1f75q6jnp0m4y7cptl9o1dwry41 2447125 2447124 2026-08-13T17:47:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447125 wikitext text/x-wiki 0acr22ireotm7jp6tr5zvnl1pqbacy9 2447126 2447125 2026-08-13T17:50:36Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1220 - ryankemper@cumin2003" 2447126 wikitext text/x-wiki hv16kimshoo2np8zdzik3uhioj2zfob 2447127 2447126 2026-08-13T17:51:00Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1220 - ryankemper@cumin2003" 2447127 wikitext text/x-wiki amof2zsz5ju253p8jip9lwrxp9t8g40 2447128 2447127 2026-08-13T17:51:03Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447128 wikitext text/x-wiki sjmszqwtren7c45rfo45k8mlwfdk2s4 2447129 2447128 2026-08-13T17:51:05Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1220.eqiad.wmnet 11.36.64.10.in-addr.arpa 1.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447129 wikitext text/x-wiki 6jnygvpj0vwrqx8erqp4nv2y2mohyuk 2447130 2447129 2026-08-13T17:51:07Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1220.eqiad.wmnet 11.36.64.10.in-addr.arpa 1.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447130 wikitext text/x-wiki h07rdwtouoer5whm21c3ioerk5q4kxj 2447131 2447130 2026-08-13T17:51:10Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1220 2447131 wikitext text/x-wiki ge0elm44xob0u2mjhu83x666hc6d97d 2447132 2447131 2026-08-13T17:51:14Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2447132 wikitext text/x-wiki rcaf4mkj8ovmbf1zxawlgcjijoo0ob1 2447133 2447132 2026-08-13T17:51:26Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2447133 wikitext text/x-wiki g77ojiymakiwxxrwy6ln0jlh5ok0n8o 2447134 2447133 2026-08-13T17:51:43Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1220 2447134 wikitext text/x-wiki 3nmfu8i2lvpk9x9xm9m0dptgmqbjpzb 2447135 2447134 2026-08-13T17:51:45Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1220 2447135 wikitext text/x-wiki 4gde6oqjx9puf6fekye4axj7wrfwgzc 2447136 2447135 2026-08-13T17:51:58Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1221 2447136 wikitext text/x-wiki qqrv2nkqja54xnhm6k1ii3s2dlsntc0 2447137 2447136 2026-08-13T17:52:16Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447137 wikitext text/x-wiki dlcvi0lpx5hw9tsyo3hevqam1x4xzvt 2447138 2447137 2026-08-13T17:52:40Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2447138 wikitext text/x-wiki 8qjbtdmobqer8wczhb53oruw5j6tls0 2447139 2447138 2026-08-13T17:53:19Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2447139 wikitext text/x-wiki jqin7o87k1ukxganj0rd1xislb81u98 2447140 2447139 2026-08-13T17:56:31Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1221 - ryankemper@cumin2003" 2447140 wikitext text/x-wiki 6ct8hc70lykzdrizn199qe652v34t1e 2447141 2447140 2026-08-13T17:57:12Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2447141 wikitext text/x-wiki o6x601x90mta5ivuxqez9jpt5muakxv 2447142 2447141 2026-08-13T17:57:17Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2447142 wikitext text/x-wiki r8ow9jaj8bey3l74vdszkaext63j54g 2447143 2447142 2026-08-13T17:58:15Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2447143 wikitext text/x-wiki 2d7t4hchyxbizdvnyi5rxy4zq2q7qg4 2447144 2447143 2026-08-13T17:58:56Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2447144 wikitext text/x-wiki iffv8tujdwzt9jbu537tvofvuuok8yz 2447145 2447144 2026-08-13T17:59:06Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2447145 wikitext text/x-wiki dsuz5vkut31m6x4uay0yoybqah239n4 2447146 2447145 2026-08-13T18:00:21Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1221 - ryankemper@cumin2003" 2447146 wikitext text/x-wiki 0jrt1llxrpv56kk6itv99khukznjpnq 2447147 2447146 2026-08-13T18:00:24Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447147 wikitext text/x-wiki 3pkr41yftfb4voh2irssfykaogg7mqp 2447148 2447147 2026-08-13T18:00:26Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1221.eqiad.wmnet 18.36.64.10.in-addr.arpa 8.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447148 wikitext text/x-wiki lltjc3k4r2pougtz7168yb7pgwbjfai 2447149 2447148 2026-08-13T18:00:29Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1221.eqiad.wmnet 18.36.64.10.in-addr.arpa 8.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447149 wikitext text/x-wiki hwqq8ikry9oxutx6u2f2gzs18ut8fsb 2447150 2447149 2026-08-13T18:00:32Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1221 2447150 wikitext text/x-wiki rvfaa1gndh3nwe74c25lwadaru3mwko 2447151 2447150 2026-08-13T18:01:29Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1221 2447151 wikitext text/x-wiki 7rjpo4rru6ar7p72yxk18d2hin2uue7 2447152 2447151 2026-08-13T18:01:31Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1221 2447152 wikitext text/x-wiki a32w3kbn1kv0fswkk5bckeionpq0kf5 2447153 2447152 2026-08-13T18:01:41Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.280.1" for 3 host(s) 2447153 wikitext text/x-wiki pm8a2gfy4fbpql67o2j2o972ruwmx9m 2447154 2447153 2026-08-13T18:01:47Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1234 2447154 wikitext text/x-wiki k4u80p9v9d11bz0xk3os3f3u3ck1i7g 2447155 2447154 2026-08-13T18:01:53Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447155 wikitext text/x-wiki qwl3oo28q7o0ujhwja5igiz3qn22juv 2447156 2447155 2026-08-13T18:02:39Z Stashbot 7414 inflatador: bking@dse-k8s-etcd2002 etcdctl member remove ${UUID of dse-k8s-etcd2001} T434681 T434793 2447156 wikitext text/x-wiki ae0lqhzttr2b0f6cn5rawu86fubj9x4 2447157 2447156 2026-08-13T18:02:53Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2447157 wikitext text/x-wiki 0vg0m1xt8om3iiw3wejrjo9m4f1jwhy 2447158 2447157 2026-08-13T18:03:41Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.280.1" completed for 3 hosts 2447158 wikitext text/x-wiki l6jq81wz2qyo1yx1g6167zbj1vm90y0 2447159 2447158 2026-08-13T18:04:06Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447159 wikitext text/x-wiki raboezy0mby63jv1bk7fz3tr9xm4tp5 2447160 2447159 2026-08-13T18:04:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2001.codfw.wmnet 2447160 wikitext text/x-wiki 5b4ak4u8b1stepwwkhf0hq5r9vmgeh6 2447161 2447160 2026-08-13T18:05:47Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1220.eqiad.wmnet with reason: host reimage 2447161 wikitext text/x-wiki 9o41ec80mnrs6nd1igp6v6i3b7l5b70 2447162 2447161 2026-08-13T18:07:06Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1234 - ryankemper@cumin2003" 2447162 wikitext text/x-wiki ngc5prtns02ghe4hj8wl4sfsvib8fpl 2447163 2447162 2026-08-13T18:07:09Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1234 - ryankemper@cumin2003" 2447163 wikitext text/x-wiki c3jioq6ws0d5anh2naldvn6edptsvse 2447164 2447163 2026-08-13T18:07:12Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447164 wikitext text/x-wiki f8m6op7q2to0q7kawi5tukc24epgueo 2447165 2447164 2026-08-13T18:07:14Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1234.eqiad.wmnet 10.36.64.10.in-addr.arpa 0.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447165 wikitext text/x-wiki eitpcph6zmx6s6uwvqysit75mx6s33n 2447166 2447165 2026-08-13T18:07:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2447166 wikitext text/x-wiki r3aiorag0zykun63io7xbn2uxx0xy3y 2447167 2447166 2026-08-13T18:07:18Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1234.eqiad.wmnet 10.36.64.10.in-addr.arpa 0.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447167 wikitext text/x-wiki nbqdv2874gjq9r4tlo62v5ohurznolp 2447168 2447167 2026-08-13T18:07:21Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1234 2447168 wikitext text/x-wiki a5m2q0cqdph3q1tbb8fqwpwte0kzfux 2447169 2447168 2026-08-13T18:07:31Z Stashbot 7414 brennen: 1.47.0-wmf.15 train status (T430834) - no current blockers, rolling to all wikis 2447169 wikitext text/x-wiki 5yvyhbasf1kys55d96zaqojjka3tnfo 2447170 2447169 2026-08-13T18:09:18Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1220.eqiad.wmnet with reason: host reimage 2447170 wikitext text/x-wiki aao25pube5k75us7pnceheqv7izghrl 2447171 2447170 2026-08-13T18:09:41Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1234 2447171 wikitext text/x-wiki 3ih7t0i4yojol3ucxn71wos4yma4mlm 2447172 2447171 2026-08-13T18:09:43Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1234 2447172 wikitext text/x-wiki 8y7g9b1ohwq89jvpb8em354q88jocij 2447173 2447172 2026-08-13T18:10:34Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2447173 wikitext text/x-wiki 2m6hlrlqwhzm1r0639qmrqa6vhi0kuu 2447174 2447173 2026-08-13T18:10:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: quash java safepoint logspam - bking@cumin2003 - T434685 2447174 wikitext text/x-wiki icti0tmzb7u5lcxtsbltd8crhpg8zvf 2447175 2447174 2026-08-13T18:12:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447175 wikitext text/x-wiki qp9hqjznqavj7p9viytpfuclw3u0005 2447176 2447175 2026-08-13T18:12:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447176 wikitext text/x-wiki f6eogi17ukihqxigovc2cwm9uembcfu 2447177 2447176 2026-08-13T18:12:34Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447177 wikitext text/x-wiki j72kedfwudteq7g258ta4w2d86qzdrd 2447178 2447177 2026-08-13T18:12:36Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors 2447178 wikitext text/x-wiki akqs4hutoenujvzx8jte8rpoaxagr4b 2447179 2447178 2026-08-13T18:12:38Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors 2447179 wikitext text/x-wiki 7z9osqom6bqs8efjac4o08po8w92jpb 2447180 2447179 2026-08-13T18:12:40Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2447180 wikitext text/x-wiki 1e5gw4ukvwtg4nbgi09c4qdygs6z3jq 2447181 2447180 2026-08-13T18:12:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2447181 wikitext text/x-wiki ox4d0pvkwi5ialspu6wk47njdo9bp7m 2447182 2447181 2026-08-13T18:14:18Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2447182 wikitext text/x-wiki 5x31oxr481zdr21sagqy58s2eyvwuwe 2447183 2447182 2026-08-13T18:15:19Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1221.eqiad.wmnet with reason: host reimage 2447183 wikitext text/x-wiki 1bzlg99yo7p9pzd54hlcvhx2cfw8ad3 2447184 2447183 2026-08-13T18:15:34Z Stashbot 7414 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.15 refs T430834 2447184 wikitext text/x-wiki 4dvhhnkd5n0xtilqagodibrb8erohk8 2447185 2447184 2026-08-13T18:17:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1166: Pool db1166.eqiad.wmnet in after cloning 2447185 wikitext text/x-wiki nezd0mrki00dgnfmz97fh2n2jxtr8gq 2447186 2447185 2026-08-13T18:19:21Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1221.eqiad.wmnet with reason: host reimage 2447186 wikitext text/x-wiki itmlziitp1oocq1i6viunyaheqpmla7 2447187 2447186 2026-08-13T18:25:21Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2447187 wikitext text/x-wiki 588y56r5zybf9efpcjicqb4mtgzj5sr 2447188 2447187 2026-08-13T18:25:39Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1234.eqiad.wmnet with reason: host reimage 2447188 wikitext text/x-wiki thycjjif8ryrpylnkouhziq0f1fnvc7 2447189 2447188 2026-08-13T18:27:22Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447189 wikitext text/x-wiki mnvq19k8xzh6vaxh2m8rn5ms9xcg8rk 2447190 2447189 2026-08-13T18:27:25Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447190 wikitext text/x-wiki 5l0gi8vz4l8ywlf4c75kl1up3odmvw6 2447191 2447190 2026-08-13T18:27:27Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447191 wikitext text/x-wiki 88s9cyvk4gyur25i10nmx1jbhq9o6xy 2447192 2447191 2026-08-13T18:27:30Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors 2447192 wikitext text/x-wiki 2ttppt5lisap1s8j50aiij9v6l21fi3 2447193 2447192 2026-08-13T18:27:33Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors 2447193 wikitext text/x-wiki e2hmm6uoyl2ui8f1jadg10ombzaj19d 2447194 2447193 2026-08-13T18:27:38Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2001.codfw.wmnet 2447194 wikitext text/x-wiki 6oer3fjdngtydbcky0q3z32qoz6x9dp 2447195 2447194 2026-08-13T18:29:07Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1234.eqiad.wmnet with reason: host reimage 2447195 wikitext text/x-wiki f1f11zjltpq75sdbbeua1rf68xl8c4a 2447196 2447195 2026-08-13T18:30:00Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1220.eqiad.wmnet with OS bookworm 2447196 wikitext text/x-wiki d2hz8tu1u54isyrewbrx4p9la2l2ebf 2447197 2447196 2026-08-13T18:35:29Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2447197 wikitext text/x-wiki 210st3d5yo2i71vwggnx89cl59zvmpy 2447198 2447197 2026-08-13T18:36:56Z Stashbot 7414 inflatador: [bking@ganeti2048] ~$ sudo gnt-instance replace-disks -n ganeti2030.codfw.wmnet aux-k8s-worker2002.codfw.wmnet T434681 2447198 wikitext text/x-wiki hzljx1xz35c2nhcglq9757r84o5uhzo 2447199 2447198 2026-08-13T18:39:54Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1221.eqiad.wmnet with OS bookworm 2447199 wikitext text/x-wiki 7t0fpu9n4639ax10xf5jpdwx138sph3 2447200 2447199 2026-08-13T18:44:21Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=yes; selector: name=dns3004.* 2447200 wikitext text/x-wiki aifb2inzvnoi13umi6xk0jepe1zjo7l 2447201 2447200 2026-08-13T18:46:18Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2447201 wikitext text/x-wiki eipj0td015un4kk9rcp5j9suhvyo40a 2447202 2447201 2026-08-13T18:46:52Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2447202 wikitext text/x-wiki m2xwu8q5ywy1a3e3z8ntss8fw3rldcg 2447203 2447202 2026-08-13T18:49:36Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447203 wikitext text/x-wiki 635wuvh5g0onbaww4hltnakxzpnxvjv 2447204 2447203 2026-08-13T18:51:22Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2447204 wikitext text/x-wiki 12l9og1kqtc4wyl4p8db9zgtp0wwpu4 2447205 2447204 2026-08-13T18:52:01Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2447205 wikitext text/x-wiki qao5nyjz2hsk56qh95sif8ybm740jef 2447206 2447205 2026-08-13T18:58:19Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1234.eqiad.wmnet with OS bookworm 2447206 wikitext text/x-wiki dn6h87rp07zjjnfvuf5qrfie1e4ai1t 2447207 2447206 2026-08-13T18:59:16Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage 2447207 wikitext text/x-wiki ngspp88lmalr4w12xocvoj2qqbcb2t8 2447208 2447207 2026-08-13T19:02:56Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: Pool db1166.eqiad.wmnet in after cloning 2447208 wikitext text/x-wiki bchixcna0phqh7zolpvtmux1054mq4d 2447209 2447208 2026-08-13T19:03:05Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage 2447209 wikitext text/x-wiki iharreaznw1vm8l8pjshtd2wcgbowov 2447210 2447209 2026-08-13T19:04:07Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1166.eqiad.wmnet onto db1280.eqiad.wmnet 2447210 wikitext text/x-wiki cnv03tq4q19fkby9hk9nit1qn0wniaq 2447211 2447210 2026-08-13T19:07:10Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1325548|Enable image lazy loading on desktop in group1 (T148047)]] 2447211 wikitext text/x-wiki 73ot6u9jejslxzxs3d8dc9odf2phy1w 2447213 2447211 2026-08-13T19:10:18Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1325548|Enable image lazy loading on desktop in group1 (T148047)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447213 wikitext text/x-wiki at4er38tenkysxi919n08mjfoto2p0b 2447214 2447213 2026-08-13T19:10:43Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2447214 wikitext text/x-wiki f28tcl0dzk5krbov7xr0kceobvyeqhd 2447215 2447214 2026-08-13T19:14:52Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325548|Enable image lazy loading on desktop in group1 (T148047)]] (duration: 07m 43s) 2447215 wikitext text/x-wiki th6ixv4tj752y9172g3unzv7hbe2z1z 2447216 2447215 2026-08-13T19:18:55Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2447216 wikitext text/x-wiki t3fz615didjbzmn2k2ksrhbx2810d8m 2447217 2447216 2026-08-13T19:20:37Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2447217 wikitext text/x-wiki m4xeoktsybzivnlhv0h7gldk1wh3vvd 2447218 2447217 2026-08-13T19:20:40Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2447218 wikitext text/x-wiki 4w0bvs4k4jm92i4ltzv5sky8anbhnzk 2447219 2447218 2026-08-13T19:24:22Z Stashbot 7414 musikanimal@deploy1003: Started scap sync-world: Backport for [[gerrit:1325551|Render the focused module view as a full-screen page (T433896)]] 2447219 wikitext text/x-wiki qq9koz9b376yy8huswnf89is9qvhv8c 2447221 2447219 2026-08-13T19:34:02Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2447221 wikitext text/x-wiki iijxs9smki05pxv8gypdwj96t5xy2po 2447222 2447221 2026-08-13T19:34:14Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2447222 wikitext text/x-wiki inml5xoyfwvn4824hzltbrvwt90a8rn 2447223 2447222 2026-08-13T19:34:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: quash java safepoint logspam - bking@cumin2003 - T434685 2447223 wikitext text/x-wiki mnnel3blb0hsr0cnlmva700byl1dq7f 2447226 2447223 2026-08-13T19:41:28Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Backport for [[gerrit:1325551|Render the focused module view as a full-screen page (T433896)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447226 wikitext text/x-wiki t3zcanuzzjuss1609bopop7ois8zgm5 2447227 2447226 2026-08-13T19:42:38Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Continuing with deployment 2447227 wikitext text/x-wiki c9zui8o5g9yvu2yyn281kg13oe96iok 2447228 2447227 2026-08-13T19:44:33Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs[2001,1016]*: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 2447228 wikitext text/x-wiki k5bgetuo794d6g6g9efbw0kz0987u34 2447231 2447228 2026-08-13T19:52:48Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs[2001,1016]*: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 2447231 wikitext text/x-wiki b24frzw8ifn8mlow2sruek5mt4scpvm 2447232 2447231 2026-08-13T19:55:00Z Stashbot 7414 musikanimal@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325551|Render the focused module view as a full-screen page (T433896)]] (duration: 30m 37s) 2447232 wikitext text/x-wiki fe0nmrg96ing0s04vs0cvs6b77hamr9 2447234 2447232 2026-08-13T19:58:21Z Stashbot 7414 robh@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs2001.codfw.wmnet with reason: updating firmware 2447234 wikitext text/x-wiki i988dkh4kfc76yee2uvb3u4mtpj313q 2447235 2447234 2026-08-13T20:07:22Z Stashbot 7414 ariel@deploy1003: Started scap sync-world: Backport for [[gerrit:1319906|Remove boilerplate language from wmf-rest and wmf-math API modules (T433736)]] 2447235 wikitext text/x-wiki 9hxvgqso45n5sucorsi2fvew7btpgo6 2447236 2447235 2026-08-13T20:11:20Z Stashbot 7414 ariel@deploy1003: ariel: Backport for [[gerrit:1319906|Remove boilerplate language from wmf-rest and wmf-math API modules (T433736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447236 wikitext text/x-wiki 2ok29d3sv77j2so2qbi25xp3zoonq0x 2447237 2447236 2026-08-13T20:14:53Z Stashbot 7414 ariel@deploy1003: ariel: Continuing with deployment 2447237 wikitext text/x-wiki 1ftp0tpgt9k8woqype2tt7htsw8htli 2447239 2447237 2026-08-13T20:21:19Z Stashbot 7414 ariel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319906|Remove boilerplate language from wmf-rest and wmf-math API modules (T433736)]] (duration: 13m 54s) 2447239 wikitext text/x-wiki 80s4uuy17ylq5bdtynls7hir58y6ynw 2447240 2447239 2026-08-13T20:23:58Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1325549|Deploy PRV to several LC wikis (T423785)]] 2447240 wikitext text/x-wiki 3ocxdkjceghbihj5s26xkr67m5xnu85 2447241 2447240 2026-08-13T20:24:37Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2447241 wikitext text/x-wiki l6n37lsej9onrzru1f5w67k0mlopp6c 2447242 2447241 2026-08-13T20:25:50Z Stashbot 7414 arlolra@deploy1003: arlolra: Backport for [[gerrit:1325549|Deploy PRV to several LC wikis (T423785)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447242 wikitext text/x-wiki d7qg8jmrcmfo3wuaw0rf8f7we2tvvky 2447243 2447242 2026-08-13T20:26:49Z Stashbot 7414 arlolra@deploy1003: arlolra: Continuing with deployment 2447243 wikitext text/x-wiki 2apvm4tooxs432g8b7mc3tqedsmlcxb 2447244 2447243 2026-08-13T20:30:56Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325549|Deploy PRV to several LC wikis (T423785)]] (duration: 06m 57s) 2447244 wikitext text/x-wiki onh3r98rg9jkpejvo6jw96idrvi1yes 2447245 2447244 2026-08-13T20:33:19Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1320163|Add configurable RestTermsOfServiceUrl (T428147)]] 2447245 wikitext text/x-wiki fzc2uefsuh9byz9p01itpzif8q28vwk 2447246 2447245 2026-08-13T20:35:11Z Stashbot 7414 samtar@deploy1003: samtar, milazg: Backport for [[gerrit:1320163|Add configurable RestTermsOfServiceUrl (T428147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447246 wikitext text/x-wiki czf2407vynwzau7l0zn1brcbsalxpfv 2447247 2447246 2026-08-13T20:50:42Z Stashbot 7414 samtar@deploy1003: samtar, milazg: Continuing with deployment 2447247 wikitext text/x-wiki 8x25fkec8byns2ldu4pg25gjh65corb 2447248 2447247 2026-08-13T20:53:04Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2001.codfw.wmnet 2447248 wikitext text/x-wiki six0nb3uuq90de2pncn717e7iqygxp7 2447249 2447248 2026-08-13T20:53:07Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2447249 wikitext text/x-wiki s8df1pssehz2efxe7ug14jhcoz2yoaq 2447250 2447249 2026-08-13T20:54:59Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1320163|Add configurable RestTermsOfServiceUrl (T428147)]] (duration: 21m 39s) 2447250 wikitext text/x-wiki bim7ce1zvalbdpb7xf562k433g8rut8 2447251 2447250 2026-08-13T20:57:19Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447251 wikitext text/x-wiki 69b8irgcgx1y6en8tutf3zlohj28id3 2447252 2447251 2026-08-13T20:57:23Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447252 wikitext text/x-wiki 7fyw0i5d6d837e1qstekq3kdxwz8ovb 2447253 2447252 2026-08-13T20:57:25Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447253 wikitext text/x-wiki 5g4vktypkrvqugu1bk7m2gezt922um9 2447254 2447253 2026-08-13T20:57:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors 2447254 wikitext text/x-wiki 7jzllyyoj9f0i1o1c1rtev9bue4qmoj 2447255 2447254 2026-08-13T20:57:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors 2447255 wikitext text/x-wiki 57vgc7nxpv0zss7fhpz9qh465ip8i2w 2447256 2447255 2026-08-13T20:57:41Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1325553|InstrumentConstructiveEdits: exclude mw-reverted as well (T431493)]] 2447256 wikitext text/x-wiki 927l0a8taowlb4ghuankbkm4on6m6mw 2447257 2447256 2026-08-13T20:57:55Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447257 wikitext text/x-wiki 483s3neby35t2tb8vbgwpdvewv3570i 2447258 2447257 2026-08-13T20:57:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" 2447258 wikitext text/x-wiki qwisendgeemu183dxcwxm8ln384qqvt 2447259 2447258 2026-08-13T20:58:20Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-etcd2001.codfw.wmnet with OS bookworm 2447259 wikitext text/x-wiki pw21u3t9sk6szlt9pnsdtv0241c7r8i 2447260 2447259 2026-08-13T20:59:32Z Stashbot 7414 kemayo@deploy1003: kemayo: Backport for [[gerrit:1325553|InstrumentConstructiveEdits: exclude mw-reverted as well (T431493)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447260 wikitext text/x-wiki l0lt73jnw3x3upenzv73yy5icfgsjj0 2447261 2447260 2026-08-13T20:59:57Z Stashbot 7414 kemayo@deploy1003: kemayo: Continuing with deployment 2447261 wikitext text/x-wiki p0arf3k5v70akv7tf1r822l0ywzivmh 2447262 2447261 2026-08-13T21:04:06Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325553|InstrumentConstructiveEdits: exclude mw-reverted as well (T431493)]] (duration: 06m 25s) 2447262 wikitext text/x-wiki jgh9os1wokcunpy5vpi2x7y5e6qp8xs 2447264 2447262 2026-08-13T21:17:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-etcd2001.codfw.wmnet with reason: host reimage 2447264 wikitext text/x-wiki clzw9rxrr18zm938gyv2wayz151n6un 2447265 2447264 2026-08-13T21:19:50Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1212.eqiad.wmnet with OS bookworm 2447265 wikitext text/x-wiki m2meg6071f50dmc63usrwydd3dttvxj 2447266 2447265 2026-08-13T21:20:11Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1212 2447266 wikitext text/x-wiki 7jy9ua86wsn6eyb9lcz7fhwlam740cj 2447267 2447266 2026-08-13T21:20:14Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1212 2447267 wikitext text/x-wiki orle0e9s9l6nrflv8mcjyme9bi9tid3 2447269 2447267 2026-08-13T21:24:58Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-etcd2001.codfw.wmnet with reason: host reimage 2447269 wikitext text/x-wiki 32dj40nt43eqjgjh8wdmagl256l5lpn 2447270 2447269 2026-08-13T21:26:58Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host aux-k8s-etcd2003.codfw.wmnet 2447270 wikitext text/x-wiki j1xcttzwibvy8kx9mbkfvjubv29ch5x 2447271 2447270 2026-08-13T21:27:00Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2447271 wikitext text/x-wiki tqe2upho8ajzcvlaavg35tfhxakhw1p 2447272 2447271 2026-08-13T21:28:29Z Stashbot 7414 maryum: Deployed security patch for T434619 2447272 wikitext text/x-wiki nlb5pg8hbbuv07vqefx0hk53bcaqmrc 2447273 2447272 2026-08-13T21:31:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003" 2447273 wikitext text/x-wiki ggct8pmqil832xyylqafdagk46dj2lp 2447274 2447273 2026-08-13T21:32:03Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003" 2447274 wikitext text/x-wiki m0dfmyerggixvfkzsj8wuh6eqf863og 2447275 2447274 2026-08-13T21:32:05Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447275 wikitext text/x-wiki qqrqmcopovdij1i078ze4b9ad02w0bc 2447276 2447275 2026-08-13T21:32:08Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache aux-k8s-etcd2003.codfw.wmnet on all recursors 2447276 wikitext text/x-wiki a7onfa8w06tuhosoy2mi0wlil7nw1hw 2447277 2447276 2026-08-13T21:32:10Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-etcd2003.codfw.wmnet on all recursors 2447277 wikitext text/x-wiki pyt2pdai827volccnhzpx8xsvp3bxub 2447278 2447277 2026-08-13T21:32:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003" 2447278 wikitext text/x-wiki lqpprd7y6hpp1m9sca6p59fs8b7l7jy 2447279 2447278 2026-08-13T21:32:45Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003" 2447279 wikitext text/x-wiki 8dsssbw3nzezp8n3chfiw27i9l18yuj 2447280 2447279 2026-08-13T21:33:42Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd2003.codfw.wmnet with OS bookworm 2447280 wikitext text/x-wiki tqbwb0kxj1n9ph51qyypso0051s7jgc 2447281 2447280 2026-08-13T21:34:43Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1212.eqiad.wmnet with reason: host reimage 2447281 wikitext text/x-wiki i3h2lznhpcro9rw4y66v4morz5z28tl 2447282 2447281 2026-08-13T21:39:16Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1212.eqiad.wmnet with reason: host reimage 2447282 wikitext text/x-wiki k77crfpnmswxdusqy4mnr4zvm173i7b 2447283 2447282 2026-08-13T21:42:39Z Stashbot 7414 maryum: Deployed security patch for T434549 2447283 wikitext text/x-wiki imwuqojhyrejrkfyjwaw69u0zcpvxki 2447284 2447283 2026-08-13T21:47:58Z Stashbot 7414 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1324295|Revert "Lazily reject pre-fix parser-cache entries for noreferrer/noopener links" (T429090)]] 2447284 wikitext text/x-wiki hhpjd3lnogtimt6ywfwngba9pf9y0ki 2447286 2447284 2026-08-13T21:48:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd2003.codfw.wmnet with reason: host reimage 2447286 wikitext text/x-wiki shzqq3fz8r9rlqp1l7890acvwfu1tqh 2447287 2447286 2026-08-13T21:49:48Z Stashbot 7414 sbassett@deploy1003: sbassett, kharlan: Backport for [[gerrit:1324295|Revert "Lazily reject pre-fix parser-cache entries for noreferrer/noopener links" (T429090)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447287 wikitext text/x-wiki ffv8knxb16p7tfumjpnfakiqbyziuut 2447288 2447287 2026-08-13T21:50:25Z Stashbot 7414 sbassett@deploy1003: sbassett, kharlan: Continuing with deployment 2447288 wikitext text/x-wiki it6mw9n7gzivghfxdd3wc8kiy1htaxm 2447289 2447288 2026-08-13T21:53:05Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-etcd2001.codfw.wmnet with OS bookworm 2447289 wikitext text/x-wiki jbov86oxwrbnlpmmv8mwgoxsb0tf8zf 2447290 2447289 2026-08-13T21:53:08Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host dse-k8s-etcd2001.codfw.wmnet 2447290 wikitext text/x-wiki kp2fz0egouzzw38b887tjtu6o3neiuv 2447291 2447290 2026-08-13T21:53:10Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd2003.codfw.wmnet with reason: host reimage 2447291 wikitext text/x-wiki d64fv182oogdvsnhr8zzpcvk024wfo7 2447292 2447291 2026-08-13T21:54:40Z Stashbot 7414 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324295|Revert "Lazily reject pre-fix parser-cache entries for noreferrer/noopener links" (T429090)]] (duration: 06m 42s) 2447292 wikitext text/x-wiki 0zni8a0s2yx5xk2ee808miui74kfs3m 2447293 2447292 2026-08-13T22:00:03Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1212.eqiad.wmnet with OS bookworm 2447293 wikitext text/x-wiki 6r8wr6h3qd0z5xute4v1la9rcati51k 2447294 2447293 2026-08-13T22:01:15Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1222.eqiad.wmnet with OS bookworm 2447294 wikitext text/x-wiki tmj6e388uodr48a11yb81hs4tbcpzqn 2447295 2447294 2026-08-13T22:01:44Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1222 2447295 wikitext text/x-wiki 2gdvpvexpbxlludgovcbatxoxie756j 2447296 2447295 2026-08-13T22:01:50Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1223.eqiad.wmnet with OS bookworm 2447296 wikitext text/x-wiki j04uhr4s98b8hobtmyau08ro1j1fzu2 2447297 2447296 2026-08-13T22:01:55Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447297 wikitext text/x-wiki 04rkl4guibjze56o82wo8gp9wdufmmk 2447298 2447297 2026-08-13T22:05:05Z Stashbot 7414 robh@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs2001.codfw.wmnet with reason: updating firmware 2447298 wikitext text/x-wiki 140nd8a8m5ld9nt6qsfsp1n253ps250 2447299 2447298 2026-08-13T22:07:41Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1222 - ryankemper@cumin2003" 2447299 wikitext text/x-wiki obr03nk4w73a7ftpx24xrhcwg833gpy 2447300 2447299 2026-08-13T22:11:03Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1222 - ryankemper@cumin2003" 2447300 wikitext text/x-wiki opajpd1yspcq43j1m61w1hwluyiuslh 2447301 2447300 2026-08-13T22:11:06Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447301 wikitext text/x-wiki 7kvlhcqu0urly9sti8fmttzybg9xzs9 2447302 2447301 2026-08-13T22:11:09Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1222.eqiad.wmnet 19.36.64.10.in-addr.arpa 9.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447302 wikitext text/x-wiki c7bzgjos93pa7cyuhwm321h5b4s2ut4 2447303 2447302 2026-08-13T22:11:12Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1222.eqiad.wmnet 19.36.64.10.in-addr.arpa 9.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447303 wikitext text/x-wiki 9in3nhicxbbnd1f1umjs310pzgejfpn 2447304 2447303 2026-08-13T22:11:14Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1222 2447304 wikitext text/x-wiki jrswfdlg630snklu1b3ff5nf2fywxu5 2447305 2447304 2026-08-13T22:11:52Z Stashbot 7414 sbassett: Deployed security updates for T429244, T434039 2447305 wikitext text/x-wiki tnhpejmdklks3ga7hvn73jjvb1bs62t 2447306 2447305 2026-08-13T22:12:22Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1222 2447306 wikitext text/x-wiki pckirh1vacg335xyl5lz10xcknuambm 2447307 2447306 2026-08-13T22:12:25Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1222 2447307 wikitext text/x-wiki nq1dft9ll704119vum36h0ba6189qgt 2447308 2447307 2026-08-13T22:12:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host kubestagemaster2005.codfw.wmnet 2447308 wikitext text/x-wiki abowyeallqccvgdk6z0o044i2wq8hp9 2447309 2447308 2026-08-13T22:12:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2447309 wikitext text/x-wiki f26vxkdmg8ncicgbgglwlwccjhcennj 2447310 2447309 2026-08-13T22:13:41Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1223 2447310 wikitext text/x-wiki 8m8r8sdizcnu1e6iuty8tjlv04m3jp8 2447311 2447310 2026-08-13T22:17:29Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447311 wikitext text/x-wiki nejyslva0lv6cwcsdpdzfixlntqq2hu 2447312 2447311 2026-08-13T22:22:34Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2005.codfw.wmnet - bking@cumin2003" 2447312 wikitext text/x-wiki e5rw9a3s699rd4rn005h5965bm17wnn 2447313 2447312 2026-08-13T22:22:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2005.codfw.wmnet - bking@cumin2003" 2447313 wikitext text/x-wiki 4mnviz4fmujj589vuli1l8x8h0edlu8 2447314 2447313 2026-08-13T22:22:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447314 wikitext text/x-wiki putoc1xzirdeubtntuseglfq8f19mgt 2447315 2447314 2026-08-13T22:22:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache kubestagemaster2005.codfw.wmnet on all recursors 2447315 wikitext text/x-wiki kvbrxljlp0ot22hsmqe0qg9ql1oao50 2447316 2447315 2026-08-13T22:22:46Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubestagemaster2005.codfw.wmnet on all recursors 2447316 wikitext text/x-wiki fvtcxl2yzw2awsmiciy5zavvepx9nqy 2447317 2447316 2026-08-13T22:23:05Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2005.codfw.wmnet - bking@cumin2003" 2447317 wikitext text/x-wiki 2cs4szj69b83gulpvplx87o7s67rae5 2447318 2447317 2026-08-13T22:23:10Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2005.codfw.wmnet - bking@cumin2003" 2447318 wikitext text/x-wiki o3zw4sjjuhgijyf92t25sy6sp9myse0 2447319 2447318 2026-08-13T22:23:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host kubestagemaster2005.codfw.wmnet with OS bookworm 2447319 wikitext text/x-wiki 9yzek71hd6l4e16jb3eidbh93e43c48 2447320 2447319 2026-08-13T22:23:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd2003.codfw.wmnet with OS bookworm 2447320 wikitext text/x-wiki 9rigu5slzcgth8wmvpu71pejs2w7hq5 2447321 2447320 2026-08-13T22:23:56Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-etcd2003.codfw.wmnet 2447321 wikitext text/x-wiki gtoyfrl15rccrdkso0x2z7ghdkky6bj 2447322 2447321 2026-08-13T22:26:15Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1222.eqiad.wmnet with reason: host reimage 2447322 wikitext text/x-wiki jpwdr3y8q60dvrkqc7z4dgs2e5v07tn 2447323 2447322 2026-08-13T22:29:40Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1222.eqiad.wmnet with reason: host reimage 2447323 wikitext text/x-wiki 7blnr2pr1iclps7ux1p052g0m2wnm0g 2447324 2447323 2026-08-13T22:42:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage 2447324 wikitext text/x-wiki nuxxb5wnv0jyg1ndii5vo7q5glplxxw 2447325 2447324 2026-08-13T22:49:40Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1222.eqiad.wmnet with OS bookworm 2447325 wikitext text/x-wiki 3owuul8syhge9kd47wm25zvkfhm0fn0 2447326 2447325 2026-08-13T22:49:43Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage 2447326 wikitext text/x-wiki 8dhntj29vrr2n5sk6fv4dkt9brk6cj1 2447327 2447326 2026-08-13T22:53:02Z Stashbot 7414 ryankemper@cumin2003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2447327 wikitext text/x-wiki afnmw7y9nhtimuscmz1cpp3uo87xr4c 2447328 2447327 2026-08-13T22:54:37Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447328 wikitext text/x-wiki l0fp79d8h03d5suuvfbr2uucs1c2rt5 2447329 2447328 2026-08-13T22:55:11Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubestagemaster2005.codfw.wmnet with OS bookworm 2447329 wikitext text/x-wiki bf275u6ibkpn7qsum4ieczaa79m3kgg 2447330 2447329 2026-08-13T22:55:13Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host kubestagemaster2005.codfw.wmnet 2447330 wikitext text/x-wiki 74sl23afzsykjeasc9xdfsybx72wcoz 2447331 2447330 2026-08-13T23:01:52Z Stashbot 7414 sbassett: Deployed security updates for T430596, T120386 2447331 wikitext text/x-wiki ak6sdo5lhjcxo67z5dmyzabls1r849p 2447332 2447331 2026-08-13T23:03:29Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1223 - ryankemper@cumin2003" 2447332 wikitext text/x-wiki 2p5c1w3dfxu57ik22jsw5omgg4zne9z 2447333 2447332 2026-08-13T23:07:36Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1223 - ryankemper@cumin2003" 2447333 wikitext text/x-wiki isetwm3b1ohagzfnaawfzznj5q8dhh1 2447334 2447333 2026-08-13T23:07:38Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447334 wikitext text/x-wiki 75rjf7ujdgax6bgj8lvwmjfi65e938l 2447335 2447334 2026-08-13T23:07:40Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1223.eqiad.wmnet 20.36.64.10.in-addr.arpa 0.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447335 wikitext text/x-wiki 6pg3eduse7dpcdom5unwm30o8d4hfdv 2447336 2447335 2026-08-13T23:07:42Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1223.eqiad.wmnet 20.36.64.10.in-addr.arpa 0.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447336 wikitext text/x-wiki 4p0x1ycn1izrmzl619p0sdr5r2qa46r 2447337 2447336 2026-08-13T23:07:44Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1223 2447337 wikitext text/x-wiki 8gl3ibx26kyx6phfmsyi0x614l8mvi5 2447338 2447337 2026-08-13T23:08:11Z Stashbot 7414 musikanimal@deploy1003: Started scap sync-world: Backport for [[gerrit:1325525|ve.ui.CodeMirror.less: ensure normal font style]] 2447338 wikitext text/x-wiki gzz6dfwvcnnkyc1igau6u08g8csg32u 2447340 2447338 2026-08-13T23:11:05Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1223 2447340 wikitext text/x-wiki 7we2p2s9jiitw7idq5tj3yygosgu5ox 2447341 2447340 2026-08-13T23:11:07Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1223 2447341 wikitext text/x-wiki rmj3t3fvunul3eq03c5sj3w0ls4scqb 2447342 2447341 2026-08-13T23:11:58Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Backport for [[gerrit:1325525|ve.ui.CodeMirror.less: ensure normal font style]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2447342 wikitext text/x-wiki 3mwsyf7qhoz1z2qbjhqdbq2tfy5nynj 2447343 2447342 2026-08-13T23:13:01Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Continuing with deployment 2447343 wikitext text/x-wiki og3fj4mquadi4zejrwus3xakt7vc6qt 2447344 2447343 2026-08-13T23:16:53Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260807000000" --end-timestamp="20260808000000" --sleep="5" --batch-size="50"` for T434688 2447344 wikitext text/x-wiki kofh1hidl9p619sn4yesuyy6yatrzun 2447345 2447344 2026-08-13T23:19:50Z Stashbot 7414 musikanimal@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325525|ve.ui.CodeMirror.less: ensure normal font style]] (duration: 11m 40s) 2447345 wikitext text/x-wiki 45sl5vgxg84bdku9x5tilz9hsemb1mn 2447346 2447345 2026-08-13T23:25:00Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1223.eqiad.wmnet with reason: host reimage 2447346 wikitext text/x-wiki 9msndidqokm3li7gfjwk48orfzfk8mn 2447347 2447346 2026-08-13T23:29:17Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1223.eqiad.wmnet with reason: host reimage 2447347 wikitext text/x-wiki g7ex1jbk52w7de9x186sxkkz4j2rzuq 2447348 2447347 2026-08-13T23:47:01Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1223.eqiad.wmnet with OS bookworm 2447348 wikitext text/x-wiki 600tmcz7c6nwxu5fx7tzq4y7mqx3bc1 2447349 2447348 2026-08-13T23:51:26Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1224.eqiad.wmnet with OS bookworm 2447349 wikitext text/x-wiki 92qfkhrovl9j1x4ttnwv60nhw000v1e 2447350 2447349 2026-08-13T23:51:45Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1224 2447350 wikitext text/x-wiki knqk69okyf7alauf94aesy76kq1cs1u 2447351 2447350 2026-08-13T23:52:56Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447351 wikitext text/x-wiki 5u819xa3vkfsdgseu9q1p7sotlraobr 2447352 2447351 2026-08-13T23:53:45Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1225.eqiad.wmnet with OS bookworm 2447352 wikitext text/x-wiki e5o3wehhrm5296npm2zabi9i5hnhg50 2447353 2447352 2026-08-13T23:54:04Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1226.eqiad.wmnet with OS bookworm 2447353 wikitext text/x-wiki osr1cud992q4jom3n8k31ua4pk3c85h 2447354 2447353 2026-08-14T00:00:31Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1224 - ryankemper@cumin2003" 2447354 wikitext text/x-wiki hislwa3s0xgedsw6zbiqsalwl7zk8z1 2447355 2447354 2026-08-14T00:00:35Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1224 - ryankemper@cumin2003" 2447355 wikitext text/x-wiki sull41abqprnwz1n11afu0gj1a9re1m 2447356 2447355 2026-08-14T00:00:38Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447356 wikitext text/x-wiki aresl2il53mryqkym4ihkd8hzr3a43e 2447357 2447356 2026-08-14T00:00:40Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1224.eqiad.wmnet 21.36.64.10.in-addr.arpa 1.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447357 wikitext text/x-wiki noku1axb4ly58wzalhybdooh2ygb29v 2447358 2447357 2026-08-14T00:00:43Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1224.eqiad.wmnet 21.36.64.10.in-addr.arpa 1.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447358 wikitext text/x-wiki sf4svjgxypmx34ttthbpbbbrsr1hpb6 2447359 2447358 2026-08-14T00:00:45Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1224 2447359 wikitext text/x-wiki koehcd6c6td5w2t2qqonqki47tel5pr 2447361 2447359 2026-08-14T00:02:16Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1224 2447361 wikitext text/x-wiki nfeuaq9c9jlz1yq5ickvrfeph7fxjce 2447362 2447361 2026-08-14T00:02:19Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1224 2447362 wikitext text/x-wiki 4px9vh4a47aqvnmbx89bkntdpbsqbzp 2447363 2447362 2026-08-14T00:02:59Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1225 2447363 wikitext text/x-wiki 5ens4v9gvlusqeugf3m3tzjoa8brljk 2447364 2447363 2026-08-14T00:03:16Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447364 wikitext text/x-wiki d91t1op7o9rwcgmc41wz4dva92izr8f 2447365 2447364 2026-08-14T00:10:04Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1225 - ryankemper@cumin2003" 2447365 wikitext text/x-wiki 49j6ftskq5t81645z4gnpnm1ir626iv 2447366 2447365 2026-08-14T00:10:09Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1225 - ryankemper@cumin2003" 2447366 wikitext text/x-wiki byc85j8tdakxnjb709samulsqryfy0u 2447367 2447366 2026-08-14T00:10:11Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447367 wikitext text/x-wiki kmyph9ckunwdu8t2rf5kmlg5ymq716w 2447368 2447367 2026-08-14T00:10:13Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1225.eqiad.wmnet 22.36.64.10.in-addr.arpa 2.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447368 wikitext text/x-wiki gbixr2te5yjxda5e8gfmwt1a9y0p8df 2447369 2447368 2026-08-14T00:10:16Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1225.eqiad.wmnet 22.36.64.10.in-addr.arpa 2.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447369 wikitext text/x-wiki ghcnihpy36cfa4pk3j5sm5707iyy4yu 2447370 2447369 2026-08-14T00:10:19Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1225 2447370 wikitext text/x-wiki 7e7fkawwdmj3swfpyggif623g1ctder 2447371 2447370 2026-08-14T00:11:02Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1225 2447371 wikitext text/x-wiki j3z22ctc94mx6yn14879o69mnvchfvh 2447372 2447371 2026-08-14T00:11:04Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1225 2447372 wikitext text/x-wiki e16zqxpn8znfktx3o91o9fwo16imhgh 2447373 2447372 2026-08-14T00:12:04Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1226 2447373 wikitext text/x-wiki 7flnj0f8n9vb283bx115vzzqbyoaagd 2447374 2447373 2026-08-14T00:12:12Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447374 wikitext text/x-wiki g8gbitk5nz6h0ssaerwxrvcq0agazdd 2447375 2447374 2026-08-14T00:16:11Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1224.eqiad.wmnet with reason: host reimage 2447375 wikitext text/x-wiki scmc1in6lpg3pqpie0o3xt2tjr2ju6c 2447376 2447375 2026-08-14T00:17:15Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1226 - ryankemper@cumin2003" 2447376 wikitext text/x-wiki 9l3sak6tjlob73f6ovageqe448m2jq6 2447377 2447376 2026-08-14T00:17:20Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1226 - ryankemper@cumin2003" 2447377 wikitext text/x-wiki qchol8i256ajj4t62mekx49gcf0k2cq 2447378 2447377 2026-08-14T00:17:22Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447378 wikitext text/x-wiki 94spc7xsfrzyz78l90s0vtep1xufbry 2447379 2447378 2026-08-14T00:17:24Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1226.eqiad.wmnet 23.36.64.10.in-addr.arpa 3.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447379 wikitext text/x-wiki jutrqqyfp2pr6yr4yc0udlzwzfptx7q 2447380 2447379 2026-08-14T00:17:27Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1226.eqiad.wmnet 23.36.64.10.in-addr.arpa 3.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447380 wikitext text/x-wiki 9v29gcldirgsj1dgbstop7wlqmcoezc 2447381 2447380 2026-08-14T00:17:29Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1226 2447381 wikitext text/x-wiki nlcxxqfamrv3lp5304xfa284a0m3zd8 2447382 2447381 2026-08-14T00:17:55Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1226 2447382 wikitext text/x-wiki mtww6u08bc5jm978kpsze6hlux3f9yr 2447383 2447382 2026-08-14T00:17:57Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1226 2447383 wikitext text/x-wiki dslap1963tph6iw1dehhuiekj1z94eb 2447384 2447383 2026-08-14T00:19:36Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1224.eqiad.wmnet with reason: host reimage 2447384 wikitext text/x-wiki dzs20m3byfqj9qim2z5zeifix8198to 2447385 2447384 2026-08-14T00:25:02Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1225.eqiad.wmnet with reason: host reimage 2447385 wikitext text/x-wiki gjl167t0ug54j6fzw0vo7qgc3pr3rf4 2447386 2447385 2026-08-14T00:28:09Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1225.eqiad.wmnet with reason: host reimage 2447386 wikitext text/x-wiki tvuv9li8is5cypsufb7031ttm1wqnnr 2447387 2447386 2026-08-14T00:31:48Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1226.eqiad.wmnet with reason: host reimage 2447387 wikitext text/x-wiki hr6f51r73wueq6ip7wixrk63a8fyr8g 2447388 2447387 2026-08-14T00:38:58Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1226.eqiad.wmnet with reason: host reimage 2447388 wikitext text/x-wiki 3vcetcprobysp8ckiu472z7af6p8rt7 2447389 2447388 2026-08-14T00:41:51Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1224.eqiad.wmnet with OS bookworm 2447389 wikitext text/x-wiki 0dcwttago39i6chertitfr5g9x9lx2z 2447390 2447389 2026-08-14T00:47:52Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1225.eqiad.wmnet with OS bookworm 2447390 wikitext text/x-wiki qigltgylff9v6a668oyyy95nkbfth9y 2447391 2447390 2026-08-14T00:59:30Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1226.eqiad.wmnet with OS bookworm 2447391 wikitext text/x-wiki bdzh7jldt0q099tisqulgw1sn2435iv 2447392 2447391 2026-08-14T01:02:24Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2447392 wikitext text/x-wiki ivm5zndiyeq24y5z778w5x6msda63vk 2447393 2447392 2026-08-14T01:06:03Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2447393 wikitext text/x-wiki hee26nxjli51hq0s0mfgcyndhvswh72 2447394 2447393 2026-08-14T01:07:09Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2447394 wikitext text/x-wiki h4vlpco8gymbnlaceu1aapcpyqdva8p 2447395 2447394 2026-08-14T01:10:33Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2447395 wikitext text/x-wiki owgzydu4x0jbg9lj8sd9zslx04vqfz0 2447397 2447395 2026-08-14T02:00:50Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2447397 wikitext text/x-wiki oze5hbzdeffqr08o8dmrhg4ddmaulug 2447399 2447397 2026-08-14T02:07:54Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 03s) 2447399 wikitext text/x-wiki 96fz68p4afz6uampi8pazggv5z2gzom 2447402 2447399 2026-08-14T05:09:08Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host apifeatureusage1001.eqiad.wmnet with OS bookworm 2447402 wikitext text/x-wiki hzthw78lhyaunqzzoltr1hqal1axxsh 2447403 2447402 2026-08-14T05:47:06Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1175.eqiad.wmnet with OS bookworm 2447403 wikitext text/x-wiki oksewc04sw056ml5qz99ux465h4aadx 2447404 2447403 2026-08-14T05:47:36Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1175 2447404 wikitext text/x-wiki 0e52v3r5xd5lk8l87nkmm5jmti1k9l8 2447405 2447404 2026-08-14T05:47:39Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447405 wikitext text/x-wiki brk5mxbpwi0ytgf9cjg5h3gylq2ucq6 2447406 2447405 2026-08-14T05:48:39Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1209.eqiad.wmnet with OS bookworm 2447406 wikitext text/x-wiki 3diacc62zfieu01um3132tjz3cm7chq 2447407 2447406 2026-08-14T05:49:02Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1227.eqiad.wmnet with OS bookworm 2447407 wikitext text/x-wiki sk3t9mayu9bnbcxo3bohrw4hml6z0ke 2447408 2447407 2026-08-14T05:49:47Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1228.eqiad.wmnet with OS bookworm 2447408 wikitext text/x-wiki m53h5ard2t33mx5mdj0zgky9u6fpnpk 2447409 2447408 2026-08-14T05:52:37Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1175 - ryankemper@cumin2003" 2447409 wikitext text/x-wiki 2cwqj3qja4ctzkk16fevy3fmjuu0m18 2447410 2447409 2026-08-14T05:52:40Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1175 - ryankemper@cumin2003" 2447410 wikitext text/x-wiki 5akhwz5i09v5wnq2kvosfclyefpwbxh 2447411 2447410 2026-08-14T05:52:43Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447411 wikitext text/x-wiki 9gkt4ugvs9h15zs1aziecbpthkfpn5p 2447412 2447411 2026-08-14T05:52:46Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1175.eqiad.wmnet 17.53.64.10.in-addr.arpa 7.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447412 wikitext text/x-wiki inl68f0dvjn0z784cirf1r7y10h137q 2447413 2447412 2026-08-14T05:52:48Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1175.eqiad.wmnet 17.53.64.10.in-addr.arpa 7.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447413 wikitext text/x-wiki n9nck94dz07tagb14x2cfevfm80b7fq 2447414 2447413 2026-08-14T05:52:50Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1175 2447414 wikitext text/x-wiki cobxe6z5qqhanw2okhuzncjuc3tk0kz 2447415 2447414 2026-08-14T05:54:19Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1175 2447415 wikitext text/x-wiki 7251hwk6qkt5iohzl2kx68cs3wmonw8 2447416 2447415 2026-08-14T05:54:21Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1175 2447416 wikitext text/x-wiki rp3gs2rpn3gv52wvfze16d28pgsjf9v 2447417 2447416 2026-08-14T05:54:38Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1209 2447417 wikitext text/x-wiki oaiadcgghfsefvx5qood3wg3v66tv9a 2447418 2447417 2026-08-14T05:54:41Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447418 wikitext text/x-wiki pwpzln1y8mwr3599v2j7ja9hp5qfze8 2447419 2447418 2026-08-14T06:00:10Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1209 - ryankemper@cumin2003" 2447419 wikitext text/x-wiki gor7l08ngxz60p2optwqve0c0c9u7no 2447420 2447419 2026-08-14T06:00:14Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1209 - ryankemper@cumin2003" 2447420 wikitext text/x-wiki jlxtlqek4jcomwos0y6t8pvx4hmzs1q 2447421 2447420 2026-08-14T06:00:17Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447421 wikitext text/x-wiki 0mwr8c13b1wcgo2njtno7by61a6k4gj 2447422 2447421 2026-08-14T06:00:19Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1209.eqiad.wmnet 15.53.64.10.in-addr.arpa 5.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447422 wikitext text/x-wiki hb6gjvg130w0rco4ib6nqj9ippbsoh4 2447423 2447422 2026-08-14T06:00:22Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1209.eqiad.wmnet 15.53.64.10.in-addr.arpa 5.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447423 wikitext text/x-wiki a1x5evwzhyrwe1kj3l7pkyl4j3bs0vd 2447424 2447423 2026-08-14T06:00:24Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1209 2447424 wikitext text/x-wiki jswiopk48135v0aod4hkddas1ikixt6 2447425 2447424 2026-08-14T06:00:38Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1209 2447425 wikitext text/x-wiki 6da9260v4k4078z1fs5x83n5v8ixof8 2447426 2447425 2026-08-14T06:00:41Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1209 2447426 wikitext text/x-wiki pjtehiu3zo7zuy8gi9y9xpynsnur2jd 2447427 2447426 2026-08-14T06:00:43Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1227 2447427 wikitext text/x-wiki o1jdz2c7efu5xoaozzx75ubxly12cg0 2447428 2447427 2026-08-14T06:00:46Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447428 wikitext text/x-wiki bdhfe7zzundzk6mdzc7ksvetmynhp8k 2447429 2447428 2026-08-14T06:06:14Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1227 - ryankemper@cumin2003" 2447429 wikitext text/x-wiki 7a7b6frnd3xdzdtpks7k7ldl5156af3 2447430 2447429 2026-08-14T06:06:18Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1227 - ryankemper@cumin2003" 2447430 wikitext text/x-wiki 2606ogihlzj3g4woqpubk1sxivmfwun 2447431 2447430 2026-08-14T06:06:20Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447431 wikitext text/x-wiki 5b0juok7it3xoxa8905lptcjctjr88f 2447432 2447431 2026-08-14T06:06:22Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1227.eqiad.wmnet 19.53.64.10.in-addr.arpa 9.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447432 wikitext text/x-wiki 8xvtqvn9rg7fypzesx6bxoc09u6jebx 2447433 2447432 2026-08-14T06:06:24Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1227.eqiad.wmnet 19.53.64.10.in-addr.arpa 9.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447433 wikitext text/x-wiki bb68vtqr29qvht70nf62y8og7o4uv1q 2447434 2447433 2026-08-14T06:06:26Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1227 2447434 wikitext text/x-wiki 7cw5z2my4qwvkcmadoir4t52y0l0zh0 2447435 2447434 2026-08-14T06:07:27Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1227 2447435 wikitext text/x-wiki 7c2fde10n4tu81nchkogi10ohl5o9eu 2447436 2447435 2026-08-14T06:07:29Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1227 2447436 wikitext text/x-wiki 7dg35zq24rthuc8b6vxg0k16fkj2t66 2447437 2447436 2026-08-14T06:07:48Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1228 2447437 wikitext text/x-wiki gcdd4vl6i1cwdmqsewwssft7l4nffm5 2447438 2447437 2026-08-14T06:07:51Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.netbox 2447438 wikitext text/x-wiki 1os4w4gmzazixjam1hqb265bgqn17qi 2447439 2447438 2026-08-14T06:09:59Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1175.eqiad.wmnet with reason: host reimage 2447439 wikitext text/x-wiki qqo7r22juss3q9j9kgek1owpa26lkgv 2447440 2447439 2026-08-14T06:12:43Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1228 - ryankemper@cumin2003" 2447440 wikitext text/x-wiki t2og0bxgbkqi5gv3capzpczmxy8ngna 2447441 2447440 2026-08-14T06:12:47Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1228 - ryankemper@cumin2003" 2447441 wikitext text/x-wiki mtpagra0jy2i2cw83v3pmnjeqsdbck1 2447442 2447441 2026-08-14T06:12:50Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447442 wikitext text/x-wiki bp8r20ekm8a5dau1opi1hv3kkl5yk7o 2447443 2447442 2026-08-14T06:12:52Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1228.eqiad.wmnet 20.53.64.10.in-addr.arpa 0.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447443 wikitext text/x-wiki twc4xv97334rrjvb4cuyu3c9ltl5cst 2447444 2447443 2026-08-14T06:12:54Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1228.eqiad.wmnet 20.53.64.10.in-addr.arpa 0.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447444 wikitext text/x-wiki sf50bxx0829rxxjsowljbofyy2wpsoe 2447445 2447444 2026-08-14T06:12:56Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1228 2447445 wikitext text/x-wiki e9t5f2rjmveu7tqpk3pxtr9yfnxjsik 2447446 2447445 2026-08-14T06:13:23Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1175.eqiad.wmnet with reason: host reimage 2447446 wikitext text/x-wiki cmu7vl6d6jbp4inwn28exug8hwx3rsc 2447447 2447446 2026-08-14T06:14:10Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1228 2447447 wikitext text/x-wiki l1y53yzpsjxywqe6wr3jqawas44o2bv 2447448 2447447 2026-08-14T06:14:12Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1228 2447448 wikitext text/x-wiki 6ofhmjnhznmplo2rs1sezjoqbxtzq6v 2447449 2447448 2026-08-14T06:14:26Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1209.eqiad.wmnet with reason: host reimage 2447449 wikitext text/x-wiki 12jgqp0ofo72yep1hrl265fdttm4l8l 2447450 2447449 2026-08-14T06:18:09Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1209.eqiad.wmnet with reason: host reimage 2447450 wikitext text/x-wiki 16xch9vtj1mcqbcp9abt4ntqkot7ehg 2447451 2447450 2026-08-14T06:21:40Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1227.eqiad.wmnet with reason: host reimage 2447451 wikitext text/x-wiki godyf85mxhh77tkrrbceffwlmi3pv91 2447452 2447451 2026-08-14T06:25:05Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1227.eqiad.wmnet with reason: host reimage 2447452 wikitext text/x-wiki j504lm376sca0j29f016icad7n9qem0 2447453 2447452 2026-08-14T06:27:56Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1228.eqiad.wmnet with reason: host reimage 2447453 wikitext text/x-wiki nujlwb942vx93l3cyokrbon60w567pg 2447454 2447453 2026-08-14T06:33:12Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1228.eqiad.wmnet with reason: host reimage 2447454 wikitext text/x-wiki 2g9oqa9et4a6ayvtqjssw77e7970ta3 2447455 2447454 2026-08-14T06:36:57Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1175.eqiad.wmnet with OS bookworm 2447455 wikitext text/x-wiki 4mfjwtv8yvl40qqj0d1zxvwkmoz1p89 2447456 2447455 2026-08-14T06:38:11Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1209.eqiad.wmnet with OS bookworm 2447456 wikitext text/x-wiki tvcblidy7yl6iftzr9nubup4n4oqz3l 2447457 2447456 2026-08-14T06:44:50Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1227.eqiad.wmnet with OS bookworm 2447457 wikitext text/x-wiki qyc897o93q83qaclqtqyvybc3vyt3dj 2447458 2447457 2026-08-14T06:53:49Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1228.eqiad.wmnet with OS bookworm 2447458 wikitext text/x-wiki ptmjoje27hl00nhe9m2xxbk3h5k03ut 2447459 2447458 2026-08-14T07:04:31Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kubestagemaster2005.codfw.wmnet with OS trixie 2447459 wikitext text/x-wiki a3cb5l12u464ytecfxrna0ctg9r6xgz 2447460 2447459 2026-08-14T07:23:09Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage 2447460 wikitext text/x-wiki k2iobqjri6zw94pwrjgn47j9z8bvbp9 2447461 2447460 2026-08-14T07:29:22Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage 2447461 wikitext text/x-wiki s1x76jbvplqyw0ofcs959hy34z30dv9 2447462 2447461 2026-08-14T07:39:45Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1152 from dbctl T434480', diff saved to https://phabricator.wikimedia.org/P96090 and previous config saved to /var/cache/conftool/dbconfig/20260814-073941-marostegui.json 2447462 wikitext text/x-wiki paxboaj16l883hci5submggpldq80ln 2447463 2447462 2026-08-14T07:41:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1230.eqiad.wmnet with OS bookworm 2447463 wikitext text/x-wiki cetjmx89ftrddkdokq1iylbup5yhx7c 2447464 2447463 2026-08-14T07:41:51Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1229.eqiad.wmnet with OS bookworm 2447464 wikitext text/x-wiki 179dgzvhm3l75dkdljt1v7dy4zb436j 2447465 2447464 2026-08-14T07:41:57Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1230 2447465 wikitext text/x-wiki 5kcn25nh988rnzmy564cqm2mtmoid46 2447466 2447465 2026-08-14T07:48:06Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2447466 wikitext text/x-wiki 9j8ci8mexhp1lgg4h5ogehdk680vehb 2447467 2447466 2026-08-14T07:51:05Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2005.codfw.wmnet with OS trixie 2447467 wikitext text/x-wiki 4iomg1r78ypp0vxag4laajhmmrymboe 2447468 2447467 2026-08-14T07:55:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1230 - btullis@cumin1003" 2447468 wikitext text/x-wiki eu44pkoqjh8gjzsa69s2ag1oy0wb35g 2447469 2447468 2026-08-14T07:55:38Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1230 - btullis@cumin1003" 2447469 wikitext text/x-wiki fvkwrsekr3yewty45j7cgh4t2nho1iu 2447470 2447469 2026-08-14T07:55:40Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447470 wikitext text/x-wiki l6p6i0rhvxk86unf5yqm5vycao22rn9 2447471 2447470 2026-08-14T07:55:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1230.eqiad.wmnet 23.53.64.10.in-addr.arpa 3.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447471 wikitext text/x-wiki twivcnbglw3tek2ffgggsxciiz3jb3f 2447472 2447471 2026-08-14T07:55:45Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1230.eqiad.wmnet 23.53.64.10.in-addr.arpa 3.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447472 wikitext text/x-wiki to3t1vrfvcvjwppux4lf5ksu47e6fnh 2447473 2447472 2026-08-14T07:55:47Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1230 2447473 wikitext text/x-wiki qt1gz7plb5bcb6l2uno64q6gy2zyu45 2447474 2447473 2026-08-14T07:58:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1230 2447474 wikitext text/x-wiki kq5upbfjp79yx47b8i8rr1lz8qmdcdc 2447475 2447474 2026-08-14T07:58:33Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1230 2447475 wikitext text/x-wiki n9s4ehlw78dl65hc7d88g2tsxk3iavg 2447476 2447475 2026-08-14T07:59:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1229 2447476 wikitext text/x-wiki qnvu6y1dx03tshtmlomwmeqsr97t1oa 2447477 2447476 2026-08-14T08:00:30Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan2002.codfw.wmnet 2447477 wikitext text/x-wiki ldtayfpexr0pzt4gfe6vlu6b0k0wrkh 2447478 2447477 2026-08-14T08:05:16Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2447478 wikitext text/x-wiki fqaj6yw6oykk3no99iwlvogd5hzfcph 2447479 2447478 2026-08-14T08:05:28Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2209: Pool back 2447479 wikitext text/x-wiki cmb7jw5t021tcr9ygo7qhqv3r5k9lae 2447480 2447479 2026-08-14T08:07:59Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan2002.codfw.wmnet 2447480 wikitext text/x-wiki egttf32v4ide0nlm2d64t2c5ihdfs3w 2447481 2447480 2026-08-14T08:10:02Z Stashbot 7414 btullis@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1201.eqiad.wmnet with reason: Fixing a disk 2447481 wikitext text/x-wiki sekzj0p25h9uugim5d9vi98gt1n3by6 2447482 2447481 2026-08-14T08:11:06Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1229 - btullis@cumin1003" 2447482 wikitext text/x-wiki 3nq9e0hp3ths8fj52sofcme3uc98w7d 2447483 2447482 2026-08-14T08:11:11Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1229 - btullis@cumin1003" 2447483 wikitext text/x-wiki 1zktfpyzvx4eidymm8hkxrla5kv89wd 2447484 2447483 2026-08-14T08:11:13Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447484 wikitext text/x-wiki n1lf7mmhtztjtuxnwyew132osyzcilz 2447485 2447484 2026-08-14T08:11:16Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1229.eqiad.wmnet 22.53.64.10.in-addr.arpa 2.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447485 wikitext text/x-wiki oryb5it9sovldzneo7h6usiwzyy1fhx 2447486 2447485 2026-08-14T08:11:18Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1229.eqiad.wmnet 22.53.64.10.in-addr.arpa 2.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447486 wikitext text/x-wiki 1v6w32t14jtfshl3h2l21bh2gku2t70 2447487 2447486 2026-08-14T08:11:20Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1229 2447487 wikitext text/x-wiki b1o4nldspoubc6fi1wvswisu9mch4r2 2447488 2447487 2026-08-14T08:11:44Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1229 2447488 wikitext text/x-wiki l37ubqm4uiso3xvthy297but1y0k9b5 2447489 2447488 2026-08-14T08:11:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1229 2447489 wikitext text/x-wiki 28wqhsfmdri7opmltwqe9is91d6ieap 2447490 2447489 2026-08-14T08:12:35Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1230.eqiad.wmnet with reason: host reimage 2447490 wikitext text/x-wiki 751v8uu2ovhlowtrnemp2snivcmwrii 2447491 2447490 2026-08-14T08:19:13Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1230.eqiad.wmnet with reason: host reimage 2447491 wikitext text/x-wiki pwxi57dp114ibyndll2puy9cnxopc0c 2447492 2447491 2026-08-14T08:21:10Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping1004.eqiad.wmnet 2447492 wikitext text/x-wiki 5hiealwt1pu1l2hnbk3ykdhcihp57k3 2447493 2447492 2026-08-14T08:24:58Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1004.eqiad.wmnet 2447493 wikitext text/x-wiki g71wl0fp83a0w8pzybmym7payur5cg8 2447494 2447493 2026-08-14T08:25:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1229.eqiad.wmnet with reason: host reimage 2447494 wikitext text/x-wiki 18o1fhk5778wq690f5ka43f9ga04cji 2447495 2447494 2026-08-14T08:25:54Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1002.eqiad.wmnet 2447495 wikitext text/x-wiki nm9ru2wvcrfx97a8mglaxuof2gmnpcy 2447496 2447495 2026-08-14T08:28:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1229.eqiad.wmnet with reason: host reimage 2447496 wikitext text/x-wiki 0xkoggg3ibxwambs8war3lxi1m54q22 2447497 2447496 2026-08-14T08:31:42Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan1002.eqiad.wmnet 2447497 wikitext text/x-wiki f6pv3xulk9yi1sy2iiylmj2xzplaytx 2447498 2447497 2026-08-14T08:34:08Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2447498 wikitext text/x-wiki jx668bc1cvk8az8bff6mvx8wzzzchkq 2447499 2447498 2026-08-14T08:34:29Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1151.eqiad.wmnet 2447499 wikitext text/x-wiki lwrmbc3u8ju28g9l0m3997ybuqgo3pz 2447500 2447499 2026-08-14T08:40:29Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2447500 wikitext text/x-wiki p2tpxmt4uro5jdhwbqr7o6nva514u7a 2447501 2447500 2026-08-14T08:40:53Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-worker1230.eqiad.wmnet with OS bookworm 2447501 wikitext text/x-wiki 4xe177o1chrs8b6de79y30ac0u3ftlq 2447502 2447501 2026-08-14T08:43:08Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping2004.codfw.wmnet 2447502 wikitext text/x-wiki s0o8b4qkzkcmzyzcypa950pr872pi4m 2447503 2447502 2026-08-14T08:46:57Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2004.codfw.wmnet 2447503 wikitext text/x-wiki sejmlq1dfnv8z9jqw2sgzwelcungahq 2447504 2447503 2026-08-14T08:47:46Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan2001.codfw.wmnet 2447504 wikitext text/x-wiki ige9xxog47538sphg44tcb87vgmmn1n 2447505 2447504 2026-08-14T08:48:48Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1229.eqiad.wmnet with OS bookworm 2447505 wikitext text/x-wiki bq90nyp215hdni8qxk7yc414uxngkw7 2447506 2447505 2026-08-14T08:50:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.remove-downtime for an-worker1201.eqiad.wmnet 2447506 wikitext text/x-wiki j19kzhumo8zxm0qrp5uzlpc2y8zln3c 2447507 2447506 2026-08-14T08:51:06Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for an-worker1201.eqiad.wmnet 2447507 wikitext text/x-wiki ppnxs3bpcl4ce95m4nra7zie5dnivcj 2447508 2447507 2026-08-14T08:51:08Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pool back 2447508 wikitext text/x-wiki tazr3npcp9mtz5mc3q9o95gscskajab 2447509 2447508 2026-08-14T08:53:36Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1152.eqiad.wmnet with OS bookworm 2447509 wikitext text/x-wiki 5leoflg5ago52qp517g2j0mqqu3giek 2447510 2447509 2026-08-14T08:53:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1231.eqiad.wmnet with OS bookworm 2447510 wikitext text/x-wiki ercmj4zy9xfxnxf5u0rbuiqd5gyj79y 2447511 2447510 2026-08-14T08:53:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1232.eqiad.wmnet with OS bookworm 2447511 wikitext text/x-wiki jz8352b5xmkv05f2zvv74xgnbgm0bj0 2447512 2447511 2026-08-14T08:54:03Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1152 2447512 wikitext text/x-wiki pgdsko1i41cmlm99u9qtxtoviyu2zfa 2447513 2447512 2026-08-14T08:54:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1151.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2447513 wikitext text/x-wiki 6bb7xkbnwm6pvnv36gyn1iftt70zh9j 2447514 2447513 2026-08-14T08:55:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2447514 wikitext text/x-wiki h41f2vojzgcpgxhhlncabuwlq3lv3me 2447515 2447514 2026-08-14T08:55:45Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan2001.codfw.wmnet 2447515 wikitext text/x-wiki etudvcccudvuwj8qowrtfqjmhu03alc 2447516 2447515 2026-08-14T09:03:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1151.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2447516 wikitext text/x-wiki tqwfle6vu37ksm2ee6vclsz3dws1ilq 2447517 2447516 2026-08-14T09:03:13Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447517 wikitext text/x-wiki geqqopx5ifed181p33wuozw6xry4n08 2447518 2447517 2026-08-14T09:03:15Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1151.eqiad.wmnet 2447518 wikitext text/x-wiki sgq1b8o4mqobfj2yg59u59tkuopot2e 2447519 2447518 2026-08-14T09:06:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1152 - btullis@cumin1003" 2447519 wikitext text/x-wiki s2itlwbkmecqhamfm9it496q7yr8ukc 2447520 2447519 2026-08-14T09:06:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1152 - btullis@cumin1003" 2447520 wikitext text/x-wiki 5qc2sprxbx7i1w9go9pqbv01z650v3b 2447521 2447520 2026-08-14T09:06:48Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447521 wikitext text/x-wiki s6viezlduz2lzws4scgzo9fa1pzo46f 2447522 2447521 2026-08-14T09:06:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1152.eqiad.wmnet 16.53.64.10.in-addr.arpa 6.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447522 wikitext text/x-wiki dfsmwj99bw2rajynw28wmkghsumzp2j 2447523 2447522 2026-08-14T09:06:52Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1152.eqiad.wmnet 16.53.64.10.in-addr.arpa 6.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447523 wikitext text/x-wiki oqzcgno59uoyj7xrw70gdpc9a5clm1t 2447524 2447523 2026-08-14T09:06:55Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1152 2447524 wikitext text/x-wiki snzx3elwtp2bqjojhcgfev0d5rqsbn5 2447525 2447524 2026-08-14T09:08:24Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1152 2447525 wikitext text/x-wiki gue8k5yr673xch4vs32ya0kmbbza5r0 2447526 2447525 2026-08-14T09:08:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1152 2447526 wikitext text/x-wiki qvt96y9vf9b3q6fg9ulbyw1isg2jgmi 2447527 2447526 2026-08-14T09:08:29Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1231 2447527 wikitext text/x-wiki k84k8s62fz740llbaeanq2f1x6g7ifq 2447528 2447527 2026-08-14T09:09:34Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2447528 wikitext text/x-wiki 134s59v5b7xk3e7rdu4no4s4ylo2bon 2447529 2447528 2026-08-14T09:11:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org 2447529 wikitext text/x-wiki 5wb2o8l6ofptbr8jsgfay521u5n61fd 2447530 2447529 2026-08-14T09:14:20Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1231 - btullis@cumin1003" 2447530 wikitext text/x-wiki 1d0n84qrdpha180cmkfagy0pywj148t 2447531 2447530 2026-08-14T09:14:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1231 - btullis@cumin1003" 2447531 wikitext text/x-wiki ievv7e9tnfbyfg6f69e2uja0haz1wtl 2447532 2447531 2026-08-14T09:14:27Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447532 wikitext text/x-wiki 9dko70en8fm9nh5wdrtgzgmcohyqo9o 2447533 2447532 2026-08-14T09:14:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1231.eqiad.wmnet 24.53.64.10.in-addr.arpa 4.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447533 wikitext text/x-wiki 637van1fi8on4uapcikntov3re44i9u 2447534 2447533 2026-08-14T09:14:32Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1001.eqiad.wmnet 2447534 wikitext text/x-wiki hk7fsxgwussvf9ccfjuu6kf9rzx4fn4 2447535 2447534 2026-08-14T09:14:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1231.eqiad.wmnet 24.53.64.10.in-addr.arpa 4.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447535 wikitext text/x-wiki ltw2gj399d1ntxnvp8oud5901hz71m4 2447536 2447535 2026-08-14T09:14:37Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1231 2447536 wikitext text/x-wiki o2j3gd1yf7vyu7od75clos3glk2l5f5 2447537 2447536 2026-08-14T09:16:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1231 2447537 wikitext text/x-wiki ogwj1zq8x33ow91z3z3vcr7h5ua8law 2447538 2447537 2026-08-14T09:16:45Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1231 2447538 wikitext text/x-wiki q8e2u9kx56lyc2ddot5e280rt3kg1ei 2447539 2447538 2026-08-14T09:17:16Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1232 2447539 wikitext text/x-wiki lr8c6ql6rs81auyksc239y2n6hymk33 2447540 2447539 2026-08-14T09:17:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org 2447540 wikitext text/x-wiki szpf1oxj2yl50y04rmmrtogfc7vk602 2447541 2447540 2026-08-14T09:17:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2447541 wikitext text/x-wiki 229uw9d0i79xjpjmf7ewk951xj3mein 2447542 2447541 2026-08-14T09:22:06Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1232 - btullis@cumin1003" 2447542 wikitext text/x-wiki pdprm1omiqgxa5bqhvq993klu03yqbt 2447543 2447542 2026-08-14T09:22:11Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1232 - btullis@cumin1003" 2447543 wikitext text/x-wiki s3329idjeax352ly6c06lsyh38mwp6p 2447544 2447543 2026-08-14T09:22:14Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447544 wikitext text/x-wiki sns71kkqwczupzk3dhjmyjua2a3zi2v 2447545 2447544 2026-08-14T09:22:16Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1232.eqiad.wmnet 25.53.64.10.in-addr.arpa 5.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447545 wikitext text/x-wiki 4buiuajdzsqfcjobu8pj0nw714csq4s 2447546 2447545 2026-08-14T09:22:19Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan1001.eqiad.wmnet 2447546 wikitext text/x-wiki eezfyc7max720399zl5kyy9l5t73wdn 2447547 2447546 2026-08-14T09:22:21Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1232.eqiad.wmnet 25.53.64.10.in-addr.arpa 5.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2447547 wikitext text/x-wiki 9huux77hpi7qlonb9u1lc975ax3jwph 2447548 2447547 2026-08-14T09:22:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1232 2447548 wikitext text/x-wiki rvroi19b9yvqkiyyslne68f0iygntg1 2447549 2447548 2026-08-14T09:22:39Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1232 2447549 wikitext text/x-wiki ofqoiivfstiuwk4pgigeg7p3jzn8ag5 2447550 2447549 2026-08-14T09:22:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1232 2447550 wikitext text/x-wiki s01fh4nb3fidnqaykc0n2be4iyutn16 2447551 2447550 2026-08-14T09:24:05Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org 2447551 wikitext text/x-wiki boj3tu6736wmcsibkysd9bg81qpsg2h 2447552 2447551 2026-08-14T09:25:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1152.eqiad.wmnet with reason: host reimage 2447552 wikitext text/x-wiki h21zcd0bar8rqynu44pxl8mcxitvrgj 2447553 2447552 2026-08-14T09:25:34Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host centrallog1002.eqiad.wmnet 2447553 wikitext text/x-wiki pbv1shpamcf0zj8wocym6qlx7yz4qfe 2447554 2447553 2026-08-14T09:28:01Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1152.eqiad.wmnet with reason: host reimage 2447554 wikitext text/x-wiki ss264q3fhq3delbjd1219ayc3rk5vn2 2447555 2447554 2026-08-14T09:30:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1231.eqiad.wmnet with reason: host reimage 2447555 wikitext text/x-wiki pgh4hoxr4dkuut24xyfp3562sv5g764 2447556 2447555 2026-08-14T09:30:46Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org 2447556 wikitext text/x-wiki jzqxf8gjsqobphwdo1bull02qzlfbe8 2447563 2447556 2026-08-14T09:32:31Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog1002.eqiad.wmnet 2447563 wikitext text/x-wiki oou6o0j637yf62m7e6alm9dzumy5dmq 2447565 2447563 2026-08-14T09:33:20Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org 2447565 wikitext text/x-wiki lh8jzmhf3q784aefb6t6b53yzfkru7h 2447566 2447565 2026-08-14T09:33:55Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1231.eqiad.wmnet with reason: host reimage 2447566 wikitext text/x-wiki tqg2srqbm9qql7ew91d673joutwabfa 2447567 2447566 2026-08-14T09:34:43Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2447567 wikitext text/x-wiki 0l6sksphwlh2pj56oxs0av1v7rbl6vl 2447568 2447567 2026-08-14T09:36:48Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2447568 wikitext text/x-wiki gf3wjjxln47s85bsj585n84pmod5ppu 2447569 2447568 2026-08-14T09:36:57Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1232.eqiad.wmnet with reason: host reimage 2447569 wikitext text/x-wiki 6q9szqao0c2ghy48ethbpk5hvdt3a15 2447570 2447569 2026-08-14T09:39:57Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org 2447570 wikitext text/x-wiki m10z4aiokgunypqci3oo0wsys68f7sa 2447572 2447570 2026-08-14T09:41:33Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host centrallog2002.codfw.wmnet 2447572 wikitext text/x-wiki 3yspjhzc5ym1tow6ckf7235z10wwzoh 2447573 2447572 2026-08-14T09:41:49Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org 2447573 wikitext text/x-wiki nrpaw5n9c1h0f59r9gfxssytimhot5f 2447574 2447573 2026-08-14T09:42:44Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1232.eqiad.wmnet with reason: host reimage 2447574 wikitext text/x-wiki q0dow0p1jzlwt9cr16hs4grhlc8sdel 2447576 2447574 2026-08-14T09:48:22Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org 2447576 wikitext text/x-wiki mpkesefr0iaqc73dlh8iw040av0t2dk 2447577 2447576 2026-08-14T09:48:24Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog2002.codfw.wmnet 2447577 wikitext text/x-wiki hjb6rkuezqm8x0pou7jx9b77asuk2it 2447578 2447577 2026-08-14T09:49:16Z Stashbot 7414 Dreamy_Jazz: `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260808000000" --end-timestamp="20260812120000" --sleep="5" --batch-size="50"` for T434688 2447578 wikitext text/x-wiki 2o7t72nc7cedlj4xmo7cxr7xr81d0bd 2447579 2447578 2026-08-14T09:50:10Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1152.eqiad.wmnet with OS bookworm 2447579 wikitext text/x-wiki 1drlbaru1s4ysxze8qn7x3qtns0juro 2447580 2447579 2026-08-14T09:53:23Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org 2447580 wikitext text/x-wiki bfwdeedb1coto6q7l29dp0etxtozv97 2447581 2447580 2026-08-14T09:53:55Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2447581 wikitext text/x-wiki m676crfpgjgqw5zaqwhcwjweg8jkp3b 2447583 2447581 2026-08-14T09:56:38Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2447583 wikitext text/x-wiki gtqkrajblim79ny66s3gq6wa6ih10pg 2447584 2447583 2026-08-14T09:56:40Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1231.eqiad.wmnet with OS bookworm 2447584 wikitext text/x-wiki 7u3zl0sbraf30r6n5powtqf94b8j991 2447585 2447584 2026-08-14T09:58:41Z Stashbot 7414 fceratto@cumin1003: Removing db1151 from zarcillo T434538 2447585 wikitext text/x-wiki m8vbq4d631lfgrmq8hbaehsaetf4kr8 2447586 2447585 2026-08-14T09:58:44Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2447586 wikitext text/x-wiki 2x1gy4baim8wj6d8jrpyawqx1j0ndmf 2447587 2447586 2026-08-14T10:00:02Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org 2447587 wikitext text/x-wiki 0gv9l1uryquyliwxot81hu8soop001k 2447588 2447587 2026-08-14T10:00:46Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org 2447588 wikitext text/x-wiki egxaxmgugngxyvhvyvsms3d8qk0inh7 2447589 2447588 2026-08-14T10:02:21Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1232.eqiad.wmnet with OS bookworm 2447589 wikitext text/x-wiki i3z5h5vd4b1g3tqd5f0m0gjietxiy2n 2447590 2447589 2026-08-14T10:03:32Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet 2447590 wikitext text/x-wiki k1h8vk9gdrk38wr5kzx8f5z8vdi8szl 2447591 2447590 2026-08-14T10:03:35Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2447591 wikitext text/x-wiki ousc2by56g1zxwfbgcqk33vyd2fp3rw 2447594 2447591 2026-08-14T10:07:21Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org 2447594 wikitext text/x-wiki 7gdg3exjp8mler7i6p3dj9wtamc26gs 2447595 2447594 2026-08-14T10:07:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1153.eqiad.wmnet with OS bookworm 2447595 wikitext text/x-wiki 2d91oqqwgktqo18fi3tqj66kon6tgd3 2447596 2447595 2026-08-14T10:07:26Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1181.eqiad.wmnet with OS bookworm 2447596 wikitext text/x-wiki eqdnifwdlnjpqiiu32eom4ltb43yr4x 2447597 2447596 2026-08-14T10:07:29Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1187.eqiad.wmnet with OS bookworm 2447597 wikitext text/x-wiki 06od6scw490sohrorbcld9t0uddujgk 2447598 2447597 2026-08-14T10:07:34Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1235.eqiad.wmnet with OS bookworm 2447598 wikitext text/x-wiki 9yxu17ce0ylt3e0czablolqbu3irjpd 2447601 2447598 2026-08-14T10:09:28Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org 2447601 wikitext text/x-wiki 3i1tlkas8uivob3bj8jjo7j176e1506 2447603 2447601 2026-08-14T10:12:02Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2447603 wikitext text/x-wiki mqb73ik3oehd6fg8yrnha0j3so24po2 2447604 2447603 2026-08-14T10:12:07Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2447604 wikitext text/x-wiki dqn6l3uu6c3jl4dx4ko61z8u8ra29ft 2447605 2447604 2026-08-14T10:12:09Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2447605 wikitext text/x-wiki m1rmbvnrmt1djibmgrm1h4k4ijowun6 2447606 2447605 2026-08-14T10:12:11Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2447606 wikitext text/x-wiki hxc7m1wuol2993lz11bmhihwhpud1mh 2447607 2447606 2026-08-14T10:12:13Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2447607 wikitext text/x-wiki qbfsoggjwqk8aod533lps3vol3n16o9 2447608 2447607 2026-08-14T10:12:41Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2447608 wikitext text/x-wiki 8hngk53v7dsrz0vlgedqfmn1ea8c47f 2447609 2447608 2026-08-14T10:12:45Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2447609 wikitext text/x-wiki hn2p41di6m87juq9j0hd3iqdh1w27s3 2447613 2447609 2026-08-14T10:16:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org 2447613 wikitext text/x-wiki 9ny04lwtdaav1xhsdd36p7h0u7pkh8d 2447614 2447613 2026-08-14T10:16:54Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock2003.codfw.wmnet with OS trixie 2447614 wikitext text/x-wiki 2q0c4yz5u72qwiblvsz83wu2fwzggq1 2447615 2447614 2026-08-14T10:21:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1187.eqiad.wmnet with reason: host reimage 2447615 wikitext text/x-wiki knxcll7eg6j2vv8e2ftwwiaf3qs3a7j 2447616 2447615 2026-08-14T10:23:48Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1235.eqiad.wmnet with reason: host reimage 2447616 wikitext text/x-wiki cas0qizs6aaqe4dhofeulb8csmfshr2 2447617 2447616 2026-08-14T10:24:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1153.eqiad.wmnet with reason: host reimage 2447617 wikitext text/x-wiki 6j4r29ljujxlcxkijqixrzdi4s10v52 2447618 2447617 2026-08-14T10:29:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1187.eqiad.wmnet with reason: host reimage 2447618 wikitext text/x-wiki hx9jl9e679htc5s07fsv5q12mda74kh 2447619 2447618 2026-08-14T10:32:56Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1235.eqiad.wmnet with reason: host reimage 2447619 wikitext text/x-wiki b6eucy1rperlwiwfld2n0l0pi9h6id4 2447620 2447619 2026-08-14T10:35:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1153.eqiad.wmnet with reason: host reimage 2447620 wikitext text/x-wiki l5mi11gtvx699nwo6i2sgozj2fv3v1v 2447621 2447620 2026-08-14T10:38:20Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock2003.codfw.wmnet with reason: host reimage 2447621 wikitext text/x-wiki gkw8duvq7y9qiiyso37gylomnndlrtk 2447622 2447621 2026-08-14T10:44:02Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock2003.codfw.wmnet with reason: host reimage 2447622 wikitext text/x-wiki 806qjtw40kbuhju84fmycu7xzufeale 2447623 2447622 2026-08-14T10:44:38Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw 2447623 wikitext text/x-wiki iw8048hzajzoiyaebe4dr4gm12ufkyl 2447624 2447623 2026-08-14T10:44:42Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet 2447624 wikitext text/x-wiki gmrnmfhpb9s9j2p818cluyx5ngmwe80 2447625 2447624 2026-08-14T10:45:17Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet 2447625 wikitext text/x-wiki d2uoiggnsle22sxnkruswygwo26l5q9 2447626 2447625 2026-08-14T10:49:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1187.eqiad.wmnet with OS bookworm 2447626 wikitext text/x-wiki lo3bv3k2dyab7swxj5dyj8prqdsg4t1 2447627 2447626 2026-08-14T10:50:31Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T434668 2447627 wikitext text/x-wiki r0i4kaz50vmeply4o5rs58qtuhqfhb6 2447628 2447627 2026-08-14T10:51:39Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet 2447628 wikitext text/x-wiki 4tngngl0tj3ny5cw8rw0j0frmyxcd40 2447629 2447628 2026-08-14T10:51:42Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet 2447629 wikitext text/x-wiki hpjwxye7il1h28ss1kwc1bwch8byesa 2447630 2447629 2026-08-14T10:51:46Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet 2447630 wikitext text/x-wiki s4njjetwc385kia76yzushwh0xt85mh 2447631 2447630 2026-08-14T10:54:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1153.eqiad.wmnet with OS bookworm 2447631 wikitext text/x-wiki e43nwed87gm8d5kohpvb1q5nsgksnt5 2447632 2447631 2026-08-14T10:56:47Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet 2447632 wikitext text/x-wiki rnlkpsypnir9cr2byowh8gkglvddy7u 2447633 2447632 2026-08-14T10:57:10Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock2003.codfw.wmnet with OS trixie 2447633 wikitext text/x-wiki mwsh5cj81etk78qz9eu3idafyb3tixq 2447634 2447633 2026-08-14T10:57:12Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock2003.codfw.wmnet 2447634 wikitext text/x-wiki 8p6ietzv86379ehn6i4juc1qyd0tiio 2447635 2447634 2026-08-14T10:58:14Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1235.eqiad.wmnet with OS bookworm 2447635 wikitext text/x-wiki pw62njbilhioid6z8cnh4da6aqkmx1g 2447636 2447635 2026-08-14T11:02:55Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet 2447636 wikitext text/x-wiki p48dvj79bd3c30ksaksp5ppjf23akb2 2447637 2447636 2026-08-14T11:02:57Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet 2447637 wikitext text/x-wiki jb0yjwfsru0u26dnzccqudc0p54i6bc 2447638 2447637 2026-08-14T11:03:01Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet 2447638 wikitext text/x-wiki pwj1qg2xqw0t2qq3djkf0fwr6um5475 2447639 2447638 2026-08-14T11:03:32Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet 2447639 wikitext text/x-wiki swadvi6rz3yz4nn79qae1kj7e16uphc 2447640 2447639 2026-08-14T11:03:53Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1181.eqiad.wmnet with OS bookworm 2447640 wikitext text/x-wiki 3uni3kxiou97uzmkkaxip0co5k8bznj 2447642 2447640 2026-08-14T11:10:01Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet 2447642 wikitext text/x-wiki 7oanteht71u6tjy66iai503vtb1dupg 2447643 2447642 2026-08-14T11:10:03Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet 2447643 wikitext text/x-wiki agcex0abz5hokv17qpwfx90gfk961a8 2447644 2447643 2026-08-14T11:10:08Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet 2447644 wikitext text/x-wiki epzsdx5ezqeik5cjq4jx54q9128h5iv 2447645 2447644 2026-08-14T11:10:43Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet 2447645 wikitext text/x-wiki kslw1v88xw7ud6sme4povg3qsnylbtv 2447646 2447645 2026-08-14T11:17:11Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet 2447646 wikitext text/x-wiki 7rj7b1bv0u0canlbe5cq4e00dtc5pac 2447647 2447646 2026-08-14T11:17:13Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet 2447647 wikitext text/x-wiki c19y1r9q2myqhlur6z89hl4cdc0c7d8 2447648 2447647 2026-08-14T11:17:16Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw 2447648 wikitext text/x-wiki mufwbojlw00ry4c2xxacduttnw8nm9f 2447649 2447648 2026-08-14T11:20:09Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T434668 2447649 wikitext text/x-wiki gkeatq0s9neqmki5fwrtcqz68qkuazz 2447650 2447649 2026-08-14T11:21:01Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org 2447650 wikitext text/x-wiki i96d67879jui76y8zpkp6n1wt6mdzzp 2447652 2447650 2026-08-14T11:24:54Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1153 from dbctl T434638', diff saved to https://phabricator.wikimedia.org/P96097 and previous config saved to /var/cache/conftool/dbconfig/20260814-112449-marostegui.json 2447652 wikitext text/x-wiki cllz7wnkgxa7z4vfq5gh2q5t42vtgrr 2447655 2447652 2026-08-14T11:27:38Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org 2447655 wikitext text/x-wiki n9evt2pmjmyrynp2rjy007s1v6spd17 2447659 2447655 2026-08-14T11:56:26Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host cuminunpriv1001.eqiad.wmnet 2447659 wikitext text/x-wiki 5rtzed7f4b2xsytd9shk97b3jml2bdo 498 14256 2447087 2443318 2026-08-13T17:08:46Z Stashbot 7414 Reedy: restart irc/znc 2447087 wikitext text/x-wiki jyesoulm0k59xsoa08maxl9fvsh8592 0 17290 2447220 2446443 2026-08-13T19:27:56Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (T401115) 2447220 wikitext text/x-wiki 1ug2jxofctvs1o26om5xm5v4w9l5d8j 2447233 2447220 2026-08-13T19:55:06Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (T401115) (redux for typo fix) 2447233 wikitext text/x-wiki qdefks6cei1e91bo6r4jsjcdgj1j2eb 0 22777 2447110 2445867 2026-08-13T17:38:36Z BKing (WMF) 29017 /* Failed hardware node */ 2447110 wikitext text/x-wiki r1t2n1cselnmoae3dzsvo80p05jzh48 2447263 2447110 2026-08-13T21:11:50Z BKing (WMF) 29017 /* VMs with DRBD-backed storage */ 2447263 wikitext text/x-wiki fpr7xv9i6cg5325ejd5t3hyle2d57ez 2447268 2447263 2026-08-13T21:23:02Z BKing (WMF) 29017 /* Failed hardware node */ 2447268 wikitext text/x-wiki 5t27lmck1o4a6xdac3fn2mr94h8e5b1 0 23934 2447285 2412938 2026-08-13T21:48:03Z BKing (WMF) 29017 /* Adding a new member to the cluster */ updated syntax 2447285 wikitext text/x-wiki a7wg5vf9gdy1rvgiiprjwuh7ukhcioo 498 30942 2446726 2446612 2026-08-13T12:23:25Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2446726 wikitext text/x-wiki axxl0umf5ki0r5236itv624td4af4iq 2446740 2446726 2026-08-13T12:34:27Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) 2446740 wikitext text/x-wiki p160lz9roy0iq6rv9h8m25stez3j25s 2447019 2446740 2026-08-13T16:00:27Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch 2447019 wikitext text/x-wiki olo92lpbf22okdu3894kshvna0bldqf 2447020 2447019 2026-08-13T16:00:59Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch 2447020 wikitext text/x-wiki 8jrkmoa7i1ftwcfk07du6fjar6vz3r7 2447021 2447020 2026-08-13T16:01:36Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2447021 wikitext text/x-wiki 85mxd8udlec0mswvdgzylw83lyrtnil 2447022 2447021 2026-08-13T16:02:17Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2447022 wikitext text/x-wiki 6j9zkqn1pmpv83i6vejlecehxe9ip32 2447025 2447022 2026-08-13T16:05:51Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2447025 wikitext text/x-wiki 4y7oeozj7p3rho5bjd5phkfhhmb2hft 2447026 2447025 2026-08-13T16:06:53Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2447026 wikitext text/x-wiki dhq08wd7ckvueka47085922ndkpvdsq 0 444551 2447641 2374192 2026-08-14T11:07:54Z Cathalmooney100 58598 /* JSON-RPC */ 2447641 wikitext text/x-wiki d0dhq920g5c9g9ab5l3xks0920e6350 2447651 2447641 2026-08-14T11:23:05Z Cathalmooney100 58598 /* JSON-RPC */ 2447651 wikitext text/x-wiki 1zmpbj9f7eesa445fr7z9u73m4ncebg 2447653 2447651 2026-08-14T11:25:17Z Cathalmooney100 58598 /* Option 2: Running Homer from your local machine (less recommended) */ 2447653 wikitext text/x-wiki oqancotjgjav39kxyuv9y9bzhojxvtv 2447654 2447653 2026-08-14T11:25:46Z Cathalmooney100 58598 /* Option 2: Running Homer from your local machine (less recommended) */ 2447654 wikitext text/x-wiki a2qs7qwxyaugoa4jjakt09f947cfuvp 2447656 2447654 2026-08-14T11:36:04Z Cathalmooney100 58598 /* JSON-RPC */ 2447656 wikitext text/x-wiki gwbqv8b7rc6a1buikr8mwhx7l70orn7 0 444622 2447015 2446214 2026-08-13T15:51:41Z Andrewbogott 33 /* Upgrade ingress nodes */ 2447015 wikitext text/x-wiki qbrlutp8le5p6e3imzgv5ze6plmp7xb 0 447343 2447582 2420681 2026-08-14T09:56:19Z ABran-WMF 37196 tunnelencabulator docs 2447582 wikitext text/x-wiki pl45alt2owauddp1ipobwvskdbfbaik 498 448635 2447657 2443551 2026-08-14T11:53:27Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'taavi' in role 'member' 2447657 wikitext text/x-wiki glh4omc0eu3tl5tgpr1nxqkeljveavc 2447658 2447657 2026-08-14T11:53:32Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'taavi' in role 'member' 2447658 wikitext text/x-wiki rxvv8i9tnezfltnggnqsisexnm4rb8v 0 448997 2447398 2434595 2026-08-14T02:05:17Z CStone (WMF) 38439 /* Deploying to payments/donorwiki */ 2447398 wikitext text/x-wiki aeiucjkaeh6nif343sju20oit48hykm 0 449160 2446997 2446527 2026-08-13T15:38:42Z Dexbot 30554 Bot: Updating the report 2446997 wikitext text/x-wiki jy3hwvzmq6nmo2d1ha1itwdu9w534u8 2447360 2446997 2026-08-14T00:01:00Z Dexbot 30554 Bot: Updating the report 2447360 wikitext text/x-wiki 06t75b1919xvrhq3rl8vncowk240g48 2 452734 2447030 2293689 2026-08-13T16:09:30Z TNTBot 39092 Copying content of Deployments for testing 2447030 wikitext text/x-wiki e25psg08gw62vuhlj6x1ho5iksfxqpg 2447031 2447030 2026-08-13T16:10:50Z TNTBot 39092 (testing) [[User:TNTBot#Updating_backport_window_deployment_statuses|Automated task]]: Updating deployment statuses [t:1/u:1/l:60] (section: deploycal-item-20260810T1300) 2447031 wikitext text/x-wiki ncc3sv61s76tl7o3wy35gaz44mopaoy 2447032 2447031 2026-08-13T16:13:00Z TNTBot 39092 (testing) [[User:TNTBot#Updating_backport_window_deployment_statuses|Automated task]]: Updating deployment statuses [t:6/u:5/l:60] (section: Monday,_August_10) 2447032 wikitext text/x-wiki pyycrl2nh629wnpjohlknn1l6v2u9kh 2447038 2447032 2026-08-13T16:18:58Z TNTBot 39092 (testing) [[User:TNTBot#Updating_backport_window_deployment_statuses|Automated task]]: Updating deployment statuses [t:29/u:1/l:1] 2447038 wikitext text/x-wiki 976aqsput4msjx79d08osew2hei0ah8 2447046 2447038 2026-08-13T16:22:18Z TNTBot 39092 (testing) [[User:TNTBot#Updating_backport_window_deployment_statuses|Automated task]]: Updating deployment statuses [t:29/u:20/l:60] 2447046 wikitext text/x-wiki kpwoxjsg39063mv1tua3og1cb7zx6i0 0 453597 2447076 2446220 2026-08-13T16:55:22Z DCaro (WMF) 18271 /* 2026 */ added note about builds cli 2447076 wikitext text/x-wiki b5sjf9wbs5sshqk27kaveq96k9lnr35 13 453703 2447400 2168092 2026-08-14T03:20:20Z Ancient9983 52221 /* Highlight this functionality more in the Toolforge documentation? */ new section 2447400 wikitext text/x-wiki mhrm5si8l00a6oh6id5t8a8stdlxkln 2447401 2447400 2026-08-14T03:22:31Z Ancient9983 52221 /* Highlight this functionality more in the Toolforge documentation? */ Reply 2447401 wikitext text/x-wiki 9va99xr08ao1ev4imksv298anpj48vi 2 458669 2447035 2292072 2026-08-13T16:18:54Z TNTBot 39092 Logging message 2447035 wikitext text/x-wiki bmtumb8eauelyb8m7bn6fn6paplpqbf 2447036 2447035 2026-08-13T16:18:55Z TNTBot 39092 Logging message 2447036 wikitext text/x-wiki jf5eaxfcsaz9btrm08a4ctic8tjmge7 2447037 2447036 2026-08-13T16:18:58Z TNTBot 39092 Logging message 2447037 wikitext text/x-wiki kzul3bgzoqfn6h23e93el2vsxa5v76s 2447039 2447037 2026-08-13T16:18:59Z TNTBot 39092 Logging message 2447039 wikitext text/x-wiki enr0zvee2uui62f83d3lt6ahl3lp0qb 2447040 2447039 2026-08-13T16:18:59Z TNTBot 39092 Logging message 2447040 wikitext text/x-wiki 4hvxfzsqiij8lqro5cyfz2d0vro10c1 2447042 2447040 2026-08-13T16:21:46Z TNTBot 39092 Logging message 2447042 wikitext text/x-wiki fc7uyt4rn8iqtpa3c96req3rq239ehx 2447043 2447042 2026-08-13T16:21:46Z TNTBot 39092 Logging message 2447043 wikitext text/x-wiki o88wk3q2qioi6n6s7xcs3vk2cfmyevm 2447045 2447043 2026-08-13T16:22:18Z TNTBot 39092 Logging message 2447045 wikitext text/x-wiki 9cyu3oyy6inja7wc7rl79pn32xgg3nv 2447047 2447045 2026-08-13T16:22:20Z TNTBot 39092 Logging message 2447047 wikitext text/x-wiki 0dtbu8ntom9mw6p21y9ohv9u92569p2 2447048 2447047 2026-08-13T16:22:20Z TNTBot 39092 Logging message 2447048 wikitext text/x-wiki m8a45fx5tjbcb3aq3lhc58zyh50qt5i 0 459699 2447339 2446438 2026-08-13T23:08:43Z Stashbot 7414 larssandergreen: civicrm upgraded from d5420716 to 105395c3 2447339 wikitext text/x-wiki cdp0b1hoeslcl4f5kyl9y5crjhm6onb 2447396 2447339 2026-08-14T01:58:02Z Stashbot 7414 danielfm: donorwiki upgraded from 80e752f7 to a06ae998 2447396 wikitext text/x-wiki 0bdi5jq6d342swu0nlqtb1yz9w01cws 0 459903 2446710 2435556 2026-08-13T12:18:06Z BJensen-WMF 48942 Add new version deployment instructions. 2446710 wikitext text/x-wiki rq4o49ydkh8x15ke0r0w2jzm73ywyvr 0 460330 2447560 2433132 2026-08-14T09:31:50Z Taavi 13997 Taavi moved page [[User:Taavi/cloud-announce]] to [[Portal:Cloud Services/Admin/cloud-announce]] 2433132 wikitext text/x-wiki 1ipyllj0jaagwpztgtioeysv4yskr1y 0 460515 2447007 2445013 2026-08-13T15:45:07Z Ottomata 88 /* 6. Update weighted tags for page discovery (optional) */ 2447007 wikitext text/x-wiki jaa1417buoa26q3k982d8dbbaclv085 2447229 2447007 2026-08-13T19:52:01Z Ottomata 88 Ottomata moved page [[User:Ottomata/Derived Data Arch Menu]] to [[Prep Pantry/Architectures]]: Promoting out of user space, final name still TBD. 2447007 wikitext text/x-wiki jaa1417buoa26q3k982d8dbbaclv085 2447238 2447229 2026-08-13T20:17:40Z Ottomata 88 2447238 wikitext text/x-wiki jc5ohav6qhv5oohiq84cf2wfs3grlke 2447571 2447238 2026-08-14T09:40:55Z GGoncalves-WMF 42848 Remove draft template and slightly simplify the lead. 2447571 wikitext text/x-wiki 8izdaolhkiqe8fihgjds819e1e6689m 0 460534 2447558 2445012 2026-08-14T09:31:34Z Taavi 13997 Taavi moved page [[User:Taavi/Deprecation playbook]] to [[Portal:Cloud Services/Admin/Deprecation playbook]] 2445012 wikitext text/x-wiki efzo142w8dzfcehrel0oth0195oznx7 2 460540 2446784 2446305 2026-08-13T13:19:08Z Ottomata 88 /* Access Options */ 2446784 wikitext text/x-wiki 04pvdt5y5fz7bzljdvsmccj1pd70pcx 2446788 2446784 2026-08-13T13:19:47Z Ottomata 88 /* Access Options */ 2446788 wikitext text/x-wiki 5sippoqzukkbwo3rmt9omhps4ygg5nq 2446891 2446788 2026-08-13T14:11:34Z Ottomata 88 added use caes section count delta TTS regeneration 2446891 wikitext text/x-wiki 7tla6y90i0dh0kr252rkhzxnxfhbufa 2446916 2446891 2026-08-13T14:25:43Z Ottomata 88 Added consultation highlights section 2446916 wikitext text/x-wiki o4rbafjdnxsjpyrplr8arlc11lq3jij 2446917 2446916 2026-08-13T14:25:52Z Ottomata 88 /* TTS dataset building use case: */ 2446917 wikitext text/x-wiki ovbqvlpqtkuawe7nda7zwqm4tqf4j1z 3 460549 2446952 2026-08-13T14:57:33Z StrikerBot 8475 Welcome to Toolforge! 2446952 wikitext text/x-wiki oh1ykv52oy1maqur2iuptm8qb1o1p1z 2 460550 2447230 2026-08-13T19:52:01Z Ottomata 88 Ottomata moved page [[User:Ottomata/Derived Data Arch Menu]] to [[Prep Pantry/Architectures]]: Promoting out of user space, final name still TBD. 2447230 wikitext text/x-wiki d4jdx78uehalacsr9f6s2ggrmdmhv7y 0 460551 2447557 2026-08-14T09:31:12Z Taavi 13997 Created page with "Generic administrative documentation about '''Cloud Services products''' that does not fit under any of the established subproducts: * [[Portal:Cloud VPS/Admin]] * [[Portal:Data Services/Admin]] * [[Portal:Toolforge/Admin]] == All subpages == {{Special:Prefixindex|prefix={{FULLPAGENAME}}/|hideredirects=1|stripprefix=1}} [[Category:Cloud Services admin]]" 2447557 wikitext text/x-wiki c5xqolecs6a97d04lwj6rd6s9tnn9cl 2 460552 2447559 2026-08-14T09:31:34Z Taavi 13997 Taavi moved page [[User:Taavi/Deprecation playbook]] to [[Portal:Cloud Services/Admin/Deprecation playbook]] 2447559 wikitext text/x-wiki lkk7o4lji8iuz0jgni9f4nrvjgvca6u 2 460553 2447561 2026-08-14T09:31:50Z Taavi 13997 Taavi moved page [[User:Taavi/cloud-announce]] to [[Portal:Cloud Services/Admin/cloud-announce]] 2447561 wikitext text/x-wiki r5ypy7zjrio99qjf6zc7it2ay8um84v 0 460554 2447562 2026-08-14T09:32:19Z Taavi 13997 Redirected page to [[Help:Cloud Services introduction]] 2447562 wikitext text/x-wiki 19hatl2o9a9v949vnx9w9pd7f4xk60d 0 460555 2447564 2026-08-14T09:33:09Z Taavi 13997 Redirected page to [[Portal:Cloud Services/Admin/Deprecation playbook]] 2447564 wikitext text/x-wiki lkk7o4lji8iuz0jgni9f4nrvjgvca6u 0 460556 2447592 2026-08-14T10:05:05Z Atsuko-WMF 54082 Created page with "== Give me the command == To run [[Data Platform/Systems/Airflow/Developer guide|airflow-devenv]], those options are needed to run Airflow 3 {{Codesample | code = airflow-devenv create \ --branch main_airflow_3.3 --extra-values values-airflow3.yaml \ -d test_k8s | lang = Shell }} Branch <code>main_airflow_3.3</code> contains fixed wmf_common and Dags, it is manually updated from <code>main</code> branch and will be eventually merged back to <code>main</code>. Direc..." 2447592 wikitext text/x-wiki m1vk6qtd9a2ly0onlx1mt8dpztft4m7 0 460557 2447612 2026-08-14T10:15:48Z PFischer-WMF 38527 Created page with "= Highlights = ''This update is an aggregate covering the work reported since [[Search Platform/Weekly Updates/2026-06-19|19 June]].'' * [https://phabricator.wikimedia.org/T431348 T431348] A commonswiki shard was lost during eqiad maintenance, leaving a 6.8 TB file index red. Brian recovered it, and rather than stopping there, re-added the snapshot repository to eqiad and cloudelastic and rehearsed the restore end to end — so the next time this happens it follows a d..." 2447612 wikitext text/x-wiki 1y0yrjkrajtgrdgkcdzgs9yjbryv4xk