Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.18 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 2161 2454443 2453780 2026-09-03T13:35:50Z Stashbot 7414 taavi: migrating proxy-5 to nftables 2454443 wikitext text/x-wiki mzcbb7esnbo0e0coffmrst4drm9wtbp 0 4108 2454462 2454304 2026-09-03T13:45:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1328202]] to Thursday, September 03 UTC afternoon backport window 2454462 wikitext text/x-wiki hs3g18vm1wpc82a2etdk3bn5h51bs7u 2454555 2454462 2026-09-03T15:38:26Z ScheduleDeploymentBot 37566 Add [[gerrit:1334777]] to Thursday, September 03 UTC late backport window 2454555 wikitext text/x-wiki p65swtuee9jvsrnd26zl4b32klj8sxa 2454567 2454555 2026-09-03T16:29:59Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-09-03}} */ UTC-late infra window 2454567 wikitext text/x-wiki hqy6zdq5kfdq9vo5238lnsfkzrnehab 2454610 2454567 2026-09-03T17:49:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1335073]] to Thursday, September 03 UTC late backport window 2454610 wikitext text/x-wiki b0ioi0ko42qvwhoufsqpb195f7794y9 2454611 2454610 2026-09-03T17:49:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1335074]] to Thursday, September 03 UTC late backport window 2454611 wikitext text/x-wiki esefj1wvypd79dd6w4uqbjwxep0ucy4 2454624 2454611 2026-09-03T18:55:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1335109]] to Thursday, September 03 UTC late backport window 2454624 wikitext text/x-wiki gx7omu2q2o9t4gmiwun3egnwndjnqn3 2454698 2454624 2026-09-04T07:36:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1335706]] to Monday, September 07 UTC morning backport window 2454698 wikitext text/x-wiki bbfxz5hcgxg91zj8vcc2urb3viq0vkl 0 7919 2454332 2454331 2026-09-03T12:04:35Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_magru 2454332 wikitext text/x-wiki dvvvsbk3w5x4aqpdrec4qbh24uepzp9 2454333 2454332 2026-09-03T12:07:29Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_ulsfo 2454333 wikitext text/x-wiki 416jugzioow97adikovtl5ptj83c5pp 2454336 2454333 2026-09-03T12:10:18Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: db2207 repool 2454336 wikitext text/x-wiki 9bznzxe4eebrage6woaz3vi4tg6jlfi 2454337 2454336 2026-09-03T12:12:06Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1020.eqiad.wmnet with OS bookworm 2454337 wikitext text/x-wiki g87c7zykrujbtoinacubmxfc1rkekzw 2454339 2454337 2026-09-03T12:15:06Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_ulsfo 2454339 wikitext text/x-wiki o7j96c96tqhsra88dja0f0519lc9mab 2454340 2454339 2026-09-03T12:19:42Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454340 wikitext text/x-wiki 3mxfyn43nbn7yoqpzzdtbu28k6w2ici 2454341 2454340 2026-09-03T12:21:43Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2454341 wikitext text/x-wiki jp8e4eaw2gp6xe8z0u1zi5fd5s0tbh9 2454342 2454341 2026-09-03T12:21:56Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2454342 wikitext text/x-wiki qciig8luufu1hqe8hhte4ganuimsfok 2454343 2454342 2026-09-03T12:21:58Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454343 wikitext text/x-wiki lap4xft0wotdv9pd2piaz8hamair513 2454344 2454343 2026-09-03T12:25:16Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1065.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454344 wikitext text/x-wiki 2bqoqz5e4a5o3d132k9lkcuuck8gqcd 2454345 2454344 2026-09-03T12:30:21Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2454345 wikitext text/x-wiki cpgpi90lnkf0lhwzsuf9xcpt0lmpkcu 2454346 2454345 2026-09-03T12:30:34Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2454346 wikitext text/x-wiki hwp6lhgfyzso2d4q2q1pelsizvzqn6v 2454347 2454346 2026-09-03T12:30:54Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2454347 wikitext text/x-wiki elin75k6n6awp5n0rnd7ecjvp7zy05o 2454348 2454347 2026-09-03T12:31:01Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2454348 wikitext text/x-wiki f7e528klzyqz7zoqr4nmfm9j8w7o8vy 2454349 2454348 2026-09-03T12:31:17Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1282: Pooling db1282 into s6 2454349 wikitext text/x-wiki fbzl2hy0se9ioq1jugkifn5x7q95ql7 2454350 2454349 2026-09-03T12:33:36Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1065.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454350 wikitext text/x-wiki cikykg425wufldc77gwi99irh8zpkht 2454351 2454350 2026-09-03T12:34:14Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS trixie 2454351 wikitext text/x-wiki 4094mbdszam3lz2ykvtjkxmjvqkfx39 2454352 2454351 2026-09-03T12:38:24Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-magru and A:durum 2454352 wikitext text/x-wiki saxsn4bd4kw1jbpkp62hixv0ttfuszh 2454353 2454352 2026-09-03T12:40:03Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-magru and A:durum 2454353 wikitext text/x-wiki 2proim1rauc4p3dx0nsvwyakpzst5fm 2454354 2454353 2026-09-03T12:41:14Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-ulsfo and A:durum 2454354 wikitext text/x-wiki ggbwgtnmesfw7ap50qp48vr3gern0xt 2454355 2454354 2026-09-03T12:42:30Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-ulsfo and A:durum 2454355 wikitext text/x-wiki 9bejyxzutr1fuu3s92j4lcl3tatlqod 2454356 2454355 2026-09-03T12:45:30Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply 2454356 wikitext text/x-wiki 5t6wkuh0gc55az3ujfv2gc0hfgc7m5c 2454357 2454356 2026-09-03T12:45:51Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply 2454357 wikitext text/x-wiki ka49pi87zxenkpeon7o824u0476be4e 2454358 2454357 2026-09-03T12:49:40Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. 2454358 wikitext text/x-wiki khtqx6annj1ynxrfntttzktyqchg8yu 2454359 2454358 2026-09-03T12:49:43Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. 2454359 wikitext text/x-wiki 96hh1cvuwg96le0xxxib7m1r09223h8 2454360 2454359 2026-09-03T12:49:52Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2454360 wikitext text/x-wiki 67xly5vx62humyi572nv5rmo4srwtjl 2454361 2454360 2026-09-03T12:50:01Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2454361 wikitext text/x-wiki 8tbr3xu52eul8ua7luzswb4iv07zhvu 2454362 2454361 2026-09-03T12:50:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2901: Test 2454362 wikitext text/x-wiki 82zvw3t976b3zryxap9k176fuq1v5j8 2454363 2454362 2026-09-03T12:50:04Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2901: Test 2454363 wikitext text/x-wiki q174mxklkfeaqgxshrq9zn52bsc4xfn 2454364 2454363 2026-09-03T12:50:10Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2454364 wikitext text/x-wiki 05knc8agwlx68lxgbmzfbg7bwgv3aun 2454365 2454364 2026-09-03T12:50:15Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2454365 wikitext text/x-wiki 1fiqq6rcy59q3n79yqstt4rr9nogra7 2454366 2454365 2026-09-03T12:50:24Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2454366 wikitext text/x-wiki 56o3zwwsnx1migcvg76yy5pul8oqbuk 2454367 2454366 2026-09-03T12:50:35Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2454367 wikitext text/x-wiki 2mue0en7k4lczt5exci3rq20nasvgdg 2454368 2454367 2026-09-03T12:50:44Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2454368 wikitext text/x-wiki g9nnsk13vz642au5lb0xar5klevwxfn 2454369 2454368 2026-09-03T12:50:50Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2454369 wikitext text/x-wiki dadni8logz9uydajbritmuk3xu5y01f 2454370 2454369 2026-09-03T12:50:54Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2454370 wikitext text/x-wiki jswsy1e8tr34u4446cdh8siqp5hmr8u 2454371 2454370 2026-09-03T12:50:58Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2454371 wikitext text/x-wiki ricn1tbltzmnpbajvdv9u6gutvuuqzi 2454372 2454371 2026-09-03T12:51:12Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2454372 wikitext text/x-wiki jtqloy720bp67suer7zg0itdf046neo 2454373 2454372 2026-09-03T12:51:20Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2454373 wikitext text/x-wiki 7zumrb6p28ni1rfnjg8d6qpydxmueop 2454374 2454373 2026-09-03T12:51:32Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2454374 wikitext text/x-wiki axqyxp4ybgph5sgs77crrg4dtoahi61 2454375 2454374 2026-09-03T12:51:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2901: Test 2454375 wikitext text/x-wiki sgbve7xocskvg3ptbmre6ogtsxqpk6f 2454376 2454375 2026-09-03T12:51:42Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. 2454376 wikitext text/x-wiki aigvb7fnjuf7bptu94v41noazsqprsx 2454377 2454376 2026-09-03T12:51:47Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2454377 wikitext text/x-wiki bs4p070dxq6i1k2s4e5d259d771o5gk 2454378 2454377 2026-09-03T12:51:54Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. 2454378 wikitext text/x-wiki ogjzz9w8bgfqx0521gjc74fmnn350c4 2454379 2454378 2026-09-03T12:52:06Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. 2454379 wikitext text/x-wiki 4m0tqgetc802yh0tdr5m53x3sosfgpt 2454380 2454379 2026-09-03T12:52:13Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. 2454380 wikitext text/x-wiki tbf6lg9f7c3fbyv1qse3yqc734f1yj7 2454381 2454380 2026-09-03T12:52:18Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2454381 wikitext text/x-wiki g7ju7w6dd2znq9hac2edk4fxo8cwdvu 2454382 2454381 2026-09-03T12:52:26Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. 2454382 wikitext text/x-wiki aqvvmjylxxzdayyijsk4hhythde9325 2454383 2454382 2026-09-03T12:52:37Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. 2454383 wikitext text/x-wiki 6evu4d3pluoig8h4p8gidmvosp2uyrf 2454384 2454383 2026-09-03T12:52:49Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Test 2454384 wikitext text/x-wiki dwnh44jp0zmnr5lwl6oy2zzyfa1fcej 2454385 2454384 2026-09-03T12:52:56Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2901: Test 2454385 wikitext text/x-wiki 0nosyy5f80qzmckwf9y1sghc5aoj780 2454386 2454385 2026-09-03T12:52:59Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Test 2454386 wikitext text/x-wiki mx0de8kavqw5c1qwor5ds5ldfql45o4 2454387 2454386 2026-09-03T12:53:10Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2901: Test 2454387 wikitext text/x-wiki ebrlwyh5anyrxvr0yqv837c0z5yz7z5 2454388 2454387 2026-09-03T12:54:43Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2454388 wikitext text/x-wiki i92aj7ro0s2lvowpbqicm9jl1curods 2454389 2454388 2026-09-03T12:56:00Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Test 2454389 wikitext text/x-wiki 2t0x9ayf3jfwp0hej227t3osv06ib2j 2454390 2454389 2026-09-03T12:56:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2901: Test 2454390 wikitext text/x-wiki 6atojsl67jq5u2pocd25ps60ribf1yv 2454391 2454390 2026-09-03T12:56:47Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Test 2454391 wikitext text/x-wiki 39iy6q9gfp5cvh9s6pyuyzhp790323d 2454392 2454391 2026-09-03T12:56:55Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2901: Test 2454392 wikitext text/x-wiki tsyvlz4yuybutrusv4uq6snslbdxj8i 2454393 2454392 2026-09-03T12:58:37Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1066] - vriley@cumin1003" 2454393 wikitext text/x-wiki 99exz697xk05qanjbr678o0j1pv638h 2454394 2454393 2026-09-03T12:58:52Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1066] - vriley@cumin1003" 2454394 wikitext text/x-wiki 4c7lfjahvw7lzvl5o7h0r6tc7zxbsh1 2454395 2454394 2026-09-03T12:58:53Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454395 wikitext text/x-wiki nyrbpclr2pcobnlfjvpgb0exkxhyopq 2454396 2454395 2026-09-03T12:59:04Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1066 2454396 wikitext text/x-wiki fxg6cklsjow4n86ur4tq3o5osei6w7m 2454397 2454396 2026-09-03T12:59:19Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1066 2454397 wikitext text/x-wiki a6bx69pn82yht4eu4chl8i7e2z7d32n 2454398 2454397 2026-09-03T13:00:49Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1066.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454398 wikitext text/x-wiki 4l7f997q2jqrx2armnb1h0hkyke2wze 2454399 2454398 2026-09-03T13:04:16Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2454399 wikitext text/x-wiki j4dg3s38sqo88owdk85y2gbi01hkek7 2454400 2454399 2026-09-03T13:04:20Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2454400 wikitext text/x-wiki 075o8cxe83hu5chdctzzydgkhrdn5o1 2454401 2454400 2026-09-03T13:04:26Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2454401 wikitext text/x-wiki k7lt89m9rv8cgcxb35e6nn7jnpy4hrs 2454402 2454401 2026-09-03T13:04:42Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2454402 wikitext text/x-wiki 9rxi10xui56w6mdu4d975lks8ldszj0 2454403 2454402 2026-09-03T13:05:13Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2454403 wikitext text/x-wiki t5k80acb5cm6yddzlbertao0kpoe4rn 2454404 2454403 2026-09-03T13:05:21Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2454404 wikitext text/x-wiki hxy41l7qj728cglrcvp9n3ic7u3sx1n 2454405 2454404 2026-09-03T13:06:02Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2454405 wikitext text/x-wiki g2la5eofwkr5to74u6gr6dr4gkvdv8p 2454406 2454405 2026-09-03T13:06:07Z Stashbot 7414 jelto@dns1004: START - running authdns-update 2454406 wikitext text/x-wiki p9090zb4qydqfktf9pp7es847yyk9th 2454408 2454406 2026-09-03T13:08:39Z Stashbot 7414 jelto@dns1004: END - running authdns-update 2454408 wikitext text/x-wiki duuzch2ltd5w3dn2o9jxw7njsnd8gf2 2454409 2454408 2026-09-03T13:09:32Z Stashbot 7414 moritzm: installing libxslt bugfix updates from Trixie point release 2454409 wikitext text/x-wiki hztu3i9w24l5gbtdw5nuu4u14yq56pn 2454410 2454409 2026-09-03T13:09:56Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454410 wikitext text/x-wiki 7o0spkkqu05tmpsxm0n0g1a0qurqofx 2454411 2454410 2026-09-03T13:10:57Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1066.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454411 wikitext text/x-wiki 1h9cdkgxi3ydpimksb8amdto6sa0bv3 2454412 2454411 2026-09-03T13:11:22Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1067] - vriley@cumin1003" 2454412 wikitext text/x-wiki cukhvmf7dt2dcn5aj42oe3hqs694mls 2454413 2454412 2026-09-03T13:11:26Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1067] - vriley@cumin1003" 2454413 wikitext text/x-wiki 9da33mtfip6qwmyn0l4e9njkpeifdvf 2454414 2454413 2026-09-03T13:11:27Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454414 wikitext text/x-wiki jplkug8eig65drk6ahmk4qslkekp492 2454415 2454414 2026-09-03T13:11:57Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: Test 2454415 wikitext text/x-wiki 79wlwm77xt4ri4pjayvxl989fwwr19l 2454416 2454415 2026-09-03T13:14:31Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1067 2454416 wikitext text/x-wiki blc32mpuim1ih1p6xpl3x5t15tnxiss 2454417 2454416 2026-09-03T13:14:32Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1067 2454417 wikitext text/x-wiki g6razn9xgd7pmf9wjyouq8bbu1w88ef 2454418 2454417 2026-09-03T13:14:34Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1066.eqiad.wmnet with OS trixie 2454418 wikitext text/x-wiki 90g4k8zmnxarxgntzjt7ishrovrnv0f 2454419 2454418 2026-09-03T13:14:35Z Stashbot 7414 andrew@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454419 wikitext text/x-wiki 2yubcvgaerkwcz5mowy4v0rlfn0svbv 2454420 2454419 2026-09-03T13:14:37Z Stashbot 7414 moritzm: installing bash updates from trixie point release 2454420 wikitext text/x-wiki 73i5gmg6l2w1vv3tql3pkmkp0lonu86 2454421 2454420 2026-09-03T13:14:38Z Stashbot 7414 andrew@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454421 wikitext text/x-wiki 458g36v1tq22n3a51jgrh1n8qy2xmnu 2454422 2454421 2026-09-03T13:14:39Z Stashbot 7414 andrew@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454422 wikitext text/x-wiki gn39cp0t2rs64hal7d1md3gx110dpmq 2454423 2454422 2026-09-03T13:14:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454423 wikitext text/x-wiki 9q0b6j3bm5lonqffhhmxmsjmy1f49b3 2454424 2454423 2026-09-03T13:14:42Z Stashbot 7414 andrew@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454424 wikitext text/x-wiki a0i42dkr4uoqrur4jbjcyi8jm0jmf73 2454425 2454424 2026-09-03T13:15:34Z Stashbot 7414 andrew@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454425 wikitext text/x-wiki sgwa4546ltgbw4im4h1p6ge1o2xff01 2454426 2454425 2026-09-03T13:15:37Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454426 wikitext text/x-wiki k4i4aw3tznotkbgta5zi4wqr8ck7j5m 2454427 2454426 2026-09-03T13:15:50Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454427 wikitext text/x-wiki 0mx4khjht3ayahvopa7ivnr31z8bjla 2454428 2454427 2026-09-03T13:15:57Z Stashbot 7414 andrew@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454428 wikitext text/x-wiki ln7n3x7o9dzx3liitax8q9wnpyzyf6e 2454429 2454428 2026-09-03T13:20:43Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454429 wikitext text/x-wiki hfx455vm5fmtgdop37qoaqozpolgibb 2454430 2454429 2026-09-03T13:22:04Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams 2454430 wikitext text/x-wiki 0p6ek83fpo41ktmdfmghs2je23epb97 2454431 2454430 2026-09-03T13:22:39Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2454431 wikitext text/x-wiki edt3rnskur7vainzqtv03gm0v7cwtkv 2454432 2454431 2026-09-03T13:24:17Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454432 wikitext text/x-wiki hfmwlp1j74wzix5l1jfhvii1rgddfs8 2454433 2454432 2026-09-03T13:24:27Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.hosts.reimage for host netflow2004.codfw.wmnet with OS trixie 2454433 wikitext text/x-wiki tpplakasq0tl1lx8kdcj49gsfwkt6ts 2454434 2454433 2026-09-03T13:27:29Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454434 wikitext text/x-wiki cetoxecd6d2brshe2rspfo8u2qs5vb6 2454435 2454434 2026-09-03T13:28:39Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1067.eqiad.wmnet with OS trixie 2454435 wikitext text/x-wiki hbue95y75no3738jmlv4kuobsdgj7ry 2454436 2454435 2026-09-03T13:28:55Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1333866|Remove revisionId from term fallback cache lines for properties (T434204)]] 2454436 wikitext text/x-wiki n1ol2bkzsfnaxhy4yn17ih3v7aomnyu 2454437 2454436 2026-09-03T13:31:26Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-eqsin and A:durum 2454437 wikitext text/x-wiki 3y5bqij2vzg3166n9308tyi0egas0xh 2454438 2454437 2026-09-03T13:32:56Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-eqsin and A:durum 2454438 wikitext text/x-wiki 8poo40g4qb2k1xq80hzs8wfo6fb5liz 2454439 2454438 2026-09-03T13:33:39Z Stashbot 7414 samtar@deploy1003: samtar, thiemowmde: Backport for [[gerrit:1333866|Remove revisionId from term fallback cache lines for properties (T434204)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2454439 wikitext text/x-wiki 3c3ls05cfb8tryv6v9wo620dsp4m67t 2454440 2454439 2026-09-03T13:33:54Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2454440 wikitext text/x-wiki bdkf7voavrtdjzmz2d7moz0xge8y96j 2454441 2454440 2026-09-03T13:34:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454441 wikitext text/x-wiki 067l0jt7s4gz9oa1ldgp63d9gf1j8bq 2454442 2454441 2026-09-03T13:35:25Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454442 wikitext text/x-wiki pufopve4ps8kkifz5dkmjn6hxns1769 2454444 2454442 2026-09-03T13:35:54Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-drmrs and A:durum 2454444 wikitext text/x-wiki ncd9qw50xzai8fh60cwpezwp25ezb3g 2454445 2454444 2026-09-03T13:37:20Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-drmrs and A:durum 2454445 wikitext text/x-wiki 8gaxzoen7q6tgnm9cb9hb0lipsai9px 2454446 2454445 2026-09-03T13:37:43Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-esams and A:durum 2454446 wikitext text/x-wiki 5ihuvroxlbikhremhys3i6hj3euqop3 2454447 2454446 2026-09-03T13:37:45Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1072] - vriley@cumin1003" 2454447 wikitext text/x-wiki 4wdlqfxjse8ckqvniiujoz75i3ahe4a 2454448 2454447 2026-09-03T13:37:50Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1072] - vriley@cumin1003" 2454448 wikitext text/x-wiki p8f8a6coiw84mqmbf1jfvfmqksukyfv 2454449 2454448 2026-09-03T13:37:50Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454449 wikitext text/x-wiki 47gcs02efqf8rtas6t80ylxxy4exdet 2454450 2454449 2026-09-03T13:38:04Z Stashbot 7414 samtar@deploy1003: samtar, thiemowmde: Continuing with deployment 2454450 wikitext text/x-wiki nftxav0593bmq2hcx2ag3kxumsbtki0 2454451 2454450 2026-09-03T13:38:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454451 wikitext text/x-wiki 3hym6h3t9gyb3slemh4pwwiupu6hwrt 2454452 2454451 2026-09-03T13:38:54Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1072 2454452 wikitext text/x-wiki 8l1ma6hassla42il8r0hcmsn4zrbq8y 2454453 2454452 2026-09-03T13:39:32Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-esams and A:durum 2454453 wikitext text/x-wiki 6n40j6y780abmpdjexdk2eawxn5zhcq 2454454 2454453 2026-09-03T13:40:08Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage 2454454 wikitext text/x-wiki qnq2moxdsjiuk5x8y31exedq0hn1uos 2454455 2454454 2026-09-03T13:40:41Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1072 2454455 wikitext text/x-wiki cn81eeycog5ksqmz9r0t31019wkig52 2454456 2454455 2026-09-03T13:41:16Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454456 wikitext text/x-wiki p2anfibn459hlgr1iq2qnnhrc2mzf49 2454457 2454456 2026-09-03T13:42:45Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1333866|Remove revisionId from term fallback cache lines for properties (T434204)]] (duration: 13m 50s) 2454457 wikitext text/x-wiki eeivn7vcw6uo0evls5gzcgsgrmze7b8 2454458 2454457 2026-09-03T13:43:04Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on netflow2004.codfw.wmnet with reason: host reimage 2454458 wikitext text/x-wiki ffxu697hqhm685pxftki6sx1ei48h7h 2454459 2454458 2026-09-03T13:43:44Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454459 wikitext text/x-wiki ghkvzd6x2rssom3ohf0fntrs3fmuaiq 2454460 2454459 2026-09-03T13:44:01Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1067.eqiad.wmnet with reason: host reimage 2454460 wikitext text/x-wiki bftajjeifno40kphrfik0yrct9pja1y 2454461 2454460 2026-09-03T13:44:33Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage 2454461 wikitext text/x-wiki pzrl2a9r9wsu1bq41i4pxog11eadafy 2454463 2454461 2026-09-03T13:47:03Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams 2454463 wikitext text/x-wiki q3pw6ooyiosep0apsc82bau1wy1pimg 2454464 2454463 2026-09-03T13:47:47Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-codfw and A:durum 2454464 wikitext text/x-wiki kgu19yn68mndhm9vf21dmav1k7bpt56 2454465 2454464 2026-09-03T13:48:26Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1067.eqiad.wmnet with reason: host reimage 2454465 wikitext text/x-wiki kl3bigc1c4w8dhg8j6468ojsa46ovk6 2454466 2454465 2026-09-03T13:49:12Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-codfw and A:durum 2454466 wikitext text/x-wiki egeg50914zor62738utet6qi73rp433 2454467 2454466 2026-09-03T13:51:20Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-eqiad and A:durum 2454467 wikitext text/x-wiki obqb1y7voptcsl85wziojx7zsti15ju 2454468 2454467 2026-09-03T13:51:24Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow2004.codfw.wmnet with reason: host reimage 2454468 wikitext text/x-wiki r353obcwda69ocnz6935e4vtxj56nft 2454469 2454468 2026-09-03T13:51:52Z Stashbot 7414 moritzm: installing sqlite3 security updates 2454469 wikitext text/x-wiki thglvafwjn7c8aqrry1wzxzj9z1k4vz 2454470 2454469 2026-09-03T13:52:35Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw 2454470 wikitext text/x-wiki 0ja2xnv03f002ex4gp3mvvdhktw3n7r 2454471 2454470 2026-09-03T13:52:57Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-eqiad and A:durum 2454471 wikitext text/x-wiki grrvumwh5c5sv4d6darkqlhzpog7zbh 2454472 2454471 2026-09-03T13:54:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454472 wikitext text/x-wiki 133sxibtb3809l1wl1uu2md53as1aw0 2454473 2454472 2026-09-03T13:55:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454473 wikitext text/x-wiki ebb76422uu8o4hh4jzk0365smlypqyy 2454474 2454473 2026-09-03T13:56:58Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" 2454474 wikitext text/x-wiki kw5l6cepodq9g4537mq0fl6iy5gjzju 2454475 2454474 2026-09-03T13:57:34Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454475 wikitext text/x-wiki 3d15jl5apm47bug9lc2fe1jcbjywi0d 2454476 2454475 2026-09-03T13:58:24Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1328202|Remove the webrequest.dumps.dev0 stream from EventStreamConfig (T425087)]] 2454476 wikitext text/x-wiki 7ef6tq4jjs6xrprffcoha3rf3n06ylc 2454477 2454476 2026-09-03T14:01:30Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" 2454477 wikitext text/x-wiki edhd53kgabjhwcvfibzqufbb9rx3nt6 2454478 2454477 2026-09-03T14:01:31Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1065.eqiad.wmnet with OS trixie 2454478 wikitext text/x-wiki kuqdon8zxv2v20n6kg7yh2i4mrv6y97 2454479 2454478 2026-09-03T14:01:33Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1072.eqiad.wmnet with OS trixie 2454479 wikitext text/x-wiki c6f53vbtjyvow0i0pufiflukiobhs0q 2454480 2454479 2026-09-03T14:02:47Z Stashbot 7414 samtar@deploy1003: btullis, samtar: Backport for [[gerrit:1328202|Remove the webrequest.dumps.dev0 stream from EventStreamConfig (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2454480 wikitext text/x-wiki rnbznbq1yxqmqyb31jbeiiudyrremt0 2454481 2454480 2026-09-03T14:03:14Z Stashbot 7414 samtar@deploy1003: btullis, samtar: Continuing with deployment 2454481 wikitext text/x-wiki m6qi082kzyqw7qf5unn4gadf67q6nfk 2454482 2454481 2026-09-03T14:03:36Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) 2454482 wikitext text/x-wiki 3f1qhtrdf5by6vhanyftfrq1fgcvr5s 2454483 2454482 2026-09-03T14:04:53Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454483 wikitext text/x-wiki 1dxnvzmovscd4j52xxe0a964b7shtfg 2454484 2454483 2026-09-03T14:05:38Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2454484 wikitext text/x-wiki iqfxixn7tzg2hriqmugbau7rf34il00 2454485 2454484 2026-09-03T14:05:52Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454485 wikitext text/x-wiki 9blbdw32qqsbauocqe5ty8vjc7gvonu 2454486 2454485 2026-09-03T14:05:53Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1067.eqiad.wmnet with OS trixie 2454486 wikitext text/x-wiki 9pym28g0bqpq4uzlgi2bnxogp5zxuyo 2454487 2454486 2026-09-03T14:06:04Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2454487 wikitext text/x-wiki k3c69wq2agm7h51w21sqvleycg26nej 2454488 2454487 2026-09-03T14:08:00Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328202|Remove the webrequest.dumps.dev0 stream from EventStreamConfig (T425087)]] (duration: 09m 36s) 2454488 wikitext text/x-wiki juoda8p0h17dx5r0yfl5wjsu05gzydz 2454489 2454488 2026-09-03T14:08:44Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2454489 wikitext text/x-wiki 058qfck4jj08jkgst7vqrap4wtnl7hp 2454490 2454489 2026-09-03T14:08:56Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2454490 wikitext text/x-wiki h3op0t1rm5peaqa9jamb2l8zqhz3xc6 2454491 2454490 2026-09-03T14:10:42Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2454491 wikitext text/x-wiki 7iam2kgfl537pth018umyc6fcv8257m 2454492 2454491 2026-09-03T14:11:10Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1073] - vriley@cumin1003" 2454492 wikitext text/x-wiki h9nxpxt9dsr3dc0vdjlj0xyg0hokb2b 2454493 2454492 2026-09-03T14:11:14Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1073] - vriley@cumin1003" 2454493 wikitext text/x-wiki 2503rxslxxvgennriv14askgz2qmakh 2454494 2454493 2026-09-03T14:11:15Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454494 wikitext text/x-wiki ah9v8snms2hq3oi2uxu3nliuu3hkrjg 2454495 2454494 2026-09-03T14:12:19Z Stashbot 7414 ayounsi@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host netflow2004.codfw.wmnet with OS trixie 2454495 wikitext text/x-wiki njwesccfcm5r12hujov2hy0sj826s8x 2454496 2454495 2026-09-03T14:15:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1073 2454496 wikitext text/x-wiki 76gg7bhodjyu73rqfbodvn02ormfz1k 2454497 2454496 2026-09-03T14:16:14Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1073 2454497 wikitext text/x-wiki rqnx7rjdpsg1mkpuyuzjiy06w7ugqbr 2454498 2454497 2026-09-03T14:16:28Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1072.eqiad.wmnet with reason: host reimage 2454498 wikitext text/x-wiki 0kg4b21pve81cuj4h9izze45q1gc5je 2454499 2454498 2026-09-03T14:16:31Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2454499 wikitext text/x-wiki mlwhe8zcq7s52ut1hec2sld7t3effss 2454500 2454499 2026-09-03T14:16:59Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1073.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454500 wikitext text/x-wiki lsheftcxg0qimfm7jyc8o0mrrbvy39i 2454501 2454500 2026-09-03T14:17:04Z Stashbot 7414 arnaudb@dns1006: START - running authdns-update 2454501 wikitext text/x-wiki t9m1fsgnqk0xt002epmv4484mhk57sz 2454502 2454501 2026-09-03T14:17:24Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw 2454502 wikitext text/x-wiki c03i57394xc81f878c3v31giskkcubx 2454503 2454502 2026-09-03T14:17:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1074] - vriley@cumin1003" 2454503 wikitext text/x-wiki 5hgyml08w7bs8tpeoghx8r9b10iy461 2454504 2454503 2026-09-03T14:17:44Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1074] - vriley@cumin1003" 2454504 wikitext text/x-wiki hjj02m2ug9l8dolwsb4za4v16dzefqy 2454505 2454504 2026-09-03T14:17:45Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454505 wikitext text/x-wiki oa5rle2jou9y90td6fzy74frfqbmjmb 2454506 2454505 2026-09-03T14:18:05Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1074 2454506 wikitext text/x-wiki 3l95iav2idn7erqv7sxwxvvzkj3pd48 2454507 2454506 2026-09-03T14:18:20Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1074 2454507 wikitext text/x-wiki sj3al2ujc7jrpmfnkmutw51mdzhk294 2454508 2454507 2026-09-03T14:18:55Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1074.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454508 wikitext text/x-wiki 0umsp5olsg52jw12p52ofqyeyg7hxti 2454509 2454508 2026-09-03T14:19:39Z Stashbot 7414 arnaudb@dns1006: END - running authdns-update 2454509 wikitext text/x-wiki jlmiznywrttn3tpdosid32kr75v7eqz 2454511 2454509 2026-09-03T14:21:03Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1072.eqiad.wmnet with reason: host reimage 2454511 wikitext text/x-wiki 52hsfj847bg2cecaw6o6t12mjls6xct 2454512 2454511 2026-09-03T14:22:37Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2002.codfw.wmnet 2454512 wikitext text/x-wiki kt3ly5b4n7lpvmny4zdzu3rnbb50quy 2454513 2454512 2026-09-03T14:27:14Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo 2454513 wikitext text/x-wiki jmkvvzp1u7o2synecmg3hs7pkqqyiu7 2454514 2454513 2026-09-03T14:29:24Z Stashbot 7414 sukhe: sudo cumin "A:cp-text" "disable-puppet 'merging CR 1334855'" T425441 2454514 wikitext text/x-wiki 43udr4f8dqkwer63jeqceowxv3x9oca 2454515 2454514 2026-09-03T14:29:25Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1073.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454515 wikitext text/x-wiki s7je95qngpxnipc7rx0xwxdhzj1lmrp 2454516 2454515 2026-09-03T14:29:43Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1066.eqiad.wmnet with OS trixie 2454516 wikitext text/x-wiki nru4tbg648lnq9xjv530kf5djrbq1bd 2454517 2454516 2026-09-03T14:30:11Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1066.eqiad.wmnet with OS trixie 2454517 wikitext text/x-wiki l3sno033j9p8mizclhqgil51gf48aie 2454518 2454517 2026-09-03T14:32:08Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1074.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2454518 wikitext text/x-wiki o2yx4qbg9oj87h5kmmf0sa1ge1s7tvx 2454519 2454518 2026-09-03T14:37:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host cassandra-dev2002.codfw.wmnet 2454519 wikitext text/x-wiki 61fj551sezi18cnedlv1el27w6wsvck 2454520 2454519 2026-09-03T14:37:38Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454520 wikitext text/x-wiki kjfox1wr8q5qabt57rkim9utfw7r6w1 2454521 2454520 2026-09-03T14:38:36Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454521 wikitext text/x-wiki 8q3hsjp4hazegpkxego7snjqchpr90j 2454522 2454521 2026-09-03T14:38:37Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1072.eqiad.wmnet with OS trixie 2454522 wikitext text/x-wiki qy646klxju824fh7a0z5jrr48ljl06d 2454523 2454522 2026-09-03T14:39:36Z Stashbot 7414 sukhe: sudo cumin "A:cp-text" "run-puppet-agent --enable 'merging CR 1334855'": T425441 2454523 wikitext text/x-wiki 65bd6hml0ckf82ojv23tq5rcu0fydc7 2454524 2454523 2026-09-03T14:45:07Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage 2454524 wikitext text/x-wiki fy4enh2m552cmy4hygm12w1t9zzfxne 2454526 2454524 2026-09-03T14:47:24Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cassandra-dev2002.codfw.wmnet 2454526 wikitext text/x-wiki 9xr40uv2eptk2di0avfm4hmonsopni5 2454527 2454526 2026-09-03T14:47:26Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2002.codfw.wmnet 2454527 wikitext text/x-wiki 7zy4xikj6bd1ny6kd1lv5rrrwlsfc78 2454528 2454527 2026-09-03T14:48:53Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage 2454528 wikitext text/x-wiki 6hqpw35fhmormm4qy65eqqivn10x8i1 2454529 2454528 2026-09-03T14:49:54Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1073.eqiad.wmnet with OS trixie 2454529 wikitext text/x-wiki i8htk5vey3c0r4b5itodtiy2vpoz756 2454530 2454529 2026-09-03T14:52:30Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo 2454530 wikitext text/x-wiki 5npxsm4uhs7grlrx1sy5r9cj4rnhzpi 2454531 2454530 2026-09-03T14:53:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2002.codfw.wmnet 2454531 wikitext text/x-wiki bj8o30b8s4fvjc7vv5igvchgg22aycy 2454532 2454531 2026-09-03T14:58:40Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-codfw 2454532 wikitext text/x-wiki ckuc8cni9irn41nvsarhxiwaceebdvl 2454533 2454532 2026-09-03T15:00:49Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: cluster=urldownloader 2454533 wikitext text/x-wiki hcuqfp12iup56msbzsewdxveqcy747k 2454534 2454533 2026-09-03T15:02:28Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) 2454534 wikitext text/x-wiki 16tnxm0s1dq2xqbtaj71n5337rtwvmy 2454535 2454534 2026-09-03T15:04:18Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454535 wikitext text/x-wiki gm5e9w2jnq7ellg4gprwng8gqw25glv 2454536 2454535 2026-09-03T15:04:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1073.eqiad.wmnet with reason: host reimage 2454536 wikitext text/x-wiki miroyhy3d8x7rh1ecwf7e7z8onm8hk7 2454537 2454536 2026-09-03T15:04:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2002.codfw.wmnet 2454537 wikitext text/x-wiki prplzlvt10v088f2wl868xjoi8vhrds 2454538 2454537 2026-09-03T15:05:31Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454538 wikitext text/x-wiki 0fpr8ubwclj8dtz4tf0u46pvze4gnpo 2454539 2454538 2026-09-03T15:05:32Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1066.eqiad.wmnet with OS trixie 2454539 wikitext text/x-wiki 2507qqj75bxjfenz7yq6hi6p097hxfj 2454540 2454539 2026-09-03T15:05:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bookworm 2454540 wikitext text/x-wiki mfw06s3oz2b3s8697kdqhkkm9jrqmp5 2454542 2454540 2026-09-03T15:09:31Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1073.eqiad.wmnet with reason: host reimage 2454542 wikitext text/x-wiki ri3osvaf7i8ey95qb1dqj1r8xuh4ntv 2454545 2454542 2026-09-03T15:13:08Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2454545 wikitext text/x-wiki jdv9xdttkqbj24i4jaqb00agx497dub 2454546 2454545 2026-09-03T15:15:56Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2454546 wikitext text/x-wiki 6m4by43drdgcwy2sx0zzq42rvc3apcv 2454547 2454546 2026-09-03T15:18:51Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2454547 wikitext text/x-wiki oxqembz4gayhe7dmenbqg6vcz4sotj3 2454548 2454547 2026-09-03T15:24:12Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage 2454548 wikitext text/x-wiki faoyomrmrr913ww4kn5cf6al4u92xu6 2454549 2454548 2026-09-03T15:24:13Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2454549 wikitext text/x-wiki jyebnppq25lzktezk3ht6uaw9hhxetc 2454550 2454549 2026-09-03T15:25:10Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454550 wikitext text/x-wiki i6bigpix3narqsiykobph3tuzxgonr3 2454551 2454550 2026-09-03T15:26:40Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2454551 wikitext text/x-wiki j65oqz6jqudbf1l4sd35xovr9zh6c7u 2454552 2454551 2026-09-03T15:26:49Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2454552 wikitext text/x-wiki serzgfumj4gr0jxtofq8r63uvqsru0d 2454553 2454552 2026-09-03T15:28:35Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage 2454553 wikitext text/x-wiki ewwxh6pu7gz1a9yileqo1v83720mj55 2454554 2454553 2026-09-03T15:29:37Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2454554 wikitext text/x-wiki di4xz3use8us2qj7usj3hlppfxhzuzy 2454556 2454554 2026-09-03T15:48:30Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2002.codfw.wmnet with OS bookworm 2454556 wikitext text/x-wiki 7mr4jxdt8619ri1jua18ebunvuyiqcn 2454557 2454556 2026-09-03T15:51:38Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1325868|Growth: Remove now removed config variable]] 2454557 wikitext text/x-wiki dtwfks0o9h0lg418ceexqe1m83t6h0m 2454558 2454557 2026-09-03T15:51:44Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=urldownloader[12]00[56].wikimedia.org [reason: depooling urldownloader trixie nodes] 2454558 wikitext text/x-wiki 291tmwavpz626fur80ur4n3b5g4tj0i 2454559 2454558 2026-09-03T16:01:07Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325868|Growth: Remove now removed config variable]] (duration: 09m 29s) 2454559 wikitext text/x-wiki 6lkh4mg49flwbb0sk711ieox02z59wq 2454560 2454559 2026-09-03T16:03:56Z Stashbot 7414 btullis@deploy1003: Started scap sync-world: Rebuilding to pick up new version of dump scripts in mediawiki-cli for T436913 2454560 wikitext text/x-wiki kbqh9t2hv9eniez4u6y6wkape8p0b13 2454561 2454560 2026-09-03T16:07:38Z Stashbot 7414 ryankemper@puppetserver1001: conftool action : set/pooled=yes; selector: name=wdqs101[1-4].eqiad.wmnet 2454561 wikitext text/x-wiki pegce2u9zs1yt7jmndbx8gmjnasy30l 2454562 2454561 2026-09-03T16:12:07Z Stashbot 7414 btullis@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.17,1.47.0-wmf.18,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/m 2454562 wikitext text/x-wiki 26hvpefjl6kc9901n4qozt7ph8ajkzg 2454563 2454562 2026-09-03T16:12:08Z Stashbot 7414 btullis@deploy1003: Started deploy [analytics/refinery@0e301af] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0e301af2] 2454563 wikitext text/x-wiki pv18c7lrzrgtdk0co5f7v6otgv3ve5i 2454564 2454563 2026-09-03T16:12:46Z Stashbot 7414 btullis@deploy1003: Finished deploy [analytics/refinery@0e301af] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0e301af2] (duration: 00m 38s) 2454564 wikitext text/x-wiki 5e4f3eq4mka4demaagco7slup06aq3v 2454565 2454564 2026-09-03T16:14:45Z Stashbot 7414 ryankemper: T430880 Stumbled across `wdqs2021` listed as inactive, looks like it was never fully re-pooled after a data xfer. Pooled. 2454565 wikitext text/x-wiki gjdiz4u5h04tbth7sawgs5ttglp3bdi 2454566 2454565 2026-09-03T16:15:00Z Stashbot 7414 ryankemper@puppetserver1001: conftool action : set/pooled=yes; selector: name=wdqs2021.codfw.wmnet,service=wdqs-main 2454566 wikitext text/x-wiki 6blnl0zes6ggisdmik2e22uohnu6ciu 2454568 2454566 2026-09-03T16:39:55Z Stashbot 7414 btullis@deploy1003: Started scap sync-world: Trying again for T436913 2454568 wikitext text/x-wiki nktspaojit6q7vysml1li8ztvcjpadi 2454570 2454568 2026-09-03T16:50:28Z Stashbot 7414 andrew@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454570 wikitext text/x-wiki de9yqlpfcyz3wbwnu5y7mwrfky5ivwr 2454571 2454570 2026-09-03T16:51:09Z Stashbot 7414 andrew@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454571 wikitext text/x-wiki fv5arv87jbx10h0a93t9dvaw7xto973 2454572 2454571 2026-09-03T16:51:26Z Stashbot 7414 andrew@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454572 wikitext text/x-wiki qg8tghqszgkyfcgduiixodzodd5unl9 2454573 2454572 2026-09-03T16:51:50Z Stashbot 7414 andrew@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454573 wikitext text/x-wiki l9vi98bxr7oargg25tlsc96qz7z9gs5 2454574 2454573 2026-09-03T16:52:46Z Stashbot 7414 btullis@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.17,1.47.0-wmf.18,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/m 2454574 wikitext text/x-wiki 8d20k1gyj7lr95625i0p1l392zgzmho 2454575 2454574 2026-09-03T16:58:31Z Stashbot 7414 dancy: Running scap clean-images on deploy1003 2454575 wikitext text/x-wiki ai07ilphm08wtddb7a8a9q9oacnjv75 2454576 2454575 2026-09-03T16:59:49Z Stashbot 7414 andrew@cumin2003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1045.eqiad.wmnet 2454576 wikitext text/x-wiki b73a4hb5rtcudayhyno954cl0rt1bre 2454577 2454576 2026-09-03T17:01:35Z Stashbot 7414 btullis@deploy1003: Started scap sync-world: Trying again for T436913 2454577 wikitext text/x-wiki lxbq8c18quhz3m4lrxvq660q398kvn0 2454578 2454577 2026-09-03T17:09:30Z Stashbot 7414 btullis@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.17,1.47.0-wmf.18,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/m 2454578 wikitext text/x-wiki goe8w8tjx3hcmfbn0ba6ctj7a8j7u60 2454579 2454578 2026-09-03T17:10:15Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2454579 wikitext text/x-wiki 3ofhcd2fhw6zdmgpy39l0q3luk1bif9 2454580 2454579 2026-09-03T17:10:28Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2454580 wikitext text/x-wiki bnf5tva5v53uf3paso3uk4nqu5v504h 2454581 2454580 2026-09-03T17:13:15Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2454581 wikitext text/x-wiki 0f347g8t5uew8lif4rzhjig078lc1fb 2454582 2454581 2026-09-03T17:14:16Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a2-eqiad 2454582 wikitext text/x-wiki ev3z00vqrevv27qi95zn5vs7byvv2ey 2454583 2454582 2026-09-03T17:15:19Z Stashbot 7414 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a2-eqiad 2454583 wikitext text/x-wiki 65z8uev1qmu9rhlc4nl98a3r8ei7u09 2454584 2454583 2026-09-03T17:16:29Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2454584 wikitext text/x-wiki 0lw7ltthnpqyx16nb8ba01uco7rrj1k 2454585 2454584 2026-09-03T17:16:54Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2454585 wikitext text/x-wiki 4xb8fkkvnbi3ile8gn8kkwrnjp6lcvv 2454586 2454585 2026-09-03T17:22:45Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2454586 wikitext text/x-wiki owa40jaxtzi3s6i962kevg6kzo5x41x 2454587 2454586 2026-09-03T17:22:55Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2454587 wikitext text/x-wiki thm2vu22bld55qdcnpubfmztskamo61 2454588 2454587 2026-09-03T17:24:32Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a2-eqiad 2454588 wikitext text/x-wiki 79xkk56kmhy0gnepr569w8ix0n19ijp 2454589 2454588 2026-09-03T17:24:52Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a2-eqiad 2454589 wikitext text/x-wiki 65ckxbgmvk8l3mmg481avqlsqnumjb7 2454590 2454589 2026-09-03T17:26:29Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.289.0" for 3 host(s) 2454590 wikitext text/x-wiki 6jlaluv1uu2zo8swa8yyjsysh073oz1 2454591 2454590 2026-09-03T17:28:25Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.289.0" completed for 3 hosts 2454591 wikitext text/x-wiki sc26n785k5rcry6219xywbmdes7f259 2454592 2454591 2026-09-03T17:30:35Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2454592 wikitext text/x-wiki fcoqqu43vr04om3a4e0mifqy9284ew1 2454593 2454592 2026-09-03T17:30:50Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2454593 wikitext text/x-wiki 08gs4xj1y3hsjdmgg8d7xgrw71m6i23 2454594 2454593 2026-09-03T17:31:34Z Stashbot 7414 andrew@cumin2003: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudcephosd1045.eqiad.wmnet 2454594 wikitext text/x-wiki 7pha8pq6hlt6sc8n9gpzs52yxlsk31h 2454595 2454594 2026-09-03T17:36:11Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2454595 wikitext text/x-wiki tn65twecl9znw1yagje1e89vg7km4k4 2454596 2454595 2026-09-03T17:36:25Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2454596 wikitext text/x-wiki oa26l86oco6dxyvjgc8r0abh2puty0b 2454597 2454596 2026-09-03T17:37:23Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2454597 wikitext text/x-wiki 4e2gyyr5i8d6thr20w06l8x7l3rqiit 2454598 2454597 2026-09-03T17:37:49Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2454598 wikitext text/x-wiki tqd0tzfk4j3u7psz559rxikuudk3ugg 2454599 2454598 2026-09-03T17:39:08Z Stashbot 7414 ryankemper: T421642 [WDQS] requestctl changes: `2026-09-03 16:23-17:33` UTC: added hard-deny pair `cache-text/wdqs_futile_sparql_sep_2026_deny(+_bots)`; extended pattern `ua/wdqs_heavy_sparql_bots_2026` and added default-scope twin `wdqs_heavy_sparql_bots_jul_2026_ratelimit_default`; added ipblock `abuse/wdqs_sparql_scanners_sep_2026` + throttle `wdqs_sparql_scanners_sep_2026_ratelimit` (needed manual `requestctl update-provenance-map`) 2454599 wikitext text/x-wiki 8yj0zzoe5krxw65zohnomshpogtp0ak 2454600 2454599 2026-09-03T17:39:24Z Stashbot 7414 ryankemper: [WDQS] Service looks healthy again, CPU load and thread count have dropped considerably over the last hour 2454600 wikitext text/x-wiki fezvby46llv0slypuq8ah4j4m962glf 2454601 2454600 2026-09-03T17:40:41Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454601 wikitext text/x-wiki 7lq1xchadlpbn1wqzrwjvndqsgkuz8s 2454602 2454601 2026-09-03T17:44:46Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2454602 wikitext text/x-wiki qm1en0a3tonkp73jmg753y8okxd6p8u 2454603 2454602 2026-09-03T17:44:58Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2454603 wikitext text/x-wiki sm7l4cfh3f1cd70bawbylu4o2382qjl 2454604 2454603 2026-09-03T17:45:18Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2454604 wikitext text/x-wiki 7ui5q4r3arpcd2309fd52uwzd7h2y4i 2454605 2454604 2026-09-03T17:45:30Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2454605 wikitext text/x-wiki b2bzlzia43li8ohhophrl21g03ugprs 2454606 2454605 2026-09-03T17:46:01Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2454606 wikitext text/x-wiki lrxxyo4cgrfqu4wcseprbm6ujacp8n6 2454607 2454606 2026-09-03T17:46:10Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2454607 wikitext text/x-wiki r6xyb08bunip0b4pb6vz1cbqqmtshy8 2454608 2454607 2026-09-03T17:46:18Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2454608 wikitext text/x-wiki rmdn9m25a1z00gybmt342g80ushcgm7 2454609 2454608 2026-09-03T17:46:49Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2454609 wikitext text/x-wiki 2q4y233re9bqv549rddpgz9scyq3q77 2454612 2454609 2026-09-03T17:52:08Z Stashbot 7414 andrew@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454612 wikitext text/x-wiki 2njab15gwfjxoy4b52vl2advnebz3zm 2454613 2454612 2026-09-03T17:53:35Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454613 wikitext text/x-wiki 70mhzml29o7vv9ln5drjfwntdaxrl0x 2454614 2454613 2026-09-03T17:54:04Z Stashbot 7414 andrew@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454614 wikitext text/x-wiki 620jrud0higuaev2hslp90h1x1amapt 2454615 2454614 2026-09-03T17:54:38Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454615 wikitext text/x-wiki mwhhhp3pdejhufu8mo3ydwtlvokb50e 2454616 2454615 2026-09-03T17:54:42Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454616 wikitext text/x-wiki 5e7pyelwhfw1q8pa5yv46bjhdu1o3jx 2454617 2454616 2026-09-03T17:55:34Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454617 wikitext text/x-wiki or86didn40z28a5q2kc1emdf8prdk2t 2454618 2454617 2026-09-03T17:55:37Z Stashbot 7414 andrew@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454618 wikitext text/x-wiki kidh0h13f6rdex2n6xure8ln66ty5aq 2454620 2454618 2026-09-03T18:16:54Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bookworm 2454620 wikitext text/x-wiki pnu6q5u7rrhgcmoe14b6lq7mwvh1khv 2454621 2454620 2026-09-03T18:20:59Z Stashbot 7414 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.18 refs T430837 2454621 wikitext text/x-wiki 6nf3edla5pl6n7hpcwdrrotuay8lmzg 2454622 2454621 2026-09-03T18:34:44Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage 2454622 wikitext text/x-wiki kn26ebxntahbs9lumd23g85yxqhv9zx 2454623 2454622 2026-09-03T18:39:52Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage 2454623 wikitext text/x-wiki kwprs8udt37hqdfotwwkk5ylczsgp6a 2454625 2454623 2026-09-03T18:57:56Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454625 wikitext text/x-wiki kms0ujszjyh2q2hd5uhcw9pllsrkpal 2454626 2454625 2026-09-03T18:58:00Z Stashbot 7414 andrew@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454626 wikitext text/x-wiki gpks4rszumlp858owtlz76n36bckg69 2454627 2454626 2026-09-03T19:00:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2002.codfw.wmnet with OS bookworm 2454627 wikitext text/x-wiki p34jt6ebykpb23m5sgcbp6oxq0dl6zs 2454628 2454627 2026-09-03T19:01:44Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454628 wikitext text/x-wiki tv9rvshlj3zkyvsa84d20du5rlz9ws4 2454629 2454628 2026-09-03T19:01:47Z Stashbot 7414 andrew@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet 2454629 wikitext text/x-wiki jo72vfikf6pzym4t6rfya6qe56g5tb7 2454630 2454629 2026-09-03T19:16:49Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Noop deployment to validate pretrain logstash check configuration - T435419 2454630 wikitext text/x-wiki oi17hawsv2e04ytsyu2ch6l16j7phz4 2454631 2454630 2026-09-03T19:19:48Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Noop deployment to validate pretrain logstash check configuration - T435419 (duration: 02m 59s) 2454631 wikitext text/x-wiki h4jfdac87zjyqv9h48lxvdh52u3zvn6 2454632 2454631 2026-09-03T19:49:59Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs1021.eqiad.wmnet 2454632 wikitext text/x-wiki 9u7kmm42j02p5mili472meoywnrx1ot 2454635 2454632 2026-09-03T20:00:45Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reboot-single for host aqs1021.eqiad.wmnet 2454635 wikitext text/x-wiki 5z8y6toqxkc818rjx3pox2ix9f6xmtn 2454636 2454635 2026-09-03T20:09:43Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1334268|Email confirmation A/A test: make registration cutoff consistent (T435135)]], [[gerrit:1335073|Instrumentation for email confirmation upfront enforcement A/A test (T435135)]], [[gerrit:1335074|Email confirmation A/A: check creation wiki, centralize logic (T435135)]] 2454636 wikitext text/x-wiki g54evoms31i3elqpgr7k2fq0yllytjn 2454637 2454636 2026-09-03T20:11:29Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1021.eqiad.wmnet 2454637 wikitext text/x-wiki qesyhyo3dc76f3ljynzxrz6284kno05 2454638 2454637 2026-09-03T20:11:31Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs1021.eqiad.wmnet 2454638 wikitext text/x-wiki r6i2hchyrosktkohnjg9jxohg3hn2rn 2454639 2454638 2026-09-03T20:13:59Z Stashbot 7414 catrope@deploy1003: catrope: Backport for [[gerrit:1334268|Email confirmation A/A test: make registration cutoff consistent (T435135)]], [[gerrit:1335073|Instrumentation for email confirmation upfront enforcement A/A test (T435135)]], [[gerrit:1335074|Email confirmation A/A: check creation wiki, centralize logic (T435135)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be 2454639 wikitext text/x-wiki 216ljfqpaxp2sxgtsg3h02yxh4r6gp1 2454640 2454639 2026-09-03T20:14:25Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1021.eqiad.wmnet with OS bookworm 2454640 wikitext text/x-wiki gq00gpsp4ef2uvh6flwxfoaoyr0ef4t 2454641 2454640 2026-09-03T20:16:39Z Stashbot 7414 catrope@deploy1003: catrope: Continuing with deployment 2454641 wikitext text/x-wiki pyqoqnnwp3lf4usa9hftzaorpiep14w 2454642 2454641 2026-09-03T20:20:50Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1046.eqiad.wmnet 2454642 wikitext text/x-wiki sg45s4lx065qfh8oth1wks7tui69nei 2454643 2454642 2026-09-03T20:23:25Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1334268|Email confirmation A/A test: make registration cutoff consistent (T435135)]], [[gerrit:1335073|Instrumentation for email confirmation upfront enforcement A/A test (T435135)]], [[gerrit:1335074|Email confirmation A/A: check creation wiki, centralize logic (T435135)]] (duration: 13m 41s) 2454643 wikitext text/x-wiki 8mibxzysxx0v38ubk2c4dqyj4e7yjtl 2454645 2454643 2026-09-03T20:28:38Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1334777|prv: Enable parsoid rendering for more wikisource wikis (T436919)]] 2454645 wikitext text/x-wiki sxssly732cx4hfgsm5ymmr7r3et97cq 2454646 2454645 2026-09-03T20:28:58Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage 2454646 wikitext text/x-wiki 02sa1wwof7fnbguy23r1qh06sh9g66d 2454648 2454646 2026-09-03T20:31:15Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1046.eqiad.wmnet 2454648 wikitext text/x-wiki pgc4pyop1uhemgo0jujga05hloo5h79 2454650 2454648 2026-09-03T20:32:45Z Stashbot 7414 arlolra@deploy1003: arlolra, jgiannelos: Backport for [[gerrit:1334777|prv: Enable parsoid rendering for more wikisource wikis (T436919)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2454650 wikitext text/x-wiki ol7ffzfuc5fmwblbykro1klacdikw5p 2454651 2454650 2026-09-03T20:33:09Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage 2454651 wikitext text/x-wiki mc3izrdaophrcqahe6851g2j9ysuapq 2454652 2454651 2026-09-03T20:33:40Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1047.eqiad.wmnet 2454652 wikitext text/x-wiki 8rwjaffe962jr83bitvtly6ju1r9b8p 2454653 2454652 2026-09-03T20:34:25Z Stashbot 7414 arlolra@deploy1003: arlolra, jgiannelos: Continuing with deployment 2454653 wikitext text/x-wiki tpqe7t5qfr5nipga4n9s2xnh7dr3yvu 2454654 2454653 2026-09-03T20:39:01Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1334777|prv: Enable parsoid rendering for more wikisource wikis (T436919)]] (duration: 10m 23s) 2454654 wikitext text/x-wiki 5o1hk7gyj2ea5xmdcd4uidzy0b9u1i5 2454655 2454654 2026-09-03T20:40:47Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a3-eqiad 2454655 wikitext text/x-wiki htvqh51qkb15tlus905f4l718nroolf 2454656 2454655 2026-09-03T20:41:04Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a3-eqiad 2454656 wikitext text/x-wiki 5zoj592ggf86flyhvcjlh8baxu1ebib 2454657 2454656 2026-09-03T20:42:45Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1335109|Add exclusions to Apple app site association file (T435363)]] 2454657 wikitext text/x-wiki 1n5n91s58xfl86husa1hnny1zoyn945 2454658 2454657 2026-09-03T20:44:15Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1047.eqiad.wmnet 2454658 wikitext text/x-wiki c42a2kfhnbhnloju495frue5nc9nfhb 2454659 2454658 2026-09-03T20:46:58Z Stashbot 7414 arlolra@deploy1003: arlolra, tsev: Backport for [[gerrit:1335109|Add exclusions to Apple app site association file (T435363)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2454659 wikitext text/x-wiki mjq41jmmn6wotf9gofybklaorsudmqg 2454660 2454659 2026-09-03T20:50:31Z Stashbot 7414 arlolra@deploy1003: arlolra, tsev: Continuing with deployment 2454660 wikitext text/x-wiki 80zpop2q6yvs7pk4di64zjcdo6p5pgn 2454661 2454660 2026-09-03T20:52:00Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1021.eqiad.wmnet with OS bookworm 2454661 wikitext text/x-wiki eqyn0iq4ep54jvfcz88fr8w6ptanzqn 2454662 2454661 2026-09-03T20:52:24Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs1021.eqiad.wmnet 2454662 wikitext text/x-wiki ddniigt9bi4pdrj762kp2nomjf1u6jl 2454663 2454662 2026-09-03T20:55:09Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1335109|Add exclusions to Apple app site association file (T435363)]] (duration: 12m 24s) 2454663 wikitext text/x-wiki e6ac64ircq6gutsz93grp8hz57z40ww 2454664 2454663 2026-09-03T21:03:57Z Stashbot 7414 eevans@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts aqs1021.eqiad.wmnet 2454664 wikitext text/x-wiki bfaamzu7y3ff6a6wtfaqdxh60nwzrj9 2454666 2454664 2026-09-03T21:46:33Z Stashbot 7414 tsev@deploy1003: mwscript-k8s job started: purgeList.php # T435363 2454666 wikitext text/x-wiki nfm5tif7exl5yhrzkwl7xymxdumq6in 2454671 2454666 2026-09-04T00:23:13Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454671 wikitext text/x-wiki m6ed78howtjqvlia4r4uiiz3uytjj42 2454672 2454671 2026-09-04T00:23:14Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1073.eqiad.wmnet with OS trixie 2454672 wikitext text/x-wiki r9jvumpyhv0ndfbqw5sal11odnviqzl 2454673 2454672 2026-09-04T00:23:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1074.eqiad.wmnet with OS trixie 2454673 wikitext text/x-wiki e593f4b2t60adwrktixjg2rpg5yil66 2454674 2454673 2026-09-04T00:39:00Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1074.eqiad.wmnet with reason: host reimage 2454674 wikitext text/x-wiki d0ocrx9e550l2hb5w3xytlo4h56pd6i 2454675 2454674 2026-09-04T00:44:01Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1074.eqiad.wmnet with reason: host reimage 2454675 wikitext text/x-wiki hncom12lf70pxwn3k34en9mqq9z0k0e 2454676 2454675 2026-09-04T00:46:59Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS trixie 2454676 wikitext text/x-wiki iega6aueu2fbgsnsa4ftaqiqfykd7te 2454677 2454676 2026-09-04T00:59:59Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454677 wikitext text/x-wiki iz6em75pqpptgk59i4930a67bmbo24i 2454678 2454677 2026-09-04T01:00:32Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2454678 wikitext text/x-wiki quf3q1rwffr4xy3hlpzh1o4miiujvoh 2454679 2454678 2026-09-04T01:00:34Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1074.eqiad.wmnet with OS trixie 2454679 wikitext text/x-wiki pxrzbzlu1lsq9740syg18ewu92rqu4l 2454680 2454679 2026-09-04T01:02:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage 2454680 wikitext text/x-wiki l09b8f9086iglrlkib5t79pe8l243xe 2454681 2454680 2026-09-04T01:08:37Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage 2454681 wikitext text/x-wiki 6uju6ai0tg0gnml08xss0aprdou48qq 2454682 2454681 2026-09-04T01:24:20Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1065.eqiad.wmnet with OS trixie 2454682 wikitext text/x-wiki b61keg929qvgf4n9j3kbp1693u4vg7y 2454683 2454682 2026-09-04T02:00:22Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2454683 wikitext text/x-wiki arqeo7i19f4229ey906au5d9v7up3j0 2454684 2454683 2026-09-04T02:01:02Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 00m 39s) 2454684 wikitext text/x-wiki gnyfwz4y3dulc7y3urnv8q93rouibt0 2454686 2454684 2026-09-04T06:20:46Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1065 2454686 wikitext text/x-wiki f27qmmz9gioqt0lkx9jod9e9irl2k0w 2454687 2454686 2026-09-04T06:21:23Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1065 2454687 wikitext text/x-wiki fbrlwdqvd3x3h6jalm7efw6zevv4h3q 2454688 2454687 2026-09-04T06:21:50Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2454688 wikitext text/x-wiki 3jxinjhwo99j0okqksu2tq7zl0n0akk 2454689 2454688 2026-09-04T06:28:46Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1065 cloud-private - filippo@cumin1003" 2454689 wikitext text/x-wiki 58en6empnayr7n9vqetd9gx0xvsan6p 2454690 2454689 2026-09-04T06:28:50Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1065 cloud-private - filippo@cumin1003" 2454690 wikitext text/x-wiki criw38qmhwywl4n09jka1zdn0wtlmlz 2454691 2454690 2026-09-04T06:28:51Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454691 wikitext text/x-wiki rpi84ik7ac8692ul9s9nl1tgsvrb0lk 2454692 2454691 2026-09-04T06:40:40Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host ldap-rw1001.wikimedia.org with OS trixie 2454692 wikitext text/x-wiki di3ztbz3h4nk46dsit6mqhvf0a149mf 2454693 2454692 2026-09-04T06:52:51Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage 2454693 wikitext text/x-wiki 9d6enk1dhgqtyyql4lemzlybvylw0e9 2454694 2454693 2026-09-04T06:57:39Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage 2454694 wikitext text/x-wiki fd9mlw91uc2enx6vwedk336x6mhfxzn 2454695 2454694 2026-09-04T07:06:18Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: down 2454695 wikitext text/x-wiki g75k7z8uan0fkpzqsr0bnbn5ux0wxb1 2454696 2454695 2026-09-04T07:06:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: down 2454696 wikitext text/x-wiki chw37uar0k6iqy5yqtugwu3mz7ptfgq 2454697 2454696 2026-09-04T07:14:00Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw1001.wikimedia.org with OS trixie 2454697 wikitext text/x-wiki j4f6io1qhoijd72wbg2z4hgjpa76s4u 2454699 2454697 2026-09-04T07:50:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454699 wikitext text/x-wiki eul108s88a9mhaajvyyek0v04o96xd0 2454700 2454699 2026-09-04T07:51:58Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454700 wikitext text/x-wiki owjz02dukn7fqkalh9qew2rw0p8n5qu 2454701 2454700 2026-09-04T07:52:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454701 wikitext text/x-wiki 1guee8lwj7z2o4z8oxun9pvkynl6el5 2454702 2454701 2026-09-04T07:55:15Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host ldap-rw2001.wikimedia.org with OS trixie 2454702 wikitext text/x-wiki t9kbeotighbf3xq9uxp4hrgyozxh9es 2454703 2454702 2026-09-04T07:55:59Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454703 wikitext text/x-wiki bon6okkoy77zjpe9wwed348lfltll40 2454704 2454703 2026-09-04T07:56:03Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host netflow5003.eqsin.wmnet with OS trixie 2454704 wikitext text/x-wiki 7w7yxwplinvwc6gekeqdrpys22nhbhw 2454705 2454704 2026-09-04T07:57:30Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454705 wikitext text/x-wiki bml8mjeawpvi684o8bdb798wfvvl36x 2454707 2454705 2026-09-04T08:02:00Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2454707 wikitext text/x-wiki dgwe8d2tourw7qoj47297pemp9kgnak 2454708 2454707 2026-09-04T08:02:30Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2454708 wikitext text/x-wiki 8uot1f1gr5koflw0p29yszkdm5xkpkv 2454709 2454708 2026-09-04T08:03:10Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2454709 wikitext text/x-wiki 3oq1pufam38k0gceh1411ve3x7j601x 2454710 2454709 2026-09-04T08:03:20Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454710 wikitext text/x-wiki 4pq1nhrn6g86mc525fmsc61xk62gonb 2454711 2454710 2026-09-04T08:03:43Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2454711 wikitext text/x-wiki njso983rxr78x8oa7p5dkj2x0d24lij 2454712 2454711 2026-09-04T08:04:31Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454712 wikitext text/x-wiki bhtf7zhnedvaig3gvok278joafzmapz 2454713 2454712 2026-09-04T08:06:19Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2454713 wikitext text/x-wiki srdcezreaosajphuvc0dtsyuj5pc2dw 2454714 2454713 2026-09-04T08:07:55Z Stashbot 7414 btullis@deploy1003: Started scap sync-world: Trying again for T436913 2454714 wikitext text/x-wiki ixolav9g3687qy7du5994bd3uln6iui 2454715 2454714 2026-09-04T08:08:20Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2003.codfw.wmnet 2454715 wikitext text/x-wiki lacgvo5236db7rd2enby2n2o1ngbsbn 2454716 2454715 2026-09-04T08:12:49Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage 2454716 wikitext text/x-wiki 18ieth18yjdljwhmkqahoghdih2dbvw 2454717 2454716 2026-09-04T08:17:52Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage 2454717 wikitext text/x-wiki l294xc1xow3vw9g6urglv17do1d57e3 2454718 2454717 2026-09-04T08:20:15Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cassandra-dev2003.codfw.wmnet 2454718 wikitext text/x-wiki so1rlz5d5b9caddl39ori359pj4jctg 2454719 2454718 2026-09-04T08:24:37Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2003.codfw.wmnet 2454719 wikitext text/x-wiki qk2cmrw5k71cl5p753wfl8n6tkcjtcm 2454720 2454719 2026-09-04T08:25:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host cassandra-dev2003.codfw.wmnet 2454720 wikitext text/x-wiki b267fuot4bg84ck4bfe53w4emae7ft8 2454721 2454720 2026-09-04T08:26:11Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db2196.codfw.wmnet with reason: Host crashed 2454721 wikitext text/x-wiki oimby7vvtzcj9wtkq2zshkyr4n8xtww 2454722 2454721 2026-09-04T08:33:31Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw2001.wikimedia.org with OS trixie 2454722 wikitext text/x-wiki anrjluppe1cjhiwe05wcasvypeecl4j 2454723 2454722 2026-09-04T08:35:13Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cassandra-dev2003.codfw.wmnet 2454723 wikitext text/x-wiki lhm41ji5284wddkran72meis27vy3kp 2454724 2454723 2026-09-04T08:35:14Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2003.codfw.wmnet 2454724 wikitext text/x-wiki 2di79ukjafpu5g3oge6xaacdlpf6aq8 2454725 2454724 2026-09-04T08:35:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm 2454725 wikitext text/x-wiki n12qtk88jlwdv4iwn1f39m3ix9cjysl 2454726 2454725 2026-09-04T08:40:43Z Stashbot 7414 btullis@deploy1003: Finished scap sync-world: Trying again for T436913 (duration: 34m 26s) 2454726 wikitext text/x-wiki 2kzxei5qacwwr9v7bjbqi4grvm2m7k3 2454727 2454726 2026-09-04T08:45:37Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on netflow5003.eqsin.wmnet with reason: host reimage 2454727 wikitext text/x-wiki 3wdjmx3st8h6bu0kkhto46l1why1ya4 2454728 2454727 2026-09-04T08:48:37Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow5003.eqsin.wmnet with reason: host reimage 2454728 wikitext text/x-wiki 35x62g4x6g0loxaz7570l6xdplekbay 2454729 2454728 2026-09-04T08:54:17Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage 2454729 wikitext text/x-wiki 1r3rgsmt4wf8la4m18a6wc1tfd5pkxz 2454730 2454729 2026-09-04T08:55:25Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.ganeti.makevm for new host pki1003.eqiad.wmnet 2454730 wikitext text/x-wiki 5g9x7v5jh2wg6b4hkw3nm0w33tuqagz 2454731 2454730 2026-09-04T08:55:27Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.netbox 2454731 wikitext text/x-wiki fe3vwi7v1er648avx1b0w10olpla1sy 2454732 2454731 2026-09-04T08:58:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage 2454732 wikitext text/x-wiki dnenspr5wlxec34p713iui052sraegy 2454733 2454732 2026-09-04T08:59:44Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki1003.eqiad.wmnet - jmm@cumin1004" 2454733 wikitext text/x-wiki csjdq378az9qbok9sphjgiroxvr2diz 2454734 2454733 2026-09-04T08:59:47Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki1003.eqiad.wmnet - jmm@cumin1004" 2454734 wikitext text/x-wiki b7jfcc13yx9tkjf8lbctzh88n742olz 2454735 2454734 2026-09-04T08:59:48Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454735 wikitext text/x-wiki hcyj0j87nstdphkknlpd8pbfxvwlwk0 2454736 2454735 2026-09-04T08:59:49Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.wipe-cache pki1003.eqiad.wmnet on all recursors 2454736 wikitext text/x-wiki 88r20ha75pohdb42rtpbfis4lyx5s46 2454737 2454736 2026-09-04T08:59:50Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki1003.eqiad.wmnet on all recursors 2454737 wikitext text/x-wiki ou0fatuli63qlsc7zimdtluw580sorg 2454738 2454737 2026-09-04T09:00:28Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki1003.eqiad.wmnet - jmm@cumin1004" 2454738 wikitext text/x-wiki nhgbo5nau615hecrsw3svp30q97e2mz 2454739 2454738 2026-09-04T09:00:29Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki1003.eqiad.wmnet - jmm@cumin1004" 2454739 wikitext text/x-wiki opf60dij469yyx3fsh6mzdgnbe371ag 2454740 2454739 2026-09-04T09:02:53Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.reimage for host pki1003.eqiad.wmnet with OS trixie 2454740 wikitext text/x-wiki auvg4bkbf7t0m3umoth4gcdxg9z1xhu 2454741 2454740 2026-09-04T09:09:25Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow5003.eqsin.wmnet with OS trixie 2454741 wikitext text/x-wiki n5kcfd2nsjqbj2nyi654kwle9cb1ikh 2454742 2454741 2026-09-04T09:17:46Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on pki1003.eqiad.wmnet with reason: host reimage 2454742 wikitext text/x-wiki d2pd7upiievlyfo1gl9qya1mmj831kz 2454743 2454742 2026-09-04T09:18:39Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm 2454743 wikitext text/x-wiki l8q9zkob6qwlhf7i07facxg0678y2pw 2454744 2454743 2026-09-04T09:23:45Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki1003.eqiad.wmnet with reason: host reimage 2454744 wikitext text/x-wiki 13mxb49dyzaqwcf2v6ax0y2x9zxymbl 2454745 2454744 2026-09-04T09:31:26Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1182: Decommissioning 2454745 wikitext text/x-wiki r4nu8ip2wf9gaycpnl7ahutsjwy8nc7 2454746 2454745 2026-09-04T09:32:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1182: Decommissioning 2454746 wikitext text/x-wiki llcw3yy7cupc3qtbb32rb95cxnxjcvq 2454747 2454746 2026-09-04T09:39:45Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pki1003.eqiad.wmnet with OS trixie 2454747 wikitext text/x-wiki ri9ozvamy0io99nrexfptz6at7jxviq 2454748 2454747 2026-09-04T09:39:47Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host pki1003.eqiad.wmnet 2454748 wikitext text/x-wiki qytcp759wvydm4o3cc32j85evmspvka 2454749 2454748 2026-09-04T09:41:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1182.eqiad.wmnet 2454749 wikitext text/x-wiki 1hje0pkgqcbuopyir1p7zgqt1xye94v 2454750 2454749 2026-09-04T09:45:30Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1182 from dbctl T434869', diff saved to https://phabricator.wikimedia.org/P96346 and previous config saved to /var/cache/conftool/dbconfig/20260904-094527-marostegui.json 2454750 wikitext text/x-wiki lrnzrzrz0wqx2evzevog15h1787hlyl 2454751 2454750 2026-09-04T09:46:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2454751 wikitext text/x-wiki lgbuahqv6e5l0mctom8tif0nyc759os 2454752 2454751 2026-09-04T09:50:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1182.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2454752 wikitext text/x-wiki omtjscyium6bx7oc73ogjzd3kd6id1n 2454753 2454752 2026-09-04T09:51:12Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1182.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2454753 wikitext text/x-wiki c27g5gb0nd4c6ndenuuqvxqm1s7cswd 2454754 2454753 2026-09-04T09:51:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454754 wikitext text/x-wiki hgn5kqnklw5zyy5yiroosilx2zta1e6 2454755 2454754 2026-09-04T09:51:14Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1182.eqiad.wmnet 2454755 wikitext text/x-wiki 2j9jgrr68vyh7y5w3ka43761425dyp5 2454756 2454755 2026-09-04T09:53:18Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.decommission 2454756 wikitext text/x-wiki bbngdqelo6efdy65xars7cldvixf45j 2454757 2454756 2026-09-04T09:53:19Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.decommission (exit_code=1) 2454757 wikitext text/x-wiki nzewf6p54ipzf4l3d0sk6em756de1ob 2454758 2454757 2026-09-04T09:53:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.decommission 2454758 wikitext text/x-wiki 4dg49glkmysn51uofw0brq8h5v1umvu 2454759 2454758 2026-09-04T09:53:47Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1182.eqiad.wmnet 2454759 wikitext text/x-wiki hdh0ie89w1lsx5i3jsd1zs1w813u9nj 2454760 2454759 2026-09-04T09:57:02Z Stashbot 7414 btullis@deploy1003: Started scap sync-world: Incorporating changes from https://gerrit.wikimedia.org/r/c/operations/dumps/+/1334846 to mediawiki-cli 2454760 wikitext text/x-wiki 7a4fggmqq4cco3eeyxyvvllfx5pihhl 2454761 2454760 2026-09-04T09:57:50Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2454761 wikitext text/x-wiki 5thk9x6s2i4sipbgl393ww0uc9czhpk 2454762 2454761 2026-09-04T10:00:15Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454762 wikitext text/x-wiki 3p83g6rpixtttvmqvv4nyeg1yq7qkbi 2454763 2454762 2026-09-04T10:00:16Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts db1182.eqiad.wmnet 2454763 wikitext text/x-wiki npkdte4qm4irl2uh0az75wwhjy8a50f 2454764 2454763 2026-09-04T10:00:24Z Stashbot 7414 marostegui@cumin1003: Removing db1182 from zarcillo T434869 2454764 wikitext text/x-wiki nhqfs0guur1taf6hi5umnph7gf7mf14 2454765 2454764 2026-09-04T10:00:28Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2454765 wikitext text/x-wiki lwuutb9v2g0s9f5ltp1i6itam69hif8 2454766 2454765 2026-09-04T10:29:24Z Stashbot 7414 btullis@deploy1003: Finished scap sync-world: Incorporating changes from https://gerrit.wikimedia.org/r/c/operations/dumps/+/1334846 to mediawiki-cli (duration: 41m 14s) 2454766 wikitext text/x-wiki 86ds39w0cii1cb1fca8jfcy8u98ix44 2454767 2454766 2026-09-04T10:44:44Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.ganeti.makevm for new host pki2003.codfw.wmnet 2454767 wikitext text/x-wiki dp8o6zbj8lbhz24rxjkt7efvdm1efn6 2454768 2454767 2026-09-04T10:44:46Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.netbox 2454768 wikitext text/x-wiki spk6utm9q732sky9mh4cwrq0i01zfrt 2454769 2454768 2026-09-04T10:49:32Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki2003.codfw.wmnet - jmm@cumin1004" 2454769 wikitext text/x-wiki 6add64kfv5ycf5c39cygiibu9rk7haj 2454770 2454769 2026-09-04T10:49:34Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki2003.codfw.wmnet - jmm@cumin1004" 2454770 wikitext text/x-wiki 4ayydvvymzl5h3j60iu6v1rr2pmp932 2454771 2454770 2026-09-04T10:49:35Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454771 wikitext text/x-wiki 017qi7122l5ej6b6zl3nwxexs7s2us8 2454772 2454771 2026-09-04T10:49:36Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.wipe-cache pki2003.codfw.wmnet on all recursors 2454772 wikitext text/x-wiki h1h1urqgwquma9kffghmvfqa2wmk12s 2454773 2454772 2026-09-04T10:49:37Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki2003.codfw.wmnet on all recursors 2454773 wikitext text/x-wiki hlf4e619wdl2p4bjyd5790b6s5qedjo 2454774 2454773 2026-09-04T10:50:12Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki2003.codfw.wmnet - jmm@cumin1004" 2454774 wikitext text/x-wiki nr68nsfyp5e0hs6cvo3i63sfu1um7ry 2454775 2454774 2026-09-04T10:50:15Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki2003.codfw.wmnet - jmm@cumin1004" 2454775 wikitext text/x-wiki 329fgclph2gr1no3yvkvgti01af3zgc 2454776 2454775 2026-09-04T10:52:22Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.reimage for host pki2003.codfw.wmnet with OS trixie 2454776 wikitext text/x-wiki q76soj1jjz9eimk3mdq5idl5ece1ra0 2454778 2454776 2026-09-04T11:06:19Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a5-eqiad 2454778 wikitext text/x-wiki rz70fk5do6sfag1h14lt25s6bzdh4vi 2454779 2454778 2026-09-04T11:06:40Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a5-eqiad 2454779 wikitext text/x-wiki 6ect8v3okhmgalew7ow7w7awt04i5q1 2454780 2454779 2026-09-04T11:13:23Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on pki2003.codfw.wmnet with reason: host reimage 2454780 wikitext text/x-wiki 75l1qdv8kz12nt8pryb56u9fg50d2pk 2454781 2454780 2026-09-04T11:19:15Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki2003.codfw.wmnet with reason: host reimage 2454781 wikitext text/x-wiki sopu7qd1koip1sdxvguwtc3ibtwnjt3 2454782 2454781 2026-09-04T11:32:55Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pki2003.codfw.wmnet with OS trixie 2454782 wikitext text/x-wiki 5defh5ea7pq72fq8e0y7en2f4d6yf83 2454783 2454782 2026-09-04T11:32:56Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host pki2003.codfw.wmnet 2454783 wikitext text/x-wiki jfbe10hzvk17c9fb93n53i1d60emu5e 2454784 2454783 2026-09-04T11:36:33Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2454784 wikitext text/x-wiki 9t7orosbuyi4ddest0nxey8sd48excs 2454785 2454784 2026-09-04T11:47:30Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a6-eqiad 2454785 wikitext text/x-wiki 1uftw3zjifmoznpvdcn43gw96vdg5my 2454786 2454785 2026-09-04T11:47:49Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a6-eqiad 2454786 wikitext text/x-wiki 2tqao5rpm6cuwfsr4aq36anoypwxi1t 498 9925 2454669 2454137 2026-09-03T23:29:05Z Stashbot 7414 Southparkfan: attach Cinder volume for /srv on deploy06, jobrunner06, mediawiki15/mediawiki16 - T435393 2454669 wikitext text/x-wiki 3gf0749cumswsg2273yn31o0awuwlug 498 15050 2454777 2412573 2026-09-04T11:00:21Z Stashbot 7414 wmbot~lokal-profil@tools-bastion-15: Deploy 3b02307 (T316627), d1447ed, 23e5c05, 1867640, e7f31b8, 7117ae0, d828084, 093e39b 2454777 wikitext text/x-wiki 7dfjgpwc7zejtt6sowd0yzt22s3ujvr 0 17290 2454668 2454314 2026-09-03T23:29:03Z Stashbot 7414 Southparkfan: attach Cinder volume for /srv on deploy06, jobrunner06, mediawiki15/mediawiki16 - T435393 2454668 wikitext text/x-wiki m1s1dcoi4m63ww6rb3invu06dcbfvf8 0 23733 2454544 2423048 2026-09-03T15:11:03Z FCeratto-WMF 39104 Link cookbook 2454544 wikitext text/x-wiki 2uv7v27xhvm7xuun83x473famhabzs1 0 152515 2454407 2422778 2026-09-03T13:08:31Z BTullis (WMF) 25295 2454407 wikitext text/x-wiki ry0wb69t7sqmx92dfitxpj4m86ehdir 0 447352 2454334 2450662 2026-09-03T12:09:19Z LToscano (WMF) 7191 /* Storage backends */ 2454334 wikitext text/x-wiki b2jrdvopeds6cbev3yglwvwp3cht8q4 2454335 2454334 2026-09-03T12:10:17Z LToscano (WMF) 7191 /* Storage backends */ 2454335 wikitext text/x-wiki j18hh1frdw2qet4icvluuk42hpemv0u 0 448386 2454338 2451841 2026-09-03T12:12:47Z LSobanski (WMF) 18783 /* Goal */ Updating the scope 2454338 wikitext text/x-wiki llkny886reowkm0y8u0u57gq8ugibxw 0 449086 2454649 2389965 2026-09-03T20:32:20Z BCornwall-WMF 31521 Porkbun redirection 2454649 wikitext text/x-wiki ijx9a4zrra8tn9ivthd7rvunhjd5psf 0 449160 2454670 2454159 2026-09-04T00:01:45Z Dexbot 30554 Bot: Updating the report 2454670 wikitext text/x-wiki fgznwnaumiytknv1fl5r3soj5u6291u 0 459076 2454665 2443966 2026-09-03T21:06:33Z CMooney (WMF) 24288 /* First login */ 2454665 wikitext text/x-wiki b2k6pwwaljji4gjm7tw2x65kqqahhhq 0 459699 2454510 2453681 2026-09-03T14:20:18Z Stashbot 7414 ejegg: fundraising civicrm upgraded from f4f09304 to e66e5440 2454510 wikitext text/x-wiki rb903f5t1mqryl0iscg9f59ge39rtxk 2454633 2454510 2026-09-03T19:51:33Z Stashbot 7414 ejegg: payments-wiki upgraded from c10380f8 to 651be6fe 2454633 wikitext text/x-wiki ozrqg8zm22skd7ghsefccujagku9pqy 2454634 2454633 2026-09-03T19:53:23Z Stashbot 7414 ejegg: payments-wiki rolled back from 651be6fe to c10380f8 2454634 wikitext text/x-wiki qquxiqloabn2dhsno1x1j6i8u1627vh 2454644 2454634 2026-09-03T20:24:01Z Stashbot 7414 ejegg: payments-wiki upgraded from c10380f8 to 50cae559 2454644 wikitext text/x-wiki 4879q41ehj2x44dr51n83ryzfsmsiv6 0 460366 2454569 2452926 2026-09-03T16:46:12Z TBurmeister (WMF) 27417 Copyedits to add structure and simplify style 2454569 wikitext text/x-wiki 16fln7gwiaeyny375zv770ghhgwy46s 2454619 2454569 2026-09-03T18:09:12Z TBurmeister (WMF) 27417 More style edits and remove rewrite-tool-specific implementation details 2454619 wikitext text/x-wiki 0i2mhkaeu1bur9f3oji1ytx0owe51qy 0 460521 2454647 2453426 2026-09-03T20:30:17Z APaskulin (WMF) 14880 Link to Phab task 2454647 wikitext text/x-wiki mj6xghpse35r0rs7cosr1nrg7co55b9 0 460600 2454667 2453947 2026-09-03T22:12:06Z AKhatun (WMF) 50461 Added doc link, fixed format issue 2454667 wikitext text/x-wiki 3mmykqg7ktxpnmlbmi94tnbxfngkxim 0 460608 2454525 2453450 2026-09-03T14:45:09Z APizzata-WMF 47941 first draft 2454525 wikitext text/x-wiki 7zfn9gwpkacqkffahida228tr6lpq44 2 460620 2454706 2454210 2026-09-04T07:57:34Z STrug-WMF 51028 /* Modify size of an existing PVC */ 2454706 wikitext text/x-wiki 8i2fxpjnhkv1bost0gr7l1hjjoal0pc 0 460629 2454541 2026-09-03T15:07:45Z FCeratto-WMF 39104 Create cookbook summary 2454541 wikitext text/x-wiki o9ij4o4l06oqv1ar18b9q9wh8y1cz7d 2454543 2454541 2026-09-03T15:09:42Z FCeratto-WMF 39104 Document cookbooks 2454543 wikitext text/x-wiki etitztug2tbddxs6c521553xk3l6qhn 2 460630 2454685 2026-09-04T03:57:33Z Pamellasilvva 59270 /* */ 2454685 wikitext text/x-wiki azdox19poiv3dbymy00m7a7ssv88eg2