Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Page MenuHomePhabricator
Projects SRE

SREGroup
ActivePublic
Watch Project

Members (29)

Watchers (13)

Details

Description

Homepage: https://www.mediawiki.org/wiki/Wikimedia_Site_Reliability_Engineering

If you're filing a task that needs to be handled by the SRE team, add a tag from the list below. If you're not sure which one to pick, add the SRE tag and the on-duty person will redirect it as appropriate.

Cross-team tags:

Tags by team:

Recent Activity
View All

Today

Maintenance_bot added a project to T373731: ManagementSSHDown: SRE.
Sat, Aug 31, 7:29 AM · SRE, DC-Ops, ops-codfw
Dzahn added a comment to T373666: Requesting access to deployment for zoe.

Hi @VPuffetMichel , looks like this is just pending your approval and then it's good to go. Cheers.

Sat, Aug 31, 3:52 AM · Patch-For-Review, SRE, SRE-Access-Requests
Dzahn triaged T373666: Requesting access to deployment for zoe as Medium priority.
Sat, Aug 31, 3:51 AM · Patch-For-Review, SRE, SRE-Access-Requests
Dzahn assigned T373666: Requesting access to deployment for zoe to VPuffetMichel.
Sat, Aug 31, 3:50 AM · Patch-For-Review, SRE, SRE-Access-Requests
Dzahn moved T373666: Requesting access to deployment for zoe from Untriaged to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.
Sat, Aug 31, 3:47 AM · Patch-For-Review, SRE, SRE-Access-Requests
Dzahn changed the status of T373666: Requesting access to deployment for zoe from Open to In Progress.
Sat, Aug 31, 3:47 AM · Patch-For-Review, SRE, SRE-Access-Requests
Dzahn triaged T373727: ganeti2009 - ManagementSSHDown as Low priority.
Sat, Aug 31, 3:42 AM · Infrastructure-Foundations, SRE, DC-Ops, ops-codfw
Dzahn added a project to T373727: ganeti2009 - ManagementSSHDown: Infrastructure-Foundations.
Sat, Aug 31, 3:42 AM · Infrastructure-Foundations, SRE, DC-Ops, ops-codfw
Dzahn renamed T373727: ganeti2009 - ManagementSSHDown from ManagementSSHDown to ganeti2009 - ManagementSSHDown.
Sat, Aug 31, 3:40 AM · Infrastructure-Foundations, SRE, DC-Ops, ops-codfw
Maintenance_bot added a project to T373727: ganeti2009 - ManagementSSHDown: SRE.
Sat, Aug 31, 3:29 AM · Infrastructure-Foundations, SRE, DC-Ops, ops-codfw

Yesterday

ops-monitoring-bot added a comment to T362841: Degraded RAID on aqs1014.

Host rebooted by eevans@cumin1002 with reason: None

Fri, Aug 30, 6:51 PM · DC-Ops, Cassandra, SRE, ops-eqiad
Dzahn added a comment to T373105: Migrate servers in codfw racks D7 & D8 from asw to lsw.

The server gerrit2002 mentioned here for Collaboration Services is a replica, not the main host. It's somewhat in production but a downtime for a few minutes is acceptable any time.

Fri, Aug 30, 5:34 PM · collaboration-services, DC-Ops, ops-codfw, netops, Infrastructure-Foundations, SRE
Dzahn added a comment to T373096: Migrate servers in codfw racks C2 & C3 from asw to lsw.

The server phab2002 mentioned here for Collaboration Services is standby and therefore ok to do anytime.

Fri, Aug 30, 5:33 PM · collaboration-services, DC-Ops, ops-codfw, netops, Infrastructure-Foundations, SRE
Dzahn added a comment to T373102: Migrate servers in codfw racks D1 & D2 from asw to lsw.

The server lists2001 mentioned here for Collaboration Services is standby and therefore ok to do anytime.

Fri, Aug 30, 5:30 PM · collaboration-services, ops-codfw, netops, Infrastructure-Foundations, SRE, DC-Ops
Stashbot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Mentioned in SAL (#wikimedia-operations) [2024-08-30T16:59:47Z] <swfrench-wmf> running homer 'cr*codfw*' commit 'T372878'

Fri, Aug 30, 4:59 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
Scott_French updated the task description for T373591: Relabel codfw kubernetes nodes.
Fri, Aug 30, 4:47 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
Stashbot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Mentioned in SAL (#wikimedia-operations) [2024-08-30T16:40:31Z] <swfrench-wmf> running homer 'lsw1-b3-codfw*' commit 'T372878'

Fri, Aug 30, 4:40 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2065.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2065 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408301619_swfrench_3962105_wikikube-worker2065.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Aug 30, 4:39 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2064.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2064 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408301612_swfrench_3955684_wikikube-worker2064.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Aug 30, 4:32 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
Stashbot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Mentioned in SAL (#wikimedia-operations) [2024-08-30T16:26:41Z] <claime> homer 'cr*eqiad*' commit 'T351074, T372878, and fix ml-serve and dse-k8s bgp'

Fri, Aug 30, 4:30 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
Maintenance_bot added a project to T373699: Relabel codfw kubernetes nodes mw237[789]: SRE.
Fri, Aug 30, 4:29 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
Clement_Goubert merged T373699: Relabel codfw kubernetes nodes mw237[789] into T373591: Relabel codfw kubernetes nodes.
Fri, Aug 30, 4:25 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
Clement_Goubert updated the task description for T373591: Relabel codfw kubernetes nodes.
Fri, Aug 30, 4:24 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
hnowlan updated the task description for T373591: Relabel codfw kubernetes nodes.
Fri, Aug 30, 4:08 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2061.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2061 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408301541_hnowlan_1236879_wikikube-worker2061.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Aug 30, 4:01 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1033.eqiad.wmnet with OS bullseye completed:

  • wikikube-worker1033 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408301538_cgoubert_1238701_wikikube-worker1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Aug 30, 3:57 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2062.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2062 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408301535_hnowlan_1236883_wikikube-worker2062.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Aug 30, 3:55 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2065.codfw.wmnet with OS bullseye

Fri, Aug 30, 3:52 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2060.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2060 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408301527_hnowlan_1237077_wikikube-worker2060.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Aug 30, 3:47 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2064.codfw.wmnet with OS bullseye

Fri, Aug 30, 3:45 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from kubernetes2057 to wikikube-worker2065 completed:

  • kubernetes2057 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Netbox updated
    • ✔️ IDRAC updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
Fri, Aug 30, 3:44 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2063.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2063 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408301520_cgoubert_1233105_wikikube-worker2063.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Aug 30, 3:40 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from kubernetes2030 to wikikube-worker2064 completed:

  • kubernetes2030 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Netbox updated
    • ✔️ IDRAC updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
Fri, Aug 30, 3:38 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
Maintenance_bot removed a project from T372432: Q1:rack/setup/install ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x): Patch-For-Review.
Fri, Aug 30, 3:32 PM · SRE, Machine-Learning-Team, ops-eqiad, DC-Ops
Maintenance_bot removed a project from T371372: ngkountas user has same SSH key for cloud/prod: Patch-For-Review.
Fri, Aug 30, 3:30 PM · SRE, SRE-Access-Requests
gerritbot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Change #1069231 merged by Scott French:

[operations/puppet@production] kubernetes: re-name/IP kubernetes20(30|57) as wikikube-worker206[45]

https://gerrit.wikimedia.org/r/1069231

Fri, Aug 30, 3:30 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
Maintenance_bot added a project to T373696: Relabel eqiad kubernetes nodes: SRE.
Fri, Aug 30, 3:29 PM · SRE, ops-eqiad, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
Clement_Goubert merged task T373669: Relabel codfw kubernetes nodes mw2295,mw2296,mw2297 into T373591: Relabel codfw kubernetes nodes.
Fri, Aug 30, 3:25 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
Clement_Goubert merged T373669: Relabel codfw kubernetes nodes mw2295,mw2296,mw2297 into T373591: Relabel codfw kubernetes nodes.
Fri, Aug 30, 3:24 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
Clement_Goubert updated the task description for T373591: Relabel codfw kubernetes nodes.
Fri, Aug 30, 3:23 PM · SRE, ops-codfw, Kubernetes, Prod-Kubernetes, DC-Ops, serviceops
gerritbot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Change #1069231 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] kubernetes: re-name/IP kubernetes20(30|57) as wikikube-worker206[45]

https://gerrit.wikimedia.org/r/1069231

Fri, Aug 30, 3:23 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1033.eqiad.wmnet with OS bullseye

Fri, Aug 30, 3:19 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw1398 to wikikube-worker1033 completed:

  • mw1398 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Netbox updated
    • ✔️ IDRAC updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
Fri, Aug 30, 3:18 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2060.codfw.wmnet with OS bullseye

Fri, Aug 30, 3:07 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2060.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2060 (FAIL)
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" wikikube-worker2060.codfw.wmnet to get a root shellbut depending on the failure this may not work.
Fri, Aug 30, 3:07 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2061.codfw.wmnet with OS bullseye

Fri, Aug 30, 3:07 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2060.codfw.wmnet with OS bullseye

Fri, Aug 30, 3:07 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2062.codfw.wmnet with OS bullseye

Fri, Aug 30, 3:07 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2383 to wikikube-worker2060 completed:

  • mw2383 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Netbox updated
    • ✔️ IDRAC updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
Fri, Aug 30, 3:05 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
ops-monitoring-bot added a comment to T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets.

Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2384 to wikikube-worker2061 completed:

  • mw2384 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Netbox updated
    • ✔️ IDRAC updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
Fri, Aug 30, 2:58 PM · Patch-For-Review, serviceops, netops, SRE, Infrastructure-Foundations
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits