We've now fully recovered from an outage that began this morning Eastern time.
Timeline
9:57 PM (June 7)
- We received an alert that the website was down.
- Connectivity issues were intermittent.
- The problem did not last long enough for any proper debugging to occur.
5:41 AM (June 8)
- We received an alert that the website was down.
- Notifications did not fire as expected, no one on our ops team was notified.
- We sleep.
- Based on our traffic data, the site came back online a few moments later.
11:41 AM
- We received an alert that the website was down.
- Investigations began.
- Private networking between our hardware was failing when going to our primary database.
- Servers were rebooted.
12:09 PM
- Our ISP provided a notice that a hardware failure had occurred on one of our racks with the private networking equipment.
- We opted to not failover the site to our secondary nodes as we had expected the failure to be resolved quickly.
5:19 PM
- Our ISP indicated the hardware had been replaced.
- Private networking between our hardware was working again.
- Out of Games came online with 50% capacity.
5:27 PM
- We started a reboot process on our hardware.
- One of our servers no longer responded to ping requests.
- Out of Games was online at 30% capacity.
5:49 PM
- A rescue operation from our ISPs datacenter team started on the down server.
6:39 PM
- Bad CMOS battery replaced on downed server.
- Server booted successfully.
- Out of Games remained at 30% capacity.
9:56 PM
- Fixed an operating system issue that was preventing Out of Games services from communicating with each other.
- Docker Swarm was having a fit with our networking cards.
- This was a previous issue we encountered upon updating our docker software and had thought it was resolved.
- Changes did not persist correctly on server reboots.
What We've Learned
- Confirm whenever you make changes to your operating systems they will actually persist on reboots.
- Just one of those things everyone should be more careful about. Document everything.
- We should have disabled CMOS checks on our hardware and instead let it simply log the issue.
- This would have enabled our hardware to come back online without any real issue.
Also, the site was quite slow during Summer Game Fest, and I never got a chance to dive into it because of how much stuff was going on. I believe the cause of this was failing network equipment at our ISP.
