Feedback

When I try to edit a post I already made on the website, the website keep displaying the message Internal Server Error. Is this something only on PC? Can this be fixed quickly?

We've now fully recovered from an outage that began this morning Eastern time.

Timeline

9:57 PM (June 7)

• We received an alert that the website was down.
• Connectivity issues were intermittent.
• The problem did not last long enough for any proper debugging to occur.

5:41 AM (June 8)

• We received an alert that the website was down.
• Notifications did not fire as expected, no one on our ops team was notified.
• We sleep.
• Based on our traffic data, the site came back online a few moments later.

11:41 AM

• We received an alert that the website was down.
• Investigations began.
• Private networking between our hardware was failing when going to our primary database.
• Servers were rebooted.

12:09 PM

• Our ISP provided a notice that a hardware failure had occurred on one of our racks with the private networking equipment.
• We opted to not failover the site to our secondary nodes as we had expected the failure to be resolved quickly.

5:19 PM

• Our ISP indicated the hardware had been replaced.
• Private networking between our hardware was working again.
• Out of Games came online with 50% capacity.

5:27 PM

• We started a reboot process on our hardware.
• One of our servers no longer responded to ping requests.
• Out of Games was online at 30% capacity.

5:49 PM

• A rescue operation from our ISPs datacenter team started on the down server.

6:39 PM

• Bad CMOS battery replaced on downed server.
• Server booted successfully.
• Out of Games remained at 30% capacity.

9:56 PM

• Fixed an operating system issue that was preventing Out of Games services from communicating with each other.
  • Docker Swarm was having a fit with our networking cards.
  • This was a previous issue we encountered upon updating our docker software and had thought it was resolved.
  • Changes did not persist correctly on server reboots.

What We've Learned

• Confirm whenever you make changes to your operating systems they will actually persist on reboots.
  • Just one of those things everyone should be more careful about. Document everything. 
• We should have disabled CMOS checks on our hardware and instead let it simply log the issue.
  • This would have enabled our hardware to come back online without any real issue.

Also, the site was quite slow during Summer Game Fest, and I never got a chance to dive into it because of how much stuff was going on. I believe the cause of this was failing network equipment at our ISP.

This thread will keep track of Known Issues on Out of Games and we'll update is as issues are resolved. After a short period of time, we'll remove fixed issues from this list.

When issues are fixed, they will be posted as Patch Notes on the Feedback Realm.

You are welcome to post new issues in this thread, however, it can be more beneficial to create a dedicated thread in the Feedback Realm instead to allow for higher visibility.

Current Known Issues

Achievements

• Achievements are not firing for activities.
  • There are new "completed x activities" achievements that aren't firing and have been removed temporarily.
  • If you complete any activities while these aren't working, you will not have credit for their completion afterward.

Editor

• The code box no longer has syntax coloring.

Search

• Realm threads are unable to be located.

Recently Fixed Issues

This list does not include issues that were not previously on the known issues list.

Discord

• Notifications are not going out for news.

Editor

• Card game related site bbcode buttons are missing.
• You cannot quote other users.

Steps:

- open site

- click on Community tab

- scroll down and click any page number

What happens:

- the main page reloads, with the "News" section active

Expected:

- the main page doesn't fully reload and the Community tab stays selected.

Might also fix the issue where the tab that you were last on is not memorized and reloaded upon going back to the main page from anywhere (News is always the active one currently).

I'm on mobile :)