On October 14th, Chromiecraft had a couple of server crashes, followed by a much longer than typical server outage from a player perspective. Before the server was taken down for investigation, players noticed when they logged back in that they had lost all their equipment. In resolving this instability, we noticed multiple issues worth discussing.
What caused the crashes?
Well, the server didn’t crash per se, but instead was forcing itself to shut down [1]. The reason was that we had accumulated too many items over the lifespan of the server. The database can only support up to 4 billion items saved, which sounds like a lot, but when you run a server as long as ChromieCraft (coming up on 6 years!), this limit becomes a lot more likely to hit. To players, this felt the same as a server crash, but it was the server trying to keep itself from causing more problems.
Issue 1: Why so many items?
When we heard players were losing items, the first thing we checked was the item_instance database table (the one responsible for holding items and anything unique about them, like their owner, enchants, gems, and their Globally Unique Identifier, or GUID), but the table was still filled with plenty of items (only around 20 million). The GUIDs of the items, however, were up in the 4 billion range. Unfortunately, Azerothcore (and by extension Chromiecraft) does not currently recycle or reorder GUIDs for items.
Imagine the following: A player goes to a vendor and purchases a Shortsword. That Shortsword would have a GUID of 1. If the player then were to purchase a Hatchet, that Hatchet would have a GUID of 2. However, if the player sells their Shortsword and then buys a Mace, that Mace would have a GUID of 3, not 1, because the server only ever increments the GUIDs that it generates.
Think of all the items that get sold, consumed, or otherwise destroyed in regular gameplay. The GUIDs of these items don’t get reused, so they sit, wasting values that could otherwise be populated by new items. This is how the server got to the point where it was shutting itself down.
Issue 2: Logging in without equipment?
When you get to the point where you’re going past the limits on what your data can store, things can get really messed up. There’s a piece of code that tries to make sure that the server won’t collide with GUIDs and fail to save changes at the database level [2]. That code checks what the highest GUID currently is in the item_instance database table, and then clears out other tables of data that has GUIDs higher than what it found (since that data shouldn’t be there to begin with). Here is a simplified version of that code:
result = DatabaseQuery("GET HIGHEST GUID FROM item_instance");
if (result)
NextGuid = (uint32(result) + 1);
// Cleanup non-existent GUIDs from other tables
DatabaseQuery("DELETE FROM character_inventory WHERE item >= 'NextGuid'");
DatabaseQuery("DELETE FROM mail_items WHERE item_guid >= 'NextGuid'");
DatabaseQuery("DELETE FROM auctionhouse WHERE itemguid >= 'NextGuid'");
DatabaseQuery("DELETE FROM guild_bank_item WHERE item_guid >= 'NextGuid'");
This is usually safe. The data it’s meant to be clearing out shouldn’t be there in the first place, and it avoids the server trying to write into spots that are already filled. Unfortunately, this operation can fail in two ways.
1. The first database query fails. If no result is read from the first query, it doesn’t properly set the next GUID value to use, so it likely picks a random value or the value 1. Either way, data’s going to be lost.
2. The database comes back with exactly the highest possible number. This is what happened on Chromiecraft. The highest GUID in the item_instance table was exactly 4,294,967,295, and the next GUID was set to be the value right above that, which overflowed to 0. The server then proceeded to wipe everything from the tables.
Not ideal.
Issue 3: What about backups?
If you have to ask this in your project, something has gone horribly wrong. We’ve been fortunate not to have needed to ask this often on Chromiecraft, but in this case, the damage was done. This is unfortunately not something we will go into detail about for security reasons, but we could not access our primary backup due to the inopportune timing of the instability. We also discovered that our secondary method of backing up was malfunctioning and hadn’t properly backed up in approximately 2 days, which meant some data was rolled back. The backup was only used to restore existing item links, so items created after the backup was taken but before the instability on the 14th weren’t linked to any player, though they can be reconciled manually.
Final takeaways
If the last time you logged in was before the 12th, your character is unchanged. If you logged in after that, you might have noticed changes to your bags, bank, mailbox and equipment, including missing items due to them being unlinked to our database tables. The team have been hard at work restoring those items which should be now in your mailbox (please be aware only 50 mails can be shown at once, after which it takes another minute or two for the next batch to come through). Should any items still be missing please do make a ticket on Discord with as much information about your case so we can try to resolve it.
We are also investigating all of our backup solutions and ensuring they are backing up properly every time going forward.
Chromiecraft’s mission is to discover and fix issues within Azerothcore, and an especially serious issue like this can easily go unnoticed without a long-running server to expose it. Thanks to this discovery, servers using Azerothcore in the future will not have to worry about this issue affecting them.
We would like to thank our community for their patience during the outage, as well as for the overwhelmingly kind words and support we received throughout. We sincerely apologise for the inconvenience caused.
As a small token of our appreciation for your patience and the incredible kindness shown by the community, we have brought back Joyous Journeys for one week, running until 21st October at 23:59 UTC (server time), giving you 50% more EXP to make up for the roughly 6 hours the server was unavailable for players.
Here’s hoping the rest of the week goes a bit smoother!