On Wednesday, 18 January 2006, the Everquest servers went down for a routine four hour patch. They came back online at 11:42 AM EST with few game changes except one - a new vendor in the Plane of Knowledge selling expensive to priceless tradeskill items for next to nothing. The Everquest developers, Sony Online Entertainment, designed these vendors for the recent betatesting of Prophecy of Ro, the eleventh Everquest expansion due for release mid February 2006. These vendors let betatesters test new tradeskill recipes with items that exist on the live servers only in much more rare quantities.
The results were quick and catastrophic. Players flooded to these new vendors, buying as many of these rare components as they could hold from the unlimited supply of the rogue vendor. Many players know that an event like this is a temporary situation so they hope to make out with what they can as long as they can. Knowing a character rollback may be likely, these players quickly transfer items to alternate characters, friends, or through their own shared bank. Players, hoping to keep their profits, might launder these rogue items through dozens of trades. Prices in the bazaar fell quickly as these new items flooded the market. All of this happened within two hours.
SOE quickly became aware of the situation and removed the rogue vendors. Yet the damage was done. Thousands of rare tradeskill items, some of them so rare that only a thousand exist across all servers, had now entered the market. While the economy of the server might balance out the error over time, many players who had never acquired a single rare tradeskill item, now might have hundreds of them. SOE made the decision to begin rolling back the characters who purchased items from the vendors and all of those who had traded with them down the line.
One can imagine the web of transactions between those who originally traded with a rogue vendor and any of those who had contact with them and so on down the line. Imagine a single piece of platinum coated with a poison that spreads to whoever holds it and anything they possess. Whenever any infected coin or item changes hands, another character and all of their possessions become infected. When it reaches the bazaar, it can flood hundreds more.
SOE contained the spread of the rogue items and all of the affected players forty eight hours after the servers originally came up. Every character touched by anyone or anything that could be traced back to the original vendors was rolled back to their last save point before the patch.
Rolling back certain characters resulted in strange lapses in the state between one character and another. One character hands an item to another character. The second character is rolled back to a previous state but the first character, having given an item and not received one, is not rolled back. This results in the loss of the original item.
Erana, an enchanter on Quellious, was one such victim. Erana, her bazaar vendor, and some friends transferred some equipment including a Manastone, an item of great power that no longer drops anywhere in Everquest. While most of these characters were rolled back including the new possesser of the Manastone, the original holder of the Manastone was not. This resulted in the Manastone becoming lost in the void between mismatched rollbacks. Like thousands of others, Erana began the painful process of navigating a complex petition form to contact customer service in the hopes of recovering her Manastone. She cares little for other items lost over the two day rollback, she can always replace those, but the Manastone is irreplaceable.
Though this example is extremely rare and most likely the exception, Erana's tale is not unique and some stories are worse. Some players lost entire corpses full of their most valuable possessions due to the paradoxical states between rollbacks. A player dies and leaves a body behind before the patch. After the patch they recover the body but become poisoned by a tainted transaction. The rollback leaves them in a state before they had recovered their body but now with no body to recover.
While these lost corpses are very rare, they are paralyzing and dismaying to those affected. A lost corpse can mean the end of a character. SOE's customer service almost always returns the body to the victim but it can take a long time for a response, a time when the player nearly convinces himself or herself that all is lost.
Most frustrating is the fact that many of the players rolled back did nothing out of the ordinary. They did not try to get rich quick on low-priced ultra-rare tradeskill items. They did not try to exploit a bug in the game. They might have accepted some coin for a buff or bought some emeralds in the bazaar. They might have transferred some gear from one character to another and lost nearly everything of value. Even if they didn't lose a body or gear, many lost all of the progress made in two days worth of play time. One player, returning to Everquest with Everquest Titanium, a $20 package including the base game and ten expansions, wrote of his confusion at losing levels and equipment for something he didn't even understand.
From the time of the initial discovery of the rogue merchants the Everquest forums exploded with posts of rage, frustration, arguments, and threats. It becomes nearly impossible to uncover the truth as angry players post rumors, assumed statistics, threats, vague details, and overdramatic outcries. One full thread of posts discussed the impact of another rogue vendor who gave out alternate advanced points when players handed them a single platinum piece. SOE developers spent time away from the real issues to dig into every script that could possibly result in a situation like this only to find out that it had never happened. No such vendor existed on the live servers, the beta servers, or anywhere else. Other players reported the losses of millions of platinum pieces, hundreds of alternate advanced points, tens of levels, and entire suits of new equipment - far greater results than any player can earn in a two-day period.
These exaggerations took much away from the real stories of those directly affected. High-end raiders lost two nights worth of victories in difficult raid zones, bazaar traders and tradeskillers lost two days worth of profits, mid-level hunters lost levels earned over two days of hunting.
At midnight on 20 January 2006, Chris Lena, the Everquest Producer, wrote a letter of apology and explained the direct results of the error and the effects. Chris stated that less than 1% of existing Everquest character were effected by the rollbacks and that customer service would help with lost corpses and guild membership status but not with lost items. Later, SOE announced that it would help with the recovery of lost epic items or epic quest pieces as well as offering a double experience bonus starting Saturday morning on 21 January and ending the following Monday. Due to some technical difficulties Saturday morning, the double experience weekend was extended until Tuesday morning.
The decision to roll back certain characters over a 48 hour period meant that many players unaffected by the poisoned transactions saw little change at all. It did, however, let the transactions continue to spread as the players rolled back. Taking down the servers and rolling back every character, affected or not, might have had many of the same problems faced already but with less of the paradoxical state for the lost 48 hours. Only SOE has the data to truly determine which method would lead to the best outcome.
Many posts questioned the 1% affected rate stated in Chris Lena's letter. It seems everyone knew someone, often more than one who was rolled back and lost time, items, money, or all three. Some entire guilds were affected as guild bankers became tainted and spread the poison out to every guild member who touched the bank. Again, no one outside of SOE knows exactly the number affected, but those who play most often, those with the widest social circles, were the most likely to become affected.
Yet the numbers matter little to those who lost much during a rollback. To the player who loses a corpse full of their hard-earned equipment or another player who loses a priceless artifact from a quick trade in the bazaar, the results are all that matter.
Now is as good a time as any for SOE to consider re-looking at their choice for equipment loss on death. At the Everquest Summit in September 2005, Craig Knapp begain the discussion of Everquest's death system. With recent games choosing alternate penalties for death instead of equipment loss, SOE should consider a new death system that lets players respawn with gear and receive another penalty - a "death effect" perhaps - to off-set the benefit of returning fully equipped.
While the psychological advantage of returning fully equipped is strong, situations like these recent rollbacks build a much stronger case. Players don't ever want to lose equipment. In today's new world of massive online games, equipment loss is too strong a penalty that effects the wrong set of players - the lower and middle level players who have few tools for corpse recovery and a smaller network of friends to assist.
If you are one of these player, what can you do? Be patient and persistant. Be polite and understanding. If you've lost a body or irreplacable gear, keep working with CS to try to recover it. Don't accept no for an answer but don't hurt your case with rudeness.
With a letter of apology and a weekend of double experience, SOE hopes to put the event back into the pages of history. While the situation will no doubt fade over time as the CS teams recover all that can be recovered and the rest of us move on with our adventures, the week of the Great Rollback will not likely become forgotten.
Loral Ciriclight
21 January 2006
loral@loralciriclight.com
Update: Erana, the enchanter in the article above, received her manastone back on 30 January 2005. Many reports came in from those who lost items in the rollback and eventually had them recovered by SOE's Customer Service staff.
This work is licensed under a Creative Commons Attribution, Share-Alike License. Contact Loral at mike@mikeshea.net.