The Electronic Arts Inc. technical team based in Redwood City, Calif., faced an unexpected obstacle getting its...
Battlefield 4 PC game out the door: the untimely demise of its cloud storage provider.
Electronic Arts (EA) game developers kept their critical game build files -- close to 100 TB of data -- on Nirvanix Cloud Storage, which suddenly went out of business six weeks before the scheduled Oct. 29 ship date of Battlefield 4. That left the technical team scrambling to save its data, which the developers needed to access in the cloud in real time.
EA Technical Director Steve Scivally said he found out about the shutdown when one of Nirvanix's directors called EA on Sept. 16, saying the cloud provider would close in two weeks. That shutdown date was subsequently extended to Oct. 15, giving EA a month to recover its data.
There's never a good time to hear your cloud storage provider is going out of business, but the timing was especially bad for EA. The company had six titles scheduled to launch over the next two months or so, beginning with Battlefield 4 for PC.
"All those products were in that [cloud] storage," Scivally said.
Scivally and his team considered their options and decided to move all the data to the Amazon Simple Storage Service (S3) cloud. EA used the same Panzura Cloud Controllers it had in 20 offices around the world to write to the Nirvanix cloud, and pointed them to S3. The trick was to do it without downtime.
"We planned all day Tuesday [Sept. 17], and by Thursday we started moving data to Amazon," Scivally said. "We swapped our cloud provider, cleared our Panzura caches and started fresh. We spent the following two weeks recovering critical data from Nirvanix, but the new data was going into the Amazon front end."
Scivally said his team prioritized data on the Nirvanix cloud. The most recent data was usually the most important because EA developers test the most recent game builds. Older files were left behind and eventually deleted.
"We staggered it according to the criticality of content," he said. "We said we had to get whatever we needed in two weeks, and decided what that would be based on performance and throughput. We started squirreling it away to another provider or internal storage. We identified other things we would not bother downloading."
Scivally said Panzura's support helped get everything finished on time. The EA team handled the data migration part of the switch, while Panzura set the devices to switch to the Amazon cloud. Panzura gave EA the option of moving data directly from Nirvanix to Amazon, but that was not optimal for EA's setup.
"Instead of doing cloud-to-cloud we did cloud-to-customer-to-cloud," he said. "Cloud-to-cloud didn't make sense for us. We would have [had] to stop everything and take a snapshot and transfer that, but we're constantly writing new data. Cloud-to-cloud would work for backup or archive, but ours is a real-time application. Stopping for a week or however long it would take was not an option."
EA has two controllers in some offices for high availability, but put them to work connecting to the cloud simultaneously to speed the migration. Developers wrote new builds to Amazon while EA pulled its files back from Nirvanix.
Scivally said his developers pulled data from the Nirvanix cloud from offices in Los Angeles, Baton Rouge, La., Japan and Romania. He said it took only three days to begin writing data to S3 instead of Nirvanix, but they needed another three-plus weeks to recover all the data EA needed from Nirvanix. That process finished Oct. 11.
Getting data off the Nirvanix cloud became more difficult as time went on because most of the cloud vendor's customers were accessing it simultaneously. EA recovered critical data and left the rest. "Our new builds went into the new cloud storage and we back-filled critical builds," Scivally said. "There was a lot of data from Nirvanix that we didn't bother recovering. After we finished recovering, we went back and deleted data [we] no longer needed."
EA already used Amazon's cloud for compute, but not storage. Scivally said EA chose Nirvanix Cloud Storage when it installed Panzura controllers in 2012 because it offered geo-replication -- the ability to replicate data between sites in different parts of the world. That allowed EA development teams in different parts of the world to work on game builds at the same time. With Amazon and other clouds, he said, the same data would have to be stored separately from multiple locations.
Nirvanix allowed EA to create a game build in its Vancouver office and use developers in other cities to test those builds.
"Nirvanix was the only company that provided geographic data replication -- that was a key feature nobody else had. You could clone data from North America to Asia," he said. "Now we just host data out of North America."
When asked if he was worried at any time about getting the critical data off Nirvanix in time to beat his deadlines, Scivally said: "Always. Now that it's over I'm impressed that it went so well, but a month ago was not a comfortable time."
Now that the migration is over, EA's developers miss Nirvanix's replication features.
"From a data migration perspective, it's back to business as usual," Scivally said. "We're a bit challenged in terms of performance, tuning and configuration because we had to choose a specific location where data is stored. Latency to Asia is a bit of a challenge. I don't know if we'll ever get back to what we had with Nirvanix."