Farm Intelligence , which provides an analytics service to help farmers grow better crops, turned to a combination of Amazon Web Services and software from Zadara Storage to achieve the scalability and flexibility it needs to process rapidly growing data.
Farm Intelligence collects data from sources that include government satellites, weather data, its own custom sensors and information gathered from customers. It then analyzes all the data with its software and advises farmers on the adjustments they can make to maximize crop yield, such as increasing water or nitrogen.
"We're really a big data company disguised as an agricultural farming company," said Steve Kickert, chief technology officer at Farm Intelligence.
The Mankato, Minn.-based company primarily analyzes corn and soybean crops, but Kickert said it will add customers that grow potatoes, sugar beets and other crops. That will extend its reach beyond the more than 1,000,000 acres of land that Farm Intelligence gathers data on today.
That adds up to a lot of information growing like the proverbial weed. Kickert said Farm Intelligence's data is growing by 10 terabytes (TB) per week, and he expects to have approximately 1 petabyte by the end of the year. Farm Intelligence started in 2011, and Kickert said the company is in rapid growth mode.
That's why he decided to replace the firm's 100 TB in-house SAN with cloud storage. "We don't want to be in the infrastructure business," Kickert said. "The ability to scale in-house wasn't there; we didn't want to be setting up servers manually and that kind of thing. Our goal is to have zero production infrastructure footprint here."
He said Farm Intelligence's sensors can move the data directly to the cloud, or it can be sent to headquarters and moved to Amazon Web Services (AWS) from there. "We added Zadara Storage to prepare us for a massive scale of data in 2014 and beyond," he said.
"I knew I wanted to be in the cloud," Kickert said, and he thought Amazon was the best fit. However, AWS Elastic Block Store has scaling and performance limitations that didn't support the type of data Farm Intelligence would send its way.
Scott Colestock, Farm Intelligence's director of cloud operations, said Amazon recommended he look at Zadara's Virtual Private Storage Array (VPSA) to complement Amazon's Elastic Compute Cloud (EC2) service.
"We needed contiguous block storage, but on a massive scale," Kickert said. "Amazon is not set up to have the type of storage that Zadara provides at the petabyte level. Some software that we develop and use for our GIS [geographic information system] couldn't use S3 [Simple Storage Service] object storage, which is where Amazon really shines in terms of massive scale.
"The other thing I like with Zadara is the ability to clone our storage at will," Kickert added. "If we decide we don't want to be on Amazon, we can bring Zadara in behind our firewall."
Farm Intelligence's processing algorithms run on EC2 and connect to servers running VPSA through AWS Direct Connect high-speed fiber service. VPSA is based on OpenStack technology and includes storage features such as Quality of Service, low-impact snapshots with read/write clones, solid-state drive read/write caching, thin provisioning and asynchronous replication. Farm Intelligence's staff accesses VPSA data through a Web portal that can add or eliminate drives, adjust controller performance and perform other management tasks on the fly.
Farm Intelligence began migrating data to AWS in January, with a deadline of April 1 to have it set up for the start of planting season. "Zadara supplied us with a set of appliances to move all of last year's data," Colestock said. "That's up and running."