Whatcha gonna do with all that data?

Ah, big data. Such a fun topic of discussion. Central to many conversations is Hadoop, and for good reason. You would be hard pressed to find a more suitable framework for analyzing large data sets. The simple idea of taking compute to the data rather than bringing data to the compute resources has changed the game.

So, of course, the promise of a tool such as Hadoop would lead one to believe that an organization should be able to run an analysis on any data set, regardless of scale. However, (don't you hate when someone writes 'However'? You know something is coming that you don't want to hear!) there are some challenges to overcome to allow Hadoop to live up to it's full potential.

As many of you most likely know, Hadoop leverages a file system approach (HDFS) to storing data to be analyzed. And, like most file system storage approaches, there are limitations. Doesn't it strike you as odd that a framework designed to analyze HUGE amounts of data is dependent on a file system that runs out of gas long before the framework does?? Well, it did strike some folks at Cleversafe and Lockheed Martin as odd.

To that end, we've announced a new offering that will address this limitation by swapping out HDFS with Cleversafe's object storage solution. Based on Information Dispersal, this offering will allow organizations to enjoy all the scalability, reliability and efficiencies of Cleversafe's technology within their Hadoop environments. 

To learn more about this, I encourage you read Bob Gourley's impressions and the article that ran in GCN on July 10th.

I'd love to hear your thoughts...