Caught between tomorrow's onslaught and today's data requirements?

The press is alive with stories predicting a huge wave of data coming. Some say within a short period of time, it's going to dwarf all the digital data created and stored to date. Wow. Like any other hot topic, the speculation is quite wild, yet, somewhere in there is a truth: the realm of digital data is growing and there's no foreseeable end to it. As a government focused guy, I know this phenomenon is top of mind with folks on all levels of government.

Now, let's swap gears for a moment. Last week, I had lunch with Adelaide O'Brien from IDC to catch up with my new gig at Cleversafe and she brought up the subject of records management within the context of the impending Data Tsunami. At first, I thought, Records Management? That's a stale conversation, all that work has already been done. Silly me. I should know better! Adelaide gave me a number of things to think about that are anything BUT stale, yet, not necessarily weighing heavily on many government people's minds (or, at the very least, not getting any press). Governance, FOIA & eDiscovery are the more compelling topics. As we talked, I began to realize there may be somewhat of a disconnect between the senior IT leaders and those a little father down in the organization. (Bear in mind, these are not the words of Adelaide or representative of her position, they are my impressions only.) The gist is this; the senior folks seem confident that their agencies are successfully handling records management, FOIA requests and are well positioned for eDiscovery, yet, the folks doing the work are not as confident, citing many examples of records being kept too long, the need to break FOIA responses down to paper, etc.

The conversation left me wondering how government can be preparing for the next wave of data without continuing to address it's current state.

I would love to hear from others who can continue to help me shape my thoughts on this issue. Am I on the right track? Am I missing the point altogether?

Whatcha gonna do with all that data?

Ah, big data. Such a fun topic of discussion. Central to many conversations is Hadoop, and for good reason. You would be hard pressed to find a more suitable framework for analyzing large data sets. The simple idea of taking compute to the data rather than bringing data to the compute resources has changed the game.

So, of course, the promise of a tool such as Hadoop would lead one to believe that an organization should be able to run an analysis on any data set, regardless of scale. However, (don't you hate when someone writes 'However'? You know something is coming that you don't want to hear!) there are some challenges to overcome to allow Hadoop to live up to it's full potential.

As many of you most likely know, Hadoop leverages a file system approach (HDFS) to storing data to be analyzed. And, like most file system storage approaches, there are limitations. Doesn't it strike you as odd that a framework designed to analyze HUGE amounts of data is dependent on a file system that runs out of gas long before the framework does?? Well, it did strike some folks at Cleversafe and Lockheed Martin as odd.

To that end, we've announced a new offering that will address this limitation by swapping out HDFS with Cleversafe's object storage solution. Based on Information Dispersal, this offering will allow organizations to enjoy all the scalability, reliability and efficiencies of Cleversafe's technology within their Hadoop environments. 

To learn more about this, I encourage you read Bob Gourley's impressions and the article that ran in GCN on July 10th.

I'd love to hear your thoughts...

If you don't like it, don't buy it

Being a relatively opinionated person myself, I've grown accustomed to using various online channels to share my thoughts on topics from technology to parenting and practically everything in between. For the most part, I generally try to keep my comments productive and positive. I figure my comments might actually be more meaningful and useful if I take such an approach.

It never ceases to amaze me when people use social media and other digital channels to beat each other up based on choices. I find myself asking, why??  Now, I can understand passion for a topic. I can understand begin steadfast in a belief. What makes me shake my head is when opinion sharing becomes snarky, rude, offensive, etc.  Again, why??

Apple vs. (name your favorite PC vendor). If you are happy with your choice, awesome! If you don't care for someone else's choice, again, awesome!

Ford vs. (whoever). Do you like the car you drive? Sweet! Don't care for your neighbor's car of choice? That's cool too, but, do you need to hate on him/her?

Don't like Facebook? Hate Twitter? YouTube? Cats? Dogs? Hot Weather? Cold Weather? Broccoli? Chicken? Football? Soccer? Rock Music? Rap Music? ????  Fine! But, why be judgmental about these things and even more importantly, why be judgmental about the people who DO happen to like them? It's not like someone's choice to use a Mac significantly alters the life of someone who prefers a PC, right??

Bottom line, we are all free to like or hate whatever we wish, but, can't we learn to express our choices and opinions in such a way that we are not being offensive? I love hearing alternative perspectives to my own, but, only when presented in a reasonable fashion. There have always been people who are more than happy to push their opinions on others to the point of being rude. However, with the relative ease of online publishing and the ability to do so anonymously, it seems many more people feel comfortable being nasty. After all, there's no consequences, so why NOT be rude?

My simple rule is this; If I don't like something, it's possible I may mention it, but, I don't buy it, use it, eat it, listen to it or endorse it. If you like whatever it is, I'm glad you're happy.

Thanks to No Haters Pictures for the above image.

Click here to get Images & No Haters Pictures - Pictures