Caught between tomorrow's onslaught and today's data requirements?

The press is alive with stories predicting a huge wave of data coming. Some say within a short period of time, it's going to dwarf all the digital data created and stored to date. Wow. Like any other hot topic, the speculation is quite wild, yet, somewhere in there is a truth: the realm of digital data is growing and there's no foreseeable end to it. As a government focused guy, I know this phenomenon is top of mind with folks on all levels of government.

Now, let's swap gears for a moment. Last week, I had lunch with Adelaide O'Brien from IDC to catch up with my new gig at Cleversafe and she brought up the subject of records management within the context of the impending Data Tsunami. At first, I thought, Records Management? That's a stale conversation, all that work has already been done. Silly me. I should know better! Adelaide gave me a number of things to think about that are anything BUT stale, yet, not necessarily weighing heavily on many government people's minds (or, at the very least, not getting any press). Governance, FOIA & eDiscovery are the more compelling topics. As we talked, I began to realize there may be somewhat of a disconnect between the senior IT leaders and those a little father down in the organization. (Bear in mind, these are not the words of Adelaide or representative of her position, they are my impressions only.) The gist is this; the senior folks seem confident that their agencies are successfully handling records management, FOIA requests and are well positioned for eDiscovery, yet, the folks doing the work are not as confident, citing many examples of records being kept too long, the need to break FOIA responses down to paper, etc.

The conversation left me wondering how government can be preparing for the next wave of data without continuing to address it's current state.

I would love to hear from others who can continue to help me shape my thoughts on this issue. Am I on the right track? Am I missing the point altogether?

Whatcha gonna do with all that data?

Ah, big data. Such a fun topic of discussion. Central to many conversations is Hadoop, and for good reason. You would be hard pressed to find a more suitable framework for analyzing large data sets. The simple idea of taking compute to the data rather than bringing data to the compute resources has changed the game.

So, of course, the promise of a tool such as Hadoop would lead one to believe that an organization should be able to run an analysis on any data set, regardless of scale. However, (don't you hate when someone writes 'However'? You know something is coming that you don't want to hear!) there are some challenges to overcome to allow Hadoop to live up to it's full potential.

As many of you most likely know, Hadoop leverages a file system approach (HDFS) to storing data to be analyzed. And, like most file system storage approaches, there are limitations. Doesn't it strike you as odd that a framework designed to analyze HUGE amounts of data is dependent on a file system that runs out of gas long before the framework does?? Well, it did strike some folks at Cleversafe and Lockheed Martin as odd.

To that end, we've announced a new offering that will address this limitation by swapping out HDFS with Cleversafe's object storage solution. Based on Information Dispersal, this offering will allow organizations to enjoy all the scalability, reliability and efficiencies of Cleversafe's technology within their Hadoop environments. 

To learn more about this, I encourage you read Bob Gourley's impressions and the article that ran in GCN on July 10th.

I'd love to hear your thoughts...

If you don't like it, don't buy it

Being a relatively opinionated person myself, I've grown accustomed to using various online channels to share my thoughts on topics from technology to parenting and practically everything in between. For the most part, I generally try to keep my comments productive and positive. I figure my comments might actually be more meaningful and useful if I take such an approach.

It never ceases to amaze me when people use social media and other digital channels to beat each other up based on choices. I find myself asking, why??  Now, I can understand passion for a topic. I can understand begin steadfast in a belief. What makes me shake my head is when opinion sharing becomes snarky, rude, offensive, etc.  Again, why??

Apple vs. (name your favorite PC vendor). If you are happy with your choice, awesome! If you don't care for someone else's choice, again, awesome!

Ford vs. (whoever). Do you like the car you drive? Sweet! Don't care for your neighbor's car of choice? That's cool too, but, do you need to hate on him/her?

Don't like Facebook? Hate Twitter? YouTube? Cats? Dogs? Hot Weather? Cold Weather? Broccoli? Chicken? Football? Soccer? Rock Music? Rap Music? ????  Fine! But, why be judgmental about these things and even more importantly, why be judgmental about the people who DO happen to like them? It's not like someone's choice to use a Mac significantly alters the life of someone who prefers a PC, right??

Bottom line, we are all free to like or hate whatever we wish, but, can't we learn to express our choices and opinions in such a way that we are not being offensive? I love hearing alternative perspectives to my own, but, only when presented in a reasonable fashion. There have always been people who are more than happy to push their opinions on others to the point of being rude. However, with the relative ease of online publishing and the ability to do so anonymously, it seems many more people feel comfortable being nasty. After all, there's no consequences, so why NOT be rude?

My simple rule is this; If I don't like something, it's possible I may mention it, but, I don't buy it, use it, eat it, listen to it or endorse it. If you like whatever it is, I'm glad you're happy.

Thanks to No Haters Pictures for the above image.

Click here to get Images & No Haters Pictures - Pictures

Faster and Faster

Ever have one of those days where you stop, turn your head around and wonder what happened when you weren't looking? I've found as I get older, this happens more and more. I don't think this has anything to do with me not paying attention, rather, I believe it's a byproduct of just how many things change and how fast. The speed of innovation, the willingness of younger generations to adopt new ideas more quickly and the sheer volume of new ideas have combined to create an environment of change that can be overwhelming.

Example: I was chatting with Rita (Rita Joseph, my long time friend, colleague and now manager) about this and she mentioned how the phrases she and I think of as normal are going away. "I heard it on the radio last night" is now "I heard it on Pandora". "Did you catch (name your TV show) last night? No? Guess you'll have to wait for reruns." is now "I caught up with the last 5 episodes of (show) on Netflix yesterday".  I make no judgement because I happen to like most new innovations, however, I get a bit nostalgic thinking about the demise of past technologies.

Now I understand how my grandparents must have felt when man went to the moon or when personal computers became prevalent. To me, this was normal. I've never known life without such things. For my kids, they've never known life without touchscreens and social media.

Sometimes I wonder, is it really going faster now or is it just my turn to notice??

Now, this is a leap in the right direction!

Last week, I put a few thoughts together regarding the newly published Digital Government Strategy. Imagine my surprise when I received a Tweet from Mr. Macon Phillips, White House Director of New Media. Mr. Phillips asked me about my thoughts on a new program called the Presidential Innovation Fellows - MyGov. I have to admit, I felt pretty special being noticed by a man who works in the Executive Office of the President. So, of course, I immediately took the time to read over the program.

Being a person who's a little hard to impress these days, I have no problem saying that the intent of this program is one of the best I've seen in a long time. For the past 7-8 years, during my tenure with Adobe and since, I've had the opportunity to share my thoughts through various channels and venues on how government could have a significant, positive impact on the lives of most Americans if only government would stop and consider what people really want, how they want it, why they want it, when they want it, where they want it, etc.

So often, citizen interactions with government are complex, tedious, confusing or seemingly not related to the outcome the citizen is seeking to achieve. Generally, this is due to the history of how IT systems have been created throughout the years, typically built from the bottom up, giving the most attention to the technical aspects rather than focusing on what the people using the system are trying to do. Government is not unique in this manner, practically all IT systems have been built this way.

So getting back to the MyGov initiative, this program is intended to reshape the way government and citizens engage by bringing together a team of experts from government, academia and, most importantly in my opinion, 'regular' people to represent the citizens of the country. Not that I flatter myself enough to believe I had anything to do with this happening, it seems as though someone was digging around in my notes. I couldn't have come up with a better plan.

My only criticism, and it is minor in the grand scheme, to become a Fellow, participants must be available on a full-time basis for 6 months. While I understand the requirement, this limitation may preclude direct and active participation from those outside government and academia who may have a harder time getting the time away from their job. In my travels over the years, I have come across many extremely talented 'engagement designers' who could contribute significantly if there was a part time role as well.

Anyway, regardless of the fact that I recently started in a new role, I intend to apply to become a Presidential Innovation Fellow. I believe in this initiative and think it is important enough to put my personal career on hold should I be honored with a spot on the team.

Mr. Phillips, thank you for bringing this to my attention. And, thank you to all those in government who have lent a hand in creating this program.

I'm impressed.

Where's the Easy Button??

The marketing folks at Staples are geniuses for coming up with the idea of the Easy Button. Really, how many times has this idea been re-purposed by others to make a point? And each time, Staples gets a mention, even if not overtly!

Well, now it's my turn to give the Easy Button a push to make a point.

For the last 28 years, I've been exposed to a wide range of enterprise technologies and systems. As a sales engineer, a product guy, a CTO, etc., I've had the opportunity to experience some really GREAT installation procedures and some really cruddy ones as well. Some installations are intuitive while others were incredibly complex. There are many reasons for such disparity, ranging from the dependencies of a system/application to the skill of the team creating the installation procedure.

Yesterday, I jumped on a plane to Chicago to visit headquarters (Cleversafe) to stage a system scheduled to be shipped to a customer for a Proof of Concept at the end of the week. Having yet to perform a staging/installation exercise from scratch, I scheduled a full two and half days to be here for this activity. I mean, we're talking about staging a system that incorporates 12 servers that are starting out in boxes with no software. The entire POC system has a storage capacity of over 2 petabytes of usable storage. This should take awhile, right?

Imagine my surprise when the entire activity was completed in under 4 hours. (BTW, I was being taught on the fly, which slowed the process down just a little bit.) This included the following:

• Bringing the boxes into the staging area
• Removing the servers from the boxes
• Placing the software image on each server
• Configuring each server with the appropriate networking information
• Connecting the machines into a network
• Setting up the actual Cleversafe dsNet software
• Configuring the dsNet
• Burn in testing
• Shutting down and reboxing the servers in preparation for shipping

When theses machines get to the customer, all that's left to do is take them out of the box, add the rack rails, slide them into the rack and connect power/network. At most, 90 minutes. Qualify for the Easy Button moniker? Oh yeah. 

It makes me wonder how many other storage providers can get a 2 petabyte system up and running from scratch this easily? 

Easy Button. Pushed.

And what is Bobby going to do with his extra time in Chicago? Well, I'll give you a hint. There's hot dogs, beer and a 7th inning stretch in store for him later tonight. Yeah, working for Cleversafe is a good thing.

Which comes first, people or information?

I just finished absorbing the new digital government strategy rolled out on Tuesday by the White House. Thinking strictly as a technologist, my first impression is the strategy is an exercise in stating the obvious. There's nothing blatantly WRONG with it, but, there's also nothing ground-breaking. This is a pretty standard and accepted approach to large scale platforms/systems/services.

Now, let's take a look at the strategy again with a different point of view, the customer. From this perspective, there is an obvious mistake in the strategy, not putting the customer/citizen front and center. Many will argue that the customer is represented, even called out in a bullet point. Yet, it is not the first bullet, it is the third. Subtle? Maybe, maybe not. It is my experience that whatever is written in a strategy or requirements document first is what gets the most focus, so, that would suggest that the 'needs' of the information come before the 'needs' of the customer. When the inevitable time comes to make trade off decisions in the design, who normally loses? Yep, the person using the system.

Forward thinking organizations who have wildly successful online applications and services have recognized this by building their digital strategies to first focus on people. Who's the target user? What do they care about? How do they use technology in their day to day lives? And most importantly, what are they trying to do?? It is only after the customer's perspective is fully understood that the technical aspects of the system should be considered. Systems and services are built from the top down, rather than the traditional approach of bottom up.

At the end of the day, what's more important, the system/data architecture, the APIs, etc. OR a citizen/customer successfully achieving a desired outcome? The answer? Both, however, it's time to get the horse before the cart.

Rehashing a point of view

Being the 'new guy on the block' with regards to the storage industry, I've given myself the license to wonder about everything, open old wounds and simply ponder the question, 'why'. This is not a new place for me, I've been curious about how everything works and why things are as they are ever since I was a little kid burning holes in my Mom's carpet. Um, little kids don't know the difference between AC and DC power. So, in my mind, a battery operated electric motor would run REAL fast if you hooked the wires to an extension cord. Oops! To this day, I don't know how I survived my curiosity as a child. But, I digress.

Most recently, I've been chewing on the notion of structured vs. unstructured data. For years, I've had a notion of what I thought constituted a structured data and all else was, by definition, unstructured. Right? Admittedly, my parameters were pretty simple. Any data stored in a database was considered structured making any data stored in some other format (can you say files?) unstructured.

But, is this really an accurate way to think of information? I figured I'm not smart enough to be the first person to ever consider this so I hit Google. Not surprisingly, I found a number of relevant hits, from blog entries to academic papers, on the subject. Really? Academic papers? Ok.

Anyway, after reading and digesting I've come to the conclusion that characterizing data as structured or unstructured is more relevant to the context of who or what is attempting to use it. For example, information stored in a database is most certainly considered structured to another computer application, yet, showing raw data in it's table format to a person, especially a non-technologist, would most likely prove to be confusing. On the other hand, a human being sitting down to read the most current corporate memo could easily argue that the information is HIGHLY structured, yet, it may not necessarily be as apparent to a computer.

So what's my point to all this. Simple. In general, 'structured vs. unstructured' is a false comparison unless it is applied to a specific point of view. What most people REALLY mean when they say 'structured vs. unstructured' is 'information stored in a database vs. information stored in a file'.

So back to being the 'new guy on the block', this seems to have some interesting implications with regards to discussing storage solutions. When a vendor says they are good with unstructured data, do they really mean information stored in files? I suspect I know the answer.

Any thoughts out there?

Garbage In - Garbage Out

Over the past few weeks, I've been engaged in a search for a new home. Part and parcel to this activity is, of course, interfacing with the financial services industry. Oh joy! Now, those of you who work in the industry, please bear with me. I know the vast majority of individuals who dedicate their time to helping the rest of us with our financial needs are competent, friendly and really do care, so, I'm not writing this post to in anyway insult the people of the industry.That said, I'd like to use the systems of the FSI industry, in particular, that of the credit scoring agencies, to make a point, so hang with me.

History lesson: Way back when in 1961, my parents opted to name me after my dad, Robert Lee Caudill, Jr.  Nice sounding name, right? I've always been proud to use it, however, like many people's names, over the years, I ended up with a number of different versions. My friends all call me "Bobby Caudill", at work, I've been "Robert Caudill", "Robert L. Caudill", "Robert Lee Caudill", "Robert Caudill, Jr." and any other twist you can imagine. And, as luck would have it, my dad's name has been similarly altered.

Why am I going on about this? Enter the credit reporting agencies. Can you imagine the havoc this situation has had on my credit report? (Yes, my dad's report is all screwed up too!) For years, we've been trying to unwind our credit profiles and after at least 25 years of trying, we are not any closer to getting it done once and for all. Every time either of us makes a life change, we both end up disputing entries to our credit report. Sad as it is, we've long since accepted this as an unfortunate reality we must deal with in this age of technology.

So, data. What an interesting thing it is. Practically every thing we do today is influenced or even controlled by data that's been collected, mined, sliced, diced, analysed to death. When the data in question is guaranteed accurate and authentic AND the questions being asked of it are appropriate and well thought out, the results can be quite useful. But what happens when the data is assumed accurate and/or the questions being asked are simply not appropriate? Well, as I can attest to based on my current credit score, anything can happen.

As the world continues down the path of storing and analyzing every single bit (yes, I mean 'bit' in the context of a 'byte') of digitally generated 'information', I often wonder just how often people (and systems) come to the wrong conclusion because the data set has incorrect information.

Working for Cleversafe, I talk a great deal about how we help customers indefinitely store huge volumes of data, keeping it reliably accessible, safe and secure for all their future processing and analytic needs. With my focus being on government customers, it is quite rewarding knowing that I am helping to preserve our nation's information through to the end of the republic. Yet, sometimes I wonder, how much of the data being stored and used for big data projects is truly accurate?

Garbage in, Garbage out? Only time will tell.

CSman Arrives

What a great day! I got up quite early today to catch a train to NYC, but, I did not make the trek alone! I now have a perpetual travel partner to keep me entertained as I traipse around the country for Cleversafe. Introducing, CSman! Who wouldn't want a little green dancing robot following them around?

So, on the train, after downing my daily allotment of caffeine, I busted out CSman, wound him up and gave him some room! My seatmate on the train was trying to ignore him, but, finally, she broke down and actually giggled! CSman scores his first smile!

CSman dancing on the Acela

A couple hours later, we arrive at the Disney/ABC facility in NYC to assist my colleague, Kevin Sullivan, install a few servers for a proof of concept system. We fully expected to be there at least 6 or 7 hours, but, wow, less than two hours later, all the servers were installed and fully configured! Where's the "Easy Button" when you need it? Of course, CSman wanted to dance again and this time, I joined him!

CSman rocking on a dsNet

Well, it's safe to say that our first outing together was a success. I'm looking forward our future exploits together. Keep an eye out for CSman. You never know where he might show up!

Transitions

So many transitions! From a big company to a small company. From a well established brand to a fledgling brand. From a pure marketing role to sales engineering/marketing/partner rep/bottle washer. From user experience solutions to storage solutions. What a ride so far.

The most fun though has been moving from representing a 'known' technology to representing a new, disruptive technology. There's nothing like a customer call where everyone is nodding their heads in agreement and sitting forward in their seats, anxious to ask the next question. Who would have ever thought talking about storage could be so exciting and interesting??  :-)

Just today, Rita (Joseph) and I had 60 minutes of time with two senior level government execs downtown. Not even 30 minutes into the meeting (which was our first visit with this team, btw), they were already identifying projects for us. Wow! I'm accustomed to it taking 2 or 3 visits before a customer begins to grasp the concepts that were being presented! And this meeting is illustrative of all the meetings we've had thus far.

As a sales engineering/marketing/partner rep/bottle washer, I feel like I'm working in the midst of the perfect storm, a customer market with definable pain and technology that offers a fresh, easy to explain solution to the pain.

Yep, transitions can be awesome.

Welcome!

Hello!

Hm, so, it's back to the dreaded 'blank page' of a start up blog! Intimidating, daunting, exciting. Will anyone care what I have to say? After being ask to change roles at my previous employer, Adobe, I opted to take a break from writing. As it turned out, the break was a little longer than I anticipated, however, I'm now feeling inspired to participate again!

I recently joined a new company, Cleversafe as the Director of Federal Solutions (I'm so happy to be back on the government side of a company!). We are a smaller company focused on solving the challenges associated with scaling systems to deal with extremely large amounts of data, systems where a petabyte is nothing more than a starting point!

So, coming from my solutions background, where much of my time has been focused on user experience and other such topics, moving into a more infrastructure-centric company has opened a whole new set of topics for me to delve into.

I'm looking forward to sharing with and learning from as many people as possible. And occasionally, because I do tend to allow my attention to wander, I may stray from the path of technology and chat about music, movies, books, dogs, kids or whatever else may come to mind!

Thanks for coming along for the ride.

Bobby