Big Data Requires IT Managers To Think About Big Databases

by drjim on July 21, 2011

Image Credit When It Comes To Databases, IT Managers Need To Think Big!

When It Comes To Databases, IT Managers Need To Think Big!

One of the biggest challenges that all IT managers face is the simple fact that often their teams expect them to know everything. In the era of massive data sets, this means that when your IT team runs into a problem that just can’t fit into a standard off-the-shelf database they’re going to show up on your doorstep with a problem that they can’t solve. When this happen, you had better know about MapReduce and Hadoop…

Welcome To The World Of NoSQL

When most IT managers think about databases, SQL comes to mind. This 40-year old standard defines databases as collections of rows and columns which can be joined using different logical criteria in order to help users find the data that they need to answer a particular question. However, as more and more firms move into the world of very, very large datasets, the limitations of SQL databases are starting to become more and more apparent.

Where companies first start to see issues is when the queries that they are trying to execute start to take longer and longer to complete. When this occurs, firms will switch and start to use massively parallel processing. However, even with this approach the complexities of the queries that will start to be done with these massive databases will eventually not work well with traditional SQL databases.

When this happens, IT managers will be asked to look for alternative database solutions. This is when MapReduce and Hadoop will start to show up in your vocabulary.

MapReduce is a programming model that was invented by Google in order to process very large data sets. Hadoop was based on MapReduce and was created by engineers at Yahoo. Hadoop has gone on and has become an open source project that is managed by the Apache organization.

Using MapReduce or Hadoop allows a firm to scale and potentially perform better. It may also allow them to see things that might not be possible if they were using a traditional SQL database. Examples of this come from McAfee who uses Hadoop to do text analysis across large collections of malware in their databases in order to find commonalities that might go unnoticed otherwise.

What Are The Downsides To Using The New Big Databases?

Although these new types of databases are very powerful, they do come with their own set of drawbacks. The first of these is the simple fact that they are brand new. SQL has been around for a long time and everyone knows just about everything that there is to know about it. The newer databases are more cutting-edge and may have drawback that nobody knows about.

Another drawback is that the way that IT developers interact with these new databases is via modern programming languages such as Java, Python, and Perl. Many of your current database programmers may only know SQL and will face a steep learning curve in order to become proficient with the new types of databases.

What All Of This Means For You

IT managers are often expected to have the answer when an IT development team runs into a problem. More and more often these days those problems have to do with traditional databases running out of steam.

The good news is that a new breed of databases has become available that has been expressly designed to work with very large datasets. The Map Reduce and Hadoop databases allow queries to be performed that are either not practical to do for time reasons or just not possible to.

A key point that all IT managers need to keep in mind is that in your management position, you are no longer required to do the investigation that will be required in order to determine if either the MapReduce or Hadoop databases are the right solution for your project. Instead it’s your job to manage your team as they do the investigation. Good luck and be happy that this time out you were able to answer your team’s technical questions!

– Dr. Jim Anderson
Blue Elephant Consulting –
Your Source For Real World IT Management Skills™

Question For You: What do you think the best way would be for your team to test if the MapRequest or Hadoop databases were the right solution for your project?

Click here to get automatic updates when The Accidental IT Leader Blog is updated.

P.S.: Free subscriptions to The Accidental IT Leader Newsletter are now available. Learn what you need to know to do the job. Subscribe now: Click Here!

What We’ll Be Talking About Next Time

Hey IT manager, just how loyal to your company are you? How loyal do you think that your team members are? I’m betting that the answer to both of these questions is “not very”. Given that that is the current situation that we find ourselves in, how did we get here and what is an IT manager to do about it?

Be Sociable, Share!

{ 0 comments… add one now }

Leave a Comment

{ 1 trackback }

Previous post:

Next post: