When to use a MapReduce based, massively parallel data management solution like HBase? Any application where realtime querying of Big Data is required.

Small, medium and large enterprises generate Big Data all the time; often it is discarded or retained on short GFS cycles because it is not cost effective to store, analyse and report on nor to query.

* Hospital records
* Supermarket purchases
* Weather station reports
* Access logs (eg. corporate web proxy, ISP clickstream etc.)
* Banking related transactions

Try running an online query for a specific transaction against your local 7-11’s nightly master cash register report, for example.

Wow, 3 hard drives. A “bunch of cores”, like what 64, 1024??, no probably 4. A “ton of RAM”? What is that really, 8GB, 16? The hardware sounds really kind of weak for something that has any I/O load. 3 drives?

Any programmer can bring an RDBMS architecture to its knees with a few badly written SQL statements, a single weak sister server will go down even faster. Maybe it is time to hire someone that knows databases or
architecture.

>> The same architecture decision that allowed the original system to run like greased lightning turns into unanticipated complexity that may well prevent the new applications from working quickly or possibly at all.

>> So I’m not the least bit surprised that this new technology appeals to start ups – they have no preexisting investment in data and often have a single clear goal.
(Nor any experience with anything more complicated than a spreadsheet image of data. Sigh)

Another voice of clarity and sanity. I have been disparing that there were none left. Except Pascal and CELKO, not that they get along.

I think you’ve missed the fundamental advantage of relational databases. I’m not going to dispute that Oracle gets very expensive as volumes go up, or that the technologies you refer to are better / very useful in certain instances. One of the consequences of Oracle’s success is that it is now used in situations where it really shouldn’t be. But I don’t see Larry Ellison crying any time soon.

The reason that the RDBMS model is still dominant is because it handles complexity well. Alternative databases force developers to make decisions about access paths at design time.

Take Amazon as an example. It’s easy enough to envisage a catalog spread over many physical machines, especially as catalog data is essentially static. But what about the inventory of books actually available for shipment now? The logical thing would be to split it the same way the catalog was split, but wouldn’t it be far more convenient for all the stuff I ordered to be on the same machine as my account records? And what about the requirement that just came in for a list of heavily browsed titles by zip code updated every hour?

This isn’t a problem if you only have to keep track of one thing such as insurance quotes. As the example above shows the problem arises when you need to share or re-use the data. Sharing happens when the business decides to create new products or integrate systems and re-use happens when a newer version replaces the first generation system. The same architecture decision that allowed the original system to run like greased lightning turns into unanticipated complexity that may well prevent the new applications from working quickly or possibly at all. When a developer tries to ’simplify’ the model so his application works well what usually happens is that more complexity appears somewhere else, usually in the interface between his model and the rest of the business.

Many real-world applications have a couple of hundred changeable tables with complex interrelationships that can not easily be split, sharded or shared. The bottom line is that complexity is inherent in the business problems we attempt to solve as well as the outside world we have to interact with. Oracle DBAs like myself have been earning a living for years by managing this complexity and attempting to create databases that reconcile the conflicting needs of different applications. RDBMS’s such as Oracle are the best tool we have available, but the real work is done in our heads.

So I’m not the least bit surprised that this new technology appeals to start ups – they have no preexisting investment in data and often have a single clear goal. I’d guess it’ll be a couple of years before they find themselves beating their heads off the wall trying to get their non-relational database to grow with the business. Neither will I be surprised when BigTable finds a use for on line catalogs and directories, but I don’t see my bank using it any time soon…

In The Age Of XP/Agile/SCRUM, “Design” is utterly obsolete, TOTALLY!!! last century, just quaintly archaic.

What we need is more code!!! Today!!! Before close of business!!!

If it does something, terrific! If “Something” is productivce, terrific!! Just make sure it’s glitzy, and cool, and in CVS by close of business!!!