Showing posts with label NoSQL. Show all posts
Showing posts with label NoSQL. Show all posts

Tuesday, May 24, 2016

Hazelcast : In-Memory NoSQL Solution

If you are evaluating high-performance NoSQL solutions such as: Redis, Riak, Couchbase, MongoDB, Cassandra etc. or in even rarer cases if you’re evaluating caching solutions such as Memcached or EHcache, it’s possible that your best choice may be Hazelcast: Hazelcast uses a considerably different approach to any of the above projects, and yet for some classes of people looking for a Key-Value store, Hazelcast may be the best option for you.




So lets look at Hazelcast and why it is better alternative for above mentioned systems, Hazelcast is an In-Memory Data Grid, not a NoSQL database.
advantages and disadvantages of using Hazelcast in-memory solution, first of all as being key-value store into the memory, it has some default advantages are speed and read efficiency but also some natural disadvantages as storing map into memory are scalability and volatility, as we know that size of RAM is always less than the total available disk space, as RAM is more expensive than the disk, so being in-memory store space is always been a constant and second one as being flash storage, RAM refreshes itself on process restart, resulting into the data loss. Hazelcast addressing this shortcomings of the in-memory stores by providing efficient and convenient solution.
In Hazelcast, scalability issue is addressed by clustering solution, as joining hundreds nodes into the cluster, we may aggregate more than terabytes of in-memory space, to accommodate Hazelcast map into the memory. Of course this not going to be compared with disk space as its been 100 times more than memory but depending upon use case some terabytes of space is sufficient for the in-memory operations or else this solution may use with some backend data store.

Volatility, in Hazelcast volatility handled by peer to peer data distribution, so every block of data has multiple copies(replication across the cluster) of it present on different locations, as any node/rack goes down because of some issue, we can recover data from other copies present on other locations. The number of backup copies can be configured as depending upon the data criticality. Too much copies of the data into the cluster may reduce the overall availability of working memory to other operations. Hazelcast addresses the tuning problem of the cluster by providing ways to make your cluster available and reliable, including setting up backup process for example servers on one rack would be set to back up on another rack, so failure of entire rack can be managed gracefully.
Hazelcast also address the Rebalancing problem of the cluster, whenever node is added/removed to/from the cluster can be lead to moving data across the cluster to rebalancing it. If the node crashes because of some issues, the data copy(primary) on dead node has to be re-owned by the another node who is having replica of that data copy(secondary) becoming primary and backed up on another node to make cluster fail-safe again. this process may consume more cluster resources like CPU, RAM, network etc. might lead to latency into the process during this whole process.
Also in addition to the above benefits, Hazelcast also make sure that Java Garbage Collection process should not be having any effect on the terabytes of data stored onto memory specifically on heap as your heap gets bigger, garbage collection might cause delay in your application response time. So memory store is Hazelcast with native storage support to avoid garbage collector to causing delay in application; resulting in more efficiency and throughput.

Friday, May 24, 2013

NoSQL brings Hadoop Live

Hadoop designed for processing the large amount of data, with different varieties and with high processing speed, but not really real time, there is some latency in the hadoop response and actual real time application request. Integration of hadoop with real time application is the more tedious and complex and of course the most important job. If we have capability of large data storage and process but we are not able to access it real time so that is of no use.
Previously for this integration, Apache HttpFS and Hoop were used but as per time goes that new WebHDFS(RESTful) services becomes active to access HDFS(Hadoop Distributed File System) over HTTP or similar protocols. 

Its a era of NoSQL databases, which are replacing the current and traditional RDBMS systems because of many more advantages over them. "NoSQL" database are designed to deal with huge amount of data in short "Bigdata", when the data is in the any form, doesn't requires a relational model, may or may not be structured, but the NoSQL is used only when there is data storage and retrieval matters not the relationship between the elements

Now think what happens when two bigdata handling giants come together and what will be their power together. We can use hadoop with NoSQL database to respond real time application.


Hadoop-NoSQL Integration with Realtime Application

In above architecture diagram you can see the frontend application can communicate with the NoSQL database (As we are replacing RDBMS with NoSQL DB) and Hadoop can Integrate with the NoSQL database, Hadoop can take a input data from NoSQL database does the processing and again stores the output data into the NoSQL database, so frontend application can easily access the processed data on UI. It is as it is simple. Here is mmion complex part is to access NoSQL data into the hadoop jobs.

Nowadays many NoSQL database provides connectors with Hadoop (e.g. MongoDB-Hadoop Connector) so we can easily get data from and stores data into the NoSQL database from hadoop jobs. 

Even we can generate a BI reports from bigdata, like we can import database tables (structured) and Application logs (unstructured) into HDFS from ETL jobs as a Hive/HBase tables using Sqoop/Flume and then we have BI connectors available to integrate with HDFS/Hive/HBase so we can generate business reports from bigdata.

Sunday, May 12, 2013

Starting with NoSQL Database

All of us know that hadoop ecosystem is designed for bigdata processing and analyzing data in batch process not really real time purpose; but we can bring hadoop live using NoSQL databases, now think what happen when tow big giants comes together and what are the powers of them together.

"NoSQL" some in the industry referred "Not only SQL" describing NoSQL systems do supports SQL like query languages but its not 100% correct. NoSQL databases are high optimized for data retrieving and accessing operations. its because of storage system of NoSQL databases are based upon key-value pair, each data item inside the NoSQL database has a unique key, because of this Runtime query complexity of traditional relational databases are removed; Made a footprint into emerging market of valuable data warehousing by gaining high scalable and optimized performance model of NoSQL database.

"NoSQL" database are designed to deal with huge amount of data in short "Bigdata", when the data is in the any form, doesn't requires a relational model, may or may not be structured, but the NoSQL is used only when there is data storage and retrieval matters not the relationship between the elements, NoSQL can able to store millions of key-value pair and access them faster than any relational database can. This system is very useful for real time statistical analysis of  growing data such as application logs.

Its not give guarantees that "NoSQL" supports full AICD operations. Perhaps only eventual consistency is possible or transactions limited to single data items, Although most of the NoSQL systems have transactions over single data objects, mutliple multiple supports transactions over the multiple data objects. Such as eXtreme Scale NoSQL system supports transaction over the single object and systems such as FoundationDB, OrientDB etc. supports transaction over multiple objects like traditional RDBMS databases.

Focusing on storing and accessing data from NoSQL database. As mentioned above NoSQL system uses a key-value pair to store data into the data store. Key-value pair allows to store data in schemaless way, so data can be stored into the store in the form of programming language like POJO classes in JAVA and accessed into the same; because of this there is no need to fixed data model to store or access the data.

Now revolution into the NoSQL databases brings it into the next step such as Graph database, in which data is stored in graphical manner, in relation with each other so we can access related data faster than any one can.

This just an overview of what is NoSQL database and how it work for detailed explanation please https://en.wikipedia.org/wiki/NoSQL 

There are 100's NoSQL databases available in market waiting for you, some of them are
1. HBase - A hadoop Database
2. MongoDB
3. Cassandra
For more related to the NoSQL dbs please do visit:

Followers