Music Analytics Opportunities

Once upon a time the words music listening habits were private as their bedrooms, music lovers used to buy the CDs, recordings and other physical copies of music and never publicly shared, Record companies were aware which radio station played their songs and where their CDs were popular, but that information painted an incomplete picture at best. Who knew what music people were sharing on tapes and CDs burnt in the privacy of their own bedrooms?

A traditional business metrics like number of CDs were sold and nothing happened after that, who purchased what and whom to assist what to buy, all this was anonymous. Thats all changed after explosion of online music sources like torrenting, music streaming sites and social media platforms, are now playing a very key role for music industry to understand their fans, spot upcoming talents like never before and anyones personal music interest nowadays becoming a public. Music analytics is now worth around $24.35 billion per year.

At the same time that the internet is taking power away from record labels, it is also giving them the ability to predict future hits.

Apache Spark, Apache Flink & Apache Strom

Apache Spark: Apache Spark is a batch processing engine that emulates streaming via microbatching. It has a well developed ecosystem and incorporates besides a Scala and Java API a Python and R library as well. Apache Spark very well integrates with Apache Hadoop Ecosystem components.

Apache Flink: Apache Flink is streaming dataflow engine. It can be programmed in Scala and Java. You can emulate batch processsing, however at its core it is a native streaming engine. Flink shines by features under the hood, such as exactly once fault management, high throughput, automated memory management and advanced streaming capabilities, Apache Flink also very well integrates with Apache Hadoop Ecosystem.

Apache Storm: Is a technology created by Nathan Marz. Compared to Flink and Spark, it has a compositional API. Meaning you build up your own topology with basic building blocks like sources or operators(spouts and bolts) and they must be tied together in order to create topology(program flow).

For more detailed comparison for all other streaming and batch processing frameworks, drop me a reply here, I'll try to reply as earliest.

Capitalizing Bigata.!!!

90% of data created today is unstructured and more difficult to manage that generating from data sources like social media(facebook, twitter), video(youtube), texts(application logs), audio(viacom), email(gmail), and documents. As all of know the social media becoming revolutionary factor for businesses.

 

Bigdata is much more than data and is already transforming the way businesses and organizations are running. It represents a new way of doing business, creating a bright path for future business world, one that is driven by data oriented decision making and new types of products and services influenced by data. The rapid explosion in Bigdata and ways to handle it, changing the landscape of not only IT industry but all over the data oriented systems, And this data is becoming so powerful and important to drive for today’s businesses, as it contains customer insight and business growth opportunities that have yet to be identified or even no one had a idea about. But due to its volume, type and speed of change, most companies are doesn't have enough resources  to address this valuable data and get business out of it. 

Its time to get together and find out the ways and patterns from bigdata that can help us to make our lives even simpler and we have the solution(Hadoop) but need to explore it more, to focus on true growth and identifying the business opportunities.

YARN Essentials

If you have a working knowledge of Hadoop 1.x but want to start afresh with YARN, this book is ideal for you. You will be able to install and administer a YARN cluster and also discover the configuration settings to fine-tune your cluster both in terms of performance and scalability. This book will help you develop, deploy, and run multiple applications/frameworks on the same shared YARN cluster.


YARN is the next generation generic resource platform used to manage resources in a typical cluster and is designed to support multi-tenancy in its core architecture. As optimal resource utilization is central to the design of YARN, learning how to fully utilize the available fine-grained resources (RAM, CPU cycles, and so on) in the cluster becomes vital.


This book is an easy-to-follow, self-learning guide to help you start working with YARN. Beginning with an overview of YARN and Hadoop, you will dive into the pitfalls of Hadoop 1.x and how YARN takes us to the next level. You will learn the concepts, terminolog

y, architecture, core components, and key interactions, and cover the installation and administration of a YARN cluster as well as learning about YARN application development with new and emerging data processing frameworks.

Follow below link for more details,
https://www.packtpub.com/big-data-and-business-intelligence/yarn-essentials

Thank you!

What is Hadoop anyway?

Hadoop will change the way businesses think about storage, processing and the value of ‘big’ data.

Apache Hadoop is an open source project governed by the Apache Software Foundation (ASF). Hadoop enables the user to extract valuable business insight from massive amounts of structured and unstructured data quickly and cost-effectively through three main functions:

Processing – MapReduce. Computation in Hadoop is based on the MapReduce paradigm that distributes tasks across a cluster of coordinated “nodes.”

Storage – HDFS. Storage is accomplished with the Hadoop Distributed File System (HDFS) – a reliable file system that allows large volumes of data to be stored and accessed across large clusters of commodity servers.

Resource Management – YARN. Coming in Hadoop 2.0, YARN performs a resource management function further increasing efficiency and extends MapReduce capabilities by supporting non-MapReduce workloads such as Graph, Steaming, In-memory, MPI processing and more.

Hadoop is designed to scale up or down without system interruption and runs on commodity hardware making the capture and processing of big data economically viable for the enterprise.

“By 2017, I believe that 50% of the world’s data will be stored and analyzed by Apache Hadoop.” 

Data Analysis from MongoDB using R

Most of us are aware of R, is a programming language and software environment for statistical computing and graphics. The R language is widely used among statisticians and data miners for developing statistical softwares and data analysis. If we empower R with proper datasets and sources it would be the icing on the cake, so in this post we are going to see how, R would be connected to the MongoDB and how one can apply R power or datasets from MongoDB.

Prerequisites for this demo, you should have MongoDB daemon up and running on server or on your local machine(pseudo distributed mode) 

Start your R instance and install "rmongodb" packages by issuing below command(s)

        $  install.packages("rmongodb")

        $  library(rmongodb)

connect R with MongoDB instance

   

       $ mongo.create(host = "127.0.0.1", name = "", username = "", password = "", db = "test", timeout = 0L)

you'll get response as below, using above connection configuration you are connecting to the mongo instance on 127.0.0.1 to the 'test' mongo database with empty username and password.

        [1] 0

        attr(,"mongo")

        <pointer: 0x0884f0a8>

        attr(,"class")

        [1] "mongo"

        attr(,"host")

        [1] "127.0.0.1"

        attr(,"name")

        [1] ""

        attr(,"username")

        [1] ""

        attr(,"password")

        [1] ""

        attr(,"db")

        [1] "test"

        attr(,"timeout")

        [1] 0   

you can check by issuing below command, whether R is connected to MongoDB or not.

        $ mongo.is.connected(mongo)

        [1] TRUE

Now your R is successfully connected to MongoDB instance to test database, so you can easily fire a simple mongo queries and use R's power to calculate analytics over mongoDB datasets.

for example to get simple one record from Mongo

        $ mongo.find.one(mongo,"test.zip",list())

we can also use filter queries to fetch records from MongoDB into R datasets,

        $ mongo.find(mongo, "test.zip", list(pop=list('$gt'=21L)))

So, this just a beginning stay tuned for the next updates.

Thanks for visiting, I'll appreciate your thoughts and comments

Data Scrapper in Python

Hello All,

Nowadays we know the data is the most valuable thing in the world, who has the more data has the more power or command over the market. This market is totally data driven and I'm sure in next couple of decades the data can also decide the future, just kidding :) 

But trust me we can power our recommendations systems to predict very much accurate results with the data. Data is directly proportional to the value.

As the data is important then the its collection is also important, so we have number of data sources available over the net, one just need to find it out and fetch the required information from.

So in this post, we are going to learn one of the very famous data collection method is Data Scrapping from world wide web. Today we are going to write data scrapper in Python(3.4.3) 

#Import the required libraries
import urllib.request
import re

#stock symbol lists, you may refer it from file
symbolslist = ["suzlon.bo","unitech.bo","spicejet.bo","idfc6.bo","powergrid6.bo"]

i=0
while i<len(symbolslist):
#scapping page url
urlstr = "https://in.finance.yahoo.com/q?s="+symbolslist[i]+""
htmfile = urllib.request.urlopen(urlstr)
htmtext = htmfile.read().decode('utf-8')
regex='<span id="yfs_l84_'+symbolslist[i]+'">(.+?)</span>'
pattern = re.compile(regex)
price = re.findall(pattern, htmtext)
#Print the scrapped data
print("The price of",symbolslist[i]," is ",price)
i+=1

This is just a basic program you can modify and extend as per your requirement.

Thanks for visiting, stay tuned for more!!!

Apache Storm Setup and Deployment

Please follow below steps for apache storm and zookeeper setup and deployment

Set up a Zookeeper cluster

Download and extract a Storm package to Nimbus and worker machines

Install dependencies on Nimbus and worker machines

Fill in mandatory configurations into storm.yaml

Launch daemons under supervision using “storm” script and a supervisor of your choice

Overall Zookeeper and Storm cluster components

Setup a Zookeeper cluster

Storm uses Zookeeper for coordinating the cluster. Zookeeper is not used for message passing, so the load Storm places on Zookeeper is quite low. Single node Zookeeper clusters should be sufficient for most cases, but if you want failover or are deploying large Storm clusters you may want larger Zookeeper clusters.

Install the Java JDK. You can use the native packaging system for your system, or download the JDK from:

http://java.sun.com/javase/downloads/index.jsp

Set the Java heap size. This is very important to avoid swapping, which will seriously degrade Zookeeper performance. To determine the correct value, use load tests, and make sure you are well below the usage limit that would cause you to swap. Be conservative - use a maximum heap size of 3GB for a 4GB machine.

Install the Zookeeper Server Package. It can be downloaded from:

http://hadoop.apache.org/zookeeper/releases.html

Create a configuration file. This file can be called anything. Use the following settings as a starting point:

tickTime=2000

dataDir=/var/zookeeper/

clientPort=2181

initLimit=5

syncLimit=2

server.1=zoo1:2888:3888

server.2=zoo2:2888:3888

server.3=zoo3:2888:3888

You can find the meanings of these and other configuration settings in the section Configuration Parameters. A word though about a few here:

Every machine that is part of the Zookeeper ensemble should know about every other machine in the ensemble. You accomplish this with the series of lines of the form server.id=host:port:port. The parameters host and port are straightforward. You attribute the server id to each machine by creating a file named myid, one for each server, which resides in that server's data directory, as specified by the configuration file parameter dataDir.

The myid file consists of a single line containing only the text of that machine's id. So myid of server 1 would contain the text "1" and nothing else. The id must be unique within the ensemble and should have a value between 1 and 255.

If your configuration file is setup, you can start a Zookeeper server:

$ java -cp zookeeper.jar:lib/log4j-1.2.15.jar:conf \ org.apache.zookeeper.server.quorum.QuorumPeerMain zoo.cfg

QuorumPeerMain starts a Zookeeper server, JMX management beans are also registered which allows management through a JMX management console. The ZooKeeper JMX document contains details on managing ZooKeeper with JMX. See the script bin/zkServer.sh, which is included in the release, for an example of starting server instances.

Test your deployment by connecting to the hosts:

In Java, you can run the following command to execute simple operations:

$ java -cp zookeeper.jar:src/java/lib/log4j-1.2.15.jar:conf:src/java/lib/jline-0.9.94.jar \ org.apache.zookeeper.ZooKeeperMain -server 127.0.0.1:2181

In C, you can compile either the single threaded client or the multithreaded client: or n the c subdirectory in the Zookeeper sources. This compiles the single threaded client:

$ make cli_st

And this compiles the multithreaded client:

$ make cli_mt

Running either program gives you a shell in which to execute simple file-system-like operations. To connect to Zookeeper with the multithreaded client, for example, you would run:

$ cli_mt 127.0.0.1:2181

Setup a Storm cluster

Environment

* OS: CentOS 6.X

* CPU Arch: x64

* Middleware: Needs JDK6 or after(Oracle JDK or Open JDK）

Installing storm package

Unzip downloaded zip archive.

https://github.com/acromusashi/storm-installer/wiki/Download

Install the ZeroMQ RPM:

If occur failed dependencies uuid, download from

http://zid-lux1.uibk.ac.at/linux/rpm2html/centos/6/os/x86_64/Packages/uuid-1.6.1-10.el6.x86_64.html

and install uuid-1.6.1-10.el6.x86_64.rpm.

# su -

# rpm -ivh zeromq-2.1.7-1.el6.x86_64.rpm

# rpm -ivh zeromq-devel-2.1.7-1.el6.x86_64.rpm

# rpm -ivh jzmq-2.1.0-1.el6.x86_64.rpm

# rpm -ivh jzmq-devel-2.1.0-1.el6.x86_64.rpm

Install the Storm RPM:

# su -

# rpm -ivh storm-0.9.0-1.el6.x86_64.rpm

# rpm -ivh storm-service-0.9.0-1.el6.x86_64.rpm

Set the zookeeper host, nimbus host and other required properties to storm configuration file.

(Reference: http://nathanmarz.github.com/storm/doc/backtype/storm/Config.html )

* storm.zookeeper.servers (STORM_ZOOKEEPER_SERVERS)

* nimbus.host (NIMBUS_HOST)

# vi /opt/storm/conf/storm.yaml

Settings Example:

Default storm.yaml example.

########### These MUST be filled in for a storm configuration##############

storm.zookeeper.servers:

- "111.222.333.444"

- "555.666.777.888" ## zookeeper hosts

storm.zookeeper.port: 2181

nimbus.host: "111.222.333.444" ## nimbus host

storm.local.dir: "/mnt/storm"

supervisor.slots.ports:

    - 6700

    - 6701

    - 6702

    - 6703

Start or stop storm cluster by following commands:

Start

# service storm-nimbus start

# service storm-ui start

# service storm-drpc start

# service storm-logviewer start

# service storm-supervisor start

Stop

# service storm-supervisor stop

# service storm-logviewer stop

# service storm-drpc stop

# service storm-ui stop

# service storm-nimbus stop

Strom Dependency libraries

Project : Storm

Version : 0.9.0

Lisence : Eclipse Public License 1.0

Source URL : http://storm-project.net/

Project : ZeroMQ

Version : 2.1.7

Lisence : LGPLv3

Source URL : http://www.zeromq.org/

Project : JZMQ

Version : 2.1.0

Lisence : LGPLv3

Source URL : https://github.com/zeromq/jzmq 

Lambda Architecture Overview

Nathan Marz and team has designed a generic, scalable and fault tolerant data (#bigdata) processing architecture named as a Lambda Architecture (LA), based on his working experiences and distributed data processing challenges with Backtype and Twitter.

Lambda Architecture has design goals like robust system that is fault tolerant, includes human errors and hardware failures, able to serve a huge range of use cases and workload in minimum time nearly real time. Should be scalable enough.

Lambda Architecture has 3 layers.

1. Batch Layer: 

It has two function managing a master dataset and pre-compute the batch views. Batch layer includes hdfs to store the master and mapreduce to precompute the batch views.

2. Speed layer: 

This layer is responsible for real time(nearly) data processing, low latency systems like Apache Storm includes in this layer to compute the data views with very minimal latency.

3. Serving layer: 

This can be any NoSQL database or indexing engine that able to index the batch view and able to merge output of batch and speed layer and query on that data, ad-hoc way.

For more details about lambda architecture do visit here

Music Analytics Opportunities

Once upon a time the words music listening habits were private as their bedrooms, music lovers used to buy the CDs, recordings and other physical copies of music and never publicly shared, Record companies were aware which radio station played their songs and where their CDs were popular, but that information painted an incomplete picture at best. Who knew what music people were sharing on tapes and CDs burnt in the privacy of their own bedrooms?

A traditional business metrics like number of CDs were sold and nothing happened after that, who purchased what and whom to assist what to buy, all this was anonymous. Thats all changed after explosion of online music sources like torrenting, music streaming sites and social media platforms, are now playing a very key role for music industry to understand their fans, spot upcoming talents like never before and anyones personal music interest nowadays becoming a public. Music analytics is now worth around $24.35 billion per year.

At the same time that the internet is taking power away from record labels, it is also giving them the ability to predict future hits.

Bigdata & TimeMachine

Powers of #Bigdata analytics, we can find out which movie gonna be blockbuster next year, not only the movie but also the future, the TimeMachine. Yesterday I saw a movie Paycheck, Michael Jennings is a reverse engineer; he analyzes his clients' competitors' technology and recreates it, often adding improvements beyond the original specifications. I think this is a best real use-case of Bigdata Implementation.

Michael creates a Time Machine with one of the his old college roommate, James Rethrick, the CEO of the successful technology company Allcom, after successful creation of TimeMachine James wipes Michael's memory, but before cleaning Michael's memory, Michael seen his future(in TimeMachine) and accordingly he sent himself a parcel(which delivers him after two years) using the things the parcel has, Michael(with lost memory) able to predict the things which he should do after two years to save himself from James.                                                      

Now we can see the things, which really correlate with Bigdata Analystics, Time Machine woks on principle of Astrology and the things we did in past gonna help us in future to survive and get the right direction, technically the data we(and off course the people who has a impact on our life) generated in our past, gets analyzed and using that analytics we are able to predict a future. Many companies now Analyzing the Bigdata generated/generating by each business vertical and designing a recommendation and decision engines to help business to survive in market.

Recommendation and decision engines, an area of predictive analytics and decision management, are going to quite active in next year, The pioneer was Amazon.com which used collaborative filtering to generate “you might also want”  or “next best offers” prompts for each product bought or page visited. 

I really appriciate your valuable comments and suggestions that guide me and you to direct our own future. Stay tunned for more updates on #TimeMachine

Bigdata & Natural Language Processing(NLP)

Natural language processing (NLP) is increasingly discussed in social media and other verticals of businesses, but often in reference to different technologies such as speech recognition, computer-assisted coding (CAC), and analytics. NLP is an enabling technology that allows computers to derive meaning from human, or natural language input.

Media is data intensive from customer satisfaction, product reviews and business perspectives. While the industry’s transition to electronic data collection and storage in recent years has increased significantly, this has not actually forced physicians to code the majority of meaningful content. Eighty percent of meaningful data remains within the unstructured text, as it does in most industries. This means that it remains in a format that cannot be easily searched or accessed electronically.

NLP can be leveraged to drive and directly impacting on improvements in financial, production, and operational aspects of business workflows:

For financial processes, automating data extraction for claims, banking transactions, financial auditing, and revenue cycle analytics can impact the top line. NLP can automatically extract underlying data, making claims more efficient and offering the potential for revenue analytics.

                                   

For production processes, automatically extracting key quality measures existing products and customer reviews, reporting and analytics. NLP can infer whether a product meets a quality measure. prelaunch response from customers, so decide a product launching stategy.

For operational processes, descriptive and predictive modeling can support more effective and efficient operations. NLP can extract hundreds of data elements similar available product rather than the 2-4 available products, producing better models and supporting business insight.

So, NLP is a powerful enabling technology, but it is not an end user application. It is not speech recognition or revenue cycle management or analytics. It can, however, enable all of these.

There is a battle underway that is increasingly recognized in the business space. Individual business divisions seek turnkey solutions and frequently purchase NLP-enabled products. But at a broader level.

We can use natural language processing for customer sentimental analysis, customer segmentation and many of the business cases, and find out the customer response and satisfaction from similar available products in market and to maintain quality of already released product, to decide business strategy to be a different in market.

Bigdata in Banking Domain

As financial industries growing with evolving business landscapes and increased information and business demands, finding efficient ways to store, organize and analyze the continuously increasing hell of data and integration and analysis is really crucial job. How effectively they can make better business decisions based on the this huge amount of data in short Bigdata they processes on a daily or weakly basis will be hurdle for the industry going forward. Nowadays banking system introduced very innovative and productive banking ideas like mobile banking, SMS banking, as we are able to carry banks in our pocket and every transactions are on our fingers. As it is increasing and having many more ideas equal proportionally the risk of banking also increasing like fraud, fake transactions, fake user accounts, miss-use of banking products by thefts and hackers.

 

Banking industries are using structural data from many years ago and finding a ways to tackle with such situations but they are not that much effective and accurate, So banks also should focus on not using more data but should use more diverse and variety of data from different data sources available on network, this includes not only the banks internal transactions and profile based data but the external information such as social networking data, application logs. Previously such data considered as none of any use but banks should use this data for customer analysis and getting more business insights out of it. Simply Banks should not only use internal structured data(traditional data) but also the external unstructured data to grow with more accurate results and effective predictions.

Bigdata plays a very important role to protect and secure end users and he’s banking activities. There are 1000’s of ways to protect our customer from theft and fraud if you have amount of data. As we can do analysis of customer transactions and monitoring its regular activities like customer salary, beneficiary transactions frequency and amount of every transaction helps banking industry to analysis of customers, customer location and transaction location analysis.

Today social networking is being very important part of every business network, we can found lots of ways customer analysis and sentimental analysis against products, as product reviews are easily available on such networking sites. There are 100s of solutions based on Hadoop available to replace banking traditional crucial analytics to new real time and less time consuming solutions to developing true relationship based analytics and finding out the true business values as per customers views.

Think again in growing business perspective take a look what data (internal plus external) we have, how we use it more effectively and where should we focus more to get more accuracy to fight in competitive market for survive and grow.

Apache Hadoop: Solution for Bigdata

Nowadays “Bigdata” is the most hitting word all over the business world, peoples are not just talking about bigdata but finding business out of it. What exactly bigdata is? Simplest definition of bigdata is nothing but a data comes with high velocity with different varieties and huge volumes. The purpose of publishing this paper to not just to talk about bigdata but how to integrate bigdata in our current solution, how to find more business insights around the bigdata and hidden bigdata dimensions around your business. 

Apache Hadoop is the open source framework provided by Apache foundation to deal with bigdata, the power of Apache Hadoop is to provide cost efficient and effective solution to businesses for focusing more on exactly what matters: extracting business values from bigdata. In this paper we will be addressing more about the technical details about Hadoop Ecosystem architecture and integration with real time application to process and analysis and to find out the various hidden dimensions of bigdata, which helps our business to grow up.

Apache Hadoop as a Team:

Consider a regular scenario; you have a project team, one project manager and ten resources under him. 

If a client comes to your project manager and asked him to sort out the ten files, each file of 100 pages record.  What will be best approach your project manager will follow? 
Exactly! what you are thinking is right, Project manager will distribute the ten files among ten resources and keep the only record track with him. This approach will reduce to work load about 1/10th, ultimately increases speed and efficiency. 

Hadoop Team Structure:

This is what hadoop is, data storage and processing team. Hadoop has data storage and processing components. Hadoop follows master-slave architecture 

Physical structure of Hadoop cluster is same as above project team we have a Manager called namenode and team members called datanodes and Data storage is the responsibility of  datanodes(slaves), controlled by name node at master level and data processing is the responsibility of task tracker(slave) and controller over task tracker is job tracker at master level.

You can see in the diagram and do map with the project team that you have already and see how interesting it is. Try to map everything with the real world things you can find many possible ways and solutions out of it.  

Capitalizing Bigata!

90% of data created today is unstructured and more difficult to manage that generating from data sources like social media(facebook, twitter), video(youtube), texts(application logs), audio(viacom), email(gmail), and documents.

Bigdata is much more than data and is already transforming the way businesses and organizations are running. It represents a new way of doing business, creating a bright path for future business world, one that is driven by data oriented decision making and new types of products and services influenced by data. The rapid explosion in Bigdata and ways to handle it, changing the landscape of not only IT industry but all over the data oriented systems, And this data is becoming so powerful and important to drive for today’s businesses, as it contains customer insight and business growth opportunities that have yet to be identified or even no one had a idea about. But due to its volume, type and speed of change, most companies are doesn't have enough resources  to address this valuable data and get business out of it. 

Its time to get together and find out the ways and patterns from bigdata that can help us to make our lives even simpler and we have the way(Hadoop) but need to explore it more, to focus on true growth and identifying a money making opportunities.

Fraud Detection and Risk Prediction in the Era of Bigdata

Fraud detection and Risk predictions is a multi-million dollar business and it is increasing proportionally every year. As mentioned on Wikipedia,  the PwC global economic crime survey of 2009 suggests that close to 30% of companies worldwide have reported being victims of fraud in the past year. 

Traditional methods of data analysis and mining have long been used to detect fraud. They require too complex architecture and time-consuming computations that deal with different domains like financial, economics and business practices, and still the results produces are not that much accurate  Fraud often consists of many instances or incidents involving repeated offences using the same method. Fraud instances can be similar in content wise and appearance wise but usually are not identical.

How exactly Bigdata helps to find out the Fraud or to predict most likely risk factors?

There are thousands of data sources with too large volumes and varieties, which are ignored by the traditional fraud analysis techniques and methods in short termed as Bigdata includes social media, transaction logs, application logs, weblogs,  geographical data etc.

For an example: A guy who has taken loan from bank say 1,00,000 with returning monthly installment of 10,000. He regularly paid installments of first four months as per policy after that he unable to pay remaining installments as unavailability of funds, But he is posting his new car, or new home or foreign trip pics on twitter. The guys who is already defaulter in banks record because of unavailability of funds and keeps posting a photos his new car on twitter or facebook. So bank officials can take immediate action on it without waiting for fraud to be happen.

Second example is like, A person whose is living in India, keeps/tries withdrawing money from Delhi, NewYark, Londan, Paris everyday, we can find out his geolocation history using google maps and  will compare with transaction location, resulting into immediate action.

There are many more use cases with bigdata to find out fraud and risk analysis, Advantage of using bigdata over traditional systems is most important is high accuracy towards results and most likely predictions, ultimately because of huge data, high accuracy and likely predictions are directly proportional to the size and sources of data.

Nowadays we have technology which can take over the bigdata analytics nearly real time, without wasting much time in computations and calculations, so action can be taken prior fraud to be happen. High performance analytics is just an technology fad, With new distributed computing options like Hadoop and in-memory processing on commodity hardware, insurers can have access to a flexible and scalable real-time big data analytics solution at a reasonable cost.

What people really thinks about Bigdata?

How much do you think people are aware of bigdata world and its advantages and disadvantages, or they are just aware of it, don't know how to use it? Bigdata analytics is really a hell? Bigdata is playing a role of hero or villain in our day today life?

Yes, these are the some headlines I found on internet while I was studying for bigdata analytics. Is that bigdata analysis is really difficult job? As per my experience I dint found such hardness and difficulties while going through. "If you know how to create a bigdata, then you should know how to bring business values out of it" this is the simple line I'm following.

Just think of end user perspective, you will get known many more dimensions and directions to analyse bigdata, do it and get successful in bigdata era.

being a simple end user is not that much difficult task I think so:) 

Bigdata and Business Verticals

As we are an active part of Bigdata ecosystems, where our day to day lifestyle and activities are responsible for data generation, and systems around us can collect the data, analyse it and consume it for their business to help our lifestyle. Nowadays world gets too much interconnected because of internet and mobile devices as never been in history, each day we are creating about 2.5 quintillion( 2.5×10¹⁸) of data, its huge amount created by different verticals in the industry, This verticals using this massive amount of information to rise above the business cloud. But before using this such huge amount of information industry must aware of the real time business scenarios, in short 'Usecases' to implement the solution for analysis of Bigdata.

We'll focus on some industry key verticals/domains which are using or most likely to use Bigdata analysis. Below are the some Bigdata value creation opportunities.

Financial Services:

-Fraud Detect

-Model and manage risk

-Improve debt recovery rates

-Personalized banking and insurance products

-Recommendation of banking products

Retail and Consumer Packaged Goods Industry:

-Customer Care Call Centers

-Customer Sentiment Analysis

-Campaign management and customer loyalty programs

-Supply Chain Management and Logistics

-Window Shoppers

-Location based Marketing

-Predicting Purchases and Recommendations

Manufacturing Industry:

-Design to value

-Consumer Sentiment Analysis

-Crowd-sourcing

-Supply Chain Management and Logistic

-Preventive Maintenance and Repairs

-Digital factory for lean manufacturing

-Improve service via product sensor data

Healthcare:

-Optimal treatment pathways

-Remote patient monitoring

-Predictive modeling for new drugs

-Personalized medicine

-Patient behavior and sentiment data

-Pharmaceutical R&D data

Web/Social/Mobile Industry:

-Location based marketing

-Social segmentation

-Sentiment analysis

-Price comparison services

-Recommendation engines

-Advertisements/promotions and Web Campaigns

Govenrment

-Reduce fraud

-Segment population, customize action

-Support open data initiatives

-Automate decision making

-Election Campaigns

Data growth in each section of each vertical is viral, speed of data generation is tremendous so needed a Bigdata capability for addressing such business problems, get ready soon and make your business to capable to hit big elephant of information.

Bigdata : Impact on day to day life

Would Bigdata really impact on our day to day life? If you asked this question 10 years before, the answer  might be No, but nowadays if you going for shopping to any mall, Google maps are tracking you, your home, you rout towards a mall and suggests the similar malls near to you. You reached to mall and  went to the mobile store, shop cameras are watching you, in which section you are spending more time and suggest you similar section to shop, Now you picked up a any gadget, they will calculate your interest and recommend you the gadgets with similar features and functionalists with discounts. (As they also want to grow up with their business:) ). Result leaving from home you decided for a-gadget and you b-gadget actually because of attractive offer on it.

From healthcare, to sports, from retails stores to the e-banking, from the business to the social networking, to the way we used to go for office, big data will making big changes to the way we live our lives. Specially internet is getting more and more importance to everyones life everyday, everyone is like to sharing his information on social site and social networking sites are becoming very popular for Business world. Businesses are becoming more and more consumer centric with the help of social networking and easily available information. Businesses are using this information to find out the customer trends and business out of it. Think of this we get an reason why E-Commerce businesses are getting more and more popularity these day. How weather forecasting is always being correct, Why healthcare programs are getting arranged in particular days of year, How fraud is detected in bank between millions of transactions per day. 

This is all about bigdata, we are surrounded by it as we are responsible for generating it and Businesses are just using it for their purpose to help us, ultimately both get benefited, We are happy because of we get better and  convenient solution even if we dint thought about it and Its impacting directly to Annual Revenue of Businesses. 

Big Business with Big Opportunities

Nowadays Businesses are struggling with abnormally growing volumes, speed and variety of information that used to generate everyday, everyday the complexity of information generation is also rapidly growing - the term to be known for as 'Bigdata'. Many companies are seeking for the technology to not only help them to bigdata storage and process but also finding many more business insights from bigdata and growing up the business strategies with bigdata. 

Arround 80% information in world is unstructured, and many businesses are not even attempting to use  that information for their advantage or not aware how to use that information. Imagine if you and your business keep afford that all data generated by you business and keep tracking and analyzing it, Imagine if know to way the handle that bigdata?

The data explosion presents great challenge to businesses, today most lack the technology and knowledge about bigdata and how to deal with it and get real business values. Many Companies are focusing on the developing skills and insights of business needs to accelerate the path of transforming larger data sets. 

What bigdata can do? Businesses are growing up with bigdata to finding more business insights and row that caries values for business with latest bigdata processing technologies like Hadoop fromework.

Its now possible to track each individual user through cell phones, wireless sensors with measurement of his interest in particular thing, where does he lives, works, plays and what is his day to day program and collect the data, analyse this huge data using bigdata processing technologies and find out the business ways with each individual user to help or make his life simpler. 

Bigdata in Social Networking, day to day millions of facebook comments, updates, twitter tweets are generating and many more so using bigdata processing to find out current market trends, what people are talking about, their likes, dislikes accordingly plan our business. 

Bigdata in Healthcare, every hospital or healthcare organization maintaining their historical records with patients records which may kind of bigdata so technology can analyse that past records and predict in future which patients, on what date, with the cause and what are the possible treatments for similar cause.

Bigdata in BFSI, In BFSI domain fault tolerance is the one of the most important pillar so, there are millions of daily banking transactions are there we want to find out the fake transactions, bigdata helps us even for product recommendations, transaction analysis bigdata plays a major role.

Bigdata in  ECommerce, somewhere and somehow on online shopping sites you might seen dialogs like 'you bought this you may like this', this is kind of recommendations calculated by bigdata processing technologies.

The information that we have today about 90% of information is generated in just last 2 years and this trend is going, I believe after 2025 there will about 70% businesses in world generated by Bigdata and Bigdata oriented. Product will be delivered to the customer if he just thinking about it, Cab will be waiting for us when decided to shopping and Discounts will already there on Shirt we might think to buy.