Internet of Things & Big data Analytics

We have entered the Digital Enterprise era.  All the businesses are aiming at reaching their Customers anywhere, anytime, any platform with any device.  All such smart devices or physical objects that are connected to the internet & are continuously emitting data and communicating with each other is called the Internet of Things (IoT).

You will find all these objects around you in your day. Your Fitbit wristband is monitoring your sleep, waking you up at the desired time, tracks your activity. When you go out for walking or jogging in the morning, your Nike shoes with built-in-sensor are collecting all the data and track your time, distance, pace & calories burned. 

All this humongous data is an ideal candidate for Bigdata Analytics. Let us see how does 3 V’s of Big data come into this scenario. All the data which is generated by these devices or things is voluminous or recurring at specific intervals.  This streaming data is “always in motion” so there is a velocity.  The variety is coming from all different sensors sending data.

Big Data will help make companies smarter, more progressive and give them a business advantage. You can have better control over your business with IoT through better tracking and better reporting. Let us see some examples.

Sigalert.com provides sensor-based analysis of traffic on highways.  This when combined with Waze, a world's largest community-based traffic and navigation app, helping drivers avoid the frustration of sitting in traffic, cluing them into a police trap or cutting 5 minutes off of their regular commute by showing them new routes they never even knew about. 

Great River Medical Center is one healthcare organization that's connecting many of its medical devices into a network using Microsoft's Windows Embedded, thereby enhancing the patient care by speeding delivery of medications, reducing an average 1.5 hour wait time, down to just 30 minutes. Getting the correct medication to patients faster has improved patient outcomes and reduced the rate of readmission.

With the Loxone iPhone app, you can access, monitor and control your home from anywhere.

A food distribution company can use sensors in trucks that send temperatures, humidity; point to point travel times back to the data center for further analysis.

Today's tech-savvy consumers have the option to shop whenever and wherever they want, including on mobile devices.

Retailers have a lot of use of IoT & Big data. They can measure the real-time customer traffic in & out of the store with video cameras, current queue lengths, historical transaction data & footfall data to predict how many more checkouts to be opened. This helps improve the customer experience.

Utility companies have installed smart meters to monitor energy, water & gas consumption.

Airbus A380 has sensors that monitor the wear and tear of the flight in real time which helps in the preventive maintenance of parts before they fail, reduce the warranty costs and increase operational efficiency.

As IoT becomes mainstream, it can play a big role in areas such as supply chain management. When customers' preferences or needs can be tracked in real time, businesses have the opportunity to react accordingly and immediately, with options such as dynamic messaging, pricing, or service delivery.

Big Data Analytics touching our lives everyday !!

The world is becoming more and more digital every day. Big Data is one of those megatrends that will impact everyone in one way or another.  

Welcome to the everyday world of "Big Data," the explosions of facts, maps, products, books, calls, references, smartphone apps, trends, videos, advertisements, surveys — all of the sense and nonsense that is literally at your fingertips, 24-7, every day from now on.

Big Data is all that everything we do in our lives to leave a digital trace (or data), which we (and others) can use and analyze for the betterment of our lives.

Here are some real-life examples of how big data is used today right from cradle to grave:

• In hospitals, pediatric unit looking after premature and sick babies is generating a live steam of every heartbeat. It then combines them with historical data to identify patterns. Based on the analysis the system can detect infections even before the baby would show any visible symptoms, which allows early intervention and treatment.
• Wearable fitness wristbands from Nike & Fitbit collects daily data about how fast we walk or jog, how many steps we have taken, the calories we have burnt each day, our sleeping patterns and other streams of data which are then combined with our health records by doctors & insurance companies for better wellness programs.
• In schools & colleges streaming videos courses and data analytics helps teachers track student’s progress, tailor the content to their abilities & predict how a student will perform.
• SmartThings, a company helps in Smart Homes with installing motion, moisture and other sensors in your home to collect data & keep you posted on what is happening at home & control all the devices via an app on the iPhone while you are away.
• While we daily drive on the roads, our smartphones send our location information & how fast we are moving, which then combined with real-time traffics to give us optimal routes to avoid traffics. Even combined with our location apps like AroundMe gives you nearby restaurants, banks, gas stations and lot more.
• In Retail, when we go shopping our loyalty card data is combined with our purchase history & social media data to give us coupons, discounts, and personalized offers.
• Finally, in IoT, companies like EarlySense are developing wellness & sleep monitoring sensors that go under the bed mattresses & automatically detects, monitors, and records heart rates, breathing rates, motion, and sleep activity as soon as a person gets into bed. The data collected by the sensor is wirelessly sent to smartphones and tablets, where it can be further analyzed.

Some other creative uses of Big data are:

• Transit Time NYC, an interactive map developed by WNYC, lets New Yorkers click a spot in any of the city's five boroughs for an estimate of subway or train travel times. They pulled data from open source itinerary platform OpentripPlanner & combined it with publicly downloadable subway schedule to create 4 million virtual trips.
• FluNearYou app developed by American Public Health Association surveys users to get a sense of their symptoms stores and analyzes the vast amount of resulting data and then produces reports to show users the flu activity in their local region.
• Buildzoom, a “one-stop-shop” for building, remodeling & renovating homes, has information about 2.5 million contractors, 50000+ customer reviews helping 500,000 users bring more objectivity & transparency in decision making.
• The FBI is combining data from social media, CCTV cameras, phone calls and texts to track down criminals and predict the next terrorist attack.

·      Presidential campaigns of Obama in 2012 used Big data Analytics to collect vast amounts of voter’s data from phone calls & surveys, coupled with top-notch analytical engines allowed him to micro-target the individual voters that were most likely going to vote in his favor.

• Google’s self-driving car is analyzing a gigantic amount of data from sensor and cameras in real time to stay on the road safely.

·       Smart TVs and set-top-boxes are able to track what you are watching, for how long and even detect how many people sit in front of the TV combined with social sentiments to determine the channel popularity.

• In Greece, the government is using Google Earth to see who can afford a swimming pool in their backyard, and then matching that against tax records.

Ultimately, you and I are going to benefit from Big data Analytics. Our economies are getting stronger when the banks have a better understanding of risk. Our taxes are lower when the government lowers its fraud expenses. Our communities are becoming healthier when disease outbreaks are pinpointed and treated earlier.

Hadoop Simplified

Today we live in the age of Big data.

Data volumes have outgrown the storage & processing capabilities of a single machine and the different types of data formats required to be analyzed have increased tremendously.  

  

This brings 2 fundamental challenges: 

• How to store and work with huge volumes & variety of data
• How to analyze these vast data points & use it for competitive advantage.

Hadoop fills this gap by overcoming both the challenges. Hadoop is based on research papers from Google & it was created by Doug Cutting, who named the framework after his son’s yellow stuffed toy elephant.

So What is Hadoop? It is a framework made up of:

• HDFS – Hadoop distributed file system
• Distributed computation tier using programming of MapReduce
• Sits on the low-cost commodity servers connected together called Cluster
• Consists of a Master Node or NameNode to control the processing
• Data Nodes to store & process the data
• JobTracker & TaskTracker to manage & monitor the jobs

Let us see why Hadoop has become so much popular now.

• Over the last decade, all the data computations were done by increasing the computing power of a single machine by adding the no of processors & increasing the RAM but they had physical limitations. 
• As the data started growing beyond these capabilities, an alternative was required to handle these storage requirements for eBay (10 PB), Facebook (30 PB), Yahoo (170 PB), JPMC (150 PB) and increasing
• With a typical 75 MB/Sec disk data transfer rate, it was impossible to process such humongous data
• Scalability was limited by physical size & no or limited fault tolerance
• Additionally, various formats of data are being added to the organizations for analysis, which is not possible with traditional databases

How Hadoop addresses these challenges?

• Data is split into small blocks of 64 or 128MB and stored onto minimum 3 machines at a time to ensure data availability & reliability
• Many machines connected in cluster work parallel for the faster crunching of data
• If anyone machine fails, the work is assigned to other automatically
• MapReduce breaks complex tasks into smaller chunks to be executed in parallel

Benefits of using Hadoop as Big data platform are:

• Cheap storage – commodity servers to decrease the cost per terabyte
• Virtually unlimited scalability – new nodes can be added without any changes to existing data gives the ability to process any amount of data, so no archival necessary
• The speed of processing – tremendous parallel processing to reduce processing time
• Flexibility – schema-less, can store any data format – structured & unstructured ( audio, video, texts, csv, pdf, images, logs, clickstream data, social media)
• Fault tolerant – any node failure is covered by another node automatically

Later multiple products & components are added to Hadoop so it is now called an eco-system.

• Hive – SQL like interface
• Pig – data management language like commercial tools AbInitio, Informatica
• HBase – column-oriented database on top of HDFS
• Flume – real-time data streaming such as credit card transaction, videos
• Sqoop – SQL interface to RDBMS and HDFS
• Zookeeper – a DBA management for Hadoop

 And multiple such products are getting added all the time from various companies like Cloudera, Hortonworks, Yahoo, etc.

How some of the world leaders are using Hadoop:

• Chevron collects large amounts of seismic data to find where they can get more oil resources
• JPMC uses it for storing more than 150 PB of data, over 3.5 Billion user log-ins for Credit scoring & Fraud detection
• eBay using it for real-time analysis and search of 9 PB data with 97 million active buyers, over 200 million items for Cross-Sell
• Nokia uses it to store data from phone, service logs to analyze how people interact with apps and usage patterns to address customer churn
• Walmart uses it to analyze customer behavior of over 200 million customer visits in a week
• UC Irvine Health hospitals are storing 9 million patients records over 22 years to build patients surveillance algorithms
• Manufacturers are using it for warranty analytics

Hadoop may not replace the existing data warehouses but it is becoming no 1 choice for Big data platform with the price/performance ratio.