How to deploy Data Science projects?

In this Digital age today, data science has become the top skill and sexiest job of the century. 

Data science projects do not have a nice clean life-cycle with well-defined steps like software development lifecycle (SDLC), but they are non-linear, highly iterative and cyclical between the data science team and various others teams in an organization.

SAS Institute, the leader in Analytics developed its own method called SEMMA (Sample, Explore, Modify, Model & Assess) for data mining.

However, many of the companies have adopted a standard workflow of a data science called CRISP-DM (CRoss Industry Standard Process for Data Mining). It was developed by a consortium of companies like SPSS, Teradata, Daimler and NCR Corporation in 1997.

With any method the process is similar which involves following steps:

• Business Understanding: This is the basic and first step as understanding business problem is extremely important for data scientist to move forward.
• Data Acquisition: Based on the business problem the next step is to understand and acquire the data which is needed. Identify the sources from where it is available, who are responsible to provide that data. It can come from various data sources like customer data, demographic data, third party data, weblogs, social media data, streaming data like sensor data, audio or video data. Main challenge is to decide whether data is up-to-date and clean for model consumption. With Internet of Things in full swing, data acquisition into Big Data platform is important step.
• Data Preparation: This is also called as data wrangling phase which takes almost 60% of overall project time. Collected data has to be formatted, treated for any missing values, any abnormalities or seasonality from the data and make it ready for model consumption.
• Modelling: This is the core activity of a data science project that requires writing, running and refining the programs to analyse and derive meaningful business insights from data. Often open sources tools like R, Python and commercial tools like SAS, IBM SPSS are used to create the statistical models. Various machine learning techniques are applied to data based on the business problem.
• Evaluation: There are several methods to compare the developed models and then use the best model for deployments. Typical comparison methods are AUC – area under curve, Confusion matrix, Gain/Life charts, Root Mean Squared Error etc.
• Deployment: Once the most suitable model is identified above, it is further tested with live data and then deployed into production environment.

There are further steps as well such as monitoring the live model performance, observe any degradation and new models are developed which are again compared with live model.

Data Science has evolved beyond normal predictive modeling into recommendation engines, text mining, deep learning, Artificial Intelligence. The foundation still remains the same of data gathering, data cleaning and then applying various algorithms.

Social Media Analytics

The Internet has taken the world by storm in the last few decades. Some of the surveys show that it has grown over 2000% in the last 10 years. Imagine how many people globally are online at any point in time. Being social animals, we like to talk about our feelings, good or bad, with our family & friends.   

Today people are using internet for communication more, than any other medium, This has given birth to social media sites like Facebook, Twitter, Orkut, myspace, digg & the list will go on. Imagine how much data is collected on these sites in one day.

So companies have started using this huge unstructured text data in these comments, sentiments of people for their product improvements, for Customer service improvements, for understanding what customers are talking about competitors. This is called Social Media Analytics.

In the recent example of Toyota recalls, they are closely watching these social sites to gauge what Customers are talking about their brand, quality & take necessary measures to correct their actions.

But social media analytics isn’t just about damage control, it can provide precise data to help you better understand your customers and discover new business opportunities.

Here are typical steps in implementing social media analytics.

• Collect the huge amount of unstructured data – comments, blogs, call center notes, twits from social sites
• Using statistical analysis & Natural Language processing (NLP) on texts & words to break up the information into good or bad
• Use categorization, classification & association methods for text processing
• Further, identify the categories on which these good or bad sentiments are applicable from the data
• Brand awareness, trust, loyalty
• Customer service
• Product improvements
• Competitor analysis
• Geographic locations
• Produce the results using visualization tools

Here is how various businesses are using Social Media Analytics:

• Manufacturing – using the warranty data combined with Customer complaints to improve products & reduce warranty costs
• Healthcare – find connections amongst the claims received to flag further frauds
• Banking & Finance – use the social network profiles data to improve credit calculations, identify reasons for Customer churn
• Insurance – claims cost prediction& fraud detection
• Telecom – improve the customer experience for new products introductions
• Hospitality & Travel – listen to guests comments to improve repeat customer rates

Social Media Analytics will continue to gain importance in near future !!

Use of Analytics in Business verticals.

In the last post, we saw how Analytics has become mainstream & how it is different from the Business Intelligence.

Let us see how businesses are using it for competitive advantage. Today businesses are more worried about survival than profitability. The very purpose of any business to exist is to be profitable & sustain it. This can happen only when there are good loyal & profitable Customers attached to the business.

Hence Customer analytics has become very prime importance in the current time and it is valid across all the business segments.

Customer Analytics:

• Customer Lifetime value – group Customers on high, medium, low value & take actions to increase revenue
• Customer Segmentation – the grouping of Customers based on demographics, or profitability, or lifetime value
• Customer Churn/attrition  - predict which Customers are likely to leave you & take suitable actions
• Customer Retention – identify most profitable Customers & then retain them
• Campaign management – selective campaigns based on segmentation or Customer’s likely behavior
• Cross-sell and Up-sell – increase the revenue proposing other products or high-end products

Apart from these, I am mentioning below some of the areas in business verticals, where Analytics is applied for foresight.

Banking & Financial Services Analytics:

• Anti-money laundering – identifying suspicious transactions to alert investigation officers
• Credit scoring – score the customer based on various parameters to arrive at a certain number and if that is above a threshold then approve the credit
• Credit Risk – predicting the risk involved due to nonpayment by borrowers in case of credit cards, loans etc
• Fraud detection & Prevention – predicting suspicious transactions which are likely to be fraud in all the transactions of the card, wire transfers, online transactions etc
• Price Optimization - Debt collection agency can predict the optimal price for the portfolio & forecast the probable recovery from defaulters

Insurance Analytics:

• Claims Fraud detection  - predicting the claims which are likely to be fraudulent
• Policy Lapse prediction – predict which are the policies that are going to lapse before completing the tenure
• Underwriting rate optimization – predicting the best price for the insurance products based on Customer profile
• Agent performance prediction – how agents are going to add revenues to the organization, improve customer satisfaction & retention
• Agent Lifetime Value – how best an agent is going to serve the organization throughout his/her tenure

Healthcare Analytics:

• Healthcare Claims Fraud detection  - predicting the claims which are likely to be fraud
• Financial recovery – predicting the payments from healthcare insurance payer which are overpayments to service providers
• Health plan analytics – allows organizations to compare & predict different benefits & risk options in terms of coverage & costs
• Condition Management – predict which of the people are likely to develop diseases like blood pressure, cholesterol etc

Retail Analytics:

• Discount or Price optimization – predict the optimal prices of discounts & normal prices of merchandizes  for today’s sensitive shopper
• Cross-Sell & Up-Sell – propose other products depending on various factors such as color, fashion, choice, location, earning patter & Customer buying behavior etc.
• Forecasting – based on demands from Customers predict how much stock is required to avoid stock-outs & excess inventory

Manufacturing Analytics:

• Predicting the parts failure – based on the history data predict  which of the mechanical parts are going to fail & when
• Issue detection – predicting the issues before they occur so preventive maintenance can be done on the parts
• Warranty Analytics – identify issues across the production period to reduce warranty costs
• Forecasting – based on demands from Customers predict how much stock is required to avoid stock-outs & excess inventory
• Inventory Optimization - to reduce inventory carrying costs & increase order fulfillment by predicting optimal inventory to be stored across warehouses

Text Analytics:

• Discover & extract meaningful patterns and relationships from the text collection from social media site such as Facebook, Twitter, Linked-in, Blogs, Call center scripts
• Understand Customer sentiments – positive & negative. Used for Product & Customer service improvements. Also for knowing what competition is good or bad at

Analytics is used in every area of life to get better insights into what is going to happen & what we can do so that the best outcome is expected !!!

So what is Business Analytics & its various components

In the first post, I talked about why Analytics is required more than ever now. In this post let us discuss, what is it all about & what are the typical components of Analytics.

Let us start with the definition of Analytics. There are multiple definitions available but as our focus is on Simplified-Analytics, I feel the one below will help you understand better.

Business Analytics is the use of statistical tools & technologies to:

• Find patterns in your data for further analysis e.g. product association
• Find out outliers from the huge data points e.g. fraud detection
• Identify relationships within the key data variables for further prediction e.g. next likely purchase from the Customer
• Provide insights as to what will happen next e.g. which of the Customers are leaving us
• Gain the competitive advantage.

So a more detailed comparison with Business Intelligence will help you understand better.

Business Intelligence	Business Analytics
What does it do?
Reports on what happened in the past or what is happening in now, in current time.	Investigate why it happened & predict what may happen in future.
How is it achieved?
Basic querying and reporting OLAP cubes, slice, and dice, drill-down Interactive display options – Dashboards, Scorecards, Charts, graphs, alerts	Applying statistical and mathematical techniques Identifying relationships between key data variables Reveal hidden patterns in data
What does your business gain?
Dashboards with “how are we doing” information Standard reports and preset KPIs Alert mechanisms when something goes wrong	Response to “what do we do next?” Proactive and planned solutions for unknown circumstances The ability to adapt and respond to changes and challenges

 

Now that you know the difference between BI & BA, let us discuss the typical components in Analytics.

There are 6 major components or categories in any analytics solution.

• Data Mining – Create models by uncovering previously unknown trends and patterns in vast amounts of data e.g. detect insurance claims frauds, Retail Market basket analysis.

         There are various statistical techniques through which data mining is achieved.

• Classification ( when we know on which variables to classify the data e.g. age, demographics)
• Regression
• Clustering ( when we don’t know on which factors to classify data)
• Associations & Sequencing Models
• Text Mining - Discover and extract meaningful patterns and relationships from text collections e.g. understand sentiments of Customers on social media sites like Twitter, Facebook, Blogs, Call center scripts etc. which are used to improve the Product or Customer service or understand how competitors are doing.
• Forecasting – Analyze & forecast processes that take place over the period of time e.g. predict seasonal energy demand using historical trends, Predict how many ice creams cones are required considering demand
• Predictive Analytics - Create, manage and deploy predictive scoring models e.g. Customer churn &  retention, Credit Scoring, predicting failure in shop floor machinery
• Optimization – Use of simulations techniques to identify scenarios which will produce best results e.g. Sale price optimization, identifying optimal Inventory for maximum fulfillment & avoid stock outs
• Visualization - Enhanced exploratory data analysis & output of modeling results with highly interactive statistical graphics

Hope this has helped you get the clarity on what is Analytics.

Next post I will cover how various business verticals are using Analytics.