How to deploy Data Science projects?

In this Digital age today, data science has become the top skill and sexiest job of the century. 

Data science projects do not have a nice clean life-cycle with well-defined steps like software development lifecycle (SDLC), but they are non-linear, highly iterative and cyclical between the data science team and various others teams in an organization.

SAS Institute, the leader in Analytics developed its own method called SEMMA (Sample, Explore, Modify, Model & Assess) for data mining.

However, many of the companies have adopted a standard workflow of a data science called CRISP-DM (CRoss Industry Standard Process for Data Mining). It was developed by a consortium of companies like SPSS, Teradata, Daimler and NCR Corporation in 1997.

With any method the process is similar which involves following steps:

• Business Understanding: This is the basic and first step as understanding business problem is extremely important for data scientist to move forward.
• Data Acquisition: Based on the business problem the next step is to understand and acquire the data which is needed. Identify the sources from where it is available, who are responsible to provide that data. It can come from various data sources like customer data, demographic data, third party data, weblogs, social media data, streaming data like sensor data, audio or video data. Main challenge is to decide whether data is up-to-date and clean for model consumption. With Internet of Things in full swing, data acquisition into Big Data platform is important step.
• Data Preparation: This is also called as data wrangling phase which takes almost 60% of overall project time. Collected data has to be formatted, treated for any missing values, any abnormalities or seasonality from the data and make it ready for model consumption.
• Modelling: This is the core activity of a data science project that requires writing, running and refining the programs to analyse and derive meaningful business insights from data. Often open sources tools like R, Python and commercial tools like SAS, IBM SPSS are used to create the statistical models. Various machine learning techniques are applied to data based on the business problem.
• Evaluation: There are several methods to compare the developed models and then use the best model for deployments. Typical comparison methods are AUC – area under curve, Confusion matrix, Gain/Life charts, Root Mean Squared Error etc.
• Deployment: Once the most suitable model is identified above, it is further tested with live data and then deployed into production environment.

There are further steps as well such as monitoring the live model performance, observe any degradation and new models are developed which are again compared with live model.

Data Science has evolved beyond normal predictive modeling into recommendation engines, text mining, deep learning, Artificial Intelligence. The foundation still remains the same of data gathering, data cleaning and then applying various algorithms.

Why Open Source is gaining momentum in Digital Transformation?

Once upon a time in IT, using open source simply meant Linux instead of Windows, or maybe MySQL instead of Oracle.

Now, there is such a huge diversity of open source tools, and almost every leading digital business and tech startups are making extensive use of them. It’s been a remarkable turnaround for open source over the last 10 years, placing the trend firmly at the heart of the digital revolution.

The explosive growth of e-commerce, mobile and social media has completely altered the customer’s lifestyle and buying habits. Today, organizations are expected to engage with customers in Omni-channel environment. They need to create a customer journey. This is the driver of digital transformation.

A key differentiator for these companies is often real-time interaction with the ability to try something out fast, to create a new experience in real time. They have two fundamental challenges – reducing costs and growing revenues.

There are various advantages of open source tools and platforms:

• Costs: The less a company spends on proprietary software, the more it can dedicate to other facets of digital transformation efforts, since there is no need to pay per server, per core or per user license fees
• Faster time to market: Less expense & time that would go with traditional, lengthy evaluations of legacy software solutions and commitment to specific technologies that may lock them in
• Fail Fast:  Quickly test and experiment with emerging technologies without the risk & see whether it succeeds or fails and move on
• Security advantage: the transparency of open source code means that bugs are more easily found and patched, and users can inspect the code they're using themselves, which is not so possible with commercial products
• Community Backing: Open source software allows companies to derive benefit not only from their own IT employees, or from the employees of proprietary software vendors, but from the whole open source community
• Customization: The flexibility provided by open source allows users to add features freely, and reformat, refactor and redevelop their platforms however they see fit

Here are some examples of open source successes:

• MongoDB a NoSQL Big data database is used by Expedia, MetLife for customer 360 view 
• Liferay portals are used by Santander Bank, Allianz Insurance & London borough
• Apache Spark has emerged very quickly as a faster successor to Hadoop's MapReduce
• LinkedIn’s developed and built Apache Kafka as the messaging backbone 
• Drupal is used by GE, P&G and other global organizations for their websites
• Google, Amazon, Facebook Microsoft and even Apple have made their platforms open to the public.

The rapid adoption of open source technologies across organizations has provided a network-effect of reliability in scalability, affordability, and adaptability. 

Not only are these traits critical to open source technology, but they are also necessary for successful digital transformations in business to bring in whole new business models and the platforms that are dominating our economy.

Digital disruption in Customer Experience

In an age of digital disruption, great customer experience has become do or die. 

Digital technologies such as Analytics, Mobile, Cloud, Gamification, Cognitive computing, Artificial Intelligence and Social media have disrupted the customer experience.

Customer satisfaction has never been a higher priority than it is today, where fierce competition for space in the market means loyalties can change for the smallest reason.

Whether a customer is visiting the store, making a purchase through a mobile app, or liking a photo on Instagram, they must be able to engage your brand in a way that is both seamless and effortless, regardless of the device or channel. Match your channels to your customers’ needs and they’ll keep coming back.

Timing is everything in this ever-changing digital world. Every customer is a digital customer.

No matter your industry, every organization must deeply consider and transform the way they handle customer service. 

Understanding customers - their behavior, frustrations and needs - will allow you to identify what unique value you can bring to them. Big data analytics enable marketers to send customer communications that are more relevant, personalized and targeted than ever before

Initially considered a gaming innovation, virtual reality is turning out to be more mainstream, and the applications for organizations and purchasers are abundant.

Here are some of the reasons customers are giving for loving top brands:

• They are easy to do business with
• Hyper personalization that makes me feel great
• They are quick to resolve my queries
• Their product are class apart and their service is super
• Technical knowledge of their customer service staff is excellent
• They know me well when I call 

Today it seems that we’re not far from having drones landing on our lawns! The first time you experience something like this, it feels like a revelation but later it  will feel like it’s the norm.

Today’s customer delights are tomorrow's customer expectations. Apple, Google, Disney, Starbucks, Airbnb, and Amazon are just a few companies delivering these every day. Are you?

Digital Transformation helping Smart Cities flourish

A smart city is simply a community that harnesses digital technologies, such as Internet of Things, Big Data Analytics, Mobility, Drones & Wearables to improve the quality of life of people, optimize available resources such as electricity, roads and water in economically sustainable manner, and reduce environmental pollution. 

They are run just like businesses to with smart and user friendly services.

In smart city, IT is the major factor and the basis for providing essential services to residents. Smart Cities depend on data available to them from all the above technological advances to enhance the quality and performance of their services.

Smart Cities integrate and analyze massive amounts of data to anticipate, mitigate, and even prevent many problems. This data is leveraged, for example, to intelligently reroute traffic and reduce accidents, identify crime hot spots and target resources for crime reduction, and connect citizens at work or out on the town.

Younger people expect to have the city in their pocket — to find a cab, locate the next bus, find a nearby restaurant or event, report a pothole or broken streetlight, apply for their driver's license, or tweet to their government representatives. They want consistent, personal service via their smart device.

Here is what future smart city can give you:

• Smart Traffic: Connected car sensors collect valuable real time data about traffic density and report it the control center, allowing to adjust traffic light sequences intelligently, as well as apply manual corrections whenever necessary.
• Smart Parking: Integrated with municipal Buses and private car owners, to build a highly efficient parking system.
• Smart Energy: Automatic lighting control, may vary from obvious night-time auto activation to dimming based on crowd density and weather conditions. Every home will have a smart meter to control the power usage and report in real time.
• Smart Buildings: Facilities including power management, lighting, physical security, fire safety and IT infrastructure.
• Smart Safety: Firefighters wearing sensors that let them know if the CO₂ in a building has reached lethal levels. Grid of connected CCTV cameras to ensure public safety. Air quality sensors detect pollution peak and warns people who are subject to allergies through their smartphones.
• Smart Charging: There are charging stations by the side of each road for electric vehicles, drivers can pay by flashing their wallet or even contact less payment.
• Smart Advertising: Personalized digital signs for people passing by.
• Smart Water System: IoT sensor enabled water systems which measure the flow, pressure, level and chemical content of the water to improve quality and usability
• Smart Views: Use of augmented and virtual reality to help people indoor and outdoor navigation.
• Smart Access: Users can open car doors or office doors by just touch of finger or even with IoT enabled sensors ensuring doors are opened when users are approaching
• Smart Commute: Connected cars and trains giving personal entertainment based on Big data Analytics insights.
• Smart Planet: Special lanes for bicycles for health conscious commuters helping zero carbon footprint

New York, London, Paris, Columbus Ohio, Dubai & Barcelona are the examples of smart cities around the world.

Smart Cities bring multiple rewards like better control of public spending, more engaged citizens in local development, improved performance of public transportation systems, lighter traffic and a reduced carbon footprint.

Smart Cities innovations have come a long way. All these digital technologies are future. Someday will come, to make Captain Kirk’s “Beam us up scotty” catchphrase true.

A to Z of Digital Transformation

Today every part of the business is subject to new expectations, competitors, channels, threats, and opportunities. Every business has the potential to be a digital business.  

Businesses that digitally transform will be able to connect more closely with customers, speed up the pace of innovation and, as a result, claim a greater share of profit in their sectors. Today digitally transformed companies have an edge; tomorrow, only digital businesses will succeed.

Here is my version of A-Z of Digital Transformation.

Artificial Intelligence: AI is the capability of a machine to imitate intelligent human behavior. BMW, Tesla, Google are using AI for self-driving cars. AI should be used to solve real-world tough problems like climate modeling to disease analysis and betterment of humanity

Big Data: This serves as a foundational backbone for digital transformation. Hyper-personalization and real-time recommendations based on the geo-location of the customers will be the key to success. Big Data Analytics is driving force for action.

Customer Centricity: The most important theme of digital transformation is it should be customer focused with a consistent, enjoyable and very personal experience to customers.

Disruption: Digital has disrupted many businesses so far who did not take it seriously like Blockbuster, Kodak, Borders and brought new leaders on the horizon like Airbnb, Uber, Netflix, Spotify.

Employee-driven: Digital Transformation will be successful only when employees are involved, educated and empowered to enhance the customer experience.

Fail Fast: Digital is all about deploying quick PoCs and check the outcome. If it does not work then drop the concept/idea and move on to a new one. Be prepared to change the pace or direction as necessary.

Growth Hacking: Focuses on building products that fit the market needs, with a viral push to market with a very simple idea & all the digital tools like email, tweets, blogs. Zynga used FB alerts and online adverts while DropBox offered extra storage space for referring friends for growth.

Hyper-personalization: Use of browsing histories, past purchases, social media data, and internal customer data to provide more personalized and targeted products, services, and content. It is about being relevant at the right place and the right time to the right person.

Internet of things: All the businesses are aiming at reaching their Customers anywhere, anytime, any platform with any device.  All such smart devices or physical objects that are connected to the internet, are continuously emitting data and communicating with each other. IoT based real-time predictive maintenance in manufacturing is becoming important to save costs and downtime.

Journey Maps: Creating customer journey maps is the prime and initial milestone for digital transformation where you understand the pitfalls and pain points with every touch point you have with customers. Once they are in place then plan with most impacting problems.

KNN: K nearest neighbor algorithm in Machine Learning used for classification problems based on distance or similarity between data points.

Leadership Support: Without actual drive from CEO and Board members, digital will not move further as it involves cultural business transformation.

Mobile First: Today we live in “always on, always connected” world, and use mobile for almost everything. We spend more time on mobile than any other digital channel. Start your digital implementation with mobile first in mind. More than half of the world population is on mobile now and

NFC: Near Field Communication used for a contactless, WiFi-style technology that could already be in your smartphone. A few years from now your credit cards, bus pass, train tickets, loyalty cards for high street coffee shops will be gone due to digital transformation and you only carry your phone.

Omni-Channel: It is about true continuity of customer’s experience even when they flip channels to complete a single transaction. Disney, Virgin Atlantic, Starbucks are great examples of Omni-Channel digital transformation.

Prioritize: Spend where you’ll make the biggest difference for the business. Small and quick successes will be drivers for digital transformation.

Quality: Quality of data collected by all the channels is extremely important in digital transformation for generating meaningful insights. Garbage in, Garbage Out still remains valid in the digital age.

Robotics: Robots are machines with programmed movements that allow them to move in certain directions or sequences. Artificial intelligence has given robots more ability to process information and to learn.

Sentiment Analytics: Sentiment analysis is all about helping companies gain better insights into their customers, and helping them to bridge the gap between insight and action by analyzing positive and negative sentiments.

Talent: To deliver on the digital transformation, organization have to get the right talent. As Jim Collins said in his book “Good to Great”, get the right people in the bus put them in right sits and then decide where the bus will go.

User Experience: This is extremely important considering the focus on the customer. Use of responsive web designs to adjust to any screen sizes with ease, make the website accessible on any platform, intuitive and pleasant to use is the key.

Virtual Reality: Virtual Reality is making a lot of impact on the world we live in today. Everything that we know about our reality comes by way of our 5 senses – Sight, Sound, Smell, Touch & Taste. Today Real Estate, Travel companies are coming up with VR gadgets to give the feel of virtual walks of houses or destinations like Eiffel tower / or Grand Canyon experience on your living room couch.

Wearables: They come in various forms, like smart watches, health trackers, Google Glass, interactive clothing, gesture controllers and the list goes on. Sooner or later, all of us are identified by the data we generate, and wearables represent a quantum leap in the type and quantity of data collected — which is both an interesting and a scary proposition

Xamarin: A modern programming language for iOS, Android & Windows mobile platforms.

Yarn: The earlier map-reduce has undergone a complete overhaul and now called yarn that allows multiple data processing engines such as interactive SQL, real-time streaming, data science and batch processing to handle data stored in a single platform.

ZooKeeper: ZooKeeper is a Hadoop centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Digital Transformation brings changes in product creation and delivery, changes in IT infrastructure, changes in the consumption and payment methods to ‘go digital’.