Road to becoming a Smart-Data-Driven-Enterprise
Blog - Analytics, Data Analytics, Data Ingestion, ESB, Middleware, Systems Integrations
Business executives analyze internal / external information to make strategic decisions. The decisions are based on information gathered from various silos where data is aggregated before executives analyze to take decisions.
Let’s look at a retail company where sales in a specialty product are going down. Is the executive presented with information from a holistic perspective (suppliers, vendors, associates, customers, competitors, etc.) on how the company performed historically within and across product departments?
Enterprise information is siloed and, unless this information is brought under one umbrella, the true picture cannot be arrived at. Data warehouses were specially built for such analytical purposes, but the growth in data volume seen in last few years is humongous and the data itself is heterogeneous (structured, semi-structured and unstructured). Digital era is the primary culprit for such growth and it will continue to be so for a few more years.
The internet revolution has ushered in the rise of many prominent companies (Google, Yahoo, Facebook etc.) which stored petabytes of consumer information. Leveraging traditional tools to analyze this data set is not an option; hence the advent of big data technologies. These successful internet companies have built tools / products to analyze massive amounts of information and open-sourced them. Technology based enterprises started leveraging these tools / products to solve similar big data issues.
Traditional businesses have slowly started to embrace these tools but are running into a conundrum because of the rapid pace at which the open-source community is pushing competing products.
We at MSRCOSMOS believe in SIMPLICITY: From data sourcing, to analysis and reporting – each one of these steps is crucial and needs to be effected in as simple, and as fast a manner as possible. This will allow enterprises to focus easily on analyzing business data leading to better and effective decision-making. But, does mere ingestion of data and providing tools to business / data-scientists solve problems faced by decision-makers? NO; IT DOESN’T!
Therefore, there are numerous vendors that promise (through their products & solutions) to enable enterprises become ‘Data-Driven’, but this doesn’t solve the problem either. We need products or solutions that can build “Smart-Data-Driven Enterprises.”
We’ve just had a peek into what a Smart-Data-Driven Enterprise is. Now, let us look at what all goes into making one. DATA INGESTION is the first step (Technical) in realizing a Smart-Data-Driven Enterprise. Traditionally, enterprises have business data either at rest (reference, master, reporting) or in motion (transactional, golden), with the advent of mobile computing, visit (Site, store, etc.) data picked up a lot of traction to identify customers’ preferences and provide them offers / coupons / discounts. In addition to the structured data, unstructured data gained prominence (paper documents, images, audio, video files etc.) from a business intelligence stand-off.
Enterprises store structured / unstructured data in files, databases or streams. Business or functional heads or data-scientists work on these types of data to derive meaningful information with short-term and long-term perspectives. These users need the ability to capture / store information through a configuration-driven (Across the website) tool to ingest data from different silos (SBUs or functions) for analytical purposes.
HCube is a data ingestion solution that enables capturing enterprise data from files, databases (traditional, NoSQL) and streams (JMS, HTTP, MQTT etc.) through a configuration-driven, easy-to-use UI approach. The data can be ingested into HDFS, Hive, or HBase. Users have the ability to leverage Zeppelin or other analytical tools of their choice (SpotFire, Tableau, etc.) for creating rich visualizations.
Ingesting structured and unstructured data through configuration driven mechanism is only the starting point in realizing a smart-data-driven enterprise. The ability to identify and resolve data quality issues ranging from cleansing, sorting, and deduplication before data is prepared for advanced analytics is a primary requirement from enterprise customers.
There are three phases in Data Quality:
- Data Profile – Data discovery phase
- Data Quality – Identify Data quality requirements
- Data Quality Assessment
During ingestion, HCube runs statistical analysis on sample datasets to build profile of data structure, content, rules and relationships. Rules executed on datasets are to identify column cardinality, NULL values, empty values, lengths and data formats. Based on generated profile analysis, data quality rules will be framed and assessment done on datasets.
Data Preparation & Analytics
However, merely capturing big-data doesn’t guarantee enterprise success. But asking the right questions on the data does. Before executing questions, data needs to be cleansed, prepared and, in some cases, summarized to an extent the questions make sense. The ingested raw data needs to be harmonized, enriched and, in some cases, standardized before it can be consumed by analytical tools.
The HCube data preparation phase involves (1) data analysis to detect anomalies with business processes (2) enrichment and transformation (3) data consolidation into one / multiple tables. Most of the enterprises are stuck with long phases of descriptive analytics instead of a short one. Predictive / Prescriptive analytics when incorporated in the early phases provide businesses with the edge to stay ahead of competition.
HCube OOTB offers real-time and historical predictive analytics with business emphasis on rapid results measured in hours instead of the days / weeks it took with traditional approaches. The product ships with analytics templates tailored to various verticals. For example, Competitor Intelligence Templates, which compare product pricing against competitors along with predictive indicator adjustment to compare forecasts (Sales / Revenue) against competitor pricing. The template uses a lot of statistical modeling, data mining, and machine learning techniques to study recent and historical data, thereby allowing to make predictions.
For a deep-dive into how MSRCosmos can help your enterprise become a truly smart-data-driven enterprise, reach out to firstname.lastname@example.org, or contact +1 925-399-4218. We’ll be happy to design the roadmap for your data journey.