{"id":2267,"date":"2020-02-19T01:16:01","date_gmt":"2020-02-18T22:16:01","guid":{"rendered":"http:\/\/kusuaks7\/?p=1872"},"modified":"2024-01-08T14:18:38","modified_gmt":"2024-01-08T14:18:38","slug":"why-you-need-data-quality-automation-to-make-data-driven-decisions-with-confidence","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/why-you-need-data-quality-automation-to-make-data-driven-decisions-with-confidence\/","title":{"rendered":"Why You Need Data Quality Automation To Make Data-Driven Decisions With Confidence"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe quarterly report showed a 40% drop in sales over the last quarter. The executive team panicked. The entire company is collapsing! Or is it? After the investigation concluded, they discovered that the sales were okay. The report was incorrect because an automated system had failed. The infrastructure that copied files with orders didn\u2019t copy 6 out of 10 files to the data lake \u2014 false alarm. The problem was not sales, but rather the quality of the data that was the basis of the dashboards.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Data quality is essential for trustworthy decisions<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData quality is critical because data is used for decision making and powering AI models. Models and decisions are only as good as the data behind them, so any lack of confidence in the data means they are less useful in predicting and providing insights, slowing down, and undermining fast decision making. Trust in data is hard to get and easy to lose, so data quality must be maintained for models and dashboards to be useful at all times. In the last 10 years, volume, variety, and velocity of business data have increased dramatically, making it impossible to control the quality of data with static testing. Therefore, To ensure confidence in analytics and predictive models at all times, data-driven firms need to be able to monitor data quality in production.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn this blog post, we review a typical data flow, causes of data corruption how to set confidence levels and data quality goals. Lastly, we go under the hood and explain our solution to monitoring data quality in production in real-time.\n\nBelow is a basic diagram of a typical data flow. Data enters the flow diagram from internal databases and other sources. It enters a central repository consisting mainly of the enterprise data warehouse and data lake. Data is then distributed to various outputs as required.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAnywhere along this data flow, corruption may occur. We look at protections at each step.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The leading causes of data corruption<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData corruption takes on many forms and has many causes. Three of the most prevalent causes of data corruption are code, data sources, and infrastructure. Catching data corruption requires data monitoring in production in realtime.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Code\u00a0corruption<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe application itself may cause some data corruption. Traditional application code debugging techniques may find some bugs, but there is a high probability it may not find all. Static testing of code in data pipelines is not enough because it is expensive to have 100% coverage, so not all defects are identified.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Data\u00a0sources\u00a0and\u00a0extraction\u00a0corruption<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData may be corrupt directly out of the data source. Sensors may fail; human error during data input could occur.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Infrastructure\u00a0corruption<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSources out of the direct control of the application can cause data corruption. Networks can go down; power glitches can affect data integrity; even sunspots have been known to cause network mishaps.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Using confidence levels to fine-tune data quality<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData quality is not a binary measurement. Some data may be difficult to resolve as valid or corrupt. For example, an expected value of a column may be between 5 and 50, but 0.3% of values in the dataset are out of range. Does this indicate value corruption or simply outliers? To better understand these marginal values, a \u201cconfidence level\u201d can be attached along with the data quality flag.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The goals of data quality monitoring<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tLet\u2019s focus on the data lake. With no data quality monitoring, the data lake would look like this:\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOn the left are jobs supplying data to the lake. Data is stored and manipulated with no quality checking. Any corrupted data would not be detected, and it can spread the corruption to adjoining modules.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn the figure below, data quality jobs get inserted between regular data processing jobs. Each data processing job is isolated from the others with the data quality jobs:\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhen the data quality job detects an anomaly, we immediately instigate action before the corruption spreads. Depending on the severity of the corruption, the remedy may be to flag the error and continue or, for more severe issues, the pipeline may be stopped to prevent further corruption.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tLet’s look at these steps in more detail\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Detect data corruption<\/h4><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThis blog deals mainly with this section. The first step in controlling data quality is always detecting the offending data point.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Prevent it from spreading<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOnce we find an issue, it should be isolated from causing further corruption. Depending on the degree of severity, this could mean tagging the offending data point, disabling a sub-system, or even shutting down the entire system.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Fix it<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOnce we identify an issue and have it isolated, the next step is to fix it. The support team responsible for the system is alerted. They may take one of several actions depending on the source of the issue. A system or subsystem may need rebooting. A sensor may need repair. The offending datapoint may need removal. These are a sampling of possible remedies.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Methods to implement data quality in real-time<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe proper way to enforce data quality in real-time is to embed data quality assurance logic in the data processing pipeline, after each step of the pipeline.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t“Methods to implement data quality in real-time Why You Need Data Quality Automation To Make Data-Driven Decisions With Confidence” \t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Control divergence from SoR at the input<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t