Data Validation Tools for Accurate Business Insights

Anomalo offers a variety of data validation testing techniques. We provide four categories of checks in the data validation process:

• Data freshness: Checks if new data has arrived on time
  Data volume: Checks if new data is complete
  Missing data: Looks for drops in segment records, increases in NULL values, or increases in zero values
  Table anomalies: Runs Anomalo’s AI-based anomaly detection to find anomalous records, and also checks for increases in duplicate data for columns that previously contained unique records

Beyond this, you can choose from 40+ kinds of user-defined validation rules in the Anomalo UI. Data engineers can write custom checks with SQL and even run them programmatically using Anomalo’s API.

Very customizable! If you need to set up a rule for validating data in Anomalo, you can use our no-code UI or integrate with our API. Within the no-code UI, there are over 40 checks that each allow you to specify a range of thresholds, options, and constraints. When defining a check in the UI, you can enter a custom SQL query at any time if you don’t see the options you’re looking for. Additionally, the API provides nearly limitless capabilities for specifying validation checks in code.

Anomalo offers one-click integrations with all major data warehouses/lakes/lakehouses. You can also run Anomalo’s checks as part of your ETL pipeline via our integrations with orchestration tools like Airflow and dbt. Additionally, you can connect Anomalo to a data catalog like Alation to ensure that the results of data validation checks are accessible to all business users.

Anomalo offers dozens of visualizations to bring you more insight into why a check might be passing or failing, as well as a high-level executive dashboard that visually tracks data quality progress and highlights problem areas across your entire data warehouse. A few specific features to highlight are our root causes analysis and data lineage tools. Our root cause analysis feature creates a severity report of each incident and lets you pull samples of good and bad rows to dig into the issue further. Our data lineage tools map out upstream and downstream tables to create a better understanding of how data assets move through your system. Additionally, our comprehensive validation features ensure that the correct data is captured and maintained throughout your workflows, helping to detect anomalies and inconsistencies before they affect business outcomes.

Anomalo uses unsupervised anomaly detection techniques. Taking on a sample of 10,000 records from the most recent data and comparison samples from previous days, our machine learning algorithms will search for ways that the existing dataset is different from those previous days. If detected changes are above a dynamically adjusted threshold, which is based on the historical data “chaos” or levels of fluctuation in the data, this check will fail.

Anomalo provides an automated and repeatable solution to validate data entry and monitor data movement processes. The machine learning algorithms need to run for 2 weeks to produce useful results. Results will continue to improve over the next 30–60 days. When a table is newly configured, the dynamically-adjusted threshold will be set to the maximum score, which requires anomalies to be extreme. Gradually, the threshold will decrease as the anomaly detection algorithms learn the expected chaos for the table.

With automated root cause analysis, you can download a sample of a “good” and “bad” data point every time Anomalo flags an issue. Anomalo also will highlight the columns and data segments that contain the largest portions of anomalous data points, compared to normal data. These tools and visualizations will help you pinpoint the anomaly in your data.