General Description

The Landscape Quality Monitoring is a quality monitor & maintenance APP, with which the quality of data can continuously be monitored - based on rules defined by the users. Thereby it comprises a Data Product that is able to generate and maintain data set quality check rules that are needed by for example Data Scientists on a regular basis. Furthermore, the check rules can be used not just for single projects, but they can be used at different places while being created and maintained in a central place. The following topics will be covered in this article:

Terminology Used in the App

In the App an own terminology is used, therefore the following terms are explained:


Check Rule

A Check Rule is an independent entity / statement which measures defined conditions of a persisted data set or its content, and returns a Boolean result. However, it is not directly associated to any data set(s).

Note: Users can create Check Rule structures based on one data set or use two data sets.

AssignmentsAssignments connect a (generalized) Check Rule with specific data set.
Quality Check
Quality Checks are a final state that will be used to monitor the quality of your data. It is a combination of selected Check Rules and their assigned data set(s). The Quality Checks will be executed and thereby generate results for monitoring the data quality.
Multiple Quality Checks

The purpose of this functionality is to create Quality Checks for multiple data sets and their respective multiple columns at the same time and can be done for simple Check Rules only (incorporating only one column in only one data set).

Use CaseBy selecting a Use Case, you add more information (metadata) to your Quality Checks. Use Case selection can be applied to register for notifications for a group of Quality Checks, and would help as a search parameter for your Quality Checks.

Understanding the Landscape Quality Monitoring App

When you open the App you will see the following welcome page:


You will be able to access and work through three main tabs: Dashboard, Quality Check, Notifications. 


Lets go through each of these tabs to see how you can create Check Rules and monitor your data quality.


Dashboard: An overview of executed assignment results along with a search functionality.


Quality Check: An overview of all available assignments and also the possibility to create new assignments. Manual execution and an option to create a schedule for execution are available. The overview table also provides a column 'URL FOR EXECUTING VIA API' with values that can be used to call/execute the Quality Check Workflows from outside the App using an API call.


Notifications: Register to receive a notification if the execution of your Quality Check fails. Get an overview of existing registered notifications. Possibility to unsubscribe (delete) a registered notification.

How to Create a Quality Check

Navigate to the QUALITY CHECK tab and click on CREATE NEW QUALITY CHECK.


Then, follow the subsequent six sub-steps as described in the App to create a Quality Check.


1. You are currently in the tab STEP 1: SELECT DATASET(S). Select a data set here from the dropdown menu. If you wish to use two data sets within your Quality Check, toggle the button on the top right corner to select the second data set via another dropdown menu.


2. Move to the tab STEP 2: SELECT CHECK RULE. Here, you can either select an existing Check Rule of your choice or create a new one by clicking on the button CREATE NEW CHECK RULE (refer to How to Create a Check Rule). 

    

3. Select a Check Rule by clicking on the arrow under the Use For Quality Check column. The selected Check Rule name appears below the table. Then, click SAVE to save your Check Rule selection.


4. After receiving a success toast message, move to the tab STEP 3: ASSIGN COLUMNS. Verify the details, like dataset selected, query skeleton, adapted Check Rule query and select or verify the column names mapping to the listed variables. The goal in this step is to assign column values to the variable value(s) of the reusable Check Rule query.


5. Move to the last tab STEP 4: ASSIGN USE CASES. You are able to select one or multiple Use Cases to which your Quality Check will be added.


6. When everything is ready, click the SAVE button to save the new Quality Check. After receieving a successful toast message, close the window and the newly created Quality Check will be listed on the overview page (QUALITY CHECK tab).


Known constraints and assumptions: Use Cases cannot be defined manually, but the whitelisted projects of the Data Catalogue are available as options to choose from.


How to Create a Check Rule

Follow these four steps to create a Check Rule.


1. Select a dataset. A previously selected dataset is pre-selected but user can change it if needed. Based on the number of datasets, the Check Rule structure gets defined and will be used in the same way in the future when reusing for another dataset(s).


2. Write a query in the Check Rule SQL Editor that defines the rule for your data. A query that defines the state you want to avoid. Your query should deliver 0 rows in case everything is fine with your dataset(s).


Important: Please make sure the query is re-usable by writing all column names and their alias between curly brackets. Example: Select * from inputTable i where {{i.column_name}} IS NULL 


3. Below the editor window, you can find a button that tests your Check Rule and displays a summary of the result in the adjoining table and as a KPI below. 


4. To save the Check Rule, specify a name and description and then hit the SAVE button.


How to Create Multiple Quality Checks

The purpose of this functionality is to create Quality Checks for multiple datasets and their respective multiple columns at the same time. This will help user get rid of redundancy and save time. The steps are similar to creating a normal 1/2 dataset Quality Check, except that these Quality Check can be done using simple Check Rules (based on one column) but therefore for multiple data sets, thus saving time. 


Navigate to the QUALITY CHECK tab and click on CREATE MULTIPLE QUALITY CHECK.


Follow the subsequent five sub-steps described in the App to create multiple Quality Checks.


1. You are currently in the tab STEP 1: SELECT DATASET(S) AND COLUMN(S). Select multiple datasets from the first dropdown, and then select multiple required columns of the selected datasets.


2. Move to the tab STEP 2: SELECT CHECK RULE. Either select the Check Rule of your choice or create a new one by clicking on the CREATE NEW CHECK RULE button (refer to How to Create a Check Rule). 


3. Select a Check Rule by clicking on the arrow under the Use For Quality Check column. The selected Check Rule name appears below.


4. Move to the tab STEP 3: ASSIGN USE CASE(S). You are able to select one or multiple Use Cases to which your Quality Checks should be added. 


5. Click SAVE to save the new Quality Checks. When a successful toast message is received, close the window and the newly created Quality Checks will be listed on the overview page (QUALITY CHECK tab).


How to Execute Quality Checks

Quality Check results can be obtained in three different ways: 

  • by manual execution 
  • by scheduling them for execution 
  • by creating or executing a Quality Check Workflow from outside the App via the API call, using the URL specified in the summary table


Manual Execution

To manually execute the Quality Check, go to the QUALITY CHECK tab and click on the run arrow in the EXECUTE column (first column in the overview table) of the desired assignment.


Scheduling Execution

1. Go to the QUALITY CHECK tab and click on the respective arrow from the EDIT SCHEDULES column (last column in the overview table) of the desired assignment.


2. Within the Scheduler window, specify the start date, start time and the frequency with which the assignments should be executed. Userscan also include a stop strategy after which the Scheduler should stop execution. After entering all details, click on SAVE button.


3. An overview table below shows all the existing and newly created Schedules. There is also a possibility to delete a Schedule. 


Note: For all the scheduled Quality Checks, the SCHEDULED column in the Quality Check overview page is marked as YES.


Via API Call

In the Quality Check Overview table, the column URL FOR EXECUTING VIA API provides a URL addition which enables the user to execute the Quality Check from outside the App. For this, Processors like for example Flexible REST API or Python can be used.


Known constraints: There is no possibility to edit the date and time of a Schedule. The user needs to delete one and then create a new Schedule with the updated values instead.


How to Register to Receive Notifications

Note: Notifications will be sent via email only when the Assignment execution fails.


1. Go to the NOTIFICATIONS tab.


2. Use the filters for Check Rule Creator and Use Case to pre-filter the available Quality Check to register for. Then, select all Quality Checks you wish to register for from the third dropdown. The subsequent dropdowns get filtered as per the previous selections.


3. Click SAVE to receive notifications in case of failure.


4. An overview table shows all the existing and newly created notification registrations. There is also the possibility to unsubscribe from a notification by deleting the respective row.


Boundaries & Current Constraints

The CREATE MULTIPLE QUALITY CHECKS functionality is only possible for simple check rules which are based on one data set and one column (see documentation).