Note: This article is an extension to the article Debug Mode - General Explanation


Overview

The main goal of this feature is to deliver an easily accessible and user-friendly visualization of debugging information to users, which is provided by the ONE DATA server. This will help have a more detailed Workflow diagnosis, thereby saving more time developing them. It may also be used to evaluate performance bottlenecks in existing and newly created Workflows.


New Features

Given below, are the new features along with a detailed description for each one of them.


Keep in mind that all these features are only accessible after performing a debug action.


Display Debug Information and Intermediate Data Samples

To provide a better debug experience, this feature enables the user to generate on-the-fly debugging information and counts on datasets 'flowing' over the edges connecting Processors.

The debugging result within the Processor no longer contains only JSON result objects (compared to previous ONE DATA versions), but in addition, it shows sample records of the data at input and output port level, displayed in different tabs.

Each tab shows data samples, row count, and execution time information.



The same tables can be seen by clicking on the input or output ports of a Processor within the workflow.


The bug icons on ports will be visible only when the job result contains "dataDebuggerResult".


Also by executing a Workflow, you can see the execution time on Processors as a tooltip (hover your mouse over the desired processor).



Note: OD has many transformation Processors. These do not cause computation on the data inside themselves. Instead, all collected transformations will be executed in a later Processor, where data gets materialized (e.g. Row Count, Result Table, Save, etc.). So while the preceding Processors don't have much execution time themselves, they contribute to the complexity (and therefore the execution time) of terminal Processors. Full Debug can give some insights into the complexity added by each Processor (by comparing the increase in execution times).





Workflow Edge Coloring for Visualized Execution Status

Showing colored Workflow edges based on the execution status generates a user-friendly visualization experience. This helps to better locate Processors that affect the job and cause a relative failure (especially in complex workflows). Usually, this is used to narrow down an issue to specific Processors, so that isolated debug actions can be performed exactly there in later stages.




Note on the priority of visualization: The decreased importance goes from ERROR to WARNING to SUCCESS.


In general, Workflow edge colors depend on the execution status of the connected Processor. More specifically, the Processor affects the edge connected to its input port. Coloring is divided as follows:


ColorStatusDefinition
GreenSuccessful

Edge has a successful Processor as target.

  • Processor has a corresponding on-the-fly result
  • Processor does not have any corresponding warnings

OrangeWarning

Edge has a warning Processor as target.

  • Processor does not have any corresponding errors
  • Processor has corresponding warnings

RedFailed

Edge has a failed Processor as target.

  • Processor has corresponding errors

GreyNot Executed

Edge has a non executed Processor as target

  • Processor does not have any corresponding errors
  • Processor does not have any corresponding results 
  • Processor does not have any corresponding warnings


Trigger the Debug Mode Directly From Within a Processor

This feature already exists in the old version of the Debug Mode (check the following link). What's new is that this functionality is now available within the Processor configuration to offer ease of using Debug Mode and reduce the user transition.




Example Scenarios

Presented below, are some example scenarios for using the Debug Mode's new features:

  • Executing the Workflow in "save and debug" mode


  • Executing Workflow showing different edge colors as per the execution status


  • Executing partially using "fast debug" or "full debug" options on a specific Processor



  • Triggering Debug Modes from within a Processor configuration




Some Constraints

  • If for some reason the server cannot provide debug data, the user obviously cannot visualize it through the UI.
  • Currently when a partial execution is performed (right-click on a Processor with any Debug Mode), the whole Workflow is validated before execution. Invalid (or missing but mandatory) configuration values anywhere in the Workflow will prevent the execution.
  • At the present, the above functionalities do not support grouped Processors yet.