NEW FEATURES AND FEATURE IMPROVEMENTS:

Classic ONE DATA

New Connection Framework

A new connection framework is introduced in OD with the Q4 release

  • The framework reduces the time required to add a new connection type in ONE DATA
  • The framework allows reading from: File, Apache Hive, MS SQL Server, MySQL, Oracle, PostgreSQL, Trino
  • New processors were introduced:
    • Metadata Load Processor provides metadata from the connection
      • your_instance /processors/processors-docs/processors/load/connectionLoad.html
      • your_instance/processors/processors-docs/processors/load/connectionMetadata.html
    • Connection Load Processor allows reading the data from the connection and execute queries 

Information about Table Partitioning

A new tab in Data Table Overview page will provide information about partitioning.

In case the tab is not displayed - there is no partitioning on this table.

Data Product Transfer and Update

Data Product Transfer and Update Improvements for Collaborative work

Data Product Transfer and update features were extended

  • Update mechanism becomes more flexible
    • Now it is possible to update projects in arbitrary directions. It is not necessary to follow the order DEV → STAGING → PROD anymore when performing an update. 
    • It is possible to have multiple projects for development. It is not necessary that updates stem only from the initial project.
    • Note: In order to benefit from the feature, please re-create the copies of your project newly. For that, export your the project using "new export logic".  Old and new export logic are incompatible.

The features will allow improvement of the existing git integration and allow pushing the state in git back in OD. Integration of the features into the existing DS script is ongoing.

  • Documentation: your_instance/project-transfer-docs-new/project-transfer-documentation/projectExport/projectExport.html

App Builder & App Viewer

App Builder: File Tree

  • App developers can group and structure components in folders and subfolders according to their needs
  • Drag & Drop / Copy & Paste of files and folders
  • File extensions are used to indicate the component type
  • Create new files by type with preset configuration
  • Components can be referenced by path and id now
  • Full text search on file names and content
  • Performance gains

Highcharts: Boxplots & Heatmap

  • Added support for Boxplots including outliers computation and visualization
  • Added support for non-numeric values as categories in Heatmaps

Pivot Tables

  • Enabling the processing of large data sets through automated server-side aggregation
  • Dark mode support
  • UI update and bug fixes

Custom Components

data-landscape


    • Double clicking a search result item selects the asset in the map
    • Interactive edge list in Edge Group (for multiple edges between assets)
    • Enhanced edge properties in sidepanel
    • Redesign of the input field of the search
    • Highlight an edge if hovering an edge in the search result list
    • Overall redesign of the search result list
    • Redesign of the edge group label (for multiple edges between nodes) to be more distinguishable from regular edges



TECHNICAL CHANGES:

Apps Server Configuration Change

  • The latest Apps Server version 2.3.1 reduces the information reported from GraphQL endpoints in error cases. This change is motivated by the discovery that some error messages from ONE DATA echo the authentication token, which could be returned via our API, for example, as part of the stack trace, which could lead to security issues.This change disables the stack trace and the error message reporting in errors in production. It can be selectively turned on by setting the configuration options introduced in this change if desired, e.g. for debugging purposes:
 apps:    server:      environment:        graphql.errors.add-stacktrace: true        graphql.errors.include-internal-messages: true

Disabled Policy Server Deployment

  • The Policy Server is included in this release but not used by any service. As a result its deployment is disabled per default.


Integrated Workflow Processor new defaults

  • Enable Integrated Workflow Execution with 3 and 5 ports by default.


URL configs in helm chart

  • Include URL configs for data-product-transfer and virtual-datatable clients in default helm chart



BREAKING CHANGES:

Area/Topic
Description of the breaking change
How to handle the breaking change
App Builder

The App Builder now provides the user with the possibility to structure content in folders. Besides some UI changes, this also implies a change in the structure of the App.json to include the folder structure.

While Apps are automatically migrated in the App Builder when edited the first time on newer versions, downgrading to older versions and Data Product Transfer/Update to older versions are not supported in an automated way. 

Note: The structure of older versions is still available in the App Builder as Viewer.json

Previously published Apps in App Viewer works as before (tick)

Editing Apps created on older instances automatically migrates App.json to ne format (tick)

DPT/DPU from old to new instance is supported. The App is transferred in old format and will be migrated automatically when edited (tick)

DPT/DPU from new to older instance will result in broken App (warning)
WORKAROUND: Additionally to DPT/DPU also the Viewer.json from the instance with new version has to be manually copied to the RAW file on the instance with the old version.

External version control: Diff/Merge will show many conflicts and will not work in a meaningful way. (warning)
WORKAROUND: If required then the old RAW file can be compared (diff) with the Viewer.json from new version, which uses the same structure. This is more a hint for a migration phase, but storing the Viewer.json in external version control is not recommended as it does not include the folder structures.

REST API Processors

In Q4 a bug fix was added to the Rest API processor, which covers following aspects:

  • previously, only 2 requests could concurrently be executed to the same target system, and a maximum of 20 concurrent requests in total; this is changed to defaults of 200 concurrent requests in total and no limitation on the number of concurrent requests per target system (both values can be overridden system-wide)
  • requests sent will not time-out from ONE DATA and can block a connection indefinitely; together with the aspect above this can lead to resource starvation; note that this behavior is only fixed for the Flexible REST API processor v1.1
  • the mentioned limitations apply to each built-in spark context (docker), stand-alone execution-context (docker) and each separate kubernetes worker (yarn, kubernetes) individually


In case the REST API processor in your workflows requires more than 120 seconds to execute, please adjust the "Socket timeout for the REST connections" setting in this Rest API processor to a higher timeout. 

data-landscape custom component

With the data-landscape, ui-lib as well as the apps-client adopting the composition api, data-landscape versions smaller 0.33.1 (that use the composition api) are no longer compatible with apps-client versions greater or equal to 8.4.0

Use data-landscape >= 0.33.1 and apps-client >= 8.4.0 in the future or apps-client < 8.4.0 for older data-landscape versions

R Serve

Disabled rserve by default as it will be deprecated in the future.

The R Script processors are not removed, they will fail with the error message `Establishing R Session with "rserve:6311" failed."` when rserve is not enabled.

It can be enabled by setting rserve.enabled = true in values.yaml.

FEATURE REMOVALS:

Feature
Removal reasons
Alternatives or replacements
Test LabDue to changed company priorities, recognized feature gaps, identified vulnerabilities and licensing issues, we do not want to spend further development resources on Test Lab