Provided functionality

  • Default options for ONE DATA API creation
  • Configurable retries (retry_timeout, retry_interval, retry_count)
  • Configurable deserializers per request

General Information

The SDK allows to set options as a general rule on initialization of the SDK. So it is possible to set a general e.g. timeout for each SDK method being called.

By default all values are set to None and therefore are not mandatory to set.

If a option parameter is set to None the SDK provides a default behaviour for each parameter.

Each command has a respective deserializer as default configured.

The default value for the timeout option is set to 10 seconds.

onedata_api = OneDataApi(base_url="https://internal.onedata.de", username="sdk.user@onelogic.de",
                         password="password", verify=True, timeout=10)

Furthermore it is possible to specify options for each SDK method call individually. This will overshadow the configuration of the SDK instance for this call.

onedata_api.workflows.jobs.get(id=job.id, timeout=5)

Configurable Retries

The provided methods for retrieving workflow jobs allows to configure a retry behaviour. This allows to e.g. "wait" for a workflow to finish by requesting the job multiple times until it reached a finished state.

For further information see documentation for Workflow Jobs

Available Options

Request Transformers

Transforms a given input Request and returns it. It is also possible to return a new Request object, but it is not advised.

Available Request Transformers

  • HeaderTransformer

    Transformer to add headers to a Request. Constructor expects either a key and a value (for a single header) or a headers dict containing multiple keys and values (for multiple headers). Existing headers with the same key will be overwritten.

Example on how to add a header for proxy authorization.

onedata_api = OneDataApi(base_url="https://random.teams.onedata.de", username="sdk.user@onelogic.de",
                         password="password", verify=False)

# Using key and value
wf: Workflow = onedata_api.datatables.get(id=wf_id,
                                          request_transformers=[
                                              HeaderTransformer(key="proxy-authorization", value="123")
                                          ]

# Using headers dict
wf: Workflow = onedata_api.datatables.get(id=wf_id,
                                          request_transformers=[
                                              HeaderTransformer(headers={"proxy-authorization": "123"})
                                          ]
  • BaseUrlTransformer

    Transforms the base URL of a Request. Constructor expects the base URL of type str. Existing base URL in the Request will be overwritten.

This example shows how to create a datatable on another ONE DATA instance. It still is encouraged to create a separate ONE DATA API though.

onedata_api = OneDataApi(base_url="https://random.teams.onedata.de", username="sdk.user@onelogic.de",
                         password="password", verify=False)

response: UploadedDataResponse = onedata_api.datatables.create(name="dt via sdk", headers=["a", "b"],
                                                               content=[{"a": 1, "b": "asd"}] * 10,
                                                               project_id=project_id, locked=DatasetLock.OPEN,
                                                               storage_format=DataTableStorageFormat.PARQUET,
                                                               create_fallback=False,
                                                               request_transformers=[
                                                                   BaseUrlTransformer("https://internal.onedata.de"),
                                                                   HeaderTransformer("authorization", new_token)
                                                               ]
  • SSlVerificationTransformer

    Transforms the SSL verification of a Request. Constructor expects a bool indicating if verification should be done or a str path to a certificate file. The existing value of the verification in the Request will be overwritten.

This example shows how to use a custom certificate

onedata_api = OneDataApi(base_url="https://random.teams.onedata.de", username="sdk.user@onelogic.de",
                         password="password", verify=True)

response: UploadedDataResponse = onedata_api.datatables.create(name="dt via sdk", headers=["a", "b"],
                                                               content=[{"a": 1, "b": "asd"}] * 10,
                                                               project_id=project_id, locked=DatasetLock.OPEN,
                                                               storage_format=DataTableStorageFormat.PARQUET,
                                                               create_fallback=False,
                                                               request_transformers=[
                                                                   BaseUrlTransformer("https://another.onedata.instance.de"),
                                                                   HeaderTransformer("authorization", new_token),
                                                                   SslVerificationTransformer("C:/certificates/another.od.instance.cert")
                                                               ]
  • TimeoutTransformer

    Transforms the timeout of a Request. Constructor expects a timeout in seconds. The existing value of timeout in the Request will be overwritten.

This example shows one way on how to configure a request to await the response for 120 seconds

latest_job_deep: WorkflowJob = onedata_api.workflows.jobs.get_latest_job_deep(wf_id=wf_id,
                                                                              job_execution_state=JobExecutionState.SUCCESS,
                                                                              request_transformers=[TimeoutTransformer(120)])
  • RequestSleepTransformer

    Introduces a sleep delay before a Request is executed. Constructor expects the sleep time in milliseconds.

    The request is not modified. This can be used to reduce the flooding of the API by introducing artificial delays

This example shows one way on how to configure a request to sleep 5 seconds before execution

job: WorkflowJob = onedata_api.workflows.execute_async(id=workflow.id, request_transformers=[RequestSleepTransformer(5000))

Response Transformers

Transforms a given input Response and returns it. It is also possible to return a new Response object, but it is not advised.

Available Response Transformers

  • ResponseSleepTransformer

    Introduces a sleep after a Request is executed and the Response is received. Constructor expects the sleep time in milliseconds. The Response is not modified. This can be used to reduce the flooding of the API by introducing artificial delays.

This example shows one way on how to configure a request to sleep after the response is received for 5 seconds

job: WorkflowJob = onedata_api.workflows.execute_async(id=workflow.id, request_transformers=[ResponseSleepTransformer(5000))

Custom Transformers

It is possible to create and use a custom Request/Response Transformer which has to provide a transform method executing the wanted behaviour and returning a Request/Response.

class CustomTransformer(RequestTransformer):
    def transform(self, request):
        print(request)
        return request

job = od.workflows.jobs.get(id=job.id, retry_count=5, request_transformers=[CustomTransformer()])

Response Deserializers

This can be used to change the deserializer for a SDK method call and therefore also its return type.

Usage Example

When retrieving a Workflow, I want the full response including the status code from the API call and not just a Workflow object.

wf: Workflow = onedata_api.workflows.get(some_id)
oneDataApi.set_deserializer(WorkflowGetCommand, IdentityDeserializer)

# note that the return type is now different
wf: Response = oneDataApi.workflows.get(some_id)
print(wf.json()["name"], wf.status_code)

Available Deserializers

General DeserializersResponse DeserializersString Deserializers
AnythingToTrueDeserializer AnythingToNoneDeserializer DictToDatabaseConnectionTableInformationDeserializer DictToDataTableContentDeserializer DictToGenericResourceDeserializer DictToOwnerInformationDeserializer DictToProjectInformationDeserializer DictToProductionLineShallowJobDeserializer DictToProductionLineShallowJobEntryDeserializer IdentityDeserializerResponseToConnectionDeserializer ResponseToCredentialsDeserializer DictToKeyInformationExposedDeserializer DictToBasicAuthInformationExposedDeserializer DictToPersonioAuthInformationExposedDeserializer DictToOneDataAuthInformationExposedDeserializer DictToMicrosoftRopcOAuthInformationExposedDeserializer ResponseToDataTableDeserializer DictToDataTableDeserializer ResponseToDictDeserializer ResponseToFunctionExecutionResultDeserializer DictToFunctionExecutionErrorDeserializer DictToFunctionExecutionErrorCauseDeserializer ResponseToJsonDeserializer ResponseToPandsDataFrameDeserializer DictToPandasDataFrameDeserializer ResponseToProductionLineDeserializer DictToProductionLineDeserializer DictToProductionLineSequenceEntryDeserializer ResponseToProductionLineJobDeserializer DictToProductionLineJobDeserializer ProductionLineJobEntryDeserializer ResponseToUploadedDataResponseDeserializer DictToModifiedDatasetNameDeserializer ResponseToUserDeserializer DictToUserDeserializer ResponseToWorkflowDeserializer ResponseToWorkflowJobDeserializer DictToWorkflowJobDeserializer ResponseToWorkflowJobShallowDeserializerStringToDatabaseTablePrivilegeDeserializer StringToDateTimeDeserializer StringToFunctionExecutionStatusDeserializer StringToGenericResourceTypeDeserializer StringToOneDataModuleDeserializer StringToPermissionDeserializer StringToProductionLineJobExecutionStateDeserializer

Sleep After Response Millis

This can be used to reduce the flooding of the API by introducing artificial delays.

wf: WorkflowJobShallow = onedata_api.workflows.jobs.get_latest_job_shallow(wf_id, sleep_after_response_millis=1000)

SSL Verification

The verify property allows to enable/disable the ssl verification for requests. Further more it is possible to reference a local certificate file by passing its location.

onedata_api = OneDataApi(base_url="https://internal.onedata.de", username="sdk.user@onelogic.de",
                         password="password", verify="C:/certificates/onedata/onedata.cer", timeout=10, sleep_after_response_millis=1000)

Timeout

Allows to set how many seconds the SDK should wait until a request is considered as failed / how long it is allowed to take.

This is especially useful to set when a request is expected to take longer. E.g. for waiting and then fetching a running job of a bigger workflow once it is done.

wf: WorkflowJobShallow = onedata_api.workflows.jobs.get_latest_job_shallow(wf_id, timeout=120)