Provided functionality
- Default options for ONE DATA API creation
- Configurable retries (retry_timeout, retry_interval, retry_count)
- Configurable deserializers per request
General Information
The SDK allows to set options as a general rule on initialization of the SDK. So it is possible to set a general e.g. timeout for each SDK method being called.
By default all values are set to None and therefore are not mandatory to set.
If a option parameter is set to None the SDK provides a default behaviour for each parameter.
Each command has a respective deserializer as default configured.
The default value for the timeout option is set to 10 seconds.
onedata_api = OneDataApi(base_url="https://internal.onedata.de", username="sdk.user@onelogic.de",
password="password", verify=True, timeout=10)
Furthermore it is possible to specify options for each SDK method call individually. This will overshadow the configuration of the SDK instance for this call.
onedata_api.workflows.jobs.get(id=job.id, timeout=5)
Configurable Retries
The provided methods for retrieving workflow jobs allows to configure a retry behaviour. This allows to e.g. "wait" for a workflow to finish by requesting the job multiple times until it reached a finished state.
For further information see documentation for Workflow Jobs
Available Options
Request Transformers
Transforms a given input Request and returns it. It is also possible to return a new Request object, but it is not advised.
Available Request Transformers
HeaderTransformer
Transformer to add headers to a Request. Constructor expects either a key and a value (for a single header) or a headers dict containing multiple keys and values (for multiple headers). Existing headers with the same key will be overwritten.
Example on how to add a header for proxy authorization.
onedata_api = OneDataApi(base_url="https://random.teams.onedata.de", username="sdk.user@onelogic.de",
password="password", verify=False)
# Using key and value
wf: Workflow = onedata_api.datatables.get(id=wf_id,
request_transformers=[
HeaderTransformer(key="proxy-authorization", value="123")
]
# Using headers dict
wf: Workflow = onedata_api.datatables.get(id=wf_id,
request_transformers=[
HeaderTransformer(headers={"proxy-authorization": "123"})
]
BaseUrlTransformer
Transforms the base URL of a Request. Constructor expects the base URL of type str. Existing base URL in the Request will be overwritten.
This example shows how to create a datatable on another ONE DATA instance. It still is encouraged to create a separate ONE DATA API though.
onedata_api = OneDataApi(base_url="https://random.teams.onedata.de", username="sdk.user@onelogic.de",
password="password", verify=False)
response: UploadedDataResponse = onedata_api.datatables.create(name="dt via sdk", headers=["a", "b"],
content=[{"a": 1, "b": "asd"}] * 10,
project_id=project_id, locked=DatasetLock.OPEN,
storage_format=DataTableStorageFormat.PARQUET,
create_fallback=False,
request_transformers=[
BaseUrlTransformer("https://internal.onedata.de"),
HeaderTransformer("authorization", new_token)
]
SSlVerificationTransformer
Transforms the SSL verification of a Request. Constructor expects a bool indicating if verification should be done or a str path to a certificate file. The existing value of the verification in the Request will be overwritten.
This example shows how to use a custom certificate
onedata_api = OneDataApi(base_url="https://random.teams.onedata.de", username="sdk.user@onelogic.de",
password="password", verify=True)
response: UploadedDataResponse = onedata_api.datatables.create(name="dt via sdk", headers=["a", "b"],
content=[{"a": 1, "b": "asd"}] * 10,
project_id=project_id, locked=DatasetLock.OPEN,
storage_format=DataTableStorageFormat.PARQUET,
create_fallback=False,
request_transformers=[
BaseUrlTransformer("https://another.onedata.instance.de"),
HeaderTransformer("authorization", new_token),
SslVerificationTransformer("C:/certificates/another.od.instance.cert")
]
TimeoutTransformer
Transforms the timeout of a Request. Constructor expects a timeout in seconds. The existing value of timeout in the Request will be overwritten.
This example shows one way on how to configure a request to await the response for 120 seconds
latest_job_deep: WorkflowJob = onedata_api.workflows.jobs.get_latest_job_deep(wf_id=wf_id,
job_execution_state=JobExecutionState.SUCCESS,
request_transformers=[TimeoutTransformer(120)])
RequestSleepTransformer
Introduces a sleep delay before a Request is executed. Constructor expects the sleep time in milliseconds.
The request is not modified. This can be used to reduce the flooding of the API by introducing artificial delays
This example shows one way on how to configure a request to sleep 5 seconds before execution
job: WorkflowJob = onedata_api.workflows.execute_async(id=workflow.id, request_transformers=[RequestSleepTransformer(5000))
Response Transformers
Transforms a given input Response and returns it. It is also possible to return a new Response object, but it is not advised.
Available Response Transformers
ResponseSleepTransformer
Introduces a sleep after a Request is executed and the Response is received. Constructor expects the sleep time in milliseconds. The Response is not modified. This can be used to reduce the flooding of the API by introducing artificial delays.
This example shows one way on how to configure a request to sleep after the response is received for 5 seconds
job: WorkflowJob = onedata_api.workflows.execute_async(id=workflow.id, request_transformers=[ResponseSleepTransformer(5000))
Custom Transformers
It is possible to create and use a custom Request/Response Transformer which has to provide a transform method executing the wanted behaviour and returning a Request/Response.
class CustomTransformer(RequestTransformer):
def transform(self, request):
print(request)
return request
job = od.workflows.jobs.get(id=job.id, retry_count=5, request_transformers=[CustomTransformer()])
Response Deserializers
This can be used to change the deserializer for a SDK method call and therefore also its return type.
Usage Example
When retrieving a Workflow, I want the full response including the status code from the API call and not just a Workflow object.
wf: Workflow = onedata_api.workflows.get(some_id)
oneDataApi.set_deserializer(WorkflowGetCommand, IdentityDeserializer)
# note that the return type is now different
wf: Response = oneDataApi.workflows.get(some_id)
print(wf.json()["name"], wf.status_code)
Available Deserializers
| General Deserializers | Response Deserializers | String Deserializers |
|---|---|---|
| AnythingToTrueDeserializer AnythingToNoneDeserializer DictToDatabaseConnectionTableInformationDeserializer DictToDataTableContentDeserializer DictToGenericResourceDeserializer DictToOwnerInformationDeserializer DictToProjectInformationDeserializer DictToProductionLineShallowJobDeserializer DictToProductionLineShallowJobEntryDeserializer IdentityDeserializer | ResponseToConnectionDeserializer ResponseToCredentialsDeserializer DictToKeyInformationExposedDeserializer DictToBasicAuthInformationExposedDeserializer DictToPersonioAuthInformationExposedDeserializer DictToOneDataAuthInformationExposedDeserializer DictToMicrosoftRopcOAuthInformationExposedDeserializer ResponseToDataTableDeserializer DictToDataTableDeserializer ResponseToDictDeserializer ResponseToFunctionExecutionResultDeserializer DictToFunctionExecutionErrorDeserializer DictToFunctionExecutionErrorCauseDeserializer ResponseToJsonDeserializer ResponseToPandsDataFrameDeserializer DictToPandasDataFrameDeserializer ResponseToProductionLineDeserializer DictToProductionLineDeserializer DictToProductionLineSequenceEntryDeserializer ResponseToProductionLineJobDeserializer DictToProductionLineJobDeserializer ProductionLineJobEntryDeserializer ResponseToUploadedDataResponseDeserializer DictToModifiedDatasetNameDeserializer ResponseToUserDeserializer DictToUserDeserializer ResponseToWorkflowDeserializer ResponseToWorkflowJobDeserializer DictToWorkflowJobDeserializer ResponseToWorkflowJobShallowDeserializer | StringToDatabaseTablePrivilegeDeserializer StringToDateTimeDeserializer StringToFunctionExecutionStatusDeserializer StringToGenericResourceTypeDeserializer StringToOneDataModuleDeserializer StringToPermissionDeserializer StringToProductionLineJobExecutionStateDeserializer |
Sleep After Response Millis
This can be used to reduce the flooding of the API by introducing artificial delays.
wf: WorkflowJobShallow = onedata_api.workflows.jobs.get_latest_job_shallow(wf_id, sleep_after_response_millis=1000)
SSL Verification
The verify property allows to enable/disable the ssl verification for requests. Further more it is possible to reference a local certificate file by passing its location.
onedata_api = OneDataApi(base_url="https://internal.onedata.de", username="sdk.user@onelogic.de",
password="password", verify="C:/certificates/onedata/onedata.cer", timeout=10, sleep_after_response_millis=1000)
Timeout
Allows to set how many seconds the SDK should wait until a request is considered as failed / how long it is allowed to take.
This is especially useful to set when a request is expected to take longer. E.g. for waiting and then fetching a running job of a bigger workflow once it is done.
wf: WorkflowJobShallow = onedata_api.workflows.jobs.get_latest_job_shallow(wf_id, timeout=120)