A data source binds external data to Chioro. It reads or receives the data from an external endpoint and converts it to Chioro’s internal format during execution.

Data sources represent the starting points of a flow, so to speak. Consequently, each flow requires at least one data source.

The configuration of a data source includes two main aspects:

  • In what format is the data available?
  • Where does the data come from?

File

Type of the data source

Currently there are four basic ways to import product data into Chioro:

  • Upload a file via the browser. Mainly intended for testing and playing.
  • Remote file. Direct (pull) access to data via a number of protocols. FTP, SFTP, S3 (various providers), and Microsoft Azure, among others. See below for more details.
  • API Push. Using the Push API, it is possible to actively send data to a Chioro data source. See below for more details.
  • Operation Using the operation it is possible to import the result of an operation from another flow into the data source and use it as a source.

Upload file

If this type is selected, a gray area appears where a file can be dragged and dropped. Alternatively, clicking on the area opens a file selection dialog.

  • While uploading the file please do not navigate to another page, close the browser or refresh the page. Otherwise, the file upload will have to be performed again.
  • Please also note that uploading is only useful for smaller files up to about 100 MB.

Remote file

For direct access to a remote file, an endpoint is selected in the Storage field. The Path field then specifies the path of the file to be used relative to this endpoint.

Endpoints represent a kind of “base URL”. Since these usually contain confidential information such as passwords or API keys, their configuration is done in the admin area. Appropriate admin rights are required for this. Please contact your Chioro administrator or our Support if you do not have the appropriate rights.

API Push

If this type is selected, a URL specific to this data source will be generated and displayed. This URL can be used to send data to the data source via HTTP PUSH.

Data sets that are sent to the Push API are first buffered in the data source. Only when a Ready-For-Processing message is sent, the data source becomes “active”. That is, only then is the data analyzed and only then is the data available for subsequent operations. Furthermore, only then the data source is recognized by a trigger as “ready with new data”.

However, a push data source can be activated immediately by starting the import manually (via the “Import” button). However, only partial data may then be available.

More about the usage in a data source can be found below at External data sources.

Operation

Using this option it is possible to select the result of another operation from another flow and use it as a source.

Datei

In this example, the data source is the output of a split named “Split” imported from “Flow number 1”. This data can then be further used for any operations.

However, it is also possible to use another operation as a source:

Datei

Here you can see that also a source of another flow can be set as source for the own flow. All operations except “Data destination” can be used as source.

Data formats

Currently Chioro supports the following import formats:

For further, e.g. user-specific data formats, please contact our Support.

External data sources

This type of data source enables the integration of external systems. To use these sources, an external program must be able to transfer records in a specific format (JSON) via “REST over HTTP”. Unlike the rest of the Chioro API, a special API key is required to access this endpoint, which must be submitted as part of the HTTP headers. The following describes the header and body format in detail.

To create an API token, please go to the admin area and create a configuration of type ‘API Key’. Please note the API key that will be displayed immediately after creation. This key is displayed only once and cannot be viewed afterwards. To use the external API, please use this key in the HTTP header in the following format (replace the <<API_TOKEN>> below):

Authorization: Token <<API_TOKEN>>

Body

The text of the request must be formatted as follows:

{
  "executionId": "an ID, not required but recommended",
  "operation": "one of CLEAR, APPEND or READY_TO_PROCESS. Description below",
  "data" : [{}, {}, {}, ...]
}

Please note that only the “POST” statement is supported.

Operation

  1. CLEAR: the data source is regenerated and cleared. All existing data will be removed.
  2. APPEND: add a record or a set of records to the data source. If the data source does not exist, it will be regenerated.
  3. READY_TO_PROCESS: indicates the logical end of the data stream. At this point the data source is ready to be used and the import will be performed.

Data

data must always be specified as an array, even if it contains only one element. For some operations, such as CLEAR, data is is optional, when used, the data source is first cleared and then filled with the data.

The format of the data is not predefined and depends on the client. However, an important limitation of Chioro is that, once a certain attribute is transmitted in a certain format, all subsequent lines must respect this format and send data only in this format. For example, if the attribute “color” is sent as the string “red”, no other line may send the same attribute “color” with a nested format like {“hue” : 12, “saturation”: “22”, …}. So “color” must not appear once as a simple string and another time as an array.

Response

Ideally, the operation returns an empty list if everything could be imported successfully. If one of the lines causes an error it will be displayed in the response list.