Dutch Analytics - Xenia documentation

Pipelines

Pipelines allow users to create sequences of Models and Connectors , by connecting the output of models to other models and connectors. Xenia manages the data flow through models, so there is no need to manually make separate model requests: the output of the first model will be the input of the subsequent model or connector.

You can use a pipeline just like you would use a model. All models and connectors can be used in multiple pipelines at once, which gives a lot of flexibility. All components in a pipeline will scale independently of each other, efficiently distributing the compute load for your process.

Creating a pipeline

A new pipeline is created by going to Pipelines in the left navigation panel and then clicking the CREATE button. The input sent to the pipeline will be send to the first model in that pipeline. For more info about inputs checkout the Models pages.

Adding objects to a pipeline

When you view the details of a pipeline, by clicking on pipeline name, you enter the scope of the pipeline. In the General tab, the visual representation of the pipeline can be seen.

The Objects menu on the right gives an overview of all models and connectors in your pipeline. At first, this tab is empty. Your Models and Connectors form a library. When you want to use them in your pipeline, you need to create a reference to these objects in your pipeline. You can add objects by clicking on Add new object in the General tab.

When you are adding an object you have to fill in the following fields:

FieldDescription
Reference typeType of the object, for example Model.
Reference nameThe name of the model or connector you want to use.
Reference version
(If Reference type = Model)
The version of the model you want to use.
NameThe name you want to give to this object. It must be unique within the pipeline.

After creating a reference, the object should appear in the Objects menu as well as in the visual representation of the pipeline.

Connecting objects

To connect models and connectors in your pipeline, click on Connect two objects. Here, you can specify the source and destination object of the connection.

📘

Drag and drop

You can also drag a line from one object to another to connect them. This will result in a connection creation form in which the source and destination are automatically selected.

Two objects can be connected if the output fields of the source object matches in data type the input fields of the destination object. All input fields of the destination object must be provided in the mapping while only the necessary fields of the source object might be selected. Multiple input fields of the destination object may be filled by the same source object output field.

To connect a model or connector to the entry point of your pipeline, you need to select your pipeline as the source object.

Currently, simultaneous input from two objects into one is not supported, meaning that two source objects cannot provide part of the destination object input fields.

The connectors can only be connected as a final object in the pipeline since they save the result to a data storage and do not produce any outputs.

Deleting objects in a pipeline

You can delete an object by clicking on it and then clicking on the trash icon. This will automatically delete all the attachments of this object too. Only the reference objects of the pipeline will be deleted, not the actual models or connectors that are being referenced.

Updated 11 days ago

Pipelines


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.