Project: kfp

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning workflows based on Docker containers within the Kubeflow project.

Use Kubeflow Pipelines to compose a multi-step workflow (pipeline) as a graph of containerized tasks using Python code and/or YAML. Then, run your pipeline with specified pipeline arguments, rerun your pipeline with new arguments or data, schedule your pipeline to run on a recurring basis, organize your runs into experiments, save machine learning artifacts to compliant artifact registries, and visualize it all through the Kubeflow Dashboard.

Installation

To install kfp, run:

pip install kfp

Getting started

The following is an example of a simple pipeline that uses the kfp v2 syntax:

from kfp import dsl
import kfp


@dsl.component
def add(a: float, b: float) -> float:
    '''Calculates sum of two arguments'''
    return a + b


@dsl.pipeline(
    name='Addition pipeline',
    description='An example pipeline that performs addition calculations.')
def add_pipeline(
    a: float = 1.0,
    b: float = 7.0,
):
    first_add_task = add(a=a, b=4.0)
    second_add_task = add(a=first_add_task.output, b=b)


client = kfp.Client(host='<my-host-url>')
client.create_run_from_pipeline_func(
    add_pipeline, arguments={
        'a': 7.0,
        'b': 8.0
    })