Migrate from KFP SDK v1
If you have existing KFP pipelines, either compiled to Argo Workflow (using the SDK v1 main namespace) or to IR YAML (using the SDK v1 v2-namespace), you can run these pipelines on the new KFP v2 backend without any changes.
If you wish to author new pipelines, there are some recommended and required steps to migrate your pipeline authoring code to the KFP SDK v2.
Terminology
- SDK v1: The
1.x.x
versions of the KFP SDK.1.8
is the highest minor version of the v1 KFP SDK that will be released. - SDK v1 v2-namespace: Refers to the v1 KFP SDK’s
v2
module (i.e.,from kfp.v2 import *
), which permits access to v2 authoring syntax and compilation to IR YAML from the v1 SDK. You should assume that references to the v1 SDK do not refer to the v2-namespace unless explicitly stated. Until the release of the v2 KFP OSS backend, these pipelines were only executable on Google Cloud Vertex AI Pipelines. - SDK v2: The
2.x.x
versions of the KFP SDK. Uses the v2 authoring syntax and compiles to IR YAML.
There are two common migration paths:
- If your existing
kfp==1.x.x
code imports from thev2
namespace (i.e.,from kfp.v2 import *
), follow the SDK v1 v2-namespace to SDK v2 migration instructions. This migration path only affects v1 SDK users that were running pipelines on Google Cloud Vertex AI Pipelines. - If your existing
kfp==1.x.x
code imports from the main namespace (i.e.,from kfp import *
), follow the SDK v1 to SDK v2 migration instructions.
SDK v1 v2-namespace to SDK v2
With few exceptions, KFP SDK v2 is backward compatible with user code that uses the KFP SDK v1 v2-namespace.
Non-breaking changes
This section documents non-breaking changes in SDK v2 relative to the SDK v1 v2-namespace. We suggest you migrate your code to the “New usage”, even though the “Previous usage” will still work with warnings.
Import namespace
KFP SDK v1 v2-namespace imports (from kfp.v2 import *
) should be converted to imports from the primary namespace (from kfp import *
).
Change: Remove the .v2
module from any KFP SDK v1 v2-namespace imports.
Previous usage | New usage |
---|---|
|
|
output_component_file parameter
In KFP SDK v2, components can be compiled to and loaded from IR YAML in the same way as pipelines.
KFP SDK v1 v2-namespace supported compiling components via the @dsl.component
decorator’s output_component_file
parameter. This is deprecated in KFP SDK v2. If you choose to still use this parameter, your pipeline will be compiled to IR YAML instead of v1 component YAML.
Change: Remove uses of output_component_file
. Replace with a call to Compiler().compile()
.
Previous usage | New usage |
---|---|
|
|
Pipeline package file extension
The KFP compiler will compile your pipeline according to the extension provided to the compiler (.yaml
or .json
).
In KFP SDK v2, YAML is the preferred serialization format.
Change: Convert package_path
arguments that use a .json
extension to use a .yaml
extension.
Previous usage | New usage |
---|---|
|
|
Breaking changes
There are only a few subtle breaking changes in SDK v2 relative to the SDK v1 v2-namespace.
Drop support for Python 3.6
KFP SDK v1 supported Python 3.6. KFP SDK v2 supports Python >=3.7.0,<3.12.0.
CLI output change
The v2 KFP CLI is more consistent, readable, and parsable. Code that parsed the v1 CLI output may fail to parse the v2 CLI output.
.after referencing upstream task in a dsl.ParallelFor loop
The following pipeline cannot be compiled in KFP SDK v2:
with dsl.ParallelFor(...):
t1 = comp()
t2 = comp().after(t1)
This usage was primarily used by KFP SDK v1 users who implemented a custom dsl.ParallelFor
fan-in. KFP SDK v2 natively supports fan-in from dsl.ParallelFor
using dsl.Collected
. See Control Flow user docs for instructions.
Importer component import statement
The location of the importer_node
object has changed.
Change: Import from kfp.dsl
.
Previous usage | New usage |
---|---|
|
|
Adding node selector constraint/accelerator
The task method .add_node_selector_constraint
is deprecated in favor of .add_node_selector_constraint
. Compared to the previous implementation of .add_node_selector_constraint
, both methods have the label_name
parameter removed and the value
parameter is replaced by the parameter accelerator
.
Change: Use task.set_accelerator_type(accelerator=...)
. Provide the previous value
argument to the accelerator
parameter. Omit the label_name
.
Previous usage | New usage |
---|---|
|
|
SDK v1 to SDK v2
KFP SDK v2 is generally not backward compatible with user code that uses the KFP SDK v1 main namespace. This section describes some of the important breaking changes and migration steps to upgrade to KFP SDK v2.
We indicate whether each breaking change affects KFP OSS backend users or Google Cloud Vertex AI Pipelines users.
Breaking changes
create_component_from_func and func_to_container_op support
Affects: KFP OSS users and Vertex AI Pipelines users
create_component_from_func
and func_to_container_op
are both used in KFP SDK v1 to create lightweight Python function-based components.
Both functions are removed in KFP SDK v2.
Change: Use the @dsl.component
decorator, as described in Lightweight Python Components and Containerized Python Components.
Previous usage | New usage |
---|---|
|
|
Keyword arguments required
Affects: KFP OSS users and Vertex AI Pipelines users
Keyword arguments are required when instantiating components as tasks within a pipeline definition.
Change: Use keyword arguments.
Previous usage | New usage |
---|---|
|
|
ContainerOp support
Affects: KFP OSS users
ContainerOp
has been deprecated since mid-2020. ContainerOp
instances do not carry a description of inputs and outputs and therefore cannot be compiled to IR YAML.
ContainerOp
is removed from v2.
Change: Use the @dsl.container_component
decorator as described in Container Components.
Previous usage | New usage |
---|---|
|
|
VolumeOp and ResourceOp support
Affects: KFP OSS users
VolumeOp
and ResourceOp
expose direct access to Kubernetes resources within a pipeline definition. There is no support for these features on a non-Kubernetes platforms.
KFP v2 enables support for platform-specific features via KFP SDK extension libraries. Kubernetes-specific features are supported in KFP v2 via the kfp-kubernetes
extension library.
v1 component YAML support
Affects: KFP OSS users and Vertex AI Pipelines users
KFP v1 supported authoring components directly in YAML via the v1 component YAML format (example). This authoring style enabled component authors to set their component’s image, command, and args directly.
In KFP v2, both components and pipelines are compiled to the same IR YAML format, which is different than the v1 component YAML format.
KFP v2 will continue to support loading existing v1 component YAML using the components.load_component_from_file
function and similar functions for backward compatibility.
Change: To author components via custom image, command, and args, use the @dsl.container_component
decorator as described in Container Components. Note that unlike when authoring v1 component YAML, Container Components do not support setting environment variables on the component itself. Environment variables should be set on the task instantiated from the component within a pipeline definition using the .set_env_variable
task configuration method.
v1 lightweight component types InputTextFile, InputBinaryFile, OutputTextFile and OutputBinaryFile support
Affects: KFP OSS users and Vertex AI Pipelines users
These types ensure files are written either in text mode or binary mode in components authored using the KFP SDK v1.
KFP SDK v2 does not support authoring with these types since users can easily do this themselves.
Change: Component authors should inputs and outputs using KFP’s artifact and parameter types.
AIPlatformClient support
Affects: Vertex AI Pipelines users
KFP SDK v1 included an AIPlatformClient
for submitting pipelines to Vertex AI Pipelines.
KFP SDK v2 does not include this client.
Change: Use the official Python Vertex SDK’s PipelineJob
class.
Previous usage | New usage |
---|---|
|
|
run_as_aiplatform_custom_job support
Affects: Vertex AI Pipelines users
KFP v1’s run_as_aiplatform_custom_job
was an experimental feature that allowed converting any component into a Vertex AI CustomJob.
KFP v2 does not include this feature.
Change: Use Google Cloud Pipeline Component’s create_custom_training_job_from_component function.
Previous usage | New usage |
---|---|
|
|
Typecasting behavior change
Affects: KFP OSS users and Vertex AI Pipelines users
KFP SDK v1 had more lenient pipeline typechecking than does KFP SDK v2. Some pipelines that utilized this leniency may not be compilable using KFP SDK v2. For example, parameters typed with float
would accept the string "0.1"
:
from kfp.v2 import compiler
from kfp.v2 import dsl
from kfp import components
@dsl.component
def train(
number_of_epochs: int,
learning_rate: float,
):
print(f"number_of_epochs={number_of_epochs}")
print(f"learning_rate={learning_rate}")
def training_pipeline(number_of_epochs: int = 1):
train(
number_of_epochs=number_of_epochs,
learning_rate="0.1", # string cannot be passed to float parameter using KFP SDK v2
)
Change: We recommend updating your components and pipelines to use types strictly.
Did we miss something?
If you believe we missed a breaking change or an important migration step, please create an issue describing the change in the kubeflow/pipelines repository.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.