Using Amazon SageMaker Components¶

In this tutorial, you run a pipeline using Amazon SageMaker Components for Kubeflow Pipelines to train a classification model using Kmeans with the MNIST dataset. This workflow uses Kubeflow pipelines as the orchestrator and Amazon SageMaker as the backend to run the steps in the workflow. For the full code for this and other pipeline examples, see the Sample AWS SageMaker Kubeflow Pipelines. For information on the components used, see the KubeFlow Pipelines GitHub repository.

Setup¶

To use Kubeflow Pipelines (KFP), you need an Amazon Elastic Kubernetes Service (Amazon EKS) cluster and a gateway node to interact with that cluster. The following sections show the steps needed to set up these resources.

Set up a gateway node¶

A gateway node is used to create an Amazon EKS cluster and access the Kubeflow Pipelines UI. Use your local machine or an Amazon EC2 instance as your gateway node. If you want to use a new EC2 instance, create one with the latest Ubuntu 18.04 DLAMI version from the AWS console using the steps in Launching and Configuring a DLAMI.

Complete the following steps to set up your gateway node. Depending on your environment, you may have certain requirements already configured.

If you don’t have an existing Amazon EKS cluster, create a user named your_credentials using the steps in Creating an IAM User in Your AWS Account. If you have an existing Amazon EKS cluster, use the credentials of the IAM role or user that has access to it.

Add the following permissions to your user using the steps in Changing Permissions for an IAM User:

CloudWatchLogsFullAccess
AWSCloudFormationFullAccess
IAMFullAccess
AmazonS3FullAccess
AmazonEC2FullAccess
AmazonEKSAdminPolicy - Create this policy using the schema from Amazon EKS Identity-Based Policy Examples

Install the following on your gateway node to access the Amazon EKS cluster and KFP UI.

AWS CLI. If you are using an IAM user, configure your Access Key ID, Secret Access Key and preferred AWS Region by running: aws configure
aws-iam-authenticator version 0.1.31 and above.
eksctl version above 0.15.
kubectl. The version needs to match your Kubernetes version within 1 minor version.

Install boto3.

pip install boto3

Set up an Amazon EKS cluster¶

Run the following steps from the command line of your gateway node to set up an Amazon EKS cluster:

If you do not have an existing Amazon EKS cluster, complete the following substeps. If you already have an Amazon EKS cluster, skip this step.

Run the following from your command line to create an Amazon EKS Cluster with version 1.14 or above. Replace <your-cluster-name> with any name for your cluster.

eksctl create cluster --name <your-cluster-name> --region us-east-1 --auto-kubeconfig --timeout=50m --managed --nodes=1

When cluster creation is complete, verify that you have access to the cluster using the following command.

kubectl get nodes

Verify that the current kubectl context is the cluster you want to use with the following command. The current context is marked with an asterisk (*) in the output.

kubectl config get-contexts

CURRENT NAME     CLUSTER
*   <username>@<clustername>.us-east-1.eksctl.io   <clustername>.us-east-1.eksctl.io

If the desired cluster is not configured as your current default, update the default with the following command.

aws eks update-kubeconfig --name <clustername> --region us-east-1

Install Kubeflow Pipelines¶

Run the following steps from the command line of your gateway node to install Kubeflow Pipelines on your cluster.

Install Kubeflow Pipelines on your cluster by following step 1 of Deploying Kubeflow Pipelines documentation. Your KFP version must be 0.5.0 or above.

Verify that the Kubeflow Pipelines service and other related resources are running.

kubectl -n kubeflow get all | grep pipeline

Your output should look like the following.

pod/ml-pipeline-6b88c67994-kdtjv                      1/1     Running            0          2d
pod/ml-pipeline-persistenceagent-64d74dfdbf-66stk     1/1     Running            0          2d
pod/ml-pipeline-scheduledworkflow-65bdf46db7-5x9qj    1/1     Running            0          2d
pod/ml-pipeline-ui-66cc4cffb6-cmsdb                   1/1     Running            0          2d
pod/ml-pipeline-viewer-crd-6db65ccc4-wqlzj            1/1     Running            0          2d
pod/ml-pipeline-visualizationserver-9c47576f4-bqmx4   1/1     Running            0          2d
service/ml-pipeline                       ClusterIP   10.100.170.170   <none>        8888/TCP,8887/TCP   2d
service/ml-pipeline-ui                    ClusterIP   10.100.38.71     <none>        80/TCP              2d
service/ml-pipeline-visualizationserver   ClusterIP   10.100.61.47     <none>        8888/TCP            2d
deployment.apps/ml-pipeline                       1/1     1            1           2d
deployment.apps/ml-pipeline-persistenceagent      1/1     1            1           2d
deployment.apps/ml-pipeline-scheduledworkflow     1/1     1            1           2d
deployment.apps/ml-pipeline-ui                    1/1     1            1           2d
deployment.apps/ml-pipeline-viewer-crd            1/1     1            1           2d
deployment.apps/ml-pipeline-visualizationserver   1/1     1            1           2d
replicaset.apps/ml-pipeline-6b88c67994                      1         1         1       2d
replicaset.apps/ml-pipeline-persistenceagent-64d74dfdbf     1         1         1       2d
replicaset.apps/ml-pipeline-scheduledworkflow-65bdf46db7    1         1         1       2d
replicaset.apps/ml-pipeline-ui-66cc4cffb6                   1         1         1       2d
replicaset.apps/ml-pipeline-viewer-crd-6db65ccc4            1         1         1       2d
replicaset.apps/ml-pipeline-visualizationserver-9c47576f4   1         1         1       2d

Access the KFP UI¶

The Kubeflow Pipelines UI is used for managing and tracking experiments, jobs, and runs on your cluster. You can use port forwarding to access the Kubeflow Pipelines UI from your gateway node.

Set up port forwarding to the KFP UI service¶

Run the following from the command line of your gateway node:

Verify that the KFP UI service is running using the following command:

kubectl -n kubeflow get service ml-pipeline-ui

NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
ml-pipeline-ui   ClusterIP   10.100.38.71   <none>        80/TCP    2d22h

Run the following command to setup port forwarding to the KFP UI service. This forwards the KFP UI to port 8080 on your gateway node and allows you to access the KFP UI from your browser.

Note

The port-forward from your remote machine drops if there is no activity. Run this command again if your dashboard is unable to get logs or updates. If the commands return an error, ensure that there is no process already running on the port you are trying to use.

kubectl port-forward -n kubeflow service/ml-pipeline-ui 8080:80

Your method of accessing the KFP UI depends on your gateway node type.

Local machine as the gateway node

Access the dashboard in your browser as follows:

http://localhost:8080

Click Pipelines to access the pipelines UI.

EC2 instance as the gateway node

You need to setup an SSH tunnel on your EC2 instance to access the Kubeflow dashboard from your local machine’s browser.

From a new terminal session in your local machine, run the following. Replace <public-DNS-of-gateway-node> with the IP address of your instance found on the EC2 console. You can also use the public DNS. Replace <path_to_key> with the path to the pem key used to access the gateway node.

public_DNS_address=<public-DNS-of-gateway-node>
key=<path_to_key>

on Ubuntu:
ssh -i ${key} -L 9000:localhost:8080 ubuntu@${public_DNS_address}

or on Amazon Linux:
ssh -i ${key} -L 9000:localhost:8080 ec2-user@${public_DNS_address}

Access the dashboard in your browser.

http://localhost:9000

Click Pipelines to access the KFP UI.

Create IAM Users/Roles for KFP pods and the Amazon SageMaker service¶

You now have a Kubernetes cluster with Kubeflow set up. To run Amazon SageMaker Components for Kubeflow Pipelines, the Kubeflow Pipeline pods need access to SageMaker. In this section, you create IAM Users/Roles to be used by Kubeflow Pipeline pods and Amazon SageMaker.

Create a KFP execution role¶

Run the following from the command line of your gateway node:

Enable OIDC support on the Amazon EKS cluster with the following command. Replace <cluster_name> with the name of your cluster and <cluster_region> with the region your cluster is in.

eksctl utils associate-iam-oidc-provider --cluster <cluster-name> \
        --region <cluster-region> --approve

Run the following to get the OIDC issuer URL. This URL is in the form https://oidc.eks.<region>.amazonaws.com/id/<OIDC_ID> .

aws eks describe-cluster --region <cluster-region> --name <cluster-name> --query "cluster.identity.oidc.issuer" --output text

Run the following to create a file named trust.json. Replace <OIDC_URL> with your OIDC issuer URL. Don’t include https:// when in your OIDC issuer URL. Replace <AWS_account_number> with your AWS account number.

OIDC_URL="<OIDC-URL>"
AWS_ACC_NUM="<AWS-account-number>"

# Run this to create trust.json file
cat <<EOF > trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACC_NUM}:oidc-provider/${OIDC_URL}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_URL}:aud": "sts.amazonaws.com",
          "${OIDC_URL}:sub": "system:serviceaccount:kubeflow:pipeline-runner"
        }
      }
    }
  ]
}
EOF

Create an IAM role named kfp-example-pod-role using trust.json using the following command. This role is used by KFP pods to create Amazon SageMaker jobs from KFP components. Note the ARN returned in the output.

aws iam create-role --role-name kfp-example-pod-role --assume-role-policy-document file://trust.json
aws iam attach-role-policy --role-name kfp-example-pod-role --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
aws iam get-role --role-name kfp-example-pod-role --output text --query 'Role.Arn'

Edit your pipeline-runner service account with the following command.

kubectl edit -n kubeflow serviceaccount pipeline-runner

In the file, add the following Amazon EKS role annotation and replace <role_arn> with your role ARN.

eks.amazonaws.com/role-arn: <role-arn>

Your file should look like the following when you’ve added the Amazon EKS role annotation. Save the file.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: <role-arn>
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"labels":{"app":"pipeline-runner","app.kubernetes.io/component":"pipelines-runner","app.kubernetes.io/instance":"pipelines-runner-0.2.0","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"pipelines-runner","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"0.2.0"},"name":"pipeline-runner","namespace":"kubeflow"}}
  creationTimestamp: "2020-04-16T05:48:06Z"
  labels:
    app: pipeline-runner
    app.kubernetes.io/component: pipelines-runner
    app.kubernetes.io/instance: pipelines-runner-0.2.0
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: pipelines-runner
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: 0.2.0
  name: pipeline-runner
  namespace: kubeflow
  resourceVersion: "11787"
  selfLink: /api/v1/namespaces/kubeflow/serviceaccounts/pipeline-runner
  uid: d86234bd-7fa5-11ea-a8f2-02934be6dc88
secrets:
- name: pipeline-runner-token-dkjrk

Create an Amazon SageMaker execution role¶

The kfp-example-sagemaker-execution-role IAM role is used by Amazon SageMaker jobs to access AWS resources. For more information, see the IAM Permissions section. You provide this role as an input parameter when running the pipeline.

Run the following to create the role. Note the ARN that is returned in your output.

SAGEMAKER_EXECUTION_ROLE_NAME=kfp-example-sagemaker-execution-role

TRUST="{ \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Principal\": { \"Service\": \"sagemaker.amazonaws.com\" }, \"Action\": \"sts:AssumeRole\" } ] }"
aws iam create-role --role-name ${SAGEMAKER_EXECUTION_ROLE_NAME} --assume-role-policy-document "$TRUST"
aws iam attach-role-policy --role-name ${SAGEMAKER_EXECUTION_ROLE_NAME} --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
aws iam attach-role-policy --role-name ${SAGEMAKER_EXECUTION_ROLE_NAME} --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

aws iam get-role --role-name ${SAGEMAKER_EXECUTION_ROLE_NAME} --output text --query 'Role.Arn'

Add access to additional IAM users or roles¶

If you use an intuitive IDE like Jupyter or want other people in your organization to use the cluster you set up, you can also give them access. The following steps run through this workflow using Amazon SageMaker notebooks. An Amazon SageMaker notebook instance is a fully managed Amazon EC2 compute instance that runs the Jupyter Notebook App. You use the notebook instance to create and manage Jupyter notebooks to create ML workflows. You can define, compile, deploy, and run your pipeline using the KFP Python SDK or CLI. If you’re not using an Amazon SageMaker notebook to run Jupyter, you need to install the AWS CLI and the latest version of kubectl.

Follow the steps in Create an Amazon SageMaker Notebook Instance to create a Amazon SageMaker notebook instance if you do not already have one. Give the IAM role for this instance the S3FullAccess permission.

Amazon EKS clusters use IAM users and roles to control access to the cluster. The rules are implemented in a config map named aws-auth. Only the user/role that has access to the cluster will be able to edit this config map. Run the following from the command line of your gateway node to get the IAM role of the notebook instance you created. Replace <instance-name> with the name of your instance.

aws sagemaker describe-notebook-instance --notebook-instance-name <instance-name> --region <region> --output text --query 'RoleArn'

This command outputs the IAM role ARN in the arn:aws:iam::<account-id>:role/<role-name> format. Take note of this ARN.

Run the following to attach the policies the IAM role. Replace <role-name> with the <role-name> in your ARN.

aws iam attach-role-policy --role-name <role-name> --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
aws iam attach-role-policy --role-name <role-name> --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name <role-name> --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

eksctl provides commands to read and edit the aws-auth config map. system:masters is one of the default user groups. You add the user to this group. The “system:masters” group has super user permissions to the cluster. You can also create a group with more restrictive permissions or you can bind permissions directly to users. Replace <IAM-Role-arn> with the ARN of the IAM role. <your_username> can be any unique username.

eksctl create iamidentitymapping \
    --cluster <cluster-name> \
    --arn <IAM-Role-arn> \
    --group system:masters \
    --username <your-username> \
    --region <region>

Open the Jupyter notebook on your Amazon SageMaker instance and run the following to verify that it has access to the cluster.

aws eks --region <region> update-kubeconfig --name <cluster-name>
kubectl -n kubeflow get all | grep pipeline

Running the Kubeflow Pipeline¶

Now that setup of your gateway node and Amazon EKS cluster is complete, you can create your classification pipeline. To create your pipeline, you need to define and compile it. You then deploy it and use it to run workflows. You can define your pipeline in Python and use the KFP dashboard, KFP CLI, or Python SDK to compile, deploy, and run your workflows. The full code for the MNIST classification pipeline example is available in the Kubeflow Github repository. To use it, clone the example Python files to your gateway node.

Prepare datasets¶

To run the pipelines, you need to upload the data extraction pre-processing script to an S3 bucket. This bucket and all resources for this example must be located in the us-east-1 Amazon Region. If you don’t have a bucket, create one using the steps in Creating a bucket.

From the mnist-kmeans-sagemaker folder of the Kubeflow repository you cloned on your gateway node, run the following command to upload the kmeans_preprocessing.py file to your S3 bucket. Change <bucket-name> to the name of the S3 bucket you created.

aws s3 cp mnist-kmeans-sagemaker/kmeans_preprocessing.py s3://<bucket-name>/mnist_kmeans_example/processing_code/kmeans_preprocessing.py

Create a Kubeflow Pipeline using Amazon SageMaker Components¶

The full code for the MNIST classification pipeline is available in the Kubeflow Github repository. To use it, clone the example Python files to your gateway node.

Input Parameters¶

The full MNIST classification pipeline has run-specific parameters that you must provide values for when creating a run. You must provide these parameters for each component of your pipeline. These parameters can also be updated when using other pipelines. We have provided default values for all parameters in the sample classification pipeline file.

The following are the only parameters you need to pass to run the sample pipelines. To pass these parameters, update their entries when creating a new run.

Role-ARN: This must be the ARN of an IAM role that has full Amazon SageMaker access in your AWS account. Use the ARN of kfp-example-pod-role.
Bucket: This is the name of the S3 bucket that you uploaded the kmeans_preprocessing.py file to.

You can adjust any of the input parameters using the KFP UI and trigger your run again.

Compile and deploy your pipeline¶

After defining the pipeline in Python, you must compile the pipeline to an intermediate representation before you can submit it to the Kubeflow Pipelines service. The intermediate representation is a workflow specification in the form of a YAML file compressed into a tar.gz file. You need the KFP SDK to compile your pipeline.

Install KFP SDK¶

Run the following from the command line of your gateway node:

Install the KFP SDK following the instructions in the Kubeflow pipelines documentation.

Verify that the KFP SDK is installed with the following command:

pip show kfp

Verify that dsl-compile has been installed correctly as follows:

which dsl-compile

Compile your pipeline¶

You have three options to interact with Kubeflow Pipelines: KFP UI, KFP CLI, or the KFP SDK. The following sections illustrate the workflow using the KFP UI and CLI.

Complete the following from your gateway node to compile your pipeline.

Modify your Python file with your S3 bucket name and IAM role ARN.

Use the dsl-compile command from the command line to compile your pipeline as follows. Replace <path-to-python-file> with the path to your pipeline and <path-to-output> with the location where you want your tar.gz file to be.

dsl-compile --py <path-to-python-file> --output <path-to-output>

Upload and run the pipeline using the KFP CLI¶

Complete the following steps from the command line of your gateway node. KFP organizes runs of your pipeline as experiments. You have the option to specify an experiment name. If you do not specify one, the run will be listed under ‘Default’ experiment.

Upload your pipeline as follows:

kfp pipeline upload --pipeline-name <pipeline-name> <path-to-output-tar.gz>

Your output should look like the following. Take note of the ID.

Pipeline 29c3ff21-49f5-4dfe-94f6-618c0e2420fe has been submitted

Pipeline Details
------------------
ID           29c3ff21-49f5-4dfe-94f6-618c0e2420fe
Name         sm-pipeline
Description
Uploaded at  2020-04-30T20:22:39+00:00
...
...

Create a run using the following command. The KFP CLI run command currently does not support specifying input parameters while creating the run. You need to update your parameters in the Python pipeline file before compiling. Replace <experiment-name> and <job-name> with any names. Replace <pipeline-id> with the ID of your submitted pipeline. Replace <your-role-arn> with the ARN of kfp-example-pod-role. Replace <your-bucket-name> with the name of the S3 bucket you created.

kfp run submit --experiment-name <experiment-name> --run-name <job-name> --pipeline-id <pipeline-id> role_arn="<your-role-arn>" bucket_name="<your-bucket-name>"

You can also directly submit a run using the compiled pipeline package created as the output of the dsl-compile command.

kfp run submit --experiment-name <experiment-name> --run-name <job-name> --package-file <path-to-output> role_arn="<your-role-arn>" bucket_name="<your-bucket-name>"

Your output should look like the following:

Creating experiment aws.
Run 95084a2c-f18d-4b77-a9da-eba00bf01e63 is submitted
+--------------------------------------+--------+----------+---------------------------+
| run id                               | name   | status   | created at                |
+======================================+========+==========+===========================+
| 95084a2c-f18d-4b77-a9da-eba00bf01e63 | sm-job |          | 2020-04-30T20:36:41+00:00 |
+--------------------------------------+--------+----------+---------------------------+

Navigate to the UI to check the progress of the job

Upload and run the pipeline using the KFP UI¶

On the left panel, choose the Pipelines tab.
In the upper-right corner, choose +UploadPipeline.
Enter the pipeline name and description.
Choose Upload a file and enter the path to the tar.gz file you created using the CLI or with the Python SDK.
On the left panel, choose the Pipelines tab.
Find the pipeline you created.
Choose +CreateRun.
Enter your input parameters.
Choose Run.

Running predictions¶

Once your classification pipeline is deployed, you can run classification predictions against the endpoint that was created by the Deploy component. Use the KFP UI to check the output artifacts for sagemaker-deploy-model-endpoint_name. Download the .tgz file to extract the endpoint name or check the Amazon SageMaker console in the region you used.

Configure permissions to run predictions¶

If you want to run predictions from your gateway node, skip this section.

If you want to use any other machine to run predictions, assign the sagemaker:InvokeEndpoint permission to the IAM role or IAM user used by the client machine. This permission is used to run predictions.

On your gateway node, run the following to create a policy file:

cat <<EoF > ./sagemaker-invoke.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sagemaker:InvokeEndpoint"
            ],
            "Resource": "*"
        }
    ]
}
EoF

Attach the policy to the client node’s IAM role or IAM user.

If your client machine has an IAM role attached, run the following. Replace <your-instance-IAM-role> with the name of the client node’s IAM role. Replace <path-to-sagemaker-invoke-json> with the path to the policy file you created.

aws iam put-role-policy --role-name <your-instance-IAM-role> --policy-name sagemaker-invoke-for-worker --policy-document file://<path-to-sagemaker-invoke-json>

If your client machine has IAM user credentials configured, run the following. Replace <your_IAM_user_name> with the name of the client node’s IAM user. Replace <path-to-sagemaker-invoke-json> with the path to the policy file you created.

aws iam put-user-policy --user-name <your-IAM-user-name> --policy-name sagemaker-invoke-for-worker --policy-document file://<path-to-sagemaker-invoke-json>

Run predictions¶

Create a Python file from your client machine named mnist-predictions.py with the following content . Replace the ENDPOINT_NAME and REGION variables. This script loads the MNIST dataset, then creates a CSV from those digits and sends it to the endpoint for prediction. It then outputs the results.

import pickle, gzip, numpy, urllib.request, json
from urllib.parse import urlparse
import json
import io
import boto3

ENDPOINT_NAME='<endpoint-name>'
REGION = '<region>'

# Load the dataset
urllib.request.urlretrieve("http://deeplearning.net/data/mnist/mnist.pkl.gz", "mnist.pkl.gz")
with gzip.open('mnist.pkl.gz', 'rb') as f:
    train_set, valid_set, test_set = pickle.load(f, encoding='latin1')

# Simple function to create a csv from our numpy array
def np2csv(arr):
    csv = io.BytesIO()
    numpy.savetxt(csv, arr, delimiter=',', fmt='%g')
    return csv.getvalue().decode().rstrip()

runtime = boto3.Session(region_name=REGION).client('sagemaker-runtime')

payload = np2csv(train_set[0][30:31])

response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                   ContentType='text/csv',
                                   Body=payload)
result = json.loads(response['Body'].read().decode())
print(result)

Run the Python file as follows:

python mnist-predictions.py

View results and logs¶

When the pipeline is running, you can click on any component to check execution details, such as inputs and outputs. This will list the names of created resources.

If the KFP request is successfully processed and an Amazon SageMaker job is created, the component logs in the KFP UI will provide a link to the job created in Amazon SageMaker. The CloudWatch logs will also be provided if the job is successfully created.

If you run too many pipeline jobs on the same cluster, you may see an error message that indicates you do not have enough pods available. To fix this, log in to your gateway node and delete the pods created by the pipelines you are not using as follows:

kubectl get pods -n kubeflow
kubectl delete pods -n kubeflow <name-of-pipeline-pod>

Cleanup¶

When you’re finished with your pipeline, you need to cleanup your resources.

From the KFP dashboard, terminate your pipeline runs if they do not exit properly by clicking Terminate.

If the Terminate option doesn’t work, log in to your gateway node and terminate all the pods created by your pipeline run manually as follows:

kubectl get pods -n kubeflow
kubectl delete pods -n kubeflow <name-of-pipeline-pod>

Using your AWS account, log in to the Amazon SageMaker service. Manually stop all training, batch transform, and HPO jobs. Delete models, data buckets and endpoints to avoid incurring any additional costs. Terminating the pipeline runs does not stop the jobs in Amazon SageMaker.