AWS SageMaker is Amazon’s fully managed cloud machine learning (ML) service. It enables data scientists and developers to quickly build, train, and deploy ML models directly into a production environment.
AWS SageMaker Model Building Pipelines represent a machine learning pipeline for a specific ML model. They consist of interconnected steps defined by a JSON pipeline definition. Here are the components involved in this process:
The image below illustrates a pipeline DAG example:
Related content: Read our guide to AWS SageMaker
In this article:
An Amazon SageMaker Model Building Pipelines instance consists of:
The SageMaker Pipelines service resolves relationships between steps in a data dependency DAG to create a series of steps that complete execution.
Here is an example of a pipeline structure:
By default, a pipeline runs all stages that can run in parallel. This behavior can be controlled using the ParallelismConfiguration property when creating or updating a pipeline and when starting or retrying pipeline execution.
Parallelism configuration is applied on a per-execution basis. For example, if you start two runs, you can run up to 50 steps for each run, for a total of 100 concurrent steps. Also, parallel configurations specified at start, retry, or update runtime take precedence over parallel configurations defined in the pipeline.
A SageMaker pipeline consists of steps that define the relationships between steps and the actions the pipeline can take. Here are the steps SageMaker pipelines support:
This step creates a processing job for data processing.
Here are the requirements for a processing step:
This step creates a training job to train your ML model.
Here are the requirements for a training step:
This step creates a hyperparameter optimization (HPO), also known as a hyperparameter tuning job. Each tuning job can run several training jobs, with each training job producing a version of your model.
This step creates a SageMaker model.
Here are the requirements for a SageMaker step:
This step registers a sagemaker.pipeline.PipelineModel or sagemaker.model.Model with the SageMaker model registry. Your PipelineModel represents an inference pipeline, a model that consists of a linear sequence of containers responsible for processing inference requests.
This step uses batch transformation to run inference on the entire dataset.
Here are the requirements for a transform step:
This step evaluates the condition of step properties to determine which action to take next in the pipeline.
Here are the requirements for a condition step:
This step incorporates additional AWS services and processes into the workflow. It enables you to add processes not directly provided by the pipeline.
This step runs an existing AWS Lambda function or a new one created by SageMaker.
This step conducts baseline drift checks against previous baselines for model explainability and bias analysis.
This step conducts drift checks and baseline suggestions against a previous baseline to evaluate your pipeline’s model quality or data quality.
Amazon EMR Step
This step processes EMR steps to a running Amazon EMR cluster.
This step stops a SageMaker Pipelines execution when a desired state or condition is not achieved. It also marks this pipeline’s execution as failed. You can use this step to specify a custom error message that indicates the cause of the failure.
Let’s see how to create a simple ML pipeline in SageMaker.
SageMaker Pipelines comes with SageMaker Python SDK integration, so you can use a Python-based interface to build each step in your pipeline. To orchestrate your workflow using an Amazon SageMaker model building pipeline, you must create a directed acyclic graph (DAG) in the format of a JSON pipeline definition.
You can create a SageMaker DAG definition in JSON format using the SageMaker Python SDK, and send it to SageMaker to start running.
Here is the general process:
In this tutorial we’ll show the first two steps. For the remaining steps, consult the documentation.
execution = pipeline.start()
2. Select Pipelines from the dropdown list.
3. Drag the right border of the SageMaker Resources pane to the right to reveal all columns. Use the search to narrow the list of pipelines.
4. Click a pipeline to view details about it.
5. From the Executions or Graph tab in the execution list, choose Start an execution.
6. Enter or update the following required information:
7. Click Submit. You can now perform the following actions via buttons on the status banner:
When using AWS SageMaker, your organization might run a large number of machine learning experiments requiring massive amounts of computing resources. Run:ai automates resource management and orchestration for machine learning infrastructure. With Run:ai, you can automatically run as many compute intensive experiments and inference workloads as needed.
Here are some of the capabilities you gain when using Run:ai:
Run:ai simplifies machine learning infrastructure pipelines, helping data scientists accelerate their productivity and the quality of their models.
Learn more about the Run:ai GPU virtualization platform.