SageMaker Pipelines Tutorial: End-to-End MLOps on AWS

Q: What is the @step decorator in SageMaker?

The @step decorator is a low-code way to convert an ordinary Python function into a pipeline step. You write and test the function locally, add @step with configuration like instance type, and it becomes a pipeline step without rewriting it into a ProcessingStep or TrainingStep object.

Q: Should I use step classes or the @step decorator?

Use the @step decorator for anything new — it converts existing functions quickly with far less boilerplate. Reach for classic step classes like ProcessingStep and TrainingStep when you need precise control over a specific SageMaker job type's configuration that the decorator doesn't expose cleanly.

Q: How do you stop a SageMaker pipeline when a metric fails?

Combine a ConditionStep checking your evaluation metric with a FailStep whose error message reports the threshold. When the condition fails, the pipeline halts with a clear error instead of silently continuing to deployment.

Contents

SageMaker Pipelines tutorial building end-to-end MLOps workflows on AWS

Figure 1: From notebook cells to a reproducible, auditable pipeline running on managed AWS infrastructure

You've got working ML code in a notebook. Turning that into a reproducible, auditable, production pipeline used to mean rewriting everything around rigid step classes. AWS changed that with the @step decorator, and now you can genuinely just decorate your existing functions and get a real pipeline. Let's walk through both the classic approach and the newer, faster one.

I'll cover the traditional step-class API first, since you'll encounter it in most existing codebases, then the @step decorator, which is the better starting point for anything new. By the end you'll know which paradigm to reach for, where each one bites, and how to wire quality gates so a bad model never silently reaches deployment.

If you've followed the MLOps for beginners guide in this series, this is the AWS-native piece that turns those lifecycle concepts into something you actually press "run" on.

What SageMaker Pipelines Actually Is

Amazon SageMaker Model Building Pipelines is described by AWS as the first purpose-built, easy-to-use continuous integration and continuous delivery service specifically for machine learning. With SageMaker Pipelines, you create, automate, and manage end-to-end ML workflows at scale, entirely within AWS's managed infrastructure.

The core building block is the same as in most ML orchestration tools: a DAG (directed acyclic graph) of steps, where each step's output can feed into another step's input, and the pipeline tracks execution, lineage, and artifacts automatically.

That last part is the real value proposition. If you've been running the Airflow tutorial's DAGs on your own infrastructure, you already know the mental model — SageMaker Pipelines is the same graph concept, except AWS runs the jobs, stores the artifacts, and records the lineage without you managing a scheduler.

Two Ways to Build: Step Classes vs the @step Decorator

This is the most important architectural decision you'll make, so let's be clear about the trade-off upfront.

Approach	Best For	Trade-off
Step classes (`ProcessingStep`, `TrainingStep`, etc.)	Fine-grained control over SageMaker job configuration	More boilerplate, steeper learning curve
`@step` decorator	Converting existing Python functions into pipeline steps quickly	Some limitations on what you can pass between steps

The @step decorator is a genuinely newer, low-code way to convert your local machine learning code into one or more pipeline steps. You write your ML function the way you would for any ordinary project, test it locally, then add @step to convert it into a pipeline step — no rewriting into a ProcessingStep or TrainingStep object required.

IMO, start with @step for anything new. Reach for the classic step classes when you need precise control over a specific SageMaker job type's configuration that the decorator doesn't expose cleanly.

Building With the @step Decorator

Here's the shape of a real pipeline using this approach:

from sagemaker.workflow.function_step import step
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.parameters import ParameterString

@step(instance_type="ml.m5.large")
def preprocess(raw_data):
    import pandas as pd
    df = pd.read_csv(raw_data)
    # cleaning logic here
    return df

@step(name="train")
def train_model(df):
    # training logic here
    return {"accuracy": 0.94}

param = ParameterString(name="raw_data_path", default_value="s3://my-bucket/data.csv")

pipeline = Pipeline(
    name="fraud-detection-pipeline",
    steps=[train_model(preprocess(param))],
    parameters=[param],
)

Notice something convenient here: you don't need to list every step explicitly. Since dependencies are defined through function calls (train_model(preprocess(param))), the pipeline object automatically retrieves all upstream steps from the final one you pass in. You only need the end step in your steps list.

That nested-call style is the whole dependency system — the graph is implied by your Python, not declared separately. It's also why the functions have to stay serializable: the wiring happens at definition time, but the data only moves between steps when the pipeline actually executes.

Configuring Steps Without Repeating Yourself

Per-step settings like instance type can live in a shared config file rather than being repeated on every decorator:

SchemaVersion: '1.0'
SageMaker:
  Pipeline:
    RoleArn: 'arn:aws:iam::555555555555:role/IMRole'
    Tags:
      - Key: 'tag_key'
        Value: 'tag_value'

Any configuration in that file applies to every step in the pipeline by default. Need to override just one step? Pass new values directly in that step's @step decorator arguments, like the instance_type="ml.m5.large" example above, and that override wins for that specific step only.

Real Limitations to Know Upfront

The @step decorator trades some flexibility for its convenience, and a few restrictions genuinely matter:

You can't directly access a pipeline variable inside a @step function. Printing a ParameterInteger directly inside a decorated function raises a SerializationError
You can't nest a pipeline variable inside another object (like a tuple) and pass it to a @step function; that also raises a SerializationError
JsonGet and Join objects aren't supported as arguments to @step-decorated functions, though DelayedReturn, Properties, Parameter, and ExecutionVariable objects are fine
You can't iterate over or unpack a DelayedReturn object (a step's return value) as a tuple or list, since the underlying length isn't known until the function actually runs. You can still index into it with [], like delayed_return[0] or delayed_return["a_key"]

These aren't arbitrary restrictions — they come from the fact that inputs and outputs get serialized to pass between steps, so anything requiring compile-time knowledge of a runtime value runs into trouble. Keep your step functions working with plain, serializable data, and you'll avoid most of these entirely.

Building With Classic Step Classes

For cases needing tighter control over a specific SageMaker job type, here's the traditional pattern, using a processing step as the example:

from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput
from sagemaker.workflow.steps import ProcessingStep

sklearn_processor = SKLearnProcessor(
    framework_version='1.0-1',
    role=role,
    instance_type='ml.m5.xlarge',
    instance_count=1
)

inputs = [ProcessingInput(source=input_data, destination="/opt/ml/processing/input")]
outputs = [
    ProcessingOutput(output_name="train", source="/opt/ml/processing/train"),
    ProcessingOutput(output_name="validation", source="/opt/ml/processing/validation"),
    ProcessingOutput(output_name="test", source="/opt/ml/processing/test"),
]

step_process = ProcessingStep(
    name="AbaloneProcess",
    step_args=sklearn_processor.run(inputs=inputs, outputs=outputs, code="preprocessing.py")
)

The pattern extends to TrainingStep, TransformStep, TuningStep, and others, each wrapping the corresponding SageMaker job type. To create data dependencies between steps, pass the properties or outputs of one step as the input to another.

The properties attribute of a step matches exactly what a Describe API call would return for that underlying SageMaker job type — so if you already know the Describe response shape for a training job, you already know what's available on a TrainingStep's properties. That's a genuinely nice property when you're reading existing AWS automation code.

This mirrors the scikit-learn approach from the feature engineering pipeline article — composable steps with explicit data flow — just executed on managed AWS infrastructure instead of in your notebook.

Adding Conditional Failure With FailStep

Real pipelines need to stop and report failure clearly when a quality gate fails. Here's a genuinely common pattern, stopping a pipeline when model error exceeds a threshold:

from sagemaker.workflow.fail_step import FailStep
from sagemaker.workflow.functions import Join
from sagemaker.workflow.parameters import ParameterInteger

mse_threshold_param = ParameterInteger(name="MseThreshold", default_value=5)

step_fail = FailStep(
    name="AbaloneMSEFail",
    error_message=Join(
        on=" ", values=["Execution failed due to MSE >", mse_threshold_param]
    ),
)

Wire this into a ConditionStep checking your evaluation metric, and you've got an automated quality gate that halts deployment rather than silently shipping a degraded model.

That's the same gate the CI/CD for machine learning article argued was the single most valuable piece of an ML delivery pipeline — here expressed as native pipeline steps instead of a script in your CI runner. The concept transfers; only the plumbing changes.

Parameterizing Your Pipeline

Both approaches lean on the same parameter types for values that aren't known until runtime:

from sagemaker.workflow.parameters import ParameterInteger, ParameterString

instance_count = ParameterInteger(name="InstanceCount", default_value=2)
process_s3_input_url = ParameterString(name="ProcessingInputUrl")

Know the limits here too. Parameterization doesn't have 100% compatibility with every corner of the SageMaker Python SDK. A commonly cited example: you can parameterize most Processor.run() arguments, but you cannot parameterize git_config when calling it. If you hit a mysterious error trying to parameterize something, check whether that specific argument is one of the SDK's known exceptions before assuming you made a mistake.

Networking Gotchas Worth Knowing

If you're running pipeline steps inside a VPC, budget extra testing time. Users have reported DHCP-related errors specifically when running SageMaker Pipelines with VPC network configuration, even when SageMaker Studio and other AWS services function correctly in that same VPC.

Networking issues in managed pipeline services are often the hardest bugs to reproduce locally, so test your VPC configuration early with a minimal pipeline before building your full workflow on top of it. A two-step pipeline that runs cleanly in your VPC is worth an hour of testing before you depend on a twelve-step one.

A Practical Decision Framework

Write and test your ML functions locally first, exactly as you would for any project
Decide your approach: @step decorator for speed, classic step classes for fine control over specific job types
Wire dependencies through function calls (@step) or properties/outputs references (classic steps)
Parameterize anything that should vary between runs, checking for SDK compatibility exceptions
Add quality gates with ConditionStep and FailStep before deployment steps
Test in a VPC early if your organization requires it, since networking issues surface unpredictably

The guiding principle: pick one paradigm per pipeline and stay in it. Mixing @step functions with hand-built step classes in the same graph works, but the readability cost is real and the benefit is usually small.

Common Mistakes People Make

Printing or branching on a pipeline variable inside a @step function

Recall the limitations section — this causes a SerializationError; keep parameter access outside the decorated function body and pass plain values in instead.

Unpacking a DelayedReturn as a tuple

Not supported — index into it instead. The length of a step's return value genuinely isn't known until the function runs, so tuple unpacking can't be resolved during pipeline definition.

Assuming every SDK argument is parameterizable

FYI: git_config and a handful of others aren't. Check the docs before you build a workflow around an assumption, because the error you'll get back won't always explain the real cause.

Mixing step-class boilerplate into a @step-based pipeline unnecessarily

Pick one paradigm per pipeline for readability, rather than mixing patterns without a clear reason.

Skipping the quality gate because the first model looks fine

A pipeline without a FailStep gate will happily deploy whatever it last trained. The gate takes an afternoon to add and is the difference between catching a regression in the pipeline and catching it in production.

Recommended Books

Designing Machine Learning Systems by Chip Huyen — the definitive guide to exactly what this article is about: getting ML models from notebook to production. Covers data pipelines, monitoring, and the organizational side most tutorials skip.
Machine Learning Engineering by Andriy Burkov — a rigorous, practical reference for the MLOps lifecycle from a practitioner's perspective, useful as a checklist against what your pipeline actually automates.
Introducing MLOps by Martin Kreuzberger, Neil Kühl, and Sebastian Hirschl — walks through the end-to-end MLOps toolchain and process, which maps directly onto the step-by-step pipeline structure in this tutorial.

Want to Go Deeper?

If you want structured practice on MLOps and cloud ML workflows, Educative's ML courses include hands-on labs that pair well with this kind of pipeline-building work. The unlimited plan is useful when you're working through several AWS-focused modules in one stretch.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is SageMaker Pipelines?

Amazon SageMaker Model Building Pipelines is AWS's purpose-built CI/CD service for machine learning. You define a DAG of steps — preprocessing, training, evaluation, registration — where each step's output feeds another's input, and the service tracks execution, lineage, and artifacts automatically on managed infrastructure.

What is the @step decorator in SageMaker?

The @step decorator is a low-code way to convert an ordinary Python function into a pipeline step. You write and test the function locally, add @step with configuration like instance type, and it becomes a pipeline step without rewriting it into a ProcessingStep or TrainingStep object.

Should I use step classes or the @step decorator?

Use the @step decorator for anything new — it converts existing functions quickly with far less boilerplate. Reach for classic step classes like ProcessingStep and TrainingStep when you need precise control over a specific SageMaker job type's configuration that the decorator doesn't expose cleanly.

Why does printing a pipeline parameter inside a @step function fail?

Inputs and outputs are serialized to pass between steps, so anything needing compile-time knowledge of a runtime value raises a SerializationError. Printing a ParameterInteger directly, or nesting a pipeline variable inside a tuple, both hit this. Keep step functions working with plain, serializable data.

How do you stop a SageMaker pipeline when a metric fails?

Combine a ConditionStep checking your evaluation metric with a FailStep whose error message reports the threshold. When the condition fails, the pipeline halts with a clear error instead of silently continuing to deployment.

Can every SageMaker SDK argument be parameterized?

No. Parameterization doesn't have 100% compatibility with every corner of the SageMaker Python SDK — git_config passed to Processor.run() is a commonly cited exception. Check the SDK docs for known exceptions before assuming you made a mistake.

Wrapping This Up

SageMaker Pipelines gives you a managed, AWS-native path from notebook code to a reproducible, auditable ML workflow. The @step decorator is the faster on-ramp for converting existing functions, while classic step classes like ProcessingStep and TrainingStep give you precise control when you need them.

Will you need every advanced feature — FailStep gates, VPC networking, cross-step property references — on day one? Probably not. But start with @step on a two-function pipeline this week, get it running end to end, and layer in the more advanced pieces as your actual production requirements demand them, not before.

Once that's running, the natural next pieces are exactly the ones the rest of this series covers: version your inputs with DVC, register and promote the artifacts the pipeline produces through a model registry, and watch the deployed model for drift with production monitoring.