Step Functions ¶

AWS Step Functions is a serverless orchestration service that lets you integrate with AWS Lambda functions and other AWS services to build business-critical applications. Through Step Functions' graphical console, you see your application’s workflow as a series of event-driven steps.

Why is step function required?

AWS Step Functions can coordinate the distributed components of your application and keep the need for orchestration out of your code. It automatically launches and tracks each step, and can retry steps when there are errors.

As your applications run, Step Functions maintains the application state, tracking exactly which workflow step your application is in, and stores an event log of data that is passed between application components. That means if the workflow is interrupted for any reason, your application can pick up right where it left off.

Use cases¶

ML: Step Functions lets you to orchestrate end-to-end machine learning workflows on SageMaker. These workflows can include data preprocessing, post-processing, feature engineering, data validation, and model evaluation
Micro-services Orchestration:
IT and Security Automation

Features of Step functions¶

Step Functions is based on state machines and tasks.
In Step Functions, a workflow is called a state machine, which is a series of event-driven steps.
Each step in a workflow is called a state.
A Task state represents a unit of work that another AWS service, such as AWS Lambda, performs. A Task state can call any AWS service or API.
Using Step Functions, you define your workflows as state machines, which transform complex code into easy to understand statements and diagrams.
Step Functions provides ready-made steps for your workflow called states that implement basic service primitives for you, which means you can remove that logic from your application.
Using Step Functions service tasks, you can configure your Step Functions workflow to call other AWS services.
Step Functions can coordinate any application that can make an HTTPS connection, regardless of where it is hosted—Amazon EC2 instances, mobile devices, or on-premises servers.
AWS Step Functions coordinates your existing Lambda functions and microservices and lets you modify them into new compositions. The tasks in your workflow can run anywhere, including on instances, containers, functions, and mobile devices.
Nesting your Step Functions workflows allows you to build larger, more complex workflows out of smaller, simpler workflows.
Step Functions keeps the logic of your application strictly separated from the implementation of your application. You can add, move, swap, and reorder steps without having to make changes to your business logic.
Step Functions maintains the state of your application during execution, including tracking what step of execution it is in, and storing data that is moving between the steps of your workflow. You won’t have to manage state yourself with data stores or by building complex state management into all of your tasks.
Step Functions automatically handles errors and exceptions with built-in try/catch and retry, whether the task takes seconds or months to complete. You can automatically retry failed or timed-out tasks, respond differently to different types of errors, and recover gracefully by falling back to the designated cleanup and recovery code.
Step Functions has built-in fault tolerance and maintains service capacity across multiple Availability Zones in each region, ensuring high availability for both the service itself and for the application workflow it operates.
Step Functions automatically scales the operations and underlying compute to run the steps of your application for you in response to changing workloads.
AWS Step Functions has a 99.9% SLA.
AWS Step Functions enables access to metadata about workflow executions so that you can easily identify related resources. For example, your workflow can retrieve its execution ID or a timestamp indicating when a task started. This makes it easier to correlate logs for faster debugging and to measure workflow performance data.
It also supports callback patterns. Callback patterns automate workflows for applications with human activities and custom integrations with third-party services.
Monitoring: AWS Step Functions supports workflow execution events, which makes it faster and easier to build and monitor event-driven, serverless workflows. Execution event notifications can be automatically delivered when a workflow starts or completes through CloudWatch Events, reaching targets like AWS Lambda, Amazon SNS, Amazon Kinesis, or AWS Step Functions for automated response to the event.

Workflows¶

For long-running workflows you can use Standard Workflows with the AWS Fargate integration to orchestrate applications running in containers. For short-duration, high-volume workflows that require an immediate response, Synchronous Express Workflows are ideal.

These can be used for web-based or mobile applications, which often have workflows of short duration, and require the completion of a series of steps before they return a response

Step Functions has 2 workflow types.

Standard workflows¶

They have exactly-once workflow execution and can run for up to 1 year. This means that each step in a Standard workflow will execute exactly once.
Designed for long-running business processes that require durable state storage
Provides automatic retries, error handling, and human approval steps.
Offers many advanced features, including execution history tracking for up to 90 days and timeouts.
Suitable for complex and fault-tolerant workflows that require coordination across multiple services.

Express workflows¶

AWS Step Functions has Express Workflows to support event rates greater than 100,000 per second, so you can build high volume, short duration workflows that has to be completed within 5 mins max.
Provides no durable state storage, no automatic retries, no error handling, and no human approval steps.
Optimized for low-latency and high-throughput use cases, such as event-driven serverless applications.
Can integrate with AWS Step Functions service integrations, AWS Lambda, and HTTP/HTTPS endpoints.
Provides a simpler and more lightweight workflow execution environment with lower per-state costs.

Concepts¶

ASL (Amazon State Language)¶

In AWS Step Functions, you define your workflows in the Amazon States Language. The Amazon States Language is a JSON-based, structured language used to define your state machine. Using Amazon States Language, you create workflows.

Transitions¶

Transitions link states together, defining the control flow for the state machine. When a state machine is invoked, the system begins with the state referenced in the top-level "StartAt" field. This field is a string value that must match the name of one of the states exactly. It is case sensitive.

States¶

State in Step Function

There are 8 types of states:

Task state – represents a single unit of work done by your state machine. It could be making API calls to AWS services, such as running an Amazon Athena query, writing to a DynamoDB table, or invoking a Lambda function to perform some custom processes.
Choice state – Make a choice between branches of execution
Fail state – Stops execution and marks it as failure
Succeed state – Stops execution and marks it as a success
Pass state – Simply pass its input to its output or inject some fixed data
Wait state – Provide a delay for a certain amount of time or until a specified time/date
Parallel state – Begin parallel branches of execution
Map state – Adds a for-each loop condition

Pass¶

A Pass state ("Type": "Pass") passes its input to its output, without performing work. Pass states are useful when constructing and debugging state machines. You can also use a Pass state to transform JSON state input using filters, and then pass the transformed data to the next state in your workflows.

Pass State Example

Here is an example of a Pass state that injects some fixed data into the state machine, probably for testing purposes.

"No-op": {
"Type": "Pass",
"Result": {
"x-datum": 0.381018,
"y-datum": 622.2269926397355
},
"ResultPath": "$.coords",
"End": true
}

Suppose the input to this state is the following.

{
"georefOf": "Home"
}

Then the output would be this.

{
"georefOf": "Home",
"coords": {
"x-datum": 0.381018,
"y-datum": 622.2269926397355
}
}

Choice¶

A customer requests a credit limit increase. Using a Choice state, you can have Step Functions make decisions based on the Choice state’s input. If the request is more than your customer’s pre-approved credit limit, you can have Step Functions send your customer's request to a manager for sign-off. If the request is less than your customer’s pre-approved credit limit, you can have Step Functions approve the request automatically.

Parallel¶

A customer converts a video file into five different display resolutions, so viewers can watch the video on multiple devices. Using a Parallel state, Step Functions inputs the video file, so Lambda can process it into the five display resolutions at the same time.

A Parallel state provides each branch with a copy of its own input data (subject to modification by the InputPath field). It generates output that is an array with one element for each branch, containing the output from that branch. There is no requirement that all elements be of the same

Map¶

Use the Map state to run a set of workflow steps for each item in a dataset. The Map state's iterations run in parallel, which makes it possible to process a dataset quickly. Map states can use a variety of input types, including a JSON array, a list of Amazon S3 objects, or a CSV file.

Wait¶

A Wait state ("Type": "Wait") delays the state machine from continuing for a specified time. You can choose either a relative time, specified in seconds from when the state begins, or an absolute end time, specified as a timestamp.

Workflow Studio¶

Workflow Studio for AWS Step Functions is a low-code visual workflow designer for Step Functions that lets you create serverless workflows by orchestrating AWS services.

States browser¶

The States browser is where you select states to drag and drop into your workflow graph. The Actions tab provides a list of AWS APIs, and the Flow tab provides a list of flow states. While the Patterns tab provides several ready-to-use, reusable building blocks that you can use for a variety of use cases. You can search all states in the States Browser using the search field at the top.

Quotas¶

AWS Step Functions places quotas on the sizes of certain state machine parameters, such as the number of API actions during a certain time period or the number of state machines that you can define. Although these quotas are designed to prevent a misconfigured state machine from consuming all of the resources of the system, many aren't hard quotas