AWS Step Functions and Node

AWS announced a new feature back in 2016 called “Step Functions”. AWS Step Functions are described as “visual workflows for modern applications” and allows the creation of custom workflows integrating several pieces of infrastructure to work together to solve interesting problems. I became intrigued by this idea only recently, and found some perfect use-cases for its functionality. On putting together my first workflow, I was pleasantly surprised with how easy they were to set up and work with.

Why Use a Step Function?

Step functions are a great way to pull several different processes together into a unified workflow. You can easily share data across steps and take what would normally be a huge process and break it down into easily consumable parts. This helps you keep each step small and targeted, with a very specific purpose. For instance, I may have a workflow where I need to:

• Grab a list of data
• Process this data in some way to generate new data, or new artifacts
• Save this data in a new way

I could, of course, have a batch process that does all this at the same time. Alternatively, I could break this into three separate “steps”, supported by individual processes that then interact with one another per a set of pre-defined rules — via AWS Step Functions.

Another common problem I saw is that I have a ton of data I need to process, normally processed via an SQS queue, and I want to be able to do something AFTER all my data is processed. This is hard to orchestrate with an SQS queue, but very easy with a Step Function workflow.

Concepts in Step Functions

When you first go to create a Step Function, it’s a little confusing, because there’s no way to create a “Step Function”. Click the left menu and you’ll find that what you’re actually creating and maintaining is a “State Machine”, or a workflow that dictates how different pieces of the AWS ecosystem work together.

A state machine will move the process from “step” to “step”, invoking your infrastructure as it goes. These steps can be anything from:

• Invoking AWS infrastructure such as a lambda function, SNS publish, ECS task, etc
• Setting up “Choice” logic, to check data and then go to different steps based on the result
• Run parallel branches that the workflow will wait on until a step is complete.
• Loop through other pieces of data
• Many more…

Setting up a Step Function “State Machine”

AWS lets you visually design a workflow or define it directly using their markup. Visually designing a workflow still generates a file defining the workflow, but is a lot easier to process. At first the markup generated is going to see pretty abstract, but it’s very straightforward and you can read up about it using the Amazon States Language documentation.

If you wanted to start by invoking a lambda, have a choice, then run another lambda, you may end up with something like this:

Name the steps something that will make sense. In the above workflow, you can get a general sense of what all happens here, even without knowing the specifics involved.

Fetch All My Data: Lambda that queries some data store to get records to be processed

Process My Data: Lambda to process some portion of the data retrieved in the first step

Do I have more data?: Checks to see if more data needs to be processed

Save My Data: Lambda that saves the data that’s been processed above

Each “Lambda: Invoke” step does exactly that, it invokes the lambda mentioned (specified in the step details by ARN) and passes the current state to it. This state is accessible via the “event” data (more on that later) and runs as a normal lambda invocation.

The “Choice state” checks a specific data field by the criteria you define, and then lets you branch logic. In this case, I said to check whether event.hasMoreRecords == true . If it is, then it’ll go back to the lambda invocation defined, as a loop. If it isn’t, then it’ll go on to the next step.

Processing Data within a Lambda

Let’s look at what a basic implementation of what Fetch All My Data may look like:

The important thing to note here is that we attach data to the event passed into our lambda handler, and then invoke the callback with that event so that it can be processed by future steps. This is how a State Machine will share state between different steps, by allowing them to mutate the event data passed around. Since this is the first step in the process, you’d want hasMoreRecords to be explicitly true.

What about the next step, Process My Data?

There are a couple of important points here:

1. We grab the data off the event that was passed previously
2. We change the length of the dataToProcess array so that we can know if we need to process any additional records
3. We evaluate whether there are additional records at the end

This effectively creates a loop that will invoke the lambda as many times as needed to process all the data. In this case, we’re going to assume that “processing” the data is labor or time intensive, so you don’t want to do it all at once.

I like having an explicit variable to flag whether there are additional records to process, as opposed to evaluating dataToProcess in the “Choice State”. No strong reason, just preference.

The final step Save My Data isn’t too terribly important here, because I think the examples above outline the important aspects. The point is that you can dynamically pass data around, and have discreet processes within each step.

Invoking a Step Function from a Lambda

Somewhere, you’ll need to kick this off. You can actually invoke a Step Function in several ways, all outlined by AWS. Say you have an API call in an existing lambda that you would like to use to start this process, it’s actually pretty easy:

A few important things to note here:

1. The only required parameter to start the execution of a state machine is the ARN.
2. To pass data to this execution, you have to pass a string, so make sure to stringify the object passed first
3. When you do pass data, it will be directly accessible on the event object.

To follow up on the final point above, if you invoke a step function with data that’s stringified representing this object:

{ dataId: 1 }

Then in your handler, you’ll be able to access the data like this:

async function handler(event, context, callback) {
  console.log(event.dataId) // logs `1`
}

Conclusion

This was a high-level overview of Lambda Step Functions, what they bring to the table, and some nuances around working with them. This is a really powerful addition to the AWS suite that I look forward to exploring in the future.

Have some other really good use-cases for Step Functions? Share your idea in a comment!