Menu

Subscribe

Forming Serverless Clouds with AWS: CloudFormation, SAM, CDK, Amplify

Author image Glenn 'devalias' Grant | | 19 minutes to read | #dev, #infrastructure, #automation, #serverless, #cloud, #aws, #amazon, #cloudformation, #sam, #cdk, #amplify, #golang, #react, #graphql, #boilerplate, #generator

Recently I have been playing around with a few little side projects, and trying out different ways of getting them IntoTheCloud(tm). If you know me, you know that I'm pretty big on increasing efficiency, reducing boilerplate/time to start, automation, infrastructure as code (IaC), and similar fun things.

With these explorations I have been looking to see how I can go from 'cool project idea' to having a PoC serverless application running InTheCloud(tm) with as little time, effort, boilerplate, and ongoing cost required; with the hope that if it is quick/easy enough, and the patterns simple enough, I will actually get around to hacking on more of my side projects (or it will be quicker and cheaper to get clients projects up and running).

AWS

For this particular exploration I have been playing around a lot in AWS (Amazon's Cloud), with a particular focus on serverless patterns. As you probably know, AWS is huge, basically runs a good chunk of the internet, and seemingly has a product line for every possible thing you could dream of.

Since I was looking to speed up my 'new project boilerplate', I decided to focus in on the following projects/services:

I'll go into a bit more detail on each of these below, but since I saw so much potential crossover/overlap between them, I opened a few issues on their respective repositories. You might find more interesting tips, tricks, and aspects in those threads too:

AWS CloudFormation

AWS CloudFormation provides a common language for you to describe and provision all the infrastructure resources in your cloud environment. CloudFormation allows you to use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This file serves as the single source of truth for your cloud environment.

Basically, CloudFormation is a bunch of JSON or YAML that defines all of the AWS resources/projects you want to use, how to configure them, and how to tie it all together. Then you can just push it ToTheCloud(tm), some kind of magic happens while you go make coffee, and you're done. It's AWS's basic Infrastructure as Code (IaC) service.

In reality, CloudFormation templates can VERY quickly get massively out of hand, huge, confusing, and pretty hard to cognitively reason about. It's great as an underlying technology layer.. but it isn't really optimised for human consumption (particularly the JSON format). Thankfully some of the other projects I will talk about a little later aim to solve that human interface problem.

Within CloudFormation there are a few high level concepts that it's good to be aware of:

  • Stack: This ties together all of your resources in an AWS Region into a single unit.
  • Nested Stack: A stack created within another stack. Allows you to seperate common patterns into their own templates and tie them all together.
  • StackSet: This ties together multiple Stacks, and allows you to manage them across multiple regions and accounts.

Since Stacks by themselves are single region, you can run into some weird problems depending on the services you want to use. For example, when I want to deploy my application in ap-southeast-2, but want to use AWS CloudFront (which runs in us-east-1) with a HTTPS certificate issued through AWS Certificate Manager, I can't natively do this within a single stack.

There are workarounds such as using custom resources to manage the deployment, or using a StackSet with exported outputs and Fn::ImportValue to deploy the related components across different regions; but sometimes it can take a little digging to figure out the best way to do it.

If you're interested in trying the Custom Resource approach, the following was how one person explained their implementation to me:

It's a bit complicated due to specifics of ACM certificate issuance. The general way it works is:

  • CloudFormation creates a custom resource that has the same "signature" as an ACM certificate. It takes the same parameters and has the same return values (Ref and attribute values).

  • The custom resource invokes a Lambda function in the account. This function requests a new certificate from ACM in us-east-1.

  • The Lambda function then sends a message to an SQS queue in the account. This queue is subscribed by the same Lambda function. The queue is effectively a "while" loop to reinvoke the function every 30 seconds to check whether the certificate has been issued.

  • Every time the Lambda function is invoked by the queued message:

    • If the certificate has been issued, the function responds with a success back to CloudFormation with the appropriate return values. The function returns successfully, which removes the message from the SQS queue.
    • If the certificate issuance failed, the function responds with a failure back to CloudFormation with an appropriate message. The function returns successfully, which removes the message from the SQS queue.
    • If the certificate is still awaiting verification, the function does nothing and throws an error. The error causes SQS to keep the message in the queue and retry 30 seconds later.

  • Meanwhile, the ACM certificate verification occurs (a human approves it via an email sent to the domain owner, or a DNS record is added to the domain to verify the certificate).

While it is pretty convoluted setup for a single project, I expect that if designed well this could be wrapped up into a simple open source/deployable component that everyone could make use of rather easily. Perhaps something for the AWS Serverless Application Repository or as a Launch Stack Button?

AWS Severless Application Model (SAM)

The AWS Serverless Application Model (AWS SAM) is a model to define serverless applications. AWS SAM is natively supported by AWS CloudFormation and defines simplified syntax for expressing serverless resources. The specification currently covers APIs, Lambda functions and Amazon DynamoDB tables.

AWS SAM (GitHub, Spec/Usage, Examples, Site, CLI, Templates) seems to have come about because using CloudFormation directly was just too verbose and time consuming for some of the more common serverless usecases. By wrapping these cases up in a simplified/abstracted way makes it easier to get started, and therefore more likely for people to use the serverless resources AWS provides. It similarly follows the CloudFormation model of defining your resources in YAML, and uses a translator (GitHub) to build the raw underlying CloudFormation template.

While AWS SAM seems great for these common usecases, there are definitely areas where you will need to fall back to using native CloudFormation (which you can thankfully use directly within a SAM template). There are also a number of areas where limitations in what SAM allows you to configure means you may not be able to use it's simplified abstractions. These are likely to improve over time as people run into the issues, and the maintainer team implements/improves features.

What is really nice is just how simple it is to get a new project off the ground:

  • Have a look at Get Started and install/upgrade the CLI: pip install --upgrade aws-sam-cli
  • Init your new application: sam init --runtime nodejs8.10 --name foo-app
    • There are MANY supported runtimes (sam init --help).. so choose your favourite: [python3.6|python2.7|python|nodejs6.10|nodejs8.10|nodejs4.3|nodejs|dotnetcore2.0|dotnetcore1.0|dotnetcore|dotnet|go1.x|go|java8|java]
  • Pull down your app dependencies: cd foo-app/hello_world && npm install
  • Run your API locally (sam local --help): cd ../ && sam local start-api
  • View your application in all of it's glory: http://127.0.0.1:3000/hello

If you have a look at the generated SAM template (template.yaml), you'll see that the entire stack is only ~45 lines (including newlines and comments), with the main function code only taking up ~15 lines. Not bad to get a PoC application running:

HelloWorldFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: hello_world/
    Handler: app.lambdaHandler
    Runtime: nodejs8.10
    Environment:
      Variables:
        PARAM1: VALUE
    Events:
      HelloWorld:
        Type: Api
        Properties:
          Path: /hello
          Method: get

Once we're ready to deploy this to the cloud, we have just a couple more commands to run:

  • Make sure our template is valid: sam validate
  • Package any external code and upload to S3 (bucket must already exist): sam package --template-file ./template.yaml --output-template-file ./packaged.yaml --s3-bucket FOO-PKGS-BUCKET
  • Deploy our stack: sam deploy --template-file ./packaged.yaml --stack-name Foo-App --capabilities CAPABILITY_IAM

Now if you're like me and enjoy writing your backend in Golang, then you may find the default template (sam init --runtime go1.x --name foo-app) a little lacking (eg. no dep, basic Makefile, etc). Thankfully we have the ability to pass a --location flag to tell it to use a different template project.

But how do we know what the template project should look like? Digging into the code we find the generate_project function, which accepts the location parameter. If the parameter is defined it will be used, otherwise it is looked up in the RUNTIME_TEMPLATE_MAPPING, which links the runtime you specified (eg. go1.x) to the template project to use (eg. cookiecutter-aws-sam-hello-golang). These templates are looked up in the _templates variable path, which after some digging I managed to locate in the repo at aws-sam-cli/samcli/local/init/templates/. There also appear to be a few more templates on the aws-samples GitHub.

Having a look at the Golang template project, it appears that these are Cookiecutter (docs) templates. So to make our own customised SAM Golang starter template, after installing Cookiecutter (pip install --upgrade cookiecutter), we can copy the existing template, make our desired changes, and save it somewhere useful for future use (such as GitHub). Then when we want to use it in a new project:

  • sam init --runtime go1.x --location gh:0xdevalias/TODO-cookiecutter-aws-sam-golang --name foo-app

While I haven't abstracted out my patterns into a custom starter template yet, this may be something I end up doing in future, so make sure to keep an eye on my GitHub.

AWS Cloud Development Kit (CDK)

The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation. The CDK integrates fully with AWS services and offers a higher level object-oriented abstraction to define AWS resources imperatively. Using the CDK’s library of infrastructure constructs, you can easily encapsulate AWS best practices in your infrastructure definition and share it without worrying about boilerplate logic. The CDK improves the end-to-end development experience because you get to use the power of modern programming languages to define your AWS infrastructure in a predictable and efficient manner. The CDK is currently available for Java, JavaScript, and TypeScript.

AWS CDK (GitHub, Changelog, Site, Reference, Examples: 1, 2) moves away from directly constructing raw YAML/JSON by hand, and takes more of a 'generator code' approach, providing a development kit of libraries that you can use to describe how your cloud infrastructure should look, connect, and interact. Once it's all defined in code, you can use it to generate the CloudFormation / AWS SAM YAML, deploy it to the cloud, and everything else you would come to expect from these sorts of tools.

The CDK is divided up into a number of libraries, with each representing an AWS service. Each of these libraries is broken up into two different levels of Constructs:

  • CloudFormation Resource: low-level constructs that provide a direct, one-to-one, mapping to an AWS CloudFormation resource, as listed in the AWS CloudFormation Resource Types Reference.
  • AWS Construct Library: handwritten by AWS and come with convenient defaults and additional knowledge about the inner workings of the AWS resources they represent. In general, you will be able to express your intent without worrying about the details too much, and the correct resources will automatically be defined for you.

Where possible you should be able to use the higher level constructs to get things done (and these will only get better over time), but it's nice to know that we have an easy way to drop down to the lower-level functionality when we need to. There also appears to be the ability to create new Construct libs (cdk init --list, template), so it's possible you could build your own custom construct abstractions with this. Another area for future exploration.

As is pretty standard by now, you define a stack which contains all of the features and services you want to use, then configure the environment to define where it should be deployed. You can define multiple stacks within your CDK App, which means we have a nice way to handle cross-region deployments. There is built in support for uploading assets (ref) that your application may require (eg. lambda code, etc), as well as applets for running custom code as part of your build (eg. compiling code/assets).

Getting started with a new project is pretty simple (note: if you don't have default creds configured, make sure to use AWS_PROFILE/--profile or things will hang):

When you're happy and think you're ready to deploy:

Following along from our previous AWS SAM example, we can create an equivalent example SAM function (ref) in ./bin/foo.ts with code such as the following:

import sam = require('@aws-cdk/aws-serverless');
import lambda = require('@aws-cdk/aws-lambda');

const helloWorld = new sam.cloudformation.FunctionResource(this, "HelloWorldFunction", {
  codeUri: "hello_world/",
  handler: "app.lambdaHandler",
  runtime: lambda.Runtime.NodeJS810.name,
  environment: {
    variables: {
      PARAM1: "VALUE"
    }
  },
  events: {
    HelloWorld: {
      type: "Api",
      properties: {
        path: "/hello",
        method: "get",
      }
    }
  }
});

Remember you will need to npm install any additional packages you need before you can use them:

npm i @aws-cdk/aws-serverless @aws-cdk/aws-lambda

Once we compile (npm run build) and synthesize (cdk synth), we can see we end up with equivalent YAML to our previous SAM example:

HelloWorldFunction:
  Type: 'AWS::Serverless::Function'
  Properties:
    CodeUri: hello_world/
    Handler: app.lambdaHandler
    Runtime: nodejs8.10
    Environment:
      Variables:
        PARAM1: VALUE
    Events:
      HelloWorld:
        Properties:
          Method: get
          Path: /hello
        Type: Api

While CDK is quite a new project (Aug 2018), we can already see that it is quite powerful to work with.

Amplify

Amplify is an open source project which is focused on mobile and web developers building applications. This consists of a library, UI components, and a CLI toolchain. The design follows a category based model allowing developers to perform advanced use cases with declarative client APIs so that they can focus on their application code (e.g. Auth.signIn() or API.graphql()). This allows developers to focus on their business use cases and less time on re-implementing the most common use cases around mobile or web app development (Auth flows, Storage and API interaction, Analytics, etc.) (Source)

AWS Amplify (GitHub) combines a number of different complementary aspects to simplify modern mobile and web development:

Of all of the projects I have explored today, this is the one I have the least experience with, so I may not have fully come to understand/appreciate the depth of it yet. In a bit of a difference from the previous projects, this seems to take more of a 'full-stack' approach to solving common application needs.

One of the nice things about the Amplify CLI is how it aims to provide simple menu-driven options for getting everything going:

  • Install the CLI: npm install -g @aws-amplify/cli
  • Init a new project: amplify init and follow the menu choices
⇒  amplify init
Note: It is recommended to run this command from the root of your app directory
? Choose your default editor: None
? Choose the type of app that you're building: javascript
Please tell us about your project
? What javascript framework are you using: react
? Source Directory Path:  src
? Distribution Directory Path: build
? Build Command:  npm run-script build
? Start Command: npm run-script start

Using default provider awscloudformation

Initializing project in the cloud...
..snip..
Your project has been successfully initialized and connected to the cloud!
  • Choose a category (feature) you want to add (amplify --help), and select it: eg. amplify function add
⇒  amplify function add
Using service: Lambda, provided by: awscloudformation
? Provide a friendly name for your resource to be used as a label for this category in the project: HelloWorld
? Provide the AWS Lambda function name: HelloWorld
? Choose the function template that you want to use: Serverless express function (Integration with Amazon API Gateway)
? Do you want to edit the local lambda function now? false
Successfully added resource HelloWorld locally.

At this point you should be able to see the generated files in ./amplify/backend/function/HelloWorld. Of particular note is the generated CloudFormation JSON (HelloWorld-cloudformation-template.json). While it is nice that it is automatically generated, using the JSON form, and not appearing to leverage SAM means that it ends up being quite a verbose file to cognitively reason about. I believe the intention is that you don't modify this directly (and I read somewhere that even if you do it may be overwritten?). If nothing else, it serves as a decent reference implementation for this kind of feature, that you could then translate back to your preferred method (eg. SAM/CDK).

Digging into the source, it appears these templates are located within the specific subpackage of the CLI, in the cloudformation provider (eg. the function template used above).

While currently there only appears to be a single 'provider' (amplify-provider-awscloudformation), language around the websites/repos implies that in future they would like to support additional providers, so it may be possible to implement CDK and/or SAM into this flow, for a 'best of all worlds' situation.

Implementing the most basic use case (function) as we did above isn't really where Amplify shines. For example, you can add an authentication system (JS Ref) to your backend with just amplify auth add, or a new GraphQL/REST api with amplify api add, and similar simplicity for other common features and patterns.

Moving from the backend infrastructure, Amplify also features libraries and UI components to consume these features in your application. For example, getting up and running with React (1, 2) can be as simple as:

create-react-app my-app
cd my-app
npm install --save aws-amplify
npm install --save aws-amplify-react

amplify init

And then a few little code changes to wire things into place.

As part of all of this, you get access to the UI Components, which should dramatically reduce the amount of boilerplate wiring up required to make use of these common application patterns.

I feel like I haven't even begun to dive deep enough into the frontend JS/UI component libraries to do them justice, so I will leave that as an excerise to the reader (or a future blog post).

As mentioned in previous sections, this is also quite a new project (Amplify (Nov 2017), CLI (Aug 2018)), so I'm sure things are going to get much better as time goes on.

Conclusion

We explored a number of different AWS serverless friendly projects and options, and how they may be able to be leveraged together synergistically, or to do similar things as each other. This is still an area I am actively exploring, and a lot of the projects are still quite young, so I'm excited to see what improvements and new efficient patterns come out of this! Maybe I will write a more specific follow up blog at some point detailing how I actually end up using some of these technologies in practice.

Where Next?

You could learn more about serverless and build a web app, put together a modern frontend with Create React App + Redux + Redux-Saga, design a serverless Golang backend with AWS SDK for Golang + Gorilla Mux + AWS Lambda Go Api Proxy, read more about Authentication with AWS Cognito, learn about GraphQL.. so many interesting things out there to learn about and play with!

What are you planning to build? Have any tips or suggestions? A story of how this helped (or hindered) you on a project? I'd love to hear about it in the comments below!