↓Skip to main content

Version deployments with S3 and CloudFront

·6 mins
Blog Aws Devops Continuous Delivery Blue/Green S3 Cloudfront
Table of Contents

Background #

In my current job my team supports a single page application (SPA) written in Angular and AngularJS. As we move to the “cloud” (mostly AWS) we wanted to take advantage of as many serverless capabilities as possible along with improving performance for our end users. Our end users are located all over the country, mostly in rural areas and they often complain about performance with our application on low bandwidth connections. To meet all of our goals and requirements we settled on using S3 and CloudFront to host the front end application.

Our second requirement falls into a team DevSecOps norm. When hosted in our data center, we had the capability to roll back a deployment if there was a problem (basically switching back to our blue servers). As we moved to S3 and CloudFront, we quickly realized we were giving up this rollback capability because when a distribution is created or invalidated, the CloudFront distribution can take some time to roll out to all edge locations. Another challenge we came across was “busting the cache” in CloudFront when a new file was published. This posed a challenge for us when we want to “stage” the new version of the application prior to release.

While doing research on how to solve these challenges, we looked at several blog posts regarding A/B testing. Most helpful was Lorenzo Nicora’s post. Lorenzo goes into excellent detail on how Lambda@Edge and CloudFront work together so I won’t repeat the explanation here. AWS also a really good example in the AWS developer guide.

Our current implementation uses S3, CloudFront, and Lambda@Edge like the A/B testing examples above, but with several twists. Let’s start by talking about S3 and the versioning. To deploy our files to the S3 bucket, we use TeamCity as our CI/CD orchestration platform. With TeamCity, we log into AWS via the CLI and use the aws s3 sync command to upload the files to the bucket. Each pipeline run in TeamCity generates a “build number” and so we prefix the object key name with that build number. For example, when looking at the bucket which hosts the application you might see something like this:

/1/index.html
/1/11-es2015.c27dad7b7c68a541fb84.js
/1/logo.jpg
/2/index.html
/2/12-es2015.1b5e6ea955f52b699b58.js
/2/logo.jpg

Those of you who are familiar with SPA applications will understand the js files, for those who aren’t Angular creates a unique file name so the js file isn’t cached by the browser.

You may be asking yourself, where does Lambda@Edge come in? First, if you haven’t already read through Lorenzo’s post I referenced above, now is a good time to dive in. I can wait. :)

Refering to the Lambda@Edge diagram, let’s look at what hooks are available to us.

Lambda@Edge

As you can see, there are 4 events:

  1. viewer-request
    • This function is executed on every request
  2. origin-request
    • This function is executed if there is a cache-miss in CloudFront
  3. origin-response
    • This function is executed if there is a cache-miss once the object is returned from the origin.
  4. viewer-response
    • This function is exectuted on every request.

For our solution, we hook into 3 of the 4 events just like Lorenzo.

Our Implementation #

Viewer Response #

Shown below is our viewer-request function.

You can see we log the event triggered by CloudFront, then we parse the request in that raised event so we can see what is in the querystring header. If there is an override_build_number in the querystring, we add that value to a Build-Number cookie and set the Override cookie to true. If the end user doesn’t supply an override_build_number querystring value, we use the Override cookie to see if this is a subsequent/dependent request. If the Override cookie is false, or not supplied we retrieve the “Active Build Number” from an the AWS SSM Parameter Store. Since there is good integration between Lambda and SSM, this is a natural location to maintain our active or production build number.

By using the querystring in this way we can validate a new deployment without “releasing” the application to our end user. It also gives us the flexibility to test versions side by side in our non production environment. This is useful if two developers want to show the customer functionality, but those features have not yet been merged into the main branch.

Retrieving the Build Number #

In the getActiveBuildNumber function we are determining our environment from our host header. We use the convention of dev.myapp.corp.tld and parse out the dev or test from the url becuase Lambda@Edge doesn’t support the use of environment variables like normal Lambda functions do. Based on the host header, we generate our ssm key and pass that to our function which retrieves the “Active Build Number” from SSM.

The getBuildNumberFromSsm function is fairly straight forward, we just use the AWS SDK to call into the parameter store based on our ssmKey.

CloudFront and OriginRequest #

Once we have our build number from SSM, our cookie, or our querystring we return our request and CloudFront does its caching magic. If CloudFront finds the object key in its cache (cache-hit), it returns that object to the user. If it is not found after looking for the object key in the CloudFront cache (cache-miss), the origin-request function is triggered.

In this function you can see we read the request from the raised event and modifies the request.origin.S3.path property with the build number value in the cookie. The request is then sent on to the origin for the actual object.

OriginResponse #

When S3 returns the object to CloudFront the third function, originResponse is invoked. This sets the Build-Number and Override cookies when then object is returned to the caller. Since this may have been the first request and the cookies aren’t yet present so we need to make sure future requests include these cookies.

Full Function #

Here is the full app.js file we use with all of the functions in one place.

CloudFront Setup #

Lastly, let’s talk a bit about the CloudFront distribution and the requirements for the cache keys and the cookies. Due to some reasons beyond the topic of this post, we create our CloudFront distribution separately from our Lambda@Edge functions. The key is to set up the proper cache policy and origin request policy. In our case we are whitelisting our specific cookies and not caching either the Headers or the Querystrings.

For our origin request policy we are passing all of our cookies and Querystrings but not our headers.

Summary #

As you can see, this implementation provides some significant functionality which is not natively available in CloudFront.

Lastly, I want to give a big thank you to Jesse Tomchak from thisDot and Josh Keel. Jesse consulted with us and was able to get a prototype running for us. Josh was instrumental in getting the final working version in order to use in a production environment.

↑