Building and deploying simple serverless functions and applications on AWS is pretty straight-forward. In general, it does not really matter how the code is structured or which conventions are used. This changes when a real applications should be build, especially when it is developed by a team. There seems to be no general convention how the codebase should be set up for more complex applications build with the Serverless Application Model (SAM).
A couple of months ago we were tasked with building a serverless application for one of our clients. This application provides the backend with a REST API for a set of mobile applications. In general, building such an application sounds like an easy task. You just take a couple of off-the-shelf AWS services and hook them up. This is described in a vast amount of blog posts all over the internet.
Therefore, building a simple service end-to-end was a pretty simple task. Once this was done the question arose how to scale what we’ve learned to be able to write many more services according to our requirements and board new team members who had no prior experience with Lambda, SAM and the other AWS services we wanted to use.
We had to come up with a convention for a homogeneous structure of the codebase for each microservice in order to make it easier for developers to navigate it. With this convention we wanted to address the following things:
In this blog post I focus on provided answers to the questions how big a single Lambda function is, how the code base can be structured and how sharing and reusing code between Lambda functions within a project is achieved.
Disclaimer: I’ve only worked with SAM in the past because it suited my needs. Of course there are other tools and frameworks available like Terraform or Serverless with a very similar feature set. Furthermore, until now I only used Node.js as a platform for building my services. Therefore, the remainder of this post assumes using SAM as well as Node.js.
Function Size
The first question is how to map a microservice to a Lambda function. Is there one big Lambda function or multiple smaller ones? We pretty soon came to the conclusion that it makes sense to have multiple smaller ones. There are various reasons for this.
First, it allows setting the IAM permissions for each function very granularly. This is in line with the least privilege principle.
Next, it provides a way to monitor each function separately. If there’s only one big function which handles requests for multiple endpoints it is only possible to get average metrics, e.g. duration, error rates, etc. It’s obvious that this provides some challenges when doing performance analysis and optimization or error analysis. The same would also apply to costs but probably is only a secondary concern.
Code Reuse
First, we tried to keep all the code for one Lambda function of a microservice separate from each other. This way a developer could easily grasp what belonged to a function - the code as well as its dependencies. We were also able to have the functions as small as possible by not installing dependencies not needed for the function. It also reduced the blast radius for an errorenous change made by a developer. It could only affect this single function.
However, it’s pretty obvious that multiple Lambda functions within one microservice share some code. One common example is loading some data from a database and encapsulating it into an object. We tried to put this shared code into an external library. Since we work with Node.js we put everything into a Node module. This module was in the same code repository like all the functions.
Unfortunately, we hit another obstacle when it came to tooling. The tools provided by AWS for packaging and deploying the SAM application are not able to deal with this kind of setup. We and others have addressed this by opening issues but a fix is not in sight.
Therefore, we settled with a different approach. We only have one code base and one set of dependencies. For each function the same package is deployed. It contains all the code and dependencies for all the functions. The only difference is that each Lambda function has a different entry point to the code by using a different handler. This way we can easily achieve the goal for reusing code across our functions.
Project Structure
Basically, the project structure for a Node.js based Serverless application looks like this:
src/
index.js
service.js
package.json
package-lock.json
src/tests/unit/
tests.js
src/tests/integration/
tests.js
sam-template.yaml
The index.js contains all the handlers for the different Lambda functions comprising a service. For a standard CRUD application this it looks similar to this:
exports.create = async (event) => {
// ...
};
exports.get = async (event) => {
// ...
};
exports.update = async (event) => {
// ...
};
exports.delete = async (event) => {
// ...
};
exports.list = async (event) => {
// ...
};
The implementation of the handlers contains all the logic which is necessary to deal with the protocol that is used to invoke the handler. In this case it would be HTTP requests. In other cases handlers would be invoked by SNS, SQS or various other AWS services. The events provided in each of the invocations is different.
The actual business logic is encapsulated in service.js and other files. The actual structure depends on the complexity of the business logic necessary for the service to work.
Having this structure in place, it’s pretty easy for developers to navigate the project and the code base. This is especially useful for new developers joining the team. After learning the basic project structure they can begin adding new features to services or fixing bugs almost immediately. They can easily grasp what goes into a service. Therefore, they can quickly understand it.
Although the example outlined above is based on Node.js it can be adapted for other platforms like Ruby, Java, etc.
Build & Test
It’s really easy to build and test the application based on this project structure. All you need to do is to run the following commands:
npm install
npm run lint
npm test
npm prune --production
It installs all the dependency of the application, checks the code style, runs all the unit tests and finally removes the dependencies which are not needed for running in production.
Summary
The proposed solution has proven to be an efficient and effective way for setting up projects based on AWS Lambda and Node.js. It provides lots of benefits regarding developer productivity:
- easy to understand the project structure
- easy to understand the code
- possibility to share and reuse code
- detailed monitoring possible
- support for least privilege principle for each individual function
The principles outlined in this blog post are shown in a sample project which can be found on GitHub. These principles can also be applied to other Lambda runtimes, like Ruby and Java.