It’s not uncommon to work with external systems from within a Lambda function: databases, web services, legacy systems, etc. Oftentimes it is necessary to authenticate against these systems via API keys or actual usernames and passwords. Now the questions is how to securely handle these secrets.
Of course, secrets should not be put anywhere in the code or configuration in plain text. Even in encrypted form this could still be dangerous. Furthermore, a common requirements is to have the Lambda functions deployed to multiple environments or stages, like development and production. For each stage the settings and credentials will in general be different.
In AWS there are at least three different ways to handle secrets and other settings for Lambda functions securely:
- Encrypted environment variable
- Parameter Store provided by Systems Manager
- Secrets Manager
In the following, each solution is presented and its benefits and drawbacks are discussed.
Encrypted Environment Variables
Environment variables have been added to Lambda functions very early in their product life cycle. These can be stored in an encrypted way. The encryption key is managed by the Key Management Service (KMS).
The secret is encrypted with a key managed by KMS. The resulting encrypted data is then provided to the Lambda function as an environment variable. The Lambda function needs to decrypt this encoded data in order to get the cleartext secret so it can actually be used. Access to KMS key can be controlled via IAM so that not everyone who has access to the encrypted data is able to actually decrypt it.
To encrypt the secret via KMS the following command can be used:
aws kms encrypt --key-id 8bbc4586-c784-421e-b50a-b911349d3097 \
--plaintext "Top Secret" \
--query CiphertextBlob \
--output text
The key-id is the ID of the key managed by KMS that should be used to encode the secret. For performing this operation a Customer Managed Key has to be used. The easiest way is to create it via the AWS Console. For such a key it can be configured who can administrate the key and who can use it. In both cases IAM roles and users can be referenced.
The resulting encoded data is base64 encoded and can be provided during as an environment variable to the Lambda function. The Lambda function needs to decrypt the data in order to get the plaintext to actually use it:
const aws = require('aws-sdk');
const EncodedSecret = process.env.SECRET;
let DecodedSecret;
const kms = new aws.KMS();
exports.handler = async (event) => {
const decodedSecret = await decodeSecret();
console.log(decodedSecret);
};
async function decodeSecret() {
if (DecodedSecret) {
return DecodedSecret;
}
const params = {
CiphertextBlob: Buffer.from(EncodedSecret, 'base64')
};
const data = await kms.decrypt(params).promise();
DecodedSecret = data.Plaintext.toString('utf-8');
return DecodedSecret;
}
In order to decode the encoded data the IAM role which is assigned to the Lambda function needs to be configured to use the KMS.
The only costs for using this solution is for managing the key by KMS. Each Customer Managed Key costs $1 per month. Additionally, each request to the KMS API is billed with $0,03 per 10.000 requests. The first 20.000 request/month are covered by the free tier.
Parameter Store provided by Systems Manager
The Parameter Store is part of the Systems Manager. It is possible to store almost any data in it, including encrypted secrets. Here again, KMS is used for managing the encryption keys.
The data can be stored in a hierarchical way. Via IAM it can be controlled in a very fine grained way who is able to access and change the data.
Additionally, for each record in the Parameter Store a history is stored. In this way it is possible to exactly know when a change to a setting or secret has been made.
Since the Parameter Store is an external service making changes to it can be made completely independent on the deployment process of the functions. However, it needs to be ensured that all the settings are available in each stage. Additionally, error handling must be put in place to handle failures of the Parameter Store or unavailability of the settings in the target environment.
Since changes to records in Parameter Store are monitored by CloudWatch Events it is easy to react to these events to put mechanisms in place for governance and compliance.
To store secrets in Parameter Store the following AWS CLI command can be used:
aws ssm put-parameter --name /my/secret \
--value 'Hello, world!' \
--type SecureString
In the Lambda function the encrypted parameter can be accessed like shown in the following code:
const aws = require('aws-sdk');
const ssm = new aws.SSM();
exports.handler = async (event) => {
const decodedSecret = await getSecret();
console.log(decodedSecret);
};
async function getSecret() {
const params = {
Name: '/my/secret',
WithDecryption: true
};
const result = await ssm.getParameter(params).promise();
return result.Parameter.Value
}
Using this solution does not cause any additional costs. If you use a Customer Managed Key the costs are the same as the one outlined for the first solution.
Secrets Manager
The Secrets Manager has been introduced at re:invent 2017. It’s another attempt to handle secrets in AWS.
A secret within Secrets Manager can be created via the AWS Console as well as this CLI command:
aws secretsmanager create-secret --name Username \
--secret-string "MyUsername"
Of course, access to the secret can controlled via IAM. To access the secret from the Lambda function the following code can be used:
const aws = require('aws-sdk');
const secretmanager = new aws.SecretsManager();
exports.handler = async (event) => {
const decodedSecret = await getSecretString();
console.log(decodedSecret);
};
async function getSecretString() {
const params = {
SecretId: 'Username'
};
const result = await secretmanager.getSecretValue(params).promise();
return result.SecretString;
}
Secrets Manager provides some additional features like automatic secrets rotation. This is especially helpful when used for databases managed by the Relational Database Service (RDS).
Managing secrets is billed with $0,40 per secret per month. Additionally, requests to the API are charged with $0,05 per 10.000 requests.
Summary
AWS offers different ways for managing secrets in a secure way for usage by Lambda. The different solutions differ in complexity for setup, usage and costs. The approach which matches the requirements best can be used. There’s no excuse to store secrets in cleartext somewhere in the source code or configuration. Furthermore, this also applies to other environment specific settings, like endpoint URLs, etc.