It’s a best practice to launch servers hosting applications and storing data only into private subnets. This way they can not be accessed from the public internet directly. Therefore, this provides a first level of security by decreasing the attack surface.
But how can anyone access the applications for using or operating them?
For providing access to the applications special purpose servers or services would be set up in the public subnets. A common service is a load balancer which routes traffic from the internet to web servers hosted in the private subnets. These load balancer are also hardened to further decrease the attack surface.
The Bastion Host
Accessing the servers for operational tasks is done through a so-called bastion host or jump server. The single purpose of this server is to allow access from the outside and allowing to access to servers inside the network. Of course, access to the bastion host must be limited as much as possible. This means that communication with this server should only be possible from certain networks or even hosts. For example, access could only be allowed from a certain part of a company network.
Furthermore, a bastion host should only be launched when it is actually needed. In simple words: If a server is not running it can not be attacked. Since it is a good practice to not rely on manual tasks for doing operations tasks on your machines it should only be necessary to actually log in to servers in rare cases. Various Configuration Management tools - like Chef, Ansible, Puppet - can be used for changing server configuration. Ideally, it should be made as inconvenient as possible for anyone to actually launch a bastion host. Additionally, a bastion host should be automatically stopped after a certain amount of time. This way no-one needs to remember to clean up.
Launch a Bastion Host
To automate the task for launching a bastion host I’ve created a simple CloudFormation template. This template assumes that the VPC to which the bastion host is launched has been set up via the CloudFormation template introduced in the previous blog post about VPC setup best practices.
The template uses the exported values for the public subnets to configure an autoscaling group. This way it is not necessary to explicitly select a subnet into which the bastion host should be launched.
Furthermore, it gets the BastionHostSecurityGroup from this CloudFormation stack. This is the security group which allows incoming SSH connections to servers in the public subnets. Additionally, it allows traffic from the bastion host to other servers in the VPC if they are associated with the SSHSecurityGroup.
Assuming that the VPC stack is called my-vpc and that the keypair which should be used for SSH access is called my-keypair the bastion host can be launched via the AWS CLI using this command:
aws cloudformation create-stack --stack-name my-bastion-host \
--template-url https://s3.eu-central-1.amazonaws.com/com.carpinuslabs.cloudformation.templates/ops/bastion-host.yaml \
--parameters ParameterKey=NetworkStackName,ParameterValue=my-vpc \
ParameterKey=KeyName,ParameterValue=my-keypair
This current version of the template has some room for improvements. For one, the instance is not teared down automatically after a certain amount of time. Next, it would be great if a spot instance could be used for the bastion host instead of a on-demand one. This way the costs for running the bastion host could be reduced significantly. Furthermore, the template refers to a certain version of an Amazon Machine Image (AMI) available in the region eu-central-1. Therefore, it’s currently not possible to launch this template without modifications to other regions.
Use a Bastion Host
As mentioned before, the primary use case of a bastion host is to jump to other servers. To log in to another machine two approaches can be used. In the following examples BastionHost and TargetHost have to be replaced with the IP addresses or domain names of the bastion host or the target host respectively.
The first approach for logging in to a target host is called Agent Forwarding. In this case we log in to bastion host via
ssh -A -i my-keypair.pem ec2-user@BastionHost
From there, we log in to the target server via
ssh TargetHost
Instead of logging in to the bastion host explicitly it is possible to use it as a proxy by using the following command
ssh -i my-keypair.pem -J ec2-user@BastionHost ec2-user@TargetHost
This can also be combined to directly run commands on the target host, like printing out the name of the operating system
ssh -i my-keypair.pem -J ec2-user@BastionHost ec2-user@TargetHost uname -a
Finally, it’s also possible to copy files from the local machine to the target host through the bastion host
scp -i my-keypair.pem \
-o "ProxyCommand ssh ec2-user@BastionHost -W %h:%p" \
file ec2-user@TargetHost:~/
With this it’s pretty easy to work on servers launched to private subnets through a bastion host in a public subnet. Again, special care must be taken to secure the bastion host itself, e.g. by only allowing connections from specific networks.