HashiCorp’s Vault with AWS-EC2 auth Backend

Recently I have been spending allot of time on HashiCorp’s Vault and trying to understand as much as I can about it. Vault provides allot of great features and is packed with stuff that I am still trying to figure out.

Recently I started to play around with the “AWS-EC2” auth backend of vault which enables you to configure vault in such a way that it can authenticate requests coming from EC2 instances based on information such as

  • EC2 identity document (details) with PKCS#7 signature
  • EC2 instance tags
  • EC2 instance IAM Role
  • any one or combination of above

I wanted to test out this feature, but wanted to take it a bit further and test the “AWS-EC2” auth backend with multitple AWS accounts using the same Vault cluster with different “Secret” paths.

In this post, I will try to share my findings and will make sure that by the time you finish reading this, you will be able to configure your AWS EC2 instance to autheticate with Vault using no extra information provided (not in case if you want to use TAGS).

I will try to keep it as close to a real environment as possible.

NOTE: This is a bit lengthy post, so grab a cup of coffee and lets start vaulting 🙂


Prerequisites / Assumptions

  • Two AWS accounts
    • AWS Account ID: 123123123123 (Prod)
    • AWS Account ID: 789789789789 (Vault Cluster)
  • VPC’s peered between accounts
    • We will not be using Vault over internet (I will not recommend that)
  • Vault -version: Vault v0.7.2 (‘d28dd5a018294562dbc9a18c95554d52b5d12390’)
  • $VAULT_ADDR variable will be used when ever I would want to communicate with Vault Server

Creating IAM Role and Instance Profile For Vault Cluster AWS Account

AWS Account ID: 789789789789

Before we proceed any further, we will create AWS IAM roles, policies and instance profile in our AWS account in which we will be setting up our Vault Cluster. The instance profile we create here will be assigned to the EC2 instances for Vault Cluster.

Create a json file named “trust-policy-for-ec2.json” with below content

[mudasir@mbp ~/AWS]$ cat trust-policy-for-ec2.json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {"Service": "ec2.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }
}

Create another json file named “vault-permissions-policy-for-ec2.json” with below content

[mudasir@mbp ~/AWS]$ cat vault-permissions-policy-for-ec2.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole",
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        }
    ]
}

The first file is to set TrustRelationship for the Role that we will create and the second file creates and attaches and InLine policy to the Role

Now lets create the IAM Role and Instance Profile

# Creating IAM Role
[mudasir@mbp ~/AWS]$ aws iam create-role --role-name vault-role-for-EC2 --assume-role-policy-document file://trust-policy-for-ec2.json
# Creating InLine Policy and attaching it to the Role created above
[mudasir@mbp ~/AWS]$ aws iam put-role-policy --role-name vault-role-for-EC2 --policy-name Vault-EC2-Permissions --policy-document file://vault-permissions-policy-for-ec2.json
# Creating the instance profile required by EC2 to contain the role
[mudasir@mbp ~/AWS]$ aws iam create-instance-profile --instance-profile-name Vault-EC2-Instance-Profile
# Finally, add the role to the instance profile
[mudasir@mbp ~/AWS]$ aws iam add-role-to-instance-profile --instance-profile Vault-EC2-Instance-Profile --role-name vault-role-for-EC2

Now when we boot EC2 instances for Vault Cluster, we have to make sure that this instance profile which we created above named “Vault-EC2-Instance-Profile” is attached to the EC2 instance(s)


Creating IAM Role and Instance Profile for Prod AWS Account

AWS Account ID: 123123123123

In this AWS account, we need only need to create a IAM role, which will be used by Vault, which is in a separate AWS account, to authenticate EC2 instances based on the METADATA provided

Create a json file named “aws-prod-trust-policy-for-ec2.json” with below content

[mudasir@mbp ~/AWS]$ cat aws-prod-trust-policy-for-ec2.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::789789789789:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create another json file named “cross-account-vault-perms-policy-for-ec2.json” with below content

[mudasir@mbp ~/AWS]$ cat cross-account-vault-perms-policy-for-ec2.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:GetInstanceProfile",
      "Resource": "*"
    }
  ]
}

Once we have both the files, now we will create the IAM Role

# Creating IAM Role
[mudasir@mbp ~/AWS]$ aws iam create-role --role-name cross-account-vault-role --assume-role-policy-document file://aws-prod-trust-policy-for-ec2.json
# Creating InLine Policy and attaching it to the Role created above
[mudasir@mbp ~/AWS]$ aws iam put-role-policy --role-name cross-account-vault-role --policy-name Cross-Account-Vault-EC2-Permissions --policy-document file://cross-account-vault-perms-policy-for-ec2.json

If we you did not get any errors, then everything from AWS IAM side is now configured and we are good to continue


Setup and Configure Vault Server

NOTE: We will not use “dev” mode to keep it close to real life environment.

Now boot up an EC2 instance with the Instance Profile named “Vault-EC2-Instance-Profile” which we created in previous step

Download the Vault binary from the link provided above and place the binary under “/usr/local/bin”

To setup vault, for this post, I will be using file base storage. Lets create a file “/opt/vault-server.hcl” (as shown below)

[mudasir@mbp ~/Vault]$ cat vault-server.hcl
backend "file" {
path = "/opt/vault-file-system"
}

listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}

disable_mlock = false
[mudasir@mbp ~/Vault]$ screen -S vault-server
[mudasir@mbp ~/Vault]$ vault -server -config=vault-server.hcl -log-level=debug
==> WARNING: mlock not supported on this system!

  An `mlockall(2)`-like syscall to prevent memory from being
  swapped to disk is not supported on this system. Running
  Vault on an mlockall(2) enabled system is much more secure.

==> Vault server configuration:

                     Cgo: disabled
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled")
               Log Level: debug
                   Mlock: supported: false, enabled: false
                 Storage: file
                 Version: Vault v0.7.2
             Version Sha: d28dd5a018294562dbc9a18c95554d52b5d12390

==> Vault server started! Log data will stream in below:

Now Vault server is up and running, but it is not yet initialized and neither unsealed. Lets initialize and unseal our Vault Server

[mudasir@mbp ~/Vault]$ export VAULT_ADDR=http://127.0.0.1:8200
[mudasir@mbp ~/Vault]$ vault init -key-shares=4 -key-threshold=2
Unseal Key 1: shiQ39wgIsGIUn9ir1vGVHfUPSLvnh1FYiwHIn8QTsez
Unseal Key 2: bLxdTFE2yXENrbU7VAmsPiUW+fCT7OHkAGsVs/k3Qr5H
Unseal Key 3: iwzZH836i0uSZdgFEFH9b314XZgOuEz7UppVpgPSzYzp
Unseal Key 4: mJKo9/lLPrTLsaqffx6qODJnKV3oWCC2aNxfnuJ+G7no
Initial Root Token: ddc45100-5b7c-a36c-3f2b-a95242de0781

Vault initialized with 4 keys and a key threshold of 2. Please
securely distribute the above keys. When the vault is re-sealed,
restarted, or stopped, you must provide at least 2 of these keys
to unseal it again.

Vault does not store the master key. Without at least 2 keys,
your vault will remain permanently sealed.

After the initialization is done, it prints out the “Unseal Key(s)” and the “root token”. Now we need to unseal our vault server so that we can start using it.

To Unseal Vault, you can use any of the 2 out of 4 keys that we generated during the initialization process.

[mudasir@mbp ~/Vault]$ vault unseal
Key (will be hidden):
Sealed: true
Key Shares: 4
Key Threshold: 2
Unseal Progress: 1
Unseal Nonce: d9eadb96-6f4c-e140-aaf9-b5fa8dc4ab86
[mudasir@mbp ~/Vault]$ vault unseal
Key (will be hidden):
Sealed: false
Key Shares: 4
Key Threshold: 2
Unseal Progress: 0
Unseal Nonce:

Once we have unsealed our Vault cluster, we will enable / mount our EC2 auth backend for each AWS account separately at different paths.

NOTE: From this place onwards, we will have to make sure that we have VAULT_TOKEN environment variable set. In this post, I will be using ROOT TOKEN.

[mudasir@mbp ~/Vault]$ export VAULT_TOKEN=ddc45100-5b7c-a36c-3f2b-a95242de0781
[mudasir@mbp ~/Vault]$ vault auth-enable -path=aws-prod -description="AWS Prod account with Account ID 123123123123" aws-ec2
Successfully enabled 'aws-ec2' at 'aws-prod'!
[mudasir@mbp ~/Vault]$ vault auth-enable -path=aws-vault-cluster -description="AWS Vault Cluster account with Account ID 789789789789" aws-ec2
Successfully enabled 'aws-ec2' at 'aws-vault-cluster'!

After enabling / mounting our “aws-ec2” auth backend, lets create some policies which our EC2 instances will be using after authenticating. We will create two policies, one for AWS account in which Vault cluster is running and one for our AWS prod account. This way we will make sure that each AWS account will only have access to certain secret paths and not everything.

The first policy we create is for our Vault Cluster AWS account.

[mudasir@mbp ~/Vault]$ cat policy-vault-cluster.hcl
path "secret/vault-cluster/*" {
  policy = "write"
}

path "auth/aws-vault-cluster/login" {
  policy = "write"
}
[mudasir@mbp ~/Vault]$ vault policy-write vault-cluster policy-vault-cluster.hcl
Policy 'vault-cluster' written.
[mudasir@mbp ~/Vault]$

Now we will create policy for our AWS Prod account

[mudasir@mbp ~/Vault]$ cat policy-aws-prod.hcl
path "secret/prod/*" {
  policy = "write"
}

path "auth/aws-prod/login" {
  policy = "write"
}
[mudasir@mbp ~/Vault]$ vault policy-write aws-prod policy-aws-prod.hcl
Policy 'aws-prod' written.

Now we have our AWS EC2 auth backend enabled and mounted and our basic policies configured, it is time to create roles in Vault associated with the policies we created above and bounded to a specific AWS account
First we will create role for our AWS account in which the Vault Cluster is running (Account ID: 789789789789).

[mudasir@mbp ~/Vault]$ vault write auth/aws-vault-cluster/role/vault-cluster-role bound_account_id=789789789789 policies=vault-cluster
Success! Data written to: auth/aws-vault-cluster/role/vault-cluster-role

Now lets configure vault to perform STS action for our AWS Cross Account EC2 authentication and create a role in Vault for our AWS Account ID: 123123123123 (prod)

[mudasir@mbp ~/Vault]$ vault write auth/aws-prod/config/sts/123123123123 sts_role=arn:aws:iam::123123123123:role/cross-account-vault-role
Success! Data written to: auth/aws-prod/config/sts/123123123123
[mudasir@mbp ~/Vault]$ vault write auth/aws-prod/role/aws-prod-role bound_account_id=123123123123 policies=aws-prod 
Success! Data written to: auth/aws-prod/role/aws-prod-role

At this stage, our Vault is configured to authenticate EC2 instances directly without the need for us to prodvide any extra information to EC2 instance.

Lets test our setup and try to authenticate from an EC2 instance in the same AWS account and also from an EC2 instance in another AWS account.

EC2 instance in same AWS Account (ID: 789789789789)

~# export VAULT_ADDR=http://10.10.10.10:8200
~# curl -X POST $VAULT_ADDR/v1/auth/aws-vault-cluster/login -d '{"role": "vault-cluster-role", "pkcs7":"'"$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/pkcs7 | tr -d \\n)"'"}'

{
	"request_id": "7e5e6b73-1eee-7004-20ba-8e4e7ec83b1f",
	"lease_id": "",
	"renewable": false,
	"lease_duration": 0,
	"data": null,
	"wrap_info": null,
	"warnings": null,
	"auth": {
		"client_token": "9a668933-3f7a-3e98-9bd5-2f6d63372e8a",
		"accessor": "d7f421ed-20e9-1561-0114-d360cbdc4c21",
		"policies": ["vault-cluster", "default"],
		"metadata": {
			"account_id": "789789789789",
			"ami_id": "ami-a8d2d7ce",
			"instance_id": "i-0cc881a4d23b5134f",
			"nonce": "2d412e91-598c-57cf-6fa8-d1e5e88c419e",
			"region": "eu-west-1",
			"role": "vault-cluster-role",
			"role_tag_max_ttl": "0s"
		},
		"lease_duration": 2764800,
		"renewable": true
	}
}

If everything is configured and you did not get any errors, you should just be able to do the same from any EC2 instance in AWS Account ID: 123123123123 (prod)

EC2 instance in another AWS Account (ID: 123123123123)

~# export VAULT_ADDR=http://10.10.10.10:8200
~# curl -X POST $VAULT_ADDR/v1/auth/aws-prod/login -d '{"role": "aws-prod", "pkcs7":"'"$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/pkcs7 | tr -d \\n)"'"}'

{
	"request_id": "c8881a39-f811-0e9c-4abe-973c49337ddd",
	"lease_id": "",
	"renewable": false,
	"lease_duration": 0,
	"data": null,
	"wrap_info": null,
	"warnings": null,
	"auth": {
		"client_token": "82efc321-abc1-6ac7-827c-d1159e8449c3",
		"accessor": "f0eb3191-d36b-9508-663c-58ef45221809",
		"policies": ["aws-prod", "default"],
		"metadata": {
			"account_id": "123123123123",
			"ami_id": "ami-a8d2d7ce",
			"instance_id": "i-0f151ebd9a5149db1",
			"nonce": "ec995582-5e69-a6d1-a151-4d3db4ca6eba",
			"region": "eu-west-1",
			"role": "aws-prod-role",
			"role_tag_max_ttl": "0s"
		},
		"lease_duration": 2764800,
		"renewable": true
	}
}

Congratulations, you have a working Vault instance which authenticates EC2 instances based on their METADATA in the same AWS account as well as in different AWS accounts (which are configured).

Please do provide your feedback in the comments section below.



Categories: Uncategorized

Tags: , , ,

1 reply

Trackbacks

  1. 1 – HashiCorp Vault with AWS EC2 auth back end with multiple AWS accounts

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: