How To Set Up Mfa For Aws

Implementing MFA for AWS

YubiKeys are one type of authentication device

1 disquisitional requirement of our efforts to enforce security best practices at Klaviyo is implementing Multi-Factor Authentication (MFA) across the organization (GitHub, Chiliad Suite, AWS, etc.) besides equally including this every bit a feature of the Klaviyo production itself. This post focuses on how we used Terraform, Python, and Fustigate to enforce MFA beyond multiple AWS accounts, and in the procedure, centralized our IAM user management and access controls.

Since Klaviyo Technology is a growing team with a strong culture of infrastructure ownership, many engineers and other technical staff manage our secure platform. This means that nosotros must balance the requirements of a secure and resilient infrastructure with enabling the broader engineering science team to self-sufficiently manage and operate their services inside AWS. Additionally, internal users needed to be able to admission and employ AWS credentials in tandem with other tooling, such as Terraform, kubectl, and boto3. When we began this project, our teammates had user accounts in multiple AWS accounts. Each user was assigned some unique combination of IAM direction policies, roles and groups. Many of these IAM resources had grown organically over the years, never been formally audited, and were entirely undocumented.

We began with an audit of AWS usage. How were developers using AWS? What services did they need access to and via what medium (e.g. the CLI vs. the panel)? After surveying and communicating with each individual team, nosotros were able to map their needs against current user permissions (IAM policies, roles, and groups).

We also audited how many users across our AWS environments already had enabled MFA devices. Being security-conscious individuals, several engineers already had MFA enabled. Nonetheless, we created a minor script, run equally a cronjob in Kubernetes, to periodically poll each AWS account and check MFA adoption over time:

Script to audit MFA usage on AWS

Our version of this script also tied into our metrics platform (StatsD + Graphite + Grafana) and then we could check adoption every bit we rolled out MFA enforcement (more than on that below).

Setting Upwards MFA Enforcement

Once we had an idea of what permissions our users needed, nosotros researched what permissions were needed to enforce MFA on AWS via IAM. Two resources were useful hither, the AWS policy example for allowing users to manage their own security credentials and the docs on how to utilise MFA in AWS.

Combining these two $.25 of noesis, we created a Terraform module that outlined the following:

An IAM policy that would allow users to manage their MFA devices, merely naught else, until they enabled MFA
Ii IAM groups, one for product developers and i for AWS administrators, each of which provides console access and enables members of the group to assume IAM roles corresponding to their group
Two IAM roles, one for product developers and one for AWS administrators, each of which provides CLI admission

Additionally, several supporting IAM policies were created to do things like deny database deletions and permit groups to presume specific roles if MFA was enabled.

Permit's accept a look at an example to illustrate what I hateful:

First, we created our programmer grouping and role:

          resource "aws_iam_group" "mfa_developer" {
            proper name = "DeveloperMFAGroup-${var.surround}"
            path = "/"
}          resource "aws_iam_role" "mfa_developer" {
            proper noun = "DeveloperMFARole-${var.environment}"
            path = "/"
            assume_role_policy = "${information.aws_iam_policy_document.allow_assume_role_if_mfa.json}"
            max_session_duration = "${var.max_session_duration}"
}

Note that the above role only allows the AWS assumeRole activeness to occur if MFA is enabled:

          data "aws_iam_policy_document" "allow_assume_role_if_mfa" {
            statement {
            sid = "AllowAssumeRoleIfMFAIsPresent"
            actions = [
            "sts:AssumeRole"
            ]
            principals {
            identifiers = ["*"]
            blazon = "AWS"
            }
            effect = "Let"
            status {
            examination = "BoolIfExists"
            variable = "aws:MultiFactorAuthPresent"
            values = [
            "true",
            ]
            }
            }
}

At present, nosotros tin first set upwardly permissions for the developer role (in this case, we gave developers PowerUserAccess):

          resource "aws_iam_role_policy_attachment" "developer_mfa_role_power_user_access" {
            function = "${aws_iam_role.mfa_developer.name}"
            policy_arn = "arn:aws:iam::aws:policy/PowerUserAccess"
}

Considering we're using Terraform, we used booleans to determine whether or not additional policies should exist applied. For instance, we could conditionally allow developers access to IAM (due east.g. in a dev/examination environs) or conditionally deny developers from deleting RDS databases:

          resources "aws_iam_role_policy_attachment" "developer_role_allow_iam_full_access" {
            count = "${var.allow_iam_full_access ? 1 : 0}"
            office = "${aws_iam_role.mfa_developer.name}"
            policy_arn = "arn:aws:iam::aws:policy/IAMFullAccess"
}          resource "aws_iam_role_policy_attachment" "developer_mfa_role_deny_db_delete" {
            count = "${var.deny_db_deletion? 1: 0}"
            function = "${aws_iam_role.mfa_developer.proper name}"
            policy_arn = "${aws_iam_policy.deny_rds_db_delete.arn}"
}

Last, nosotros immune our DeveloperMFAGroup to assumeRole to the DeveloperMFARole, every bit well as lock it downward to require MFA and provide read only access to resource in the console:

          resources "aws_iam_group_policy_attachment" "force_mfa_developer_group" {
            group = "${aws_iam_group.mfa_developer.name}"
            policy_arn = "${aws_iam_policy.force_mfa.arn}"
}          resource "aws_iam_group_policy_attachment" "allow_assume_developer_role_developer_group" {
            grouping = "${aws_iam_group.mfa_developer.proper noun}"
            policy_arn = "${aws_iam_policy.allow_assume_developer_role_if_mfa.arn}"
}          resource "aws_iam_group_policy_attachment" "ec2_access_developer_group" {
            group = "${aws_iam_group.mfa_developer.name}"
            policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess"
}

After applying these changes to our AWS environments, we had the user permissions we wanted to exist able to give to engineers.

Centralizing IAM User Management and Admission Control

We were now managing all of a user's permissions in Terraform. We idea to ourselves, "why not also manage users themselves via Terraform?". To make this change, we had to practice two things:

Import all existing users into Terraform
Create a Terraform module to standardize new user creation

The import was fairly straightforward, nosotros needed to:

Create a new master.tf that will deed as the source of truth for users
Add backend and provider blocks to the file so a statefile can be created

          terraform {
            backend "s3" {
            saucepan = "an-case-bucket"
            key    = "path/to/user-mgmt"
            }
}          provider "aws" {
}

After creating the file, we wrote a small script to:

Get a list of all users in an AWS account
Initialize a new Terraform statefile in S3 (where user state will alive)
Import the user into Terraform

Script to import AWS IAM users into Terraform

Next, we had to standardize new user creation. Nosotros created some other module for this, which handles all of the components required for onboarding a new IAM user. We settled on a modified version of the terraform examples IAM user module, which adds a user, can give console and/or CLI access, and optionally adds SSH keys. Notably, we crave each user to create a PGP key that is used to encrypt the access and secret keys generated past Terraform. We do this to ensure that this sensitive information is not stored in plain text in s3.

We had a great first use-case for this new module. We noticed that AWS does not currently back up CLI MFA using YubiKey, but many engineers at Klaviyo use YubiKeys, and nosotros wanted to go on to enable this option every bit much as possible. So, we worked around this issue past encouraging engineers to use a separate AWS user with TOTP for CLI access, while maintaining their original user for console access via YubiKey.

MFA Enforcement

At present, we were managing all users via Terraform, but had not yet placed our staff into the groups we had created to enforce MFA. The implementation of this was like shooting fish in a barrel, when nosotros wanted to enforce MFA for a particular engineer, we only moved forrad with adding new group permissions to that user:

          resource "aws_iam_user_group_membership" "test_user_group" {
            user = "${aws_iam_user.test_user.proper name}"          groups = [
            "DeveloperMFAGroup",
            ]
}

While nosotros establish this more repetitive than using count, automation fabricated this change fairly piece of cake and ultimately will be less piece of work down the line as we add and remove users (equally opposed to attempting to remove parts of a list, which is what would occur with count).

However, we did not cutting everyone over at once. Nosotros started by dogfooding our new policies on the Site Reliability Engineering (SRE) squad. Then, we convinced a small number of production engineers to pilot using the new groups. After getting feedback from them and tweaking their permissions, we let the rest of the arrangement know we would exist cut them over.

In guild for the cutover to go smoothly, we created documentation for engineers to follow walking them through the procedure of setting up their local machines for MFA enablement. In order to achieve this, each user must:

Create and enable an MFA device
Edit their AWS credentials file (~/.aws/credentials on UNIX systems)
Create or add to their AWS config file (~/.aws/config on UNIX systems)

For more than details, bank check out the AWS CLI Profiles section of this blog postal service: https://blog.jayway.com/2017/11/22/aws-cli-mfa/

The Developer Experience

When rolling out this change, we first enforced MFA for ourselves (within the SRE team). It turns out, having to enter a TOTP token every fourth dimension yous want to utilize the AWS CLI gets kind of annoying! So, we constitute a tool that helps securely manage and store AWS credentials: aws-vault. This tool manages AWS sessions and also helped manage the usage of all of our automation tooling. We too made modest changes to existing automation tooling to utilise aws-vault or appropriately handle sessions using MFA via boto3.

Lastly, we found several helper tools to complement the usage of aws-vault. These include a custom Bash wrapper for Linux Fustigate users since 1Password is not available on Linux as an amanuensis, Bash functions to assistance get credentials from 1Password for OSX Users, and the oh-my-zsh aws-vault plugin for ZSH users.

The Bash wrapper can exist added to your ~/.bashrc:

Bash helpers when using aws-vault to manage AWS credentials

Similarly, 1Password AWS MFA functions grab signin data from 1Password and use information technology to run aws-vault:

          avprod() {
            # If you are not authenticated for 1pass then authenticate first
            if ! op listing items; then
            eval $(op signin ${AWS_ACCOUNT})
            fi
            aws-vault exec --mfa-token="$(op go totp aws-prod)" prod -- zsh
}          avdev() {
            # If you are not authenticated for 1pass then cosign commencement
            if ! op list items; then
            eval $(op signin ${AWS_ACCOUNT)
            fi
            aws-vault exec --mfa-token="$(op get totp aws-dev)" dev -- zsh
}

Lastly, the aws-vault oh-my-zsh plugin is simple and shortens the amount of typing required to use aws-vault.

Lessons Learned

In that location were several of import learnings that came out of this projection.

Start, managing IAM user management and access control via Terraform seriously upped our security game on this front end. At present, we can ensure that all changes have an audit trail (in GitHub) and Terraform beingness checked into source control follows change management best practices.

Second, focusing on developer educational activity smoothed over the transition for developers. Letting users know they could enable MFA far before they were forced to helped people adopt the workflow change at their own step. Due to the MFA rollout being messaged equally part of a larger push toward security across the organization, many engineers immediately saw the benefit.

3rd, a boring rollout was a good approach to making this modify. Dogfooding the experience was a great empathy-builder and drove us to notice helper tooling that had the added benefit of some other layer of security. Soliciting feedback from a pilot group was likewise a great way to ensure the developer feel stayed equally good every bit possible. We defenseless issues early, and it enabled us to get more 1-on-i time with developers to walk them through the process. They could then pay this cognition forwards and help united states of america when developers beingness onboarded later had questions.

Lastly, having a variety of access needs was some other strong programmer experience feature. We support both TOTP and YubiKey-based MFA equally much as possible (with the exception of the AWS CLI, which currently does not support YubiKey-based MFA). This helped developers with a uniform MFA experience (as they are also using MFA for other tools they use at Klaviyo).

That said, if we were to exercise information technology once more, at that place are iii things I would alter:

Having a pilot group for the rollout caught some bug, but it didn't assist with team-specific permissions issues. In the future, I would curl this change out past team.
Although our access controls are significantly more standardized than they used to be, nosotros are now running into problems that would exist solved past more than granularity for developer permissions. In the hereafter, we should go on to refactor our groups and roles, perhaps implementing team-based access patterns.
It really sucks to need to demand to create ii users for folks who want to use YubiKeys — hopefully AWS implements YubiKey for CLI access soon

If this sounds interesting, we're hiring!