Secrets

QuickCart needs its database password in exactly one place that it can control, with the app fetching it when it needs it.

A secret is any value that lets someone in: a database password, an API key for a payment provider, a signing key. If someone else gets a copy, they can do what your app can do.

The trouble with secrets is not storing them. It’s that they get copied. Every copy is a place it can leak from, and every copy has to change when the secret changes.

Where secrets usually end up

The two places AWS gives you

AWS has two services that store secrets encrypted and hand them out only to callers whose IAM policy allows it.

QuickCart puts the database password in Secrets Manager, because it should rotate. Settings like the database’s hostname go in plain parameters or ordinary environment variables. A hostname is not a secret. Anyone who has it still can’t log in, and putting it in a secret only means one more thing to keep in step.

Who may read it: one role, one secret

The API servers get an IAM role, attached to each instance through an instance profile. Servers on EC2 covers how that works. The role’s policy allows reading one secret, and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadTheDatabaseSecret",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:quickcart/dev/db-*"
    }
  ]
}

A server with this role can read the database password. It can’t read any other secret, and it can’t even list which secrets exist. If it’s ever compromised, the damage stops at that one password.

Fetching it at run time

When the app starts, it asks Secrets Manager for the value, using its role’s credentials. Here is all it takes in Python:

import json

import boto3

client = boto3.client("secretsmanager", region_name="us-east-1")


def db_credentials():
    response = client.get_secret_value(SecretId="quickcart/dev/db")
    return json.loads(response["SecretString"])


if __name__ == "__main__":
    creds = db_credentials()
    print("Loaded credentials for user", creds["username"])

The code contains the secret’s name, and that’s fine. The name is useless without a role that’s allowed to read it.

Two habits make this work well in a real app:

• Keep it in memory for a while. Don’t fetch it on every request. Every read costs a little, and it’s slower. Fetching once at start, or every few minutes, is plenty.
• When the database rejects the password, fetch it again. That’s what lets rotation happen while the app is running.

Rotation

Rotation means changing a secret on a schedule, say every 30 days, so that a copy that leaked months ago stops working by itself.

Secrets Manager rotates a secret by running a small function. For a database, that function sets a new password in the database and saves it in the secret, as one job. For RDS, you can skip writing any of this: tick Manage master credentials in AWS Secrets Manager when you create the database, and RDS creates the secret and rotates it for you. Managed databases: RDS shows where that option is.

Rotation is only safe if the app follows the second habit above. Straight after a rotation, any server with the old password cached gets rejected once, fetches the new one, and carries on.

When a secret has leaked

The order matters:

1. Change it first. Once a secret has been pushed or pasted anywhere, you can’t know who has it. A new value makes every copy useless.
2. Then move it out of the code and into Secrets Manager, so it doesn’t happen again.
3. Then, if you like, clean it out of git history. That’s tidying up, not the fix: anyone who cloned before you cleaned still has it.

Below is the path QuickCart’s database password travels, from the editor where the code is written to the running app. Choose where the password lives, then follow it. Then press Change the password to see how much work a change is.

What to notice: the app needs the password in memory whatever you choose. The difference is how many other places it gets written down on the way. With Secrets Manager, there are none, which is also why changing it takes one step instead of a redeploy.

Pennies — one secret in Secrets Manager (about $0.40 a month, charged for the time it exists, so a few cents for this lab), a free Parameter Store parameter, and a t3.micro server for under an hour, about $0.01 plus about $0.005 an hour for its public address. New secrets may also fall under a free trial. Prices change; these are for us-east-1.

You’ll store QuickCart’s database credentials in Secrets Manager, give a server a role that can read exactly that one secret, read it from the server, change it, and compare it with Parameter Store. About 30 minutes.

Check the region selector at the top right of the console says N. Virginia (us-east-1) before you start.

1. Store the secret

   AWSIn the AWS console: Secrets Manager→Secrets→Store a new secret

   On the first page, Choose secret type:

   Field	Enter
   Secret type	Other type of secret
   Key/value pairs, row 1	Key username, value quickcart_app
   Key/value pairs, row 2	Choose Add row. Key password, value practice-only-change-me
   Encryption key	aws/secretsmanager — the default

   This lab uses a made-up practice password on purpose. For a real database you’d use a long random one.

   Choose Next. On Configure secret:

   Field	Enter
   Secret name	quickcart/dev/db
   Description	QuickCart dev database login

   Leave everything else as it is. Choose Next. On Configure rotation, leave Automatic rotation off; there’s no real database to rotate yet. Choose Next, check the review page, and choose Store.

   You should see

   quickcart/dev/db in the list of secrets. Open it: the Secret ARN ends in quickcart/dev/db- followed by six random characters. Those six characters are why the policy in the next step ends with -*.

2. Write a policy that can read only this secret

   AWSIn the AWS console: IAM→Policies→Create policy

   Switch the Policy editor from Visual to JSON. Delete what’s there and paste this, with your own 12-digit account ID in place of 111122223333. You can copy your account ID from the account menu at the top right of the console.

   quickcart-read-db-secret.jsonCopy

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "ReadTheDatabaseSecret",
         "Effect": "Allow",
         "Action": "secretsmanager:GetSecretValue",
         "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:quickcart/dev/db-*"
       }
     ]
   }

   Every line is explained in the idea, above. Choose Next.

   Field	Enter
   Policy name	quickcart-read-db-secret

   Choose Create policy.

3. Create the role for the server

   AWSIn the AWS console: IAM→Roles→Create role

   Field	Enter
   Trusted entity type	AWS service
   Service or use case	EC2, then use case EC2

   Choose Next. On Add permissions, search for and tick both of these:

   Policy	Why
   quickcart-read-db-secret	Your policy: read the one secret
   AmazonSSMManagedInstanceCore	AWS’s policy that lets Session Manager connect to the server, so you need no SSH key

   Choose Next.

   Field	Enter
   Role name	quickcart-app-role

   Choose Create role. Because you chose EC2, the console also makes an instance profile with the same name. That’s the thing you attach to a server.

4. Launch a small server with the role

   AWSIn the AWS console: EC2→Instances→Launch instances

   Field	Enter
   Name	quickcart-secrets-lab
   Application and OS Images	Amazon Linux 2023 AMI — the default
   Instance type	t3.micro
   Key pair (login)	Proceed without a key pair
   Network settings → VPC	The default VPC
   Network settings → Auto-assign public IP	Enable
   Network settings → Firewall	Create security group, and untick Allow SSH traffic from. The server needs no inbound rules at all
   Advanced details → IAM instance profile	quickcart-app-role
   Advanced details → Metadata version	V2 only (token required)

   Choose Launch instance.

   The public address is only there so the server can reach AWS’s services from the default VPC. Nothing can connect in, because the security group allows nothing in.

   You should see

   The instance in the list, with Instance state: Running after a minute or so.

5. Read the secret from the server

   Wait two or three minutes after launch, so the Session Manager agent can register. Then:

   AWSIn the AWS console: EC2→Instances→quickcart-secrets-lab→Connect→Session Manager→Connect

   A terminal opens in your browser. Run:

   TerminalCopy

   aws secretsmanager get-secret-value --secret-id quickcart/dev/db --query SecretString --output text --region us-east-1

   Part	What it means
   aws secretsmanager get-secret-value	ask Secrets Manager for a secret’s value. The AWS CLI comes installed on Amazon Linux
   --secret-id quickcart/dev/db	which secret, by its name
   --query SecretString	from the reply, keep only the secret itself, not the extra details like its ARN and dates
   --output text	print it as plain text, not wrapped in more JSON
   --region us-east-1	the region the secret is in

   You should see

   {"username":"quickcart_app","password":"practice-only-change-me"}

   There are no access keys on this server. The CLI got short-lived credentials for quickcart-app-role from the instance profile, and they renew by themselves.

6. Prove the role can do nothing else

   Still on the server, ask for the list of every secret in the account:

   TerminalCopy

   aws secretsmanager list-secrets --region us-east-1

   Part	What it means
   aws secretsmanager list-secrets	ask for every secret in the account, by name
   --region us-east-1	in this region

   It's refused

   An error occurred (AccessDeniedException) when calling the ListSecrets operation: User: arn:aws:sts::111122223333:assumed-role/quickcart-app-role/i-0abc123example is not authorized to perform: secretsmanager:ListSecrets because no identity-based policy allows the secretsmanager:ListSecrets action

   That refusal is what you want. The server can read its one password, and can’t even find out what other secrets exist.

7. Change the password in one place

   Back in the console:

   AWSIn the AWS console: Secrets Manager→Secrets→quickcart/dev/db→Retrieve secret value→Edit

   Field	Enter
   password	practice-only-second-value

   Choose Save. Now, in the Session Manager terminal, run the same command as before:

   TerminalCopy

   aws secretsmanager get-secret-value --secret-id quickcart/dev/db --query SecretString --output text --region us-east-1

   Part	What it means
   aws secretsmanager get-secret-value	read the secret again
   --secret-id quickcart/dev/db	the same secret
   --query SecretString	keep only the value
   --output text	as plain text
   --region us-east-1	in this region

   You should see

   {"username":"quickcart_app","password":"practice-only-second-value"}

   You changed nothing on the server. The next read simply got the new value. That’s what rotation relies on.

   Secrets Manager kept the old value too, labelled AWSPREVIOUS. Read it:

   TerminalCopy

   aws secretsmanager get-secret-value --secret-id quickcart/dev/db --version-stage AWSPREVIOUS --query SecretString --output text --region us-east-1

   Part	What it means
   aws secretsmanager get-secret-value	read the secret
   --secret-id quickcart/dev/db	the same secret
   --version-stage AWSPREVIOUS	the value before the latest change, instead of the current one
   --query SecretString	keep only the value
   --output text	as plain text
   --region us-east-1	in this region

   You should see

   {"username":"quickcart_app","password":"practice-only-change-me"}

   If a new password ever turns out to be wrong, the previous one is still there to go back to.

8. Compare with Parameter Store

   AWSIn the AWS console: Systems Manager→Parameter Store→Create parameter

   Field	Enter
   Name	/quickcart/dev/db-password
   Tier	Standard
   Type	SecureString
   KMS key source	My current account
   KMS Key ID	alias/aws/ssm — the default
   Value	practice-only-change-me

   Choose Create parameter. Then, on the server:

   TerminalCopy

   aws ssm get-parameter --name /quickcart/dev/db-password --with-decryption --query Parameter.Value --output text --region us-east-1

   Part	What it means
   aws ssm get-parameter	read one parameter from Parameter Store
   --name /quickcart/dev/db-password	which one
   --with-decryption	decrypt the SecureString. Without it you get the encrypted text
   --query Parameter.Value	keep only the value
   --output text	as plain text
   --region us-east-1	in this region

   You should see

   practice-only-change-me

   Look at what just happened. You never gave this role permission to read that parameter. It worked because AmazonSSMManagedInstanceCore, the policy that lets Session Manager connect, also allows reading every parameter in the account. So on any server that uses Session Manager, every SecureString parameter is readable.

9. Clean-up step: Clean up

   The server

   AWSIn the AWS console: EC2→Instances→quickcart-secrets-lab→Instance state→Terminate (delete) instance

   When it shows Terminated, delete the security group the wizard made: EC2 → Security Groups, tick the one named like launch-wizard-1, then Actions → Delete security groups.

   The secret

   AWSIn the AWS console: Secrets Manager→Secrets→quickcart/dev/db→Actions→Delete secret

   Secrets Manager won’t delete a secret straight away. It makes you choose a waiting period of 7 to 30 days, in case you deleted the wrong one. Enter 7 and choose Schedule deletion.

   You should see

   The secret’s status shows it is scheduled for deletion, with a date. Until then you can cancel.

   The parameter

   AWSIn the AWS console: Systems Manager→Parameter Store→/quickcart/dev/db-password→Delete

   Confirm with Delete parameters.

   The role and the policy

   AWSIn the AWS console: IAM→Roles→quickcart-app-role→Delete

   Type the role’s name to confirm. Then IAM → Policies, tick quickcart-read-db-secret, choose Actions → Delete, and type its name to confirm.

   You should see

   No instance running, no parameter, no role, and one secret waiting to be deleted. Nothing else is left.