Answers: <p>When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.</p>
  
  <p>The two most common methods I've seen for doing this are:</p>
  
  <ol>
  <li><p>If your provisioning tool (<a href="https://www.terraform.io/docs/providers/aws/r/kms_key.html" rel="nofollow noreferrer">terraform</a>, <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kms-key.html" rel="nofollow noreferrer">cloudformation</a>, <a href="https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.boto_kms.html" rel="nofollow noreferrer">salt cloud</a>) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.</p></li>
  <li><p>You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use <a href="https://www.vaultproject.io/" rel="nofollow noreferrer">Vault</a>. It's not <em>that</em> hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.</p></li>
  </ol>
  
Answers: CC BY-SA 4.0
Answers: 2019-02-22T14:22:03.560Z
Answers: 2019-02-22T14:22:03.560Z
Answers: Briansbum
Body: <p>Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.</p>
  
  <p>My understanding is that I have a few ways to pass those secret data to Spot instances:</p>
  
  <ul>
  <li>Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)</li>
  <li>Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)</li>
  <li>Use Secrets Manager, but that could quickly cost a lot...</li>
  <li>Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)</li>
  </ul>
  
  <p>And I am not sure I could individually identify spot instances?</p>
  
  <p>Am I missing something there? How do you solve those challenges?</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-02-22T06:26:25.463Z
Comments: A variant of instance Ebs is to share an EFS file system
Comments: jdog
Comments: CC BY-SA 4.0
Comments: 2019-02-22T08:03:21.900Z
Comments: *"lot of engineering there"* Are you sure? https://docs.aws.amazon.com/kms/latest/developerguide/services-parameter-store.html
Comments: Michael - sqlbot
Comments: CC BY-SA 4.0
Comments: 2019-02-22T11:51:33.243Z
Comments: @Michael-sqlbot Thanks for your feedback, I rephrased.
Comments: Philippe
Comments: CC BY-SA 4.0
Comments: 2019-02-22T11:52:24.733Z
Comments: But maybe I am just wrong to think about node-specific data and I should accept that all nodes are the _same_? It's very hard for me as I cannot, for example, allow myself to blindly trust an unknown SSH host key.
Comments: Philippe
Comments: CC BY-SA 4.0
Comments: 2019-02-22T14:02:32.283Z
Comments: A quick one with the ssh keys, you don't need to trust unknown ssh keys. I've used the approach linked below a lot and it works _really_ well, it's simple to implement and secure enough. Also it's worlds easier to maintain than a key per instance or user.
  https://code.fb.com/security/scalable-and-secure-access-with-ssh/
Comments: Briansbum
Comments: CC BY-SA 4.0
Comments: 2019-02-22T15:11:29.650Z
Comments: @Philippe I see what you mean, now.  Thank you for the clarification.  But it does feel like you might 
  be overcomplicating the issue.  SSM Parameter Store allows granular access control of *which* parameters (by path prefix) any given set of credentials can access.  If your instances are doing the same work, there should be no need for them to have individual "personalities" -- they can share one IAM role that allows them to access the relevant parameters, by path, and SSMPS/KMS integration can use the shared KMS key.  You can't decrypt what you can't access.
Comments: Michael - sqlbot
Tags: <amazon-ec2><security><devsecops>
Title: How to securely pass sensitive data to EC2 Spot instances?
UserDisplayName: Philippe
ThreadText: # How to securely pass sensitive data to EC2 Spot instances?
  
  (Asked by: username_0 on 2019-02-22)
  
  Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.
  
  My understanding is that I have a few ways to pass those secret data to Spot instances:
  
  * Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)
  * Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)
  * Use Secrets Manager, but that could quickly cost a lot...
  * Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)
  
  And I am not sure I could individually identify spot instances?
  
  Am I missing something there? How do you solve those challenges?
  
  **Tags:** amazon-ec2, security, devsecops
  
  > **Comments:**
  > **username_1**: A variant of instance Ebs is to share an EFS file system  
  > **username_2**: *"lot of engineering there"* Are you sure? https://docs.aws.amazon.com/kms/latest/developerguide/services-parameter-store.html  
  > **username_0**: @username_2 Thanks for your feedback, I rephrased.  
  > **username_0**: But maybe I am just wrong to think about node-specific data and I should accept that all nodes are the _same_? It's very hard for me as I cannot, for example, allow myself to blindly trust an unknown SSH host key.  
  > **username_3**: A quick one with the ssh keys, you don't need to trust unknown ssh keys. I've used the approach linked below a lot and it works _really_ well, it's simple to implement and secure enough. Also it's worlds easier to maintain than a key per instance or user. https://code.fb.com/security/scalable-and-secure-access-with-ssh/  
  > **username_2**: @username_0 I see what you mean, now.  Thank you for the clarification.  But it does feel like you might  be overcomplicating the issue.  SSM Parameter Store allows granular access control of *which* parameters (by path prefix) any given set of credentials can access.  If your instances are doing the same work, there should be no need for them to have individual "personalities" -- they can share one IAM role that allows them to access the relevant parameters, by path, and SSMPS/KMS integration can use the shared KMS key.  You can't decrypt what you can't access.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-02-22. Score: 3 | Accepted answer
  
  When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.
  
  The two most common methods I've seen for doing this are:
  
  1. If your provisioning tool ([terraform](https://www.terraform.io/docs/providers/aws/r/kms_key.html), [cloudformation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kms-key.html), [salt cloud](https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.boto_kms.html)) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.
  2. You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use [Vault](https://www.vaultproject.io/). It's not *that* hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.
  
  
  ---
  
  