Answers: <p>So what happens in first case is, as Dan's already said, there's no shell used and as such nothing to expand the <code>~</code>. Quoting the <a href="https://www.terraform.io/docs/providers/external/data_source.html#program" rel="noreferrer">documentation about program</a>:</p>
  
  <blockquote>
    <p>Terraform does not execute the program through a shell</p>
  </blockquote>
  
  <p>On the second case, bash receive in stdin something like this:</p>
  
  <pre><code>{ "-c": "ssh-keygen -y -f ~/.ssh/id_rsa" }
  </code></pre>
  
  <p>And this looks like a command block for bash, but -c is not a valid command.</p>
  
  <p>What could work could be this kind of program (assuming no specific input):</p>
  
  <pre><code>jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{"pubkey":$pubkey}'
  </code></pre>
  
  <p>So something like this should work to get the key in <code>local_key["pubkey"]</code> if I understand the documentation properly:</p>
  
  <pre><code>data "external" "local_key" {
    program = [
      "bash", "-c jq -n --arg pubkey \"$(ssh-keygen -y -f ~/.ssh/id_rsa)\" '{\"pubkey\":$pubkey}'"
    ]
  }
  </code></pre>
  
  <p>There's a need to use bash for a one liner to take advantage of command subsitution. You can also do a .sh script like:</p>
  
  <pre><code>#!/bin/sh
  jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{\"pubkey\":$pubkey}'
  </code></pre>
  
  <p>And call this script in the <code>program</code> parameter.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-22T09:02:19.363Z
Answers: 2018-02-22T09:02:19.363Z
Answers: Tensibai
Answers: <p>Note: this answer is based solely on documentation, I didn't try it.</p>
  
  <p>From <a href="https://www.terraform.io/docs/providers/external/data_source.html" rel="nofollow noreferrer">External Data Source</a> (emphasis mine):</p>
  
  <blockquote>
    <p>The following arguments are supported:</p>
    
    <ul>
    <li><a href="https://www.terraform.io/docs/providers/external/data_source.html#program" rel="nofollow noreferrer">program</a> - (Required) A list of strings, whose first element is the program to run and whose subsequent elements are optional command
    line arguments to the program. <strong>Terraform does not execute the
    program through a shell</strong>, so it is not necessary to escape shell
    metacharacters nor add quotes around arguments containing spaces.</li>
    </ul>
  </blockquote>
  
  <p>Expanding <code>~</code> to the home directory is a capability of the shell, not of <code>ssh-keygen</code> and since in the 1st example <code>ssh-keygen</code> is invoked, it literally attempts to open a file named <code>~/.ssh/id_rsa</code>. Which fails because a <code>~</code> directory doesn't exist (I'm not sure if that's what you mean by <code>presumably happens because ~ expansion doesn't</code>).</p>
  
  <p>In the 2nd example you're passing the <code>ssh-keygen</code> command as a query, not as arguments to <code>bash</code>. Probably what's executed is not what you're expecting.</p>
  
  <p>I'd try:</p>
  
  <pre><code>data "external" "local_key" {
    program = [
      "bash", "-c", "ssh-keygen", "-y", "-f ~/.ssh/id_rsa"
    ]
  }
  </code></pre>
  
  <p>But I'm not sure if <code>bash</code> itself (or <code>ssh-keygen</code> for that matter) follows the External Program Protocol required  (also on the above referenced page). You may need to write your own script to wrap the cmd you desire while also providing the data according to the protocol. Invoke that script as the program instead of <code>bash</code> or <code>ssh-keygen</code>.</p>
  
  <p>Finally, the entire external data source page doesn't appear to suggest that <code>export</code> would be designed for interactive programs and <code>ssh-keygen -y -f ~/.ssh/id_rsa</code> asks for the user passphrase interactively. YMMV.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-22T08:26:48.883Z
Answers: It is not compatible, terraform give input in json and waits for a json output to be used in the query block. But I don't get the question goal...
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-02-22T08:29:07.707Z
Answers: Yep, that's what I suspect as well - see my 2nd to last paragraph.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2018-02-22T08:34:44.850Z
Answers: It was a confirmation of your suspicion :) the terraform documentation is clear on this point:  "The program must then produce a valid JSON object on stdout, ..."
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-02-22T08:17:56.200Z
Answers: 2018-02-22T08:17:56.200Z
Answers: Dan Cornilescu
Answers: <p>Have you considered other options? e.g. A discussion my peers and I engaged recently was "time and place". We use mostly Ansible for deploying servers. Yet some of our team wanted to use Terraform and others argued Cloud Formation. Our discussion concluded why not let Terraform do what it does best and leave the rest to Ansible? In that within the <a href="https://www.terraform.io/docs/providers/template/index.html" rel="nofollow noreferrer">Terraform user_data</a> section you can call your Ansible Playbooks.  My '<strong>limited</strong>' understanding of Terraform is that its purpose is to deploy services to a limited extent and not be a full on automation drop-in replacement for services like Ansible, Chef, Puppet, etc. As a side note, Ansible has a module to handle the ssh keys generation and plenty more.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-22T16:50:52.733Z
Answers: I wasn't the person who down-voted you, but I think I can see why you were. Your points are valid, Terraform should be used in conjunction with a configuration management (CM) tool like Ansible. Terraform is not a replacement. However, the OP's question was about a specific function of the Terraform program. There's nothing from the OP that tells us _why_ they are running this ssh-keygen or how its used in their pipeline. Its entirely possible that running this with `terraform` _actually is_ the best place for their use case.
Answers: BoomShadow
Answers: CC BY-SA 3.0
Answers: 2018-02-23T07:11:27.887Z
Answers: I use terraform for orchestration and ansible configuration management, this particular issue is just about having my dropping in the initial SSH key in an automated fashion so ansible can connect. :P
Answers: Kit Sunde
Answers: CC BY-SA 3.0
Answers: 2018-03-01T17:04:02.027Z
Answers: @KitSunde; In my use case of Ansible we have it create the SSH key(s) for all servers and store them in out management server. Is it possible to use Ansible to create this and possibly pass that via file back to Terraform? Apologies for the inexperienced question.
Answers: Steven K7FAQ
Answers: CC BY-SA 3.0
Answers: 2018-02-22T16:17:52.567Z
Answers: 2018-02-22T16:17:52.567Z
Answers: Steven K7FAQ
Answers: <p>The best way to achieve what you're trying to do (generate an openssh public key from an <code>id_rsa</code> private key) is using the <a href="https://www.terraform.io/docs/providers/tls/d/public_key.html" rel="nofollow noreferrer"><code>tls_public_key</code></a> data provider built-in to terraform. Using the <code>external</code> provider should be considered an option of last resort because it introduces dependencies on the local OS which may not be portable. It's like an escape hatch. </p>
  
  <p>You can use it like this:</p>
  
  <pre><code>data "tls_public_key" "example" {
    private_key_pem = "${file("~/.ssh/id_rsa")}"
  }
  </code></pre>
  
  <p>Then to get the public key suitable for openssh, you can access</p>
  
  <pre><code>"${data.tls_public_key.example.public_key_openssh}" 
  </code></pre>
  
  <p>Lastly, if you are open to generating a new SSH key pair, you can use the <a href="https://github.com/cloudposse/terraform-tls-ssh-key-pair" rel="nofollow noreferrer"><code>terraform-tls-ssh-key-pair</code></a> module which will write the keys to a new file and exposes a <code>public_key</code> output with the openssh public key.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-01-20T18:49:26.703Z
Answers: Just keep in mind that Terraform does not support DSA and ED25519 certs. DSA probably will never be supported but ED25519 seems is on the roadmap.
Answers: Dzintars
Answers: CC BY-SA 4.0
Answers: 2018-07-27T04:30:30.983Z
Answers: 2018-07-27T05:06:31.547Z
Answers: 2018-07-27T05:06:31.547Z
Answers: Erik Osterman
Body: <p>I seem to be confused about how <a href="https://www.terraform.io/docs/providers/external/index.html" rel="noreferrer">external</a> works. I tried:</p>
  
  <pre><code>data "external" "local_key" {
    program = [
      "ssh-keygen", "-y", "-f ~/.ssh/id_rsa"
    ]
  }
  </code></pre>
  
  <p>This gives me:</p>
  
  <blockquote>
    <p>failed to execute "ssh-keygen":  ~/.ssh/id_rsa: No such file or directory</p>
  </blockquote>
  
  <p>Which presumably happens because <code>~</code> expansion doesn't. <code>ssh-keygen -y -f ~/.ssh/id_rsa</code> works normally. So instead I tried invoking bash like:</p>
  
  <pre><code>data "external" "local_key" {
    program = [
      "bash"
    ]
  
    query {
      "-c" = "ssh-keygen -y -f ~/.ssh/id_rsa"
    }
  }
  </code></pre>
  
  <p>I'm still getting the same issue. I realize I can just pass the output as a var to terraform from the outside, but I'm still curious what the solution is.</p>
  
Comments: CC BY-SA 3.0
Comments: 2018-02-22T05:36:16.207Z
Comments: Why insisting in using ~ where you can use $HOME?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-22T06:29:12.297Z
Comments: @Tensibai Good point, it's just habit. :P I still don't understand why calling bash won't end up doing the expansion though.
Comments: Kit Sunde
Comments: CC BY-SA 3.0
Comments: 2018-02-22T07:14:05.887Z
Comments: Maybe because there's no or invalid login when Terraform execute the command? But external is supposed to be an interaction with the remote machine, not running a command on the remote machine from what I understand from the documentation.
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-22T07:16:32.560Z
Comments: I'm not sure to really understand what you're trying to achieve in fact.
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-22T08:28:32.487Z
Comments: @Tensibai External runs local programs, it doesn't imply interaction with anything remote. It's an external data source relative to terraforms execution environment, not relative to the machine terraform runs on. "external is a special provider that exists to provide an interface between Terraform and external programs." I'm just getting the public key for the local private key, the public key doesn't exist on the machine I'm running terraform on.
Comments: Kit Sunde
Comments: CC BY-SA 3.0
Comments: 2018-02-22T08:33:19.867Z
Comments: So that's just a XY problem, getting the key in a difficult manner instead of extracting it once for all in a file and using this file as usual...
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-22T08:39:14.773Z
Comments: @Tensibai What is a XY problem? Like I mentioned in my question, I can solve the issue just fine by passing it as a variable from the outside. I wanted to understand *why* I couldn't solve it in the manner I first tried.
Comments: Kit Sunde
Comments: CC BY-SA 3.0
Comments: 2018-02-22T08:43:38.820Z
Comments: The why is simply because the external protocol waits a json output to be used later: "The program must then produce a valid JSON object on stdout, which will be used to populate the result attribute exported to the rest of the Terraform configuration. " (quoted from the [documentation](https://www.terraform.io/docs/providers/external/data_source.html)). ssh-keygen doesn't output a json object
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-22T08:46:41.517Z
Comments: I mean that running `ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub` outside of terraform and using this .pub file in terraform would be far easier IMHO.
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-02-22T08:48:44.827Z
Comments: And your second try doesn't work because the query block content will be passed as a json file to your program, bash doesn't read arguments from json, so as the documentation show you should create a script to aprse the input json, run the command and then format the output to be json format.
Comments: Tensibai
Tags: <terraform>
Title: How can I get terraforms extern to execute `ssh-keygen -y -f ~/.ssh/id_rsa`?
UserDisplayName: Kit Sunde
ThreadText: # How can I get terraforms extern to execute `ssh-keygen -y -f ~/.ssh/id_rsa`?
  
  (Asked by: username_0 on 2018-02-22)
  
  I seem to be confused about how [external](https://www.terraform.io/docs/providers/external/index.html) works. I tried:
  
  ```
  data "external" "local_key" {
    program = [
      "ssh-keygen", "-y", "-f ~/.ssh/id_rsa"
    ]
  }
  
  ```
  
  This gives me:
  
  > failed to execute "ssh-keygen": ~/.ssh/id\_rsa: No such file or directory
  
  Which presumably happens because `~` expansion doesn't. `ssh-keygen -y -f ~/.ssh/id_rsa` works normally. So instead I tried invoking bash like:
  
  ```
  data "external" "local_key" {
    program = [
      "bash"
    ]
  
    query {
      "-c" = "ssh-keygen -y -f ~/.ssh/id_rsa"
    }
  }
  
  ```
  
  I'm still getting the same issue. I realize I can just pass the output as a var to terraform from the outside, but I'm still curious what the solution is.
  
  **Tags:** terraform
  
  > **Comments:**
  > **username_1**: Why insisting in using ~ where you can use $HOME?  
  > **username_0**: @username_1 Good point, it's just habit. :P I still don't understand why calling bash won't end up doing the expansion though.  
  > **username_1**: Maybe because there's no or invalid login when Terraform execute the command? But external is supposed to be an interaction with the remote machine, not running a command on the remote machine from what I understand from the documentation.  
  > **username_1**: I'm not sure to really understand what you're trying to achieve in fact.  
  > **username_0**: @username_1 External runs local programs, it doesn't imply interaction with anything remote. It's an external data source relative to terraforms execution environment, not relative to the machine terraform runs on. "external is a special provider that exists to provide an interface between Terraform and external programs." I'm just getting the public key for the local private key, the public key doesn't exist on the machine I'm running terraform on.  
  > **username_1**: So that's just a XY problem, getting the key in a difficult manner instead of extracting it once for all in a file and using this file as usual...  
  > **username_0**: @username_1 What is a XY problem? Like I mentioned in my question, I can solve the issue just fine by passing it as a variable from the outside. I wanted to understand *why* I couldn't solve it in the manner I first tried.  
  > **username_1**: The why is simply because the external protocol waits a json output to be used later: "The program must then produce a valid JSON object on stdout, which will be used to populate the result attribute exported to the rest of the Terraform configuration. " (quoted from the [documentation](https://www.terraform.io/docs/providers/external/data_source.html)). ssh-keygen doesn't output a json object  
  > **username_1**: I mean that running `ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub` outside of terraform and using this .pub file in terraform would be far easier IMHO.  
  > **username_1**: And your second try doesn't work because the query block content will be passed as a json file to your program, bash doesn't read arguments from json, so as the documentation show you should create a script to aprse the input json, run the command and then format the output to be json format.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2018-07-27. Score: 4 | Accepted answer
  
  The best way to achieve what you're trying to do (generate an openssh public key from an `id_rsa` private key) is using the [`tls_public_key`](https://www.terraform.io/docs/providers/tls/d/public_key.html) data provider built-in to terraform. Using the `external` provider should be considered an option of last resort because it introduces dependencies on the local OS which may not be portable. It's like an escape hatch.
  
  You can use it like this:
  
  ```
  data "tls_public_key" "example" {
    private_key_pem = "${file("~/.ssh/id_rsa")}"
  }
  
  ```
  
  Then to get the public key suitable for openssh, you can access
  
  ```
  "${data.tls_public_key.example.public_key_openssh}" 
  
  ```
  
  Lastly, if you are open to generating a new SSH key pair, you can use the [`terraform-tls-ssh-key-pair`](https://github.com/cloudposse/terraform-tls-ssh-key-pair) module which will write the keys to a new file and exposes a `public_key` output with the openssh public key.
  
  > **Comments:**
  > **username_6**: Just keep in mind that Terraform does not support DSA and ED25519 certs. DSA probably will never be supported but ED25519 seems is on the roadmap.  
  
  ---
  
  ### Answer by username_1 on 2018-02-22. Score: 6
  
  So what happens in first case is, as Dan's already said, there's no shell used and as such nothing to expand the `~`. Quoting the [documentation about program](https://www.terraform.io/docs/providers/external/data_source.html#program):
  
  > Terraform does not execute the program through a shell
  
  On the second case, bash receive in stdin something like this:
  
  ```
  { "-c": "ssh-keygen -y -f ~/.ssh/id_rsa" }
  
  ```
  
  And this looks like a command block for bash, but -c is not a valid command.
  
  What could work could be this kind of program (assuming no specific input):
  
  ```
  jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{"pubkey":$pubkey}'
  
  ```
  
  So something like this should work to get the key in `local_key["pubkey"]` if I understand the documentation properly:
  
  ```
  data "external" "local_key" {
    program = [
      "bash", "-c jq -n --arg pubkey \"$(ssh-keygen -y -f ~/.ssh/id_rsa)\" '{\"pubkey\":$pubkey}'"
    ]
  }
  
  ```
  
  There's a need to use bash for a one liner to take advantage of command subsitution. You can also do a .sh script like:
  
  ```
  #!/bin/sh
  jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{\"pubkey\":$pubkey}'
  
  ```
  
  And call this script in the `program` parameter.
  
  
  ---
  
  ### Answer by username_2 on 2018-02-22. Score: 1
  
  Note: this answer is based solely on documentation, I didn't try it.
  
  From [External Data Source](https://www.terraform.io/docs/providers/external/data_source.html) (emphasis mine):
  
  > The following arguments are supported:
  >
  > * [program](https://www.terraform.io/docs/providers/external/data_source.html#program) - (Required) A list of strings, whose first element is the program to run and whose subsequent elements are optional command
  >   line arguments to the program. **Terraform does not execute the
  >   program through a shell**, so it is not necessary to escape shell
  >   metacharacters nor add quotes around arguments containing spaces.
  
  Expanding `~` to the home directory is a capability of the shell, not of `ssh-keygen` and since in the 1st example `ssh-keygen` is invoked, it literally attempts to open a file named `~/.ssh/id_rsa`. Which fails because a `~` directory doesn't exist (I'm not sure if that's what you mean by `presumably happens because ~ expansion doesn't`).
  
  In the 2nd example you're passing the `ssh-keygen` command as a query, not as arguments to `bash`. Probably what's executed is not what you're expecting.
  
  I'd try:
  
  ```
  data "external" "local_key" {
    program = [
      "bash", "-c", "ssh-keygen", "-y", "-f ~/.ssh/id_rsa"
    ]
  }
  
  ```
  
  But I'm not sure if `bash` itself (or `ssh-keygen` for that matter) follows the External Program Protocol required (also on the above referenced page). You may need to write your own script to wrap the cmd you desire while also providing the data according to the protocol. Invoke that script as the program instead of `bash` or `ssh-keygen`.
  
  Finally, the entire external data source page doesn't appear to suggest that `export` would be designed for interactive programs and `ssh-keygen -y -f ~/.ssh/id_rsa` asks for the user passphrase interactively. YMMV.
  
  > **Comments:**
  > **username_1**: It is not compatible, terraform give input in json and waits for a json output to be used in the query block. But I don't get the question goal...  
  > **username_2**: Yep, that's what I suspect as well - see my 2nd to last paragraph.  
  > **username_1**: It was a confirmation of your suspicion :) the terraform documentation is clear on this point:  "The program must then produce a valid JSON object on stdout, ..."  
  
  ---
  
  ### Answer by username_3 on 2018-02-22. Score: 0
  
  Have you considered other options? e.g. A discussion my peers and I engaged recently was "time and place". We use mostly Ansible for deploying servers. Yet some of our team wanted to use Terraform and others argued Cloud Formation. Our discussion concluded why not let Terraform do what it does best and leave the rest to Ansible? In that within the [Terraform user\_data](https://www.terraform.io/docs/providers/template/index.html) section you can call your Ansible Playbooks. My '**limited**' understanding of Terraform is that its purpose is to deploy services to a limited extent and not be a full on automation drop-in replacement for services like Ansible, Chef, Puppet, etc. As a side note, Ansible has a module to handle the ssh keys generation and plenty more.
  
  > **Comments:**
  > **username_4**: I wasn't the person who down-voted you, but I think I can see why you were. Your points are valid, Terraform should be used in conjunction with a configuration management (CM) tool like Ansible. Terraform is not a replacement. However, the OP's question was about a specific function of the Terraform program. There's nothing from the OP that tells us _why_ they are running this ssh-keygen or how its used in their pipeline. Its entirely possible that running this with `terraform` _actually is_ the best place for their use case.  
  > **username_0**: I use terraform for orchestration and ansible configuration management, this particular issue is just about having my dropping in the initial SSH key in an automated fashion so ansible can connect. :P  
  > **username_3**: @username_0; In my use case of Ansible we have it create the SSH key(s) for all servers and store them in out management server. Is it possible to use Ansible to create this and possibly pass that via file back to Terraform? Apologies for the inexperienced question.  
  
  ---
  
  