Body: <p>I want to add eight public keys via instance metadata to avoid adding them manually (i.e.: ssh to VMs, pasting the keys to <code>.ssh/authorized_keys</code>, etc.).<br />
  I added the keys in Terraform (four distinct keys for two users) using the metadata attribute of the <code>google_compute_instance</code>:</p>
  <pre><code>resource &quot;google_compute_instance&quot; &quot;host&quot; {
    count         = var.number_of_hosts
    
    // vm details...
  
    metadata = {
      &quot;ssh-keys&quot; = &lt;&lt;EOF
      user1:${file(&quot;${path.root}/key1.pub&quot;)}
      user1:${file(&quot;${path.root}/key2.pub&quot;)}
      user1:${file(&quot;${path.root}/key3.pub&quot;)}
      user1:${file(&quot;${path.root}/key4.pub&quot;)}
      user2:${file(&quot;${path.root}/key1.pub&quot;)}
      user2:${file(&quot;${path.root}/key2.pub&quot;)}
      user2:${file(&quot;${path.root}/key3.pub&quot;)}
      user2:${file(&quot;${path.root}/key4.pub&quot;)}
      EOF
    }
  </code></pre>
  <p>I ran <code>terraform apply</code>. I opened the GCP console and clicked on one of the deployed machines. In the &quot;Details&quot; tab, I can see all eight keys in the SSH Keys tab.
  Now, when I ssh from my local computer, i.e., <code>ssh user2@EXTERNAL_IP</code> (I deliberately started with user2, not user1 - not a typo) and then <code>cat ~/.ssh/authorized_keys</code>, I can only see the following:</p>
  <pre><code>user1 : key1
  user2 : key4
  </code></pre>
  <p>Thus, I can't ssh to VM2 because the public part of the key pair that USER 2 has access to is not <code>~/.ssh/authorized_keys</code> even though it is declared in the instance metadata.</p>
  <p>On the other hand, when I do <code>user1@EXTERNAL_IP</code> and <code>cat ~/.ssh/authorized_keys</code>, I can see:</p>
  <pre><code>user1 : key1
  user2 : key4
  user1 : key1 (duplicate)
  </code></pre>
  <p>Since the private key that corresponds to <code>user1 : key1</code> is there, I can ssh to VM2 successfully.</p>
  <p>What baffles me:</p>
  <ol>
  <li>Why are not all keys declared in the instance metadata added to the <code>authorized_keys</code>?</li>
  <li>Why is there a difference in the content of the <code>authorized_keys</code> depending on the user?</li>
  <li>Where does the duplicate come from?</li>
  </ol>
  <hr />
  <p>Edit - some additional information:</p>
  <ol>
  <li>the image used - <code>ubuntu-minimal-2004-focal-v20230427</code></li>
  <li><code>ssh_config</code> (only uncommented lines):</li>
  </ol>
  <pre><code>Include /etc/ssh/ssh_config.d/*.conf
  Host *
  SendEnv LANG LC_*
  HashKnownHosts yes
  GSSAPIAuthentication yes
  </code></pre>
  <ol start="3">
  <li><code>sshd_config</code> (only uncommented lines):</li>
  </ol>
  <pre><code>Include /etc/ssh/sshd_config.d/*.conf
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  UsePAM yes
  X11Forwarding yes
  PrintMotd no
  AcceptEnv LANG LC_*
  Subsystem       sftp    /usr/lib/openssh/sftp-server
  
  </code></pre>
  
Comments: CC BY-SA 4.0
Comments: 2023-05-25T15:51:46.787Z
Comments: What software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files? I expect it's that software which is making the decision about which files to write and what to write into each of them. From Terraform's perspective and GCP's perspective this is just an opaque string, so how it gets interpreted is up to the software running inside your VM.
Comments: Martin Atkins
Comments: CC BY-SA 4.0
Comments: 2023-05-25T17:21:02.057Z
Comments: I edited my answer with the `sshd_config` and `ssh_config`, but I'm not sure that is what you meant with "software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files".
Comments: SkogensKonung
Comments: CC BY-SA 4.0
Comments: 2023-05-25T23:25:44.133Z
Comments: The way this metadata concept works is that Google Compute Engine just exposes [an internal API that returns the metadata values](https://cloud.google.com/compute/docs/metadata/querying-metadata) and then some software running in your VM -- typically during the boot process -- will make requests to that API to fetch the data and then reconfigure your system based on what it reads. It's that software that would be responsible for generating these `authorized_keys` files. If you're using a generic Linux distribution image then perhaps you can name that distribution so we can find docs about it.
Comments: Martin Atkins
Comments: CC BY-SA 4.0
Comments: 2023-05-26T06:19:59.717Z
Comments: Here's the info from the description tab in the GCP console: `Canonical, Ubuntu, 20.04 LTS Minimal, amd64 focal minimal image built on 2023-04-27, supports Shielded VM features`
Comments: SkogensKonung
Comments: CC BY-SA 4.0
Comments: 2023-05-26T15:19:48.723Z
Comments: According to Ubuntu's documentation, [official Ubuntu cloud images since 18.04 use CloudInit for boot-time initialization](https://help.ubuntu.com/community/CloudInit). That suggests that this is a CloudInit question, and so I'm going to add a label for that in the hope that it will be more visible to those who have CloudInit expertise.
Comments: Martin Atkins
Tags: <terraform><google-cloud-platform><ssh><cloud-init>
Title: SSH Keys not propagating correctly from instance metadata to authorized_keys: missing keys, user discrepancy, and duplicate key
UserDisplayName: SkogensKonung
ThreadText: # SSH Keys not propagating correctly from instance metadata to authorized_keys: missing keys, user discrepancy, and duplicate key
  
  (Asked by: username_0 on 2023-05-25)
  
  I want to add eight public keys via instance metadata to avoid adding them manually (i.e.: ssh to VMs, pasting the keys to `.ssh/authorized_keys`, etc.).  
  I added the keys in Terraform (four distinct keys for two users) using the metadata attribute of the `google_compute_instance`:
  
  ```
  resource "google_compute_instance" "host" {
    count         = var.number_of_hosts
    
    // vm details...
  
    metadata = {
      "ssh-keys" = <<EOF
      user1:${file("${path.root}/key1.pub")}
      user1:${file("${path.root}/key2.pub")}
      user1:${file("${path.root}/key3.pub")}
      user1:${file("${path.root}/key4.pub")}
      user2:${file("${path.root}/key1.pub")}
      user2:${file("${path.root}/key2.pub")}
      user2:${file("${path.root}/key3.pub")}
      user2:${file("${path.root}/key4.pub")}
      EOF
    }
  
  ```
  
  I ran `terraform apply`. I opened the GCP console and clicked on one of the deployed machines. In the "Details" tab, I can see all eight keys in the SSH Keys tab.
  Now, when I ssh from my local computer, i.e., `ssh user2@EXTERNAL_IP` (I deliberately started with user2, not user1 - not a typo) and then `cat ~/.ssh/authorized_keys`, I can only see the following:
  
  ```
  user1 : key1
  user2 : key4
  
  ```
  
  Thus, I can't ssh to VM2 because the public part of the key pair that USER 2 has access to is not `~/.ssh/authorized_keys` even though it is declared in the instance metadata.
  
  On the other hand, when I do `user1@EXTERNAL_IP` and `cat ~/.ssh/authorized_keys`, I can see:
  
  ```
  user1 : key1
  user2 : key4
  user1 : key1 (duplicate)
  
  ```
  
  Since the private key that corresponds to `user1 : key1` is there, I can ssh to VM2 successfully.
  
  What baffles me:
  
  1. Why are not all keys declared in the instance metadata added to the `authorized_keys`?
  2. Why is there a difference in the content of the `authorized_keys` depending on the user?
  3. Where does the duplicate come from?
  
  ---
  
  Edit - some additional information:
  
  1. the image used - `ubuntu-minimal-2004-focal-v20230427`
  2. `ssh_config` (only uncommented lines):
  
  ```
  Include /etc/ssh/ssh_config.d/*.conf
  Host *
  SendEnv LANG LC_*
  HashKnownHosts yes
  GSSAPIAuthentication yes
  
  ```
  
  3. `sshd_config` (only uncommented lines):
  
  ```
  Include /etc/ssh/sshd_config.d/*.conf
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  UsePAM yes
  X11Forwarding yes
  PrintMotd no
  AcceptEnv LANG LC_*
  Subsystem       sftp    /usr/lib/openssh/sftp-server
  
  ```
  
  **Tags:** terraform, google-cloud-platform, ssh, cloud-init
  
  > **Comments:**
  > **username_1**: What software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files? I expect it's that software which is making the decision about which files to write and what to write into each of them. From Terraform's perspective and GCP's perspective this is just an opaque string, so how it gets interpreted is up to the software running inside your VM.  
  > **username_0**: I edited my answer with the `sshd_config` and `ssh_config`, but I'm not sure that is what you meant with "software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files".  
  > **username_1**: The way this metadata concept works is that Google Compute Engine just exposes [an internal API that returns the metadata values](https://cloud.google.com/compute/docs/metadata/querying-metadata) and then some software running in your VM -- typically during the boot process -- will make requests to that API to fetch the data and then reconfigure your system based on what it reads. It's that software that would be responsible for generating these `authorized_keys` files. If you're using a generic Linux distribution image then perhaps you can name that distribution so we can find docs about it.  
  > **username_0**: Here's the info from the description tab in the GCP console: `Canonical, Ubuntu, 20.04 LTS Minimal, amd64 focal minimal image built on 2023-04-27, supports Shielded VM features`  
  > **username_1**: According to Ubuntu's documentation, [official Ubuntu cloud images since 18.04 use CloudInit for boot-time initialization](https://help.ubuntu.com/community/CloudInit). That suggests that this is a CloudInit question, and so I'm going to add a label for that in the hope that it will be more visible to those who have CloudInit expertise.  
  
  ---
  
  ## 0 answers
  
  