Answers: <p>The problem here is caused by the random salt in ansible-vault encryption. You can hack VaultEditor class to pass the salt to it from an argument in ansible-vault. The random salt is generated in <code>lib/ansible/parsing/vault/__init__.py</code> on this <a href="https://github.com/ansible/ansible/blob/stable-2.2/lib/ansible/parsing/vault/__init__.py#L711" rel="noreferrer">line</a>. It is called from <a href="https://github.com/ansible/ansible/blob/stable-2.2/lib/ansible/cli/vault.py" rel="noreferrer">lib/ansible/cli/vault.py</a> where you can easily add the argument for fixed salt. If you do change it, please submit an upstream patch to Ansible, I'd love to use it.</p>
  
  <p>This problem is further discussed here on <a href="https://news.ycombinator.com/item?id=8264496" rel="noreferrer">hacker news</a>. And there are other implementations with tools that take fixed salt, namely <a href="https://github.com/kinnalru/gitcrypt" rel="noreferrer">gitcrypt</a>, <a href="https://github.com/elasticdog/transcrypt" rel="noreferrer">transcrypt</a>. Here is also a link to one more implementation using ansible-vault called <a href="https://github.com/building5/ansible-vault-tools" rel="noreferrer">ansible-vault-tools</a>, but this one has the same salt issue as far as I know.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-14T08:29:24.633Z
Answers: If you check the code, I am using a checksum to workaround the variable salt issue, ie. decrypt the HEAD vault in a tmp folder first and compare the checksums of plain text files before generating the new binary blob. That's a bit slow but actually ok. My problem is on merges now; in certain situations it works, in others I get the blob automerged before i can decrypt it and it breaks.
Answers: ᴳᵁᴵᴰᴼ
Answers: CC BY-SA 3.0
Answers: 2017-03-14T13:53:41.817Z
Answers: If you look into the three examples I have linked, there are some workarounds on the merges as well. And it is being discussed in the hacker news comments too.
Answers: Jiri Klouda
Answers: CC BY-SA 3.0
Answers: 2017-03-14T13:58:27.873Z
Answers: BTW Merging is tricky. What you need to realize is that in case where you pick all your changes or all changes from upstream during the merge, git would figure that out through hash comparison, which would work if the salt was right. The temp file is not enough on clean/smudge. You would need to do the same on merge and in case of non-conflict merge checkout the correct already encrypted version from git and use that one as opposed to re-encrypting with new random salt.
Answers: Jiri Klouda
Answers: CC BY-SA 3.0
Answers: 2017-03-14T16:51:17.287Z
Answers: Not sure i understand what you are saying here; merging would happen on the plain text of vaults (as it goes thru diff-ing), and having the secrets always be marked as a conflict even for auto-merges, thus including a merged re-encrypted secrets within any merge commit,  would not really represent a problem (for me).
Answers: ᴳᵁᴵᴰᴼ
Answers: CC BY-SA 3.0
Answers: 2017-03-14T18:18:13.340Z
Answers: Can you be then specific on the merge issues? You should provide reproducible case. But I would still suggest to look for ideas into the 3 projects mentioned above. As for the merging issues, when you merge content A with content B and you have all resolved to take always A or always B, for version control systems that is a special case and they will sometimes do that by linking the version together. Git does it through hash on the content, so it will assume the hash will be the same, but if you re-encrypt, even if content is all A, the hash will not be the same. But you could have other issue
Answers: Jiri Klouda
Answers: CC BY-SA 4.0
Answers: 2020-10-12T18:30:41.463Z
Answers: Since this answer was written, someone did [submit a patch to make vault encryption deterministic](https://github.com/ansible/ansible/pull/43689), but it looks like discussion petered out a while ago.
Answers: larsks
Answers: CC BY-SA 4.0
Answers: 2023-02-13T09:16:43.103Z
Answers: ... the traction the bubbled around a bit, flowed over into a new issue and finally a [change made it into 2.1.5](https://github.com/ansible/ansible/commit/fcea07bbef71683362c4761cade109f5ffe2f7a3) /cc @larsks thx for the reference!
Answers: hakre
Answers: CC BY-SA 3.0
Answers: 2017-03-14T05:10:35.270Z
Answers: 2017-03-15T18:45:15.577Z
Answers: 2017-03-15T18:45:15.577Z
Answers: Jiri Klouda
Body: <p>I am trying to setup <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes" rel="noreferrer">clean/smudge filter</a> in git to have automatic encrypting and decrypting of files containing secrets thru <a href="http://docs.ansible.com/ansible/playbooks_vault.html#encrypting-unencrypted-files" rel="noreferrer">ansible-vault</a> command.</p>
  
  <p>Peculiarity of the ansible-vault command is that it is not idempotent (it creates a different binary each time it is invoked on the same data).</p>
  
  <p>I started with the implementation suggested in <a href="https://leucos.github.io/articles/transparent-vault-revisited/" rel="noreferrer">this blog page</a>. Unfortunately it did not work correctly, as whenever smudge is called (be it a git checkout , or just git status), the secret files looks as modified for git, even if it is not.</p>
  
  <p>So I wondered if git would be comparing the binary he has in the index with the clean filtered current file, and I tried to build on those script like follows:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/sh -x
  # clean filter, it is invoked with %f
  
  if [ ! -r "$HOME/.vault_password" ]; then
    exit 1
  fi
  
  tmp=`mktemp`
  cat &gt; $tmp
  
  # get the plain text from the binary in the index
  tmphead=`mktemp`
  git show HEAD:$1 &gt; $tmphead
  contenthead=`echo "embedded" | ansible-vault view $tmphead --vault-password-file=$HOME/.vault_password`
  export PAGER=cat
  echo -n "$contenthead" | tee $tmphead
  
  # if current and index plain text version differ
  if [ "`md5sum $tmp | cut -d' ' -f1`" != "`md5sum $tmphead | cut -d' ' -f1`" ]; then
    tmpcrypt=`mktemp`
    cp $tmp $tmpcrypt
    # generate a new crypted blob
    echo "embedded" | ansible-vault encrypt $tmpcrypt --vault-password-file=$HOME/.vault_password &gt; /dev/null 2&gt;&amp;1
    cat "$tmpcrypt"
  else
    # just return the HEAD version
    cat "$tmphead"
  fi
  
  rm $tmp $tmphead $tmpcrypt
  </code></pre>
  
  <p>The difference here is that it tries to compare the current and HEAD versions of the plain text (unencrypted) secret files, and only in case they differ output a new binary blob encrypted with ansible-vault.</p>
  
  <p>Unfortunately, after this change git continues to think the secret file is always modified. Even after <code>git add</code>ing the file again, so that the git blob is computed, git thinks the file is different and let the change go into the commit. Note that <code>git diff</code> return empty changes, as it should.</p>
  
  <p>For reference, this is smudge:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/sh
  
  if [ ! -r "$HOME/.vault_password" ]; then
    exit 1
  fi
  
  tmp=`mktemp`
  cat &gt; $tmp
  
  export PAGER='cat'
  CONTENT="`echo "embedded" | ansible-vault view "$tmp" --vault-password-file=$HOME/.vault_password 2&gt; /dev/null`"
  
  if echo "$CONTENT" | grep 'ERROR: data is not encrypted' &gt; /dev/null; then
    echo "Looks like one file was commited clear text"
    echo "Please fix this before continuing !"
    exit 1
  else
    echo -n "$CONTENT"
  fi
  
  rm $tmp
  </code></pre>
  
  <p>and this is diff:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/sh
  
  if [ ! -r "$HOME/.vault_password" ]; then
    exit 1
  fi
  
  export PAGER='cat'
  CONTENT=`echo "embedded" | ansible-vault view "$1" --vault-password-file=$HOME/.vault_password 2&gt; /dev/null`
  
  if echo "$CONTENT" | grep 'ERROR: data is not encrypted' &gt; /dev/null; then
    cat "$1"
  else
    echo "$CONTENT"
  fi
  </code></pre>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-10T09:24:30.543Z
Comments: Yet another idea: https://github.com/dellis23/ansible-toolkit (I will dig more deeply into this one day)
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-01T10:30:55.770Z
Comments: I have updated scripts which behave correctly except when git tries to automerge conflicts on vaults which I'll post shortly
Comments: ᴳᵁᴵᴰᴼ
Comments: CC BY-SA 3.0
Comments: 2017-03-01T13:41:29.373Z
Comments: Throwing a bottle at the sea but: could the file be different because of different line endings or different code page ?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-01T13:50:44.130Z
Comments: I'd try removing the `-n` from the smudge echo, but that's a guess. No hidden option for git diff telling it to ignore single line endings ?
Comments: Tensibai
Tags: <ansible><git><automation><ansible-vault>
Title: Git clean/smudge filters for ansible vault secrets
UserDisplayName: ᴳᵁᴵᴰᴼ
ThreadText: # Git clean/smudge filters for ansible vault secrets
  
  (Asked by: username_0 on 2017-02-28)
  
  I am trying to setup [clean/smudge filter](https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes) in git to have automatic encrypting and decrypting of files containing secrets thru [ansible-vault](http://docs.ansible.com/ansible/playbooks_vault.html#encrypting-unencrypted-files) command.
  
  Peculiarity of the ansible-vault command is that it is not idempotent (it creates a different binary each time it is invoked on the same data).
  
  I started with the implementation suggested in [this blog page](https://leucos.github.io/articles/transparent-vault-revisited/). Unfortunately it did not work correctly, as whenever smudge is called (be it a git checkout , or just git status), the secret files looks as modified for git, even if it is not.
  
  So I wondered if git would be comparing the binary he has in the index with the clean filtered current file, and I tried to build on those script like follows:
  
  ```sh
  #!/bin/sh -x
  # clean filter, it is invoked with %f
  
  if [ ! -r "$HOME/.vault_password" ]; then
    exit 1
  fi
  
  tmp=`mktemp`
  cat > $tmp
  
  # get the plain text from the binary in the index
  tmphead=`mktemp`
  git show HEAD:$1 > $tmphead
  contenthead=`echo "embedded" | ansible-vault view $tmphead --vault-password-file=$HOME/.vault_password`
  export PAGER=cat
  echo -n "$contenthead" | tee $tmphead
  
  # if current and index plain text version differ
  if [ "`md5sum $tmp | cut -d' ' -f1`" != "`md5sum $tmphead | cut -d' ' -f1`" ]; then
    tmpcrypt=`mktemp`
    cp $tmp $tmpcrypt
    # generate a new crypted blob
    echo "embedded" | ansible-vault encrypt $tmpcrypt --vault-password-file=$HOME/.vault_password > /dev/null 2>&1
    cat "$tmpcrypt"
  else
    # just return the HEAD version
    cat "$tmphead"
  fi
  
  rm $tmp $tmphead $tmpcrypt
  
  ```
  
  The difference here is that it tries to compare the current and HEAD versions of the plain text (unencrypted) secret files, and only in case they differ output a new binary blob encrypted with ansible-vault.
  
  Unfortunately, after this change git continues to think the secret file is always modified. Even after `git add`ing the file again, so that the git blob is computed, git thinks the file is different and let the change go into the commit. Note that `git diff` return empty changes, as it should.
  
  For reference, this is smudge:
  
  ```sh
  #!/bin/sh
  
  if [ ! -r "$HOME/.vault_password" ]; then
    exit 1
  fi
  
  tmp=`mktemp`
  cat > $tmp
  
  export PAGER='cat'
  CONTENT="`echo "embedded" | ansible-vault view "$tmp" --vault-password-file=$HOME/.vault_password 2> /dev/null`"
  
  if echo "$CONTENT" | grep 'ERROR: data is not encrypted' > /dev/null; then
    echo "Looks like one file was commited clear text"
    echo "Please fix this before continuing !"
    exit 1
  else
    echo -n "$CONTENT"
  fi
  
  rm $tmp
  
  ```
  
  and this is diff:
  
  ```sh
  #!/bin/sh
  
  if [ ! -r "$HOME/.vault_password" ]; then
    exit 1
  fi
  
  export PAGER='cat'
  CONTENT=`echo "embedded" | ansible-vault view "$1" --vault-password-file=$HOME/.vault_password 2> /dev/null`
  
  if echo "$CONTENT" | grep 'ERROR: data is not encrypted' > /dev/null; then
    cat "$1"
  else
    echo "$CONTENT"
  fi
  
  ```
  
  **Tags:** ansible, git, automation, ansible-vault
  
  > **Comments:**
  > **username_0**: I have updated scripts which behave correctly except when git tries to automerge conflicts on vaults which I'll post shortly  
  > **username_1**: Throwing a bottle at the sea but: could the file be different because of different line endings or different code page ?  
  > **username_1**: I'd try removing the `-n` from the smudge echo, but that's a guess. No hidden option for git diff telling it to ignore single line endings ?  
  > **username_1**: Yet another idea: https://github.com/dellis23/ansible-toolkit (I will dig more deeply into this one day)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-03-14. Score: 9
  
  The problem here is caused by the random salt in ansible-vault encryption. You can hack VaultEditor class to pass the salt to it from an argument in ansible-vault. The random salt is generated in `lib/ansible/parsing/vault/__init__.py` on this [line](https://github.com/ansible/ansible/blob/stable-2.2/lib/ansible/parsing/vault/__init__.py#L711). It is called from [lib/ansible/cli/vault.py](https://github.com/ansible/ansible/blob/stable-2.2/lib/ansible/cli/vault.py) where you can easily add the argument for fixed salt. If you do change it, please submit an upstream patch to Ansible, I'd love to use it.
  
  This problem is further discussed here on [hacker news](https://news.ycombinator.com/item?id=8264496). And there are other implementations with tools that take fixed salt, namely [gitcrypt](https://github.com/kinnalru/gitcrypt), [transcrypt](https://github.com/elasticdog/transcrypt). Here is also a link to one more implementation using ansible-vault called [ansible-vault-tools](https://github.com/building5/ansible-vault-tools), but this one has the same salt issue as far as I know.
  
  > **Comments:**
  > **username_0**: If you check the code, I am using a checksum to workaround the variable salt issue, ie. decrypt the HEAD vault in a tmp folder first and compare the checksums of plain text files before generating the new binary blob. That's a bit slow but actually ok. My problem is on merges now; in certain situations it works, in others I get the blob automerged before i can decrypt it and it breaks.  
  > **username_2**: If you look into the three examples I have linked, there are some workarounds on the merges as well. And it is being discussed in the hacker news comments too.  
  > **username_2**: BTW Merging is tricky. What you need to realize is that in case where you pick all your changes or all changes from upstream during the merge, git would figure that out through hash comparison, which would work if the salt was right. The temp file is not enough on clean/smudge. You would need to do the same on merge and in case of non-conflict merge checkout the correct already encrypted version from git and use that one as opposed to re-encrypting with new random salt.  
  > **username_0**: Not sure i understand what you are saying here; merging would happen on the plain text of vaults (as it goes thru diff-ing), and having the secrets always be marked as a conflict even for auto-merges, thus including a merged re-encrypted secrets within any merge commit,  would not really represent a problem (for me).  
  > **username_2**: Can you be then specific on the merge issues? You should provide reproducible case. But I would still suggest to look for ideas into the 3 projects mentioned above. As for the merging issues, when you merge content A with content B and you have all resolved to take always A or always B, for version control systems that is a special case and they will sometimes do that by linking the version together. Git does it through hash on the content, so it will assume the hash will be the same, but if you re-encrypt, even if content is all A, the hash will not be the same. But you could have other issue  
  > **username_3**: Since this answer was written, someone did [submit a patch to make vault encryption deterministic](https://github.com/ansible/ansible/pull/43689), but it looks like discussion petered out a while ago.  
  > **username_4**: ... the traction the bubbled around a bit, flowed over into a new issue and finally a [change made it into 2.1.5](https://github.com/ansible/ansible/commit/fcea07bbef71683362c4761cade109f5ffe2f7a3) /cc @username_3 thx for the reference!  
  
  ---
  
  