<p>It was the case that python-bitcoinlib's script evaluation code did not handle  SCRIPT_VERIFY_STRICTENC flag.</p>

<p>I had to create additional method, verify_nonstrict(), and make the scripteval use it when STRICTENC flag is not specified.</p>

<p>Handled in <a href="https://github.com/Simplexum/python-bitcointx/commit/d8ab8ff76de412f90480c3c68addbbb1791d3413" rel="nofollow noreferrer">https://github.com/Simplexum/python-bitcointx/commit/d8ab8ff76de412f90480c3c68addbbb1791d3413</a></p>

CC BY-SA 4.0
2018-12-01T16:54:53.400Z
2018-12-01T16:59:56.220Z
2018-12-01T16:59:56.220Z
Dmitry Petukhov
<p>If you want to guarantee strict DER encoded signatures which are accepted by Bitcoind then they need to be bip66 compliant(low s values(ecdsa malleability), remove leading nulls (encoding malleability)). I am not sure you are guarding for that with your decode-encode-decode cycle. These do not enforce r,s value-space constraints.</p>

CC BY-SA 4.0
2018-12-01T15:59:42.360Z
low s values are enforced separately, with secp256k1_ecdsa_signature_normalize(). This is about DER encoding (https://en.wikipedia.org/wiki/X.690#DER_encoding) that openssl and bitcoind may treat differently
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T16:06:29.920Z
So you need to ensure for bip66 compliant DER encoding, which is a subset of the standard you linked above, no?
James C.
CC BY-SA 4.0
2018-12-01T16:11:51.913Z
looks like the ecdsa_signature_parse_der_lax() is only relevant to the signatures before activation of BIP66, and I can just go with requiring strict DER encoding. There is some tests in that may fail as a result, but since it is irrelevant, I may just disable these tests.
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T16:17:56.513Z
Still, I do not see where secp256k1_ecdsa_signature_parse_der() is used by bitcoind, and only ecdsa_signature_parse_der_lax(). Need to investigate furter.
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T16:22:18.423Z
Peter Wuille could definitely help, he wrote bip66...
James C.
CC BY-SA 4.0
2018-12-01T16:35:38.697Z
It may be the case of just correcting the tests to use special non-strict verification method. python-bitcoinlib test cases seem to ignore STRICTENC modifiers in test data.
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T17:54:35.813Z
note on my previous comment that secp256k1_ecdsa_signature_parse_der() is not used by bitcoind: it does the checking in [IsValidSignatureEncoding()](https://github.com/bitcoin/bitcoin/blob/ed12fd83ca7999a896350197533de5e9202bc2fe/src/script/interpreter.cpp#L107)
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T17:57:49.990Z
The parsing code used is always the same, before and after BIP66 activated. The difference is in an extra sanity check before the parser is invoked.
Pieter Wuille
CC BY-SA 4.0
2018-12-01T18:46:08.593Z
If the signature passes  IsValidSignatureEncoding(), it should always be accepted by secp256k1_ecdsa_signature_parse_der(), or there might be corner cases when such signature might not be accepted by secp256k1_ecdsa_signature_parse_der() ?
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T20:33:25.807Z
It seems that checks are slightly different b/w secp256k1_ecdsa_signature_parse_der() and  IsValidSignatureEncoding():

secp256k1_ecdsa_signature_parse_der have check for "Excessive 0xFF padding", but allows negative integers. IsValidSignatureEncoding do not allow negative integers, and checks only for excessive 0x00 padding.
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T20:33:38.173Z
I will need to use ported IsValidSignatureEncoding() to check for STRICTENC, and cannot just rely on secp256k1_ecdsa_signature_parse_der(), to be compatible with bitcoin's script interpreter.
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T15:56:13.997Z
2018-12-01T15:56:13.997Z
James C.
<p>It isn't clear to me from your question what your goal is.  If you want to accept all transactions and only transactions that Bitcoin would accept, libsecp256k1 alone is what you want. (You may need to use the signature normalize call if you want to match consensus acceptance instead of standardness acceptance).</p>

<p>If you want to validate historical signatures from the blockchain then either openssl or the code in contrib is sufficient. If you use either of these things on current transactions you will accept things that Bitcoin will reject.</p>

<p>We, the authors of libsecp256k1, are currently of the belief that no application should use a lax parser, except for historical compatibility. Unfortunately, no function is likely to be universally compatible with OpenSSL's parser except OpenSSL itself... but for cases where the set of signatures that you need to be compatible with is known a specific alternative lax parser might work, as is the case for Bitcoin's history and the lax parser in contrib.</p>

<p>The contrib lax parser isn't included in the library because it's only provably sufficient for Bitcoin's history. Someone trying to lax-parse a different set of signatures may need to modify that lax parser to accommodate the specific weird signatures that they're dealing with.  We also, in general, don't think anyone should be lax parsing unless its strictly needed since it can result in surprising vulnerabilities.</p>

<p>As an aside, in the current versions of OpenSSL the ECDSA validator attempts to validate signature encodings by checking that the serialization is unchanged under decode and encode and it rejects signatures that aren't.  This likely means they reject any signature that libsecp256k1 (with normalize) rejects.  [And annoyingly, OpenSSL made this breaking change in a security update.]</p>

CC BY-SA 4.0
2018-12-02T05:35:05.603Z
I mistakenly thought that lax parser may be still relevant, and this question arose out of that. This was because python-bitcoinlib tests also test script evaluation code for historical, lax-encoded signatures. The tests failed after I introduced strict decoding. Now I added handling for STRICTENC flag (that was missing) and do lax verification when STRICTENC is not specified. Tests with historical signatures now pass. I also added a warning to README that people should not use python-bitcointx's VerifyScript as authoritative source on input validity, because it may not be in-sync with core.
Dmitry Petukhov
CC BY-SA 4.0
2018-12-02T06:16:07.990Z
Because I thought that python-bitcointx's VerifyScript might be relied upon by someone to check input validity, I was worried that the differences in handling strict encoding might bite them. I thought that this is the only serious difference b/w this lib and the core. The goal of the question was to understand the scope of the differences to decide how to handle this. It turned out that the differences are huge, and all bets are off - so the warning in README is necessary.
Dmitry Petukhov
CC BY-SA 4.0
2018-12-01T20:45:36.087Z
2018-12-01T23:28:13.270Z
2018-12-01T23:28:13.270Z
G. Maxwell
<p>I have recently changed verification code in <a href="https://github.com/Simplexum/python-bitcointx/blob/b998a5a66f9698f252fc5ab9e6556cbd0ac412d8/bitcointx/core/key.py#L309" rel="nofollow noreferrer">python-bitcointx</a> to use libsecp256k1 library.</p>

<p>In the python-bitcoinlib code (that python-bitcointx is a derivative of), openssl functions were used for signature verification.</p>

<p>I want to be able to remove dependency on openssl in the future, and use only libsecp256k1 in python-bitcointx, but there is a problem that I noticed when I re-wrote CECKey.verify() method:</p>

<p>libsecp256k1 function secp256k1_ecdsa_signature_parse_der() only allows strictly valid DER-encoded signatures.</p>

<p>bitcoind uses <a href="https://github.com/bitcoin/bitcoin/blob/ed12fd83ca7999a896350197533de5e9202bc2fe/src/pubkey.cpp#L27" rel="nofollow noreferrer">ecdsa_signature_parse_der_lax()</a> function, copied from libsecp256k1's contrib/ directory. This function is not compiled in a normal compilation cycle of libsecp256k1.</p>

<p>It is expected for bitcoin-focused library to be able to verify signatures that bitcoind will also verify.</p>

<p>To be able to verify non-strict-encoded signatures with libsecp256k1, I see 3 options:</p>

<blockquote>
  <p>EDIT: I might just implement two verification methods, strict and non-strict, and then use them in tests. openssl dependency will stay because of verify_nonstrict().</p>
</blockquote>

<ul>
<li>get ecdsa_signature_parse_der_lax() function from libsecp256k1/contrib, add it to python-bitcointx repository, compile at python package build time, install with a package.</li>
</ul>

<p>This is not desireable, because it adds requirement to have C compiler to build the python package. In some environments this may be inconvenient.</p>

<ul>
<li>re-implement ecdsa_signature_parse_der_lax() in python</li>
</ul>

<p>I would like to avoid this, to reduce the size of the code (reduce the surface where bug may be introduced). Porting this function correctly will require to invest some time.</p>

<p>It will also make verification slower to some degree, because more operations will be done in python, vs calling C functions.</p>

<ul>
<li>do a cycle of decode-with-openssl / encode-with-openssl / decode-with-libsecp256k1</li>
</ul>

<p>This way all encoding violations that openssl allows are also allowed by first decode-with-openssl. encode-with-openssl then creates valid encoding, and decode-with-libsecp256k1 is happy.</p>

<p>The problem with this approach (in addition to this decode/encode cycle) is that it will allow some format violations that bitcoind won't allow, because ecdsa_signature_parse_der_lax() only allows an arbitrary subset of violations allowed by openssl.</p>

<p>python-bitcoinlib was using openssl to decode, too, so I went with the later approach - decode/encode/decode. This means I cannot drop openssl dependency.</p>

<p>My question is: what may be the implications of this choice ?</p>

<p>What I can see is that when someone uses python-bitcointx or python-bitcoinlib to verify inputs in some transaction, and these inputs contain signatures that is accepted by openssl, but not by bitcoind, they might be convinced that tx is valid, but will have unpleasant surprise when they try to broadcast it.</p>

<p>With some protocols that require parties to store signed transactions, that might be relevant.</p>

<p>As this is historical behavior of python-bitcoinlib, I am inclined to just document this in the README.md as a 'gotcha', and leave it as it is now.</p>

<p>But maybe there is some other potential problems that could arise out of this issue, that may be more serious, and the first or second options described above should be considered ? </p>

<signature><encoding>
allowed DER encoding format violations for signatures in bitcoin: implications for libraries?
Dmitry Petukhov
# allowed DER encoding format violations for signatures in bitcoin: implications for libraries?

(Asked by: username_0 on 2018-12-01)

I have recently changed verification code in [python-bitcointx](https://github.com/Simplexum/python-bitcointx/blob/b998a5a66f9698f252fc5ab9e6556cbd0ac412d8/bitcointx/core/key.py#L309) to use libsecp256k1 library.

In the python-bitcoinlib code (that python-bitcointx is a derivative of), openssl functions were used for signature verification.

I want to be able to remove dependency on openssl in the future, and use only libsecp256k1 in python-bitcointx, but there is a problem that I noticed when I re-wrote CECKey.verify() method:

libsecp256k1 function secp256k1\_ecdsa\_signature\_parse\_der() only allows strictly valid DER-encoded signatures.

bitcoind uses [ecdsa\_signature\_parse\_der\_lax()](https://github.com/bitcoin/bitcoin/blob/ed12fd83ca7999a896350197533de5e9202bc2fe/src/pubkey.cpp#L27) function, copied from libsecp256k1's contrib/ directory. This function is not compiled in a normal compilation cycle of libsecp256k1.

It is expected for bitcoin-focused library to be able to verify signatures that bitcoind will also verify.

To be able to verify non-strict-encoded signatures with libsecp256k1, I see 3 options:

> EDIT: I might just implement two verification methods, strict and non-strict, and then use them in tests. openssl dependency will stay because of verify\_nonstrict().

* get ecdsa\_signature\_parse\_der\_lax() function from libsecp256k1/contrib, add it to python-bitcointx repository, compile at python package build time, install with a package.

This is not desireable, because it adds requirement to have C compiler to build the python package. In some environments this may be inconvenient.

* re-implement ecdsa\_signature\_parse\_der\_lax() in python

I would like to avoid this, to reduce the size of the code (reduce the surface where bug may be introduced). Porting this function correctly will require to invest some time.

It will also make verification slower to some degree, because more operations will be done in python, vs calling C functions.

* do a cycle of decode-with-openssl / encode-with-openssl / decode-with-libsecp256k1

This way all encoding violations that openssl allows are also allowed by first decode-with-openssl. encode-with-openssl then creates valid encoding, and decode-with-libsecp256k1 is happy.

The problem with this approach (in addition to this decode/encode cycle) is that it will allow some format violations that bitcoind won't allow, because ecdsa\_signature\_parse\_der\_lax() only allows an arbitrary subset of violations allowed by openssl.

python-bitcoinlib was using openssl to decode, too, so I went with the later approach - decode/encode/decode. This means I cannot drop openssl dependency.

My question is: what may be the implications of this choice ?

What I can see is that when someone uses python-bitcointx or python-bitcoinlib to verify inputs in some transaction, and these inputs contain signatures that is accepted by openssl, but not by bitcoind, they might be convinced that tx is valid, but will have unpleasant surprise when they try to broadcast it.

With some protocols that require parties to store signed transactions, that might be relevant.

As this is historical behavior of python-bitcoinlib, I am inclined to just document this in the README.md as a 'gotcha', and leave it as it is now.

But maybe there is some other potential problems that could arise out of this issue, that may be more serious, and the first or second options described above should be considered ?

**Tags:** signature, encoding


---

## 3 answers

### Answer by username_3 on 2018-12-01. Score: 6 | Accepted answer

It isn't clear to me from your question what your goal is. If you want to accept all transactions and only transactions that Bitcoin would accept, libsecp256k1 alone is what you want. (You may need to use the signature normalize call if you want to match consensus acceptance instead of standardness acceptance).

If you want to validate historical signatures from the blockchain then either openssl or the code in contrib is sufficient. If you use either of these things on current transactions you will accept things that Bitcoin will reject.

We, the authors of libsecp256k1, are currently of the belief that no application should use a lax parser, except for historical compatibility. Unfortunately, no function is likely to be universally compatible with OpenSSL's parser except OpenSSL itself... but for cases where the set of signatures that you need to be compatible with is known a specific alternative lax parser might work, as is the case for Bitcoin's history and the lax parser in contrib.

The contrib lax parser isn't included in the library because it's only provably sufficient for Bitcoin's history. Someone trying to lax-parse a different set of signatures may need to modify that lax parser to accommodate the specific weird signatures that they're dealing with. We also, in general, don't think anyone should be lax parsing unless its strictly needed since it can result in surprising vulnerabilities.

As an aside, in the current versions of OpenSSL the ECDSA validator attempts to validate signature encodings by checking that the serialization is unchanged under decode and encode and it rejects signatures that aren't. This likely means they reject any signature that libsecp256k1 (with normalize) rejects. [And annoyingly, OpenSSL made this breaking change in a security update.]

> **Comments:**
> **username_0**: I mistakenly thought that lax parser may be still relevant, and this question arose out of that. This was because python-bitcoinlib tests also test script evaluation code for historical, lax-encoded signatures. The tests failed after I introduced strict decoding. Now I added handling for STRICTENC flag (that was missing) and do lax verification when STRICTENC is not specified. Tests with historical signatures now pass. I also added a warning to README that people should not use python-bitcointx's VerifyScript as authoritative source on input validity, because it may not be in-sync with core.  
> **username_0**: Because I thought that python-bitcointx's VerifyScript might be relied upon by someone to check input validity, I was worried that the differences in handling strict encoding might bite them. I thought that this is the only serious difference b/w this lib and the core. The goal of the question was to understand the scope of the differences to decide how to handle this. It turned out that the differences are huge, and all bets are off - so the warning in README is necessary.  

---

### Answer by username_0 on 2018-12-01. Score: 1

It was the case that python-bitcoinlib's script evaluation code did not handle SCRIPT\_VERIFY\_STRICTENC flag.

I had to create additional method, verify\_nonstrict(), and make the scripteval use it when STRICTENC flag is not specified.

Handled in <https://github.com/Simplexum/python-bitcointx/commit/d8ab8ff76de412f90480c3c68addbbb1791d3413>


---

### Answer by username_1 on 2018-12-01. Score: 1

If you want to guarantee strict DER encoded signatures which are accepted by Bitcoind then they need to be bip66 compliant(low s values(ecdsa malleability), remove leading nulls (encoding malleability)). I am not sure you are guarding for that with your decode-encode-decode cycle. These do not enforce r,s value-space constraints.

> **Comments:**
> **username_0**: low s values are enforced separately, with secp256k1_ecdsa_signature_normalize(). This is about DER encoding (https://en.wikipedia.org/wiki/X.690#DER_encoding) that openssl and bitcoind may treat differently  
> **username_1**: So you need to ensure for bip66 compliant DER encoding, which is a subset of the standard you linked above, no?  
> **username_0**: looks like the ecdsa_signature_parse_der_lax() is only relevant to the signatures before activation of BIP66, and I can just go with requiring strict DER encoding. There is some tests in that may fail as a result, but since it is irrelevant, I may just disable these tests.  
> **username_0**: Still, I do not see where secp256k1_ecdsa_signature_parse_der() is used by bitcoind, and only ecdsa_signature_parse_der_lax(). Need to investigate furter.  
> **username_1**: Peter Wuille could definitely help, he wrote bip66...  
> **username_0**: It may be the case of just correcting the tests to use special non-strict verification method. python-bitcoinlib test cases seem to ignore STRICTENC modifiers in test data.  
> **username_0**: note on my previous comment that secp256k1_ecdsa_signature_parse_der() is not used by bitcoind: it does the checking in [IsValidSignatureEncoding()](https://github.com/bitcoin/bitcoin/blob/ed12fd83ca7999a896350197533de5e9202bc2fe/src/script/interpreter.cpp#L107)  
> **username_2**: The parsing code used is always the same, before and after BIP66 activated. The difference is in an extra sanity check before the parser is invoked.  
> **username_0**: If the signature passes  IsValidSignatureEncoding(), it should always be accepted by secp256k1_ecdsa_signature_parse_der(), or there might be corner cases when such signature might not be accepted by secp256k1_ecdsa_signature_parse_der() ?  
> **username_0**: It seems that checks are slightly different b/w secp256k1_ecdsa_signature_parse_der() and  IsValidSignatureEncoding():  secp256k1_ecdsa_signature_parse_der have check for "Excessive 0xFF padding", but allows negative integers. IsValidSignatureEncoding do not allow negative integers, and checks only for excessive 0x00 padding.  
> **username_0**: I will need to use ported IsValidSignatureEncoding() to check for STRICTENC, and cannot just rely on secp256k1_ecdsa_signature_parse_der(), to be compatible with bitcoin's script interpreter.  

---

