Question: Given an arbitrary (valid!) private or public key encoded inside of a PEM, with the `pre-encapsulation boundary` and `post-encapsulation boundaries` intact, is it possible to know exactly what format the bytes take (i.e. are they OpenSSL traditional, PKCS8, X.509 `SubjectPublicKeyInfo`, etc.), or does one need some a priori information to properly decode them?

Answer: 
This is the problem

```
int X[X_len][4];

```

With X\_len greater than 1000000 you do not have enough memory for the *(stack)* array.

Try dynamic allocation *(heap)* instead

```
int (*X)[4];
X = malloc(X_len * sizeof *X);
if (X == NULL) /* error */;
// ...
free(X);

```
I'd say the problem is here:

```
 int X[X_len][4];

```

this allocates memory **on stack** which is probably only 4MB, that would explain failing after 1 000 000 lines.

I suggest allocating it on heap:

```
int *X = (int*)malloc(X_len * sizeof(int) * 4);

```