<p>Blocks are never split across files: the whole block, from magic to length, to header, to transaction data, is always stored in a single file, continuously.</p>
<p>However you may need to take into account:</p>
<ul>
<li>Blocks in the blk*.dat files are stored in the order they are received from the network.l, which may not be the blocks' logical order.</li>
<li>It is possible that a block is in file N, and its descendant block is in file N-1.</li>
<li>The files contain all blocks ever received, including ones that are not part of the main chain but were reorged out.</li>
<li>There can be &quot;gaps&quot; of zeroes or garbage data between blocks in the blk*.dat files.</li>
</ul>

CC BY-SA 4.0
2022-08-07T03:53:27.410Z
Thanks for your answer! Just one follow up, is there anyway we can pre-sort the blk files in anyway ? Because I'd like to process them into a CSV on individual nodes on a distributed environment and then find out the input address by looking backwards if that makes sense
Rohan Maheshwari
CC BY-SA 4.0
2022-08-07T12:12:57.107Z
Bitcoin Core has a linearize script: https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize. This can be used to construct a single file with all blocks in the active chain (no reorged forks), in order, without gaps.
Pieter Wuille
CC BY-SA 4.0
2022-08-05T12:00:07.243Z
2022-08-05T12:20:48.460Z
2022-08-05T12:20:48.460Z
Pieter Wuille
<p>As a kind of addendum to the first bullet-point in <a href="https://bitcoin.stackexchange.com/a/114743/13866">Pieter's answer</a>, I had written a program to parse <code>blkxxxxx.dat</code> files and, in my case at least, the blocks are very obviously <em>not</em> in logical order. Just as Pieter asserted.</p>
<p>Notice the day number in the timestamps in the first ten blocks:</p>
<pre><code>C&gt; blockchain -file blk02100.dat -blocks -longstats
Block      1, 2020-06-03 03:48, target 171297F6,   2273 transactions
Block      2, 2020-06-03 04:29, target 171297F6,   2768 transactions
Block      3, 2020-06-04 02:00, target 171297F6,   2177 transactions
Block      4, 2020-06-05 12:37, target 17147F35,   2243 transactions
Block      5, 2020-06-01 07:15, target 171297F6,   3150 transactions
Block      6, 2020-06-02 21:48, target 171297F6,   2911 transactions
Block      7, 2020-06-02 21:49, target 171297F6,   2722 transactions
Block      8, 2020-06-05 16:23, target 17147F35,    182 transactions
Block      9, 2020-06-03 05:34, target 171297F6,   2002 transactions
Block     10, 2020-06-03 04:36, target 171297F6,   1958 transactions
...
Block    111, 2020-06-06 03:39, target 17147F35,    259 transactions

Statistics for blk02100.dat (2020-06-01 to 2020-06-06)

   111 blocks
222519 transactions (2004.7 per block)
109173 Segwit tx    (49.1%)
622026 inputs       (mean 2.8, max 1030 per transaction)
614553 outputs      (mean 2.8, max 3799 per transaction)

Processed file in 322.5756ms
</code></pre>
<p>I guess this scattered order is because my node was receiving blocks from multiple peers and concurrently receiving them from fairly disjoint ranges.</p>
<p>I can confirm that every <code>blkxxxxx.dat</code> file I tried, started and ended at a block boundary. There were no partial blocks.</p>
<hr />
<p>Footnote: My program isn't extensively tested so maybe take the above with a pinch of salt. I believe the timestamps to be parsed correctly.</p>

CC BY-SA 4.0
2022-08-07T03:54:56.467Z
Yes ! I have found the same to be true and the python parser i am currently using [parser](https://github.com/alecalve/python-bitcoin-blockchain-parser) also has the option to obtain blocks in an ordered/ unordered form.
Rohan Maheshwari
CC BY-SA 4.0
2022-08-06T19:56:09.457Z
2022-08-06T20:45:49.420Z
2022-08-06T20:45:49.420Z
RedGrittyBrick
<p>I am trying to parse out the bitcoin data using Big Data and wanted to know more about the blk files themselves. I was wondering how does bitcoin split the transactions across the files. I know there is a 128 MiB limit on the files. Does that mean it is possible for transaction data to be split across a blk ? For instance the merkle root of the transaction is in one file and the output index is in another ? Or is block data retained in a single file ?</p>

<bitcoin-core><block><blockchain-data-storage>
Splitting of data across blkxxxxx.dat files
Rohan Maheshwari
# Splitting of data across blkxxxxx.dat files

(Asked by: username_0 on 2022-08-05)

I am trying to parse out the bitcoin data using Big Data and wanted to know more about the blk files themselves. I was wondering how does bitcoin split the transactions across the files. I know there is a 128 MiB limit on the files. Does that mean it is possible for transaction data to be split across a blk ? For instance the merkle root of the transaction is in one file and the output index is in another ? Or is block data retained in a single file ?

**Tags:** bitcoin-core, block, blockchain-data-storage


---

## 2 answers

### Answer by username_1 on 2022-08-05. Score: 3 | Accepted answer

Blocks are never split across files: the whole block, from magic to length, to header, to transaction data, is always stored in a single file, continuously.

However you may need to take into account:

* Blocks in the blk\*.dat files are stored in the order they are received from the network.l, which may not be the blocks' logical order.
* It is possible that a block is in file N, and its descendant block is in file N-1.
* The files contain all blocks ever received, including ones that are not part of the main chain but were reorged out.
* There can be "gaps" of zeroes or garbage data between blocks in the blk\*.dat files.

> **Comments:**
> **username_0**: Thanks for your answer! Just one follow up, is there anyway we can pre-sort the blk files in anyway ? Because I'd like to process them into a CSV on individual nodes on a distributed environment and then find out the input address by looking backwards if that makes sense  
> **username_1**: Bitcoin Core has a linearize script: https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize. This can be used to construct a single file with all blocks in the active chain (no reorged forks), in order, without gaps.  

---

### Answer by username_2 on 2022-08-06. Score: 0

As a kind of addendum to the first bullet-point in Pieter's answer, I had written a program to parse blkxxxxx.dat files and, in my case at least, the blocks are very obviously not in logical order. Just as Pieter asserted.
Notice the day number in the timestamps in the first ten blocks:
C> blockchain -file blk02100.dat -blocks -longstats
Block      1, 2020-06-03 03:48, target 171297F6,   2273 transactions
Block      2, 2020-06-03 04:29, target 171297F6,   2768 transactions
Block      3, 2020-06-04 02:00, target 171297F6,   2177 transactions
Block      4, 2020-06-05 12:37, target 17147F35,   2243 transactions
Block      5, 2020-06-01 07:15, target 171297F6,   3150 transactions
Block      6, 2020-06-02 21:48, target 171297F6,   2911 transactions
Block      7, 2020-06-02 21:49, target 171297F6,   2722 transactions
Block      8, 2020-06-05 16:23, target 17147F35,    182 transactions
Block      9, 2020-06-03 05:34, target 171297F6,   2002 transactions
Block     10, 2020-06-03 04:36, target 171297F6,   1958 transactions
...
Block    111, 2020-06-06 03:39, target 17147F35,    259 transactions

Statistics for blk02100.dat (2020-06-01 to 2020-06-06)

   111 blocks
222519 transactions (2004.7 per block)
109173 Segwit tx    (49.1%)
622026 inputs       (mean 2.8, max 1030 per transaction)
614553 outputs      (mean 2.8, max 3799 per transaction)

Processed file in 322.5756ms

I guess this scattered order is because my node was receiving blocks from multiple peers and concurrently receiving them from fairly disjoint ranges.
I can confirm that every blkxxxxx.dat file I tried, started and ended at a block boundary. There were no partial blocks.

Footnote: My program isn't extensively tested so maybe take the above with a pinch of salt. I believe the timestamps to be parsed correctly.

> **Comments:**
> **username_0**: Yes ! I have found the same to be true and the python parser i am currently using [parser](https://github.com/alecalve/python-bitcoin-blockchain-parser) also has the option to obtain blocks in an ordered/ unordered form.  

---

