<p dir="auto">We convert our RDF data set to Nodes and Relationships. And then we use <code class="notranslate">neo4j-import</code> to import this data set.<br>
Since we have change the <em>JVM_size</em> and <em>dbms.memory.heap.initial_size</em> and <em>dbms.memory.heap.max_size</em> in <code class="notranslate">neo4j.conf</code>, and the available resources output is shown below:</p>
<blockquote>
<p dir="auto">Total machine memory: 125.52 GB<br>
Free machine memory: 122.06 GB<br>
Max heap memory: 26.67 GB<br>
Processors: 32<br>
Configured max memory: 85.86 GB</p>
</blockquote>
<p dir="auto">However, when it comes to <em>Relationship counts</em>, <strong>import failed</strong> because of the following Error:</p>
<blockquote>
<p dir="auto">Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded</p>
</blockquote>
<p dir="auto">We know it's about Garbage Collection and Heap Memory, and we try our methods to overcome this error but failed. Also, when the import is running, the memory used is full all the time.</p>
<p dir="auto"><strong>So, how could we overcome this problem? Or neo4j with 128GB memory can't import so much a data set?</strong></p>
<ul dir="auto">
<li>Neo4j version: Docker container neo4j:3.3.2</li>
<li>Operating system: Container-Optimized OS on GCP (based on Chromium OS)</li>
</ul>
<p dir="auto">Hello,<br>
I have a daily database dump exported via the following command:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="docker run \
--publish=7473:7473 \
--publish=7687:7687 \
--volume=$HOME/neo4j/data:/data \
--volume=$HOME/neo4j/logs:/logs \
--volume=$HOME/neo4j/conf:/conf \
--volume=$HOME/neo4j/plugins:/plugins \
--volume=$HOME/neo4j/import:/import \
--volume=$HOME/neo4j/scripts:/scripts \
--volume=$HOME/neo4j/ssl:/ssl \
--volume=$TEMP/graph.db:/graph.db \
--ulimit=nofile=40000:40000 \
--name=neo4j-dump \
-id \
neo4j:3.3.2 \
/bin/bash -c &quot;bin/neo4j-admin dump --database=graph.db --to=/graph.db/$TAG.dump&quot;"><pre class="notranslate"><code class="notranslate">docker run \
--publish=7473:7473 \
--publish=7687:7687 \
--volume=$HOME/neo4j/data:/data \
--volume=$HOME/neo4j/logs:/logs \
--volume=$HOME/neo4j/conf:/conf \
--volume=$HOME/neo4j/plugins:/plugins \
--volume=$HOME/neo4j/import:/import \
--volume=$HOME/neo4j/scripts:/scripts \
--volume=$HOME/neo4j/ssl:/ssl \
--volume=$TEMP/graph.db:/graph.db \
--ulimit=nofile=40000:40000 \
--name=neo4j-dump \
-id \
neo4j:3.3.2 \
/bin/bash -c "bin/neo4j-admin dump --database=graph.db --to=/graph.db/$TAG.dump"
</code></pre></div>
<p dir="auto">And when I try to load it in another instance (same version of the container) with the following command:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="docker run \
--publish=7474:7474 \
--publish=7473:7473 \
--publish=7687:7687 \
--volume=$HOME/neo4j/data:/data \
--volume=$HOME/neo4j/logs:/logs \
--volume=$HOME/neo4j/ssl:/ssl \
--volume=$HOME/neo4j/backups:/backups \
--volume=$HOME/neo4j/conf:/conf \
--ulimit=nofile=40000:40000 \
--env-file=$HOME/neo4j.env \
--name=neo4j-admin \
-it \
neo4j:3.3.2 \
/bin/bash -c &quot;bin/neo4j-admin load --from=/backups/graph.db/$TAG.dump --database=graph.db --force=true&quot;"><pre class="notranslate"><code class="notranslate">docker run \
--publish=7474:7474 \
--publish=7473:7473 \
--publish=7687:7687 \
--volume=$HOME/neo4j/data:/data \
--volume=$HOME/neo4j/logs:/logs \
--volume=$HOME/neo4j/ssl:/ssl \
--volume=$HOME/neo4j/backups:/backups \
--volume=$HOME/neo4j/conf:/conf \
--ulimit=nofile=40000:40000 \
--env-file=$HOME/neo4j.env \
--name=neo4j-admin \
-it \
neo4j:3.3.2 \
/bin/bash -c "bin/neo4j-admin load --from=/backups/graph.db/$TAG.dump --database=graph.db --force=true"
</code></pre></div>
<p dir="auto">I systematically get the following error : <code class="notranslate">command failed: unable to load database: EOFException: null</code></p>
<p dir="auto">What could be the issue? How can I test if my daily backup dump is corrupted?<br>
I can still duplicate my database by copy/pasting the graph.db folder but it is not recommended so I'd rather use the dump and load method...</p>
<p dir="auto">Thanks</p>