<p dir="auto">I'm trying to import a fairly large CSV file (~300MB) with the following structure:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="id,type,internal,data
1,a,true,abc
2,b,false,def
1,a,true,ghi
3,a,true,jkl
4,c,true,mno
..."><pre class="notranslate"><code class="notranslate">id,type,internal,data
1,a,true,abc
2,b,false,def
1,a,true,ghi
3,a,true,jkl
4,c,true,mno
...
</code></pre></div>
<p dir="auto">(note: based on the ID value there might be duplicates with different values in the data column, which we are not using)</p>
<p dir="auto">I'm using the following script:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE CONSTRAINT ON (t:Thread) ASSERT t.id IS UNIQUE;
CREATE INDEX ON :Thread(type);

USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM 'file:///thread.csv' AS line
MERGE (t:Thread{id:toint(line.id)})
ON CREATE SET
  t.type = line.type,
  t.internal = case line.internal WHEN 'true' THEN true ELSE false END;"><pre class="notranslate"><code class="notranslate">CREATE CONSTRAINT ON (t:Thread) ASSERT t.id IS UNIQUE;
CREATE INDEX ON :Thread(type);

USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM 'file:///thread.csv' AS line
MERGE (t:Thread{id:toint(line.id)})
ON CREATE SET
  t.type = line.type,
  t.internal = case line.internal WHEN 'true' THEN true ELSE false END;
</code></pre></div>
<p dir="auto">For some reason no matter what I try, the periodic commit doesn't seem to work, and I usually get a java out of memory error. Also none of the inserts get commited.</p>
<p dir="auto">If I limit the csv size, for example with a <code class="notranslate">WITH line LIMIT 100000</code> then the code works fine, and I get the first 100k lines imported. The same code also works, and can import the whole file with neo4j 2.3 as well.</p>
<p dir="auto">I'm using the default docker versions for both 2.3 and 3.0 for testing</p>
<p dir="auto"><strong>Neo4j Version:</strong> neo4j docker, tag 3.0.0<br>
<strong>API:</strong> neo4j-shell, through docker exec</p>
<h3 dir="auto">Steps to reproduce</h3>
<ol dir="auto">
<li>Pull the image: <code class="notranslate">docker pull neo4j:3.0.0</code></li>
<li>Create a directory on the host that will be a mount for the data: <code class="notranslate">mkdir /home/neo4j-data</code></li>
<li>Start a new container that mounts to this directory: <code class="notranslate">docker run -d --name neo4j-test -p 7474:7474 -v /home/neo4j:/data neo4j:3.0.0</code></li>
<li>Generate a large CSV file, with the above mentioned schema</li>
<li>Copy the file and the import script over: <code class="notranslate">docker cp thread.csv neo4j:/var/lib/neo4j/import; docker cp load.cyp neo4j:/var/lib/neo4j/import</code></li>
<li>Start the import process: <code class="notranslate">docker exec -ti neo4j /var/lib/neo4j/bin/neo4j-shell -file /var/lib/neo4j/import/load.cyp</code></li>
</ol>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">The data should be imported</p>
<h3 dir="auto">Actual behavior</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR (-v for expanded information):
        Error occurred in server thread; nested exception is:
        java.lang.OutOfMemoryError: Java heap space"><pre class="notranslate"><code class="notranslate">ERROR (-v for expanded information):
        Error occurred in server thread; nested exception is:
        java.lang.OutOfMemoryError: Java heap space
</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nigelsmall/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nigelsmall">@nigelsmall</a>: 'Hope I'm not raising a duplicate issue here (can't find it anywhere else) but am still having issues with CREATE UNIQUE (née RELATE) and array properties passed through parameters.</p>
<p dir="auto">The error scenario can be recreated with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -X POST -H &quot;Content-Type: application/json&quot; -H &quot;Accept: application/json&quot; -d '{&quot;query&quot;:&quot;START a0=node(120),b0=node(121) CREATE UNIQUE a0-[r0:`LIKES` {D0}]-&gt;b0 RETURN r0&quot;,&quot;params&quot;:{&quot;D0&quot;:{&quot;colours&quot;:[&quot;red&quot;,&quot;green&quot;]}}}' http://localhost:7474/db/data/cypher"><pre class="notranslate"><code class="notranslate">curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" -d '{"query":"START a0=node(120),b0=node(121) CREATE UNIQUE a0-[r0:`LIKES` {D0}]-&gt;b0 RETURN r0","params":{"D0":{"colours":["red","green"]}}}' http://localhost:7474/db/data/cypher
</code></pre></div>
<p dir="auto">But works fine with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -X POST -H &quot;Content-Type: application/json&quot; -H &quot;Accept: application/json&quot; -d '{&quot;query&quot;:&quot;START a0=node(120),b0=node(121) CREATE UNIQUE a0-[r0:`LIKES` {D0}]-&gt;b0 RETURN r0&quot;,&quot;params&quot;:{&quot;D0&quot;:{&quot;colours&quot;:&quot;red&quot;}}}' http://localhost:7474/db/data/cypher"><pre class="notranslate"><code class="notranslate">curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" -d '{"query":"START a0=node(120),b0=node(121) CREATE UNIQUE a0-[r0:`LIKES` {D0}]-&gt;b0 RETURN r0","params":{"D0":{"colours":"red"}}}' http://localhost:7474/db/data/cypher
</code></pre></div>
<p dir="auto">The actual error returned is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;message&quot; : &quot;Unknown property type on: [red, green], class java.util.ArrayList&quot;,
  &quot;exception&quot; : &quot;IllegalArgumentException&quot;,
  &quot;stacktrace&quot; : [ &quot;org.neo4j.kernel.impl.nioneo.store.PropertyStore.encodeValue(PropertyStore.java:570)&quot;,
    .......
  ]
}"><pre class="notranslate"><code class="notranslate">{
  "message" : "Unknown property type on: [red, green], class java.util.ArrayList",
  "exception" : "IllegalArgumentException",
  "stacktrace" : [ "org.neo4j.kernel.impl.nioneo.store.PropertyStore.encodeValue(PropertyStore.java:570)",
    .......
  ]
}
</code></pre></div>
<p dir="auto">Cheers<br>
Nige'</p>