<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/peterneubauer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/peterneubauer">@peterneubauer</a>: 'Probably that is not right, especially if we are expecting nodes coming back in the batch API, see <a href="https://groups.google.com/forum/?fromgroups=#!topic/neo4j/KYQo5LIVoQg" rel="nofollow">https://groups.google.com/forum/?fromgroups=#!topic/neo4j/KYQo5LIVoQg</a> fro details'</p>
<p dir="auto">Sorry for the long title, feel free to rename :)</p>
<p dir="auto">The <a href="https://neo4j.com/guides/upgrade/" rel="nofollow">upgrade docs for 3.3</a> state that <code class="notranslate">3.0.any</code> -&gt; <code class="notranslate">3.3.1</code> is a supported path, but mentions no caveats. Additionally, the <a href="https://neo4j.com/guides/upgrade/" rel="nofollow">3.3 Upgrade FAQ</a> says upgrading any <code class="notranslate">3.x version will be very straightforward</code>, with no mention of caveats.</p>
<p dir="auto">However, I was unable to upgrade from 3.0.4 to 3.3.1 using the following steps and received several error messages, none of which were indicative of the underlying problem.</p>
<p dir="auto">These steps were executed within a docker environment, using the <code class="notranslate">3.0.4-enterprise</code> tag and the <code class="notranslate">3.3.1-enterprise</code> tag.</p>
<p dir="auto"><strong>Steps to reproduce</strong></p>
<ol dir="auto">
<li>In a fresh 3.0.4 enterprise container, insert some data and take a supported, online backup using <a href="https://neo4j.com/docs/operations-manual/3.0/backup/perform-backup/" rel="nofollow">these instructions</a>.</li>
<li>Duplicate that backup into 3 directories.</li>
<li>Start a causal cluster of 3 nodes, mounting the directories in step 2 into <code class="notranslate">/data</code> of each container</li>
<li>Wait for cluster members to acknowledge eachother.</li>
</ol>
<p dir="auto"><strong>Expected behavior</strong><br>
Neo4j upgrades database and runs as a causal cluster</p>
<p dir="auto"><strong>Actual behavior</strong><br>
The database fails startup with the following error message:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="...
Caused by: org.neo4j.kernel.impl.storemigration.StoreUpgrader$DatabaseNotCleanlyShutDownException: The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
    at org.neo4j.kernel.impl.storemigration.UpgradableDatabase.checkUpgradeable(UpgradableDatabase.java:124)
    at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:132)
    at org.neo4j.kernel.impl.storemigration.DatabaseMigrator.migrate(DatabaseMigrator.java:101)
    at org.neo4j.kernel.NeoStoreDataSource.upgradeStore(NeoStoreDataSource.java:573)
    at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:435)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)"><pre class="notranslate"><code class="notranslate">...
Caused by: org.neo4j.kernel.impl.storemigration.StoreUpgrader$DatabaseNotCleanlyShutDownException: The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
    at org.neo4j.kernel.impl.storemigration.UpgradableDatabase.checkUpgradeable(UpgradableDatabase.java:124)
    at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:132)
    at org.neo4j.kernel.impl.storemigration.DatabaseMigrator.migrate(DatabaseMigrator.java:101)
    at org.neo4j.kernel.NeoStoreDataSource.upgradeStore(NeoStoreDataSource.java:573)
    at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:435)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
</code></pre></div>
<p dir="auto">Sometimes, and I wasn't able to isolate the conditions, the error message would instead be something like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@f88bfbe' was successfully initialized, but failed to start. Please see the attached cause exception &quot;Unable to find transaction 23289 in any of my logical logs: Couldn't find any log containing 23289&quot;. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@f88bfbe' was successfully initialized, but failed to start. Please see the attached cause exception &quot;Unable to find transaction 23289 in any of my logical logs: Couldn't find any log containing 23289&quot;.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@f88bfbe' was successfully initialized, but failed to start. Please see the attached cause exception &quot;Unable to find transaction 23289 in any of my logical logs: Couldn't find any log containing 23289&quot;."><pre class="notranslate"><code class="notranslate">ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@f88bfbe' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to find transaction 23289 in any of my logical logs: Couldn't find any log containing 23289". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@f88bfbe' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to find transaction 23289 in any of my logical logs: Couldn't find any log containing 23289".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@f88bfbe' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to find transaction 23289 in any of my logical logs: Couldn't find any log containing 23289".
</code></pre></div>
<p dir="auto">I was able to start up the causal cluster as expected without the volume mount.</p>
<p dir="auto"><strong>Fix/Workaround</strong><br>
I was able to successfully start the causal cluster if I first upgraded the 3.0.4 backup by starting a 3.3.1 node in single instance mode and letting it perform the upgrade, then using that upgraded DB as the seed for the causal cluster.</p>
<p dir="auto">It seem like that step could be an acceptable upgrade path, but it should be well documented, and perhaps a descriptive error message should be added in the next patch release.</p>