title: How to fix Gitlab CE 14.1.6 upgrade failed
body: <p>I wanted to update my GitLab CE installed on a EC2 instance following the recommendation. I went from 13.0.x to 13.9.2 -&gt; 13.12.12 -&gt; 14.0.11 -&gt; 14.1.6 as the documentation recommend.</p>
  <p>Every time I check if the server was responding. I just had to restart the process when I jump to 14.X.
  But when I went to 14.1.6 I got an error on the update :</p>
  <pre><code>[...]
  System Info:
      ------------
      chef_version=15.17.4
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client
  
  Recipe: gitlab::gitlab-rails
    * execute[clear the gitlab-rails cache] action run
      - execute /opt/gitlab/bin/gitlab-rake cache:clear
  Recipe: gitaly::enable
    * runit_service[gitaly] action restart (up to date)
    * runit_service[gitaly] action hup
      - send hup to runit_service[gitaly]
  
  Running handlers:
  There was an error running gitlab-ctl reconfigure:
  
  rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
  ---- Begin output of &quot;bash&quot;  &quot;/tmp/chef-script20211028-68712-eysjor&quot; ----
  STDOUT: rake aborted!
  StandardError: An error has occurred, all later migrations canceled:
  
  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':   {:job_class_name=&gt;&quot;CopyColumnUsingBackgroundMigrationJob&quot;, :table_name=&gt;&quot;events&quot;, :column_name=&gt;&quot;id&quot;, :job_arguments=&gt;[[&quot;id&quot;], [&quot;id_convert_to_bigint&quot;]]}
  
  Finalize it manualy by running
  
      sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[[&quot;id&quot;]\, [&quot;id_convert_to_bigint&quot;]]']
  
  For more information, check the documentation
  
      https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
  /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
  /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in &lt;top (required)&gt;'
  /opt/gitlab/embedded/bin/bundle:23:in `load'
  /opt/gitlab/embedded/bin/bundle:23:in `&lt;main&gt;'
  
  Caused by:
  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':   {:job_class_name=&gt;&quot;CopyColumnUsingBackgroundMigrationJob&quot;, :table_name=&gt;&quot;events&quot;, :column_name=&gt;&quot;id&quot;, :job_arguments=&gt;[[&quot;id&quot;], [&quot;id_convert_to_bigint&quot;]]}
  
  Finalize it manualy by running
  
      sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[[&quot;id&quot;]\, [&quot;id_convert_to_bigint&quot;]]']
  
  For more information, check the documentation
  
      https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
  /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
  /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in &lt;top (required)&gt;'
  /opt/gitlab/embedded/bin/bundle:23:in `load'
  /opt/gitlab/embedded/bin/bundle:23:in `&lt;main&gt;'
  Tasks: TOP =&gt; db:migrate
  (See full trace by running task with --trace)
  == 20210611082822 AddPagesFileEntriesToPlanLimits: migrating ==================
  -- add_column(:plan_limits, :pages_file_entries, :integer, {:default=&gt;200000, :null=&gt;false})
     -&gt; 0.0021s
  == 20210611082822 AddPagesFileEntriesToPlanLimits: migrated (0.0022s) =========
  
  == 20210622045705 FinalizeEventsBigintConversion: migrating ===================
  STDERR:
  ---- End output of &quot;bash&quot;  &quot;/tmp/chef-script20211028-68712-eysjor&quot; ----
  Ran &quot;bash&quot;  &quot;/tmp/chef-script20211028-68712-eysjor&quot; returned 1
  
  Running handlers complete
  Chef Infra Client failed. 27 resources updated in 51 seconds
  ===
  There was an error running gitlab-ctl reconfigure. Please check the output above for more
  details.
  ===
  
  dpkg: error processing package gitlab-ce (--configure):
   installed gitlab-ce package post-installation script subprocess returned error exit status 1
  Errors were encountered while processing:
   gitlab-ce
  E: Sub-process /usr/bin/dpkg returned an error code (1)
  </code></pre>
  <p>So as the output told me I tried :
  <code>sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[[&quot;id&quot;]\, [&quot;id_convert_to_bigint&quot;]]']</code></p>
  <p>and got</p>
  <pre><code>rake aborted!
  JSON::ParserError: unexpected character () at line 1, column 1 [parse.c:714]
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:87:in `rescue in adapter_load'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:82:in `adapter_load'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
  /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in &lt;top (required)&gt;'
  /opt/gitlab/embedded/bin/bundle:23:in `load'
  /opt/gitlab/embedded/bin/bundle:23:in `&lt;main&gt;'
  
  Caused by:
  Oj::ParseError: unexpected character () at line 1, column 1 [parse.c:714]
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `load'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `adapter_load'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
  /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in &lt;top (required)&gt;'
  /opt/gitlab/embedded/bin/bundle:23:in `load'
  /opt/gitlab/embedded/bin/bundle:23:in `&lt;main&gt;'
  Tasks: TOP =&gt; gitlab:background_migrations:finalize
  (See full trace by running task with --trace)
  </code></pre>
  <p>Any idea ?</p>