Can not update from 18.4.1 to 18.5.x

[Basti-Fantasti]

Hi all,

I've tried to update my gitlab to 18.5.x but everytime I try to start the new stack loops at this point:

gitlab-gitlab-1      | Missing Rails.application.credentials.openid_connect_signing_key for production environment. The secret will be generated and stored in config/secrets.yml.
gitlab-gitlab-1      | Creating a backup of secrets file /home/git/gitlab/config/secrets.yml at /home/git/data/backups/secrets.yml.orig.1761199974
gitlab-gitlab-1      | 2025-10-23 08:13:01,315 WARN exited: sidekiq (exit status 1; not expected)
gitlab-gitlab-1      | rake aborted!
gitlab-gitlab-1      | NameError: uninitialized constant API::Entities::PackageVersion
gitlab-gitlab-1      | /home/git/gitlab/lib/api/entities/package.rb:53:in `<class:Package>'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/entities/package.rb:5:in `<module:Entities>'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/entities/package.rb:4:in `<module:API>'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/entities/package.rb:3:in `<top (required)>'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.7.3/lib/zeitwerk/core_ext/kernel.rb:26:in `require'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/group_packages.rb:23:in `block (2 levels) in <class:GroupPackages>'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/util/strict_hash_configuration.rb:17:in `instance_exec'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/util/strict_hash_configuration.rb:17:in `configure'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/desc.rb:70:in `desc'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/group_packages.rb:21:in `block in <class:GroupPackages>'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:118:in `instance_eval'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:118:in `block in evaluate_as_instance_with_configuration'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/util/lazy_block.rb:11:in `evaluate_from'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:125:in `evaluate_as_instance_with_configuration'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `block in nest'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `each'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `nest'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/routing.rb:175:in `block in namespace'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/settings.rb:156:in `within_namespace'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/routing.rb:172:in `namespace'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/group_packages.rb:20:in `<class:GroupPackages>'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/group_packages.rb:4:in `<module:API>'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/group_packages.rb:3:in `<top (required)>'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.7.3/lib/zeitwerk/core_ext/kernel.rb:26:in `require'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/api.rb:278:in `block (2 levels) in <class:API>'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:118:in `instance_eval'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:118:in `block in evaluate_as_instance_with_configuration'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/util/lazy_block.rb:11:in `evaluate_from'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:125:in `evaluate_as_instance_with_configuration'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `block in nest'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `each'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `nest'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/routing.rb:175:in `block in namespace'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/settings.rb:156:in `within_namespace'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/routing.rb:172:in `namespace'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/api.rb:210:in `block in <class:API>'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:118:in `instance_eval'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:118:in `block in evaluate_as_instance_with_configuration'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/util/lazy_block.rb:11:in `evaluate_from'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:125:in `evaluate_as_instance_with_configuration'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `block in nest'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `each'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:107:in `nest'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/routing.rb:175:in `block in namespace'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/settings.rb:156:in `within_namespace'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/grape-2.0.0/lib/grape/dsl/routing.rb:172:in `namespace'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/api.rb:204:in `<class:API>'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/api.rb:4:in `<module:API>'
gitlab-gitlab-1      | /home/git/gitlab/lib/api/api.rb:3:in `<top (required)>'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.7.3/lib/zeitwerk/core_ext/kernel.rb:26:in `require'
gitlab-gitlab-1      | /home/git/gitlab/config/routes/api.rb:7:in `draw_route'
gitlab-gitlab-1      | /home/git/gitlab/lib/gitlab/patch/draw_route.rb:30:in `instance_eval'
gitlab-gitlab-1      | /home/git/gitlab/lib/gitlab/patch/draw_route.rb:30:in `draw_route'
gitlab-gitlab-1      | /home/git/gitlab/lib/gitlab/patch/draw_route.rb:17:in `draw_ce'
gitlab-gitlab-1      | /home/git/gitlab/lib/gitlab/patch/draw_route.rb:11:in `draw'
gitlab-gitlab-1      | /home/git/gitlab/config/routes.rb:312:in `block (2 levels) in <top (required)>'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/actionpack-7.1.5.2/lib/action_dispatch/routing/route_set.rb:448:in `instance_exec'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/actionpack-7.1.5.2/lib/action_dispatch/routing/route_set.rb:448:in `eval_block'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/actionpack-7.1.5.2/lib/action_dispatch/routing/route_set.rb:430:in `draw'
gitlab-gitlab-1      | /home/git/gitlab/config/routes.rb:7:in `block in <top (required)>'
gitlab-gitlab-1      | /home/git/gitlab/lib/initializer_connections.rb:20:in `block in raise_if_new_database_connection'
gitlab-gitlab-1      | /home/git/gitlab/lib/initializer_connections.rb:34:in `debug_database_queries'
gitlab-gitlab-1      | /home/git/gitlab/lib/initializer_connections.rb:19:in `raise_if_new_database_connection'
gitlab-gitlab-1      | /home/git/gitlab/config/routes.rb:6:in `<top (required)>'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/routes_reloader.rb:50:in `load'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/routes_reloader.rb:50:in `block in load_paths'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/routes_reloader.rb:50:in `each'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/routes_reloader.rb:50:in `load_paths'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/routes_reloader.rb:24:in `reload!'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/routes_reloader.rb:38:in `block in updater'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.1.5.2/lib/active_support/file_update_checker.rb:85:in `execute'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/routes_reloader.rb:13:in `execute'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application/finisher.rb:161:in `block in <module:Finisher>'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/initializable.rb:32:in `instance_exec'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/initializable.rb:32:in `run'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/initializable.rb:61:in `block in run_initializers'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/initializable.rb:50:in `each'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/initializable.rb:50:in `tsort_each_child'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/initializable.rb:60:in `run_initializers'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application.rb:426:in `initialize!'
gitlab-gitlab-1      | /home/git/gitlab/config/environment.rb:7:in `<top (required)>'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.7.3/lib/zeitwerk/core_ext/kernel.rb:34:in `require'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application.rb:402:in `require_environment!'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/railties-7.1.5.2/lib/rails/application.rb:591:in `block in run_tasks_blocks'
gitlab-gitlab-1      | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
gitlab-gitlab-1      | /usr/local/bin/bundle:25:in `<main>'
gitlab-gitlab-1      | Tasks: TOP => db:migrate => db:load_config => environment
gitlab-gitlab-1      | (See full trace by running task with --trace)
gitlab-gitlab-1 exited with code 0
gitlab-gitlab-1      | Loading /etc/docker-gitlab/runtime/env-defaults
gitlab-gitlab-1      | Initializing logdir...
gitlab-gitlab-1      | Initializing datadir...

and then it starts over...

Do I manually need to trigger some db migration steps manually or what could cause this behaviour.

[Basti-Fantasti]

I've made now a smaller step from 18.4.1 to 18.4.2 and the Missing Rails error is coming there up as well but seems not to be the problem...

Additional info: I'm on postgres 16 as being the default for the latest 18.x releases

[kkimurak]

Could not reproduced but found similar issue on upstream : https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/9408

[Basti-Fantasti]

Thanks for the link.
Yes this exactly seems to be the problem and it seems as if they are still unsure what could cause this bug.. I'll keep an eye on that issue over at gitlab.com

[Basti-Fantasti]

Ah they're working on it. They just pushed a MR Link
It seems to be related to an update Zeitwerk dependency which was now downgraded again.
So maybe it will be fixed in the next bugfix release...