mission_control-jobs fails with new Rails 8.0.0.beta1 #173

kbachl · 2024-10-09T09:49:11Z

Create new rails app with 8.0.0.beta1, add dummy job;
Add gem "solid_queue", ">= 1.0.0"
Mount under e.g. /jobs -> mount MissionControl::Jobs::Engine, at: "/jobs"
Open up /jobs

Error comes:

MissionControl::Jobs::Errors::IncompatibleAdapter in MissionControl::Jobs::QueuesController#index
Adapter Async must implement queue_names

private
def raise_incompatible_adapter_error_from(method_name)
raise MissionControl::Jobs::Errors::IncompatibleAdapter, "Adapter #{ActiveJob.adapter_name(self)} must implement #{method_name}"
end
end

mission_control-jobs (0.3.2) lib/mission_control/jobs/adapter.rb:154:in raise_incompatible_adapter_error_from' mission_control-jobs (0.3.2) lib/mission_control/jobs/adapter.rb:91:in queues'
mission_control-jobs (0.3.2) lib/active_job/querying.rb:24:in `fetch_queues'

The text was updated successfully, but these errors were encountered:

Roupiye · 2024-10-10T01:05:42Z

having the same problem

Roupiye · 2024-10-10T01:06:20Z

but i am using rails 7.2.1

Funcke · 2024-10-10T17:42:30Z

In config/envornments/development.rb what's your config.active_job.queue_adapter value? If it's not set it will use :async.
If you are setting it manually like this
config.active_job.queue_adapter = :solid_queue
It should work.
Additionally, make sure you have configured the database connection you want to use for your queue. In rails 8.0.0.beta1 it could look something like this:

# database.yml
default: &default
  adapter: sqlite3
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  timeout: 5000

development:
  primary:
    <<: *default
    database: storage/development.sqlite3
  cache:
    <<: *default
    database: storage/development_cache.sqlite3
    migrations_paths: db/cache_migrate
  queue:
    <<: *default
    database: storage/development_queue.sqlite3
    migrations_paths: db/queue_migrate
  cable:
    <<: *default
    database: storage/development_cable.sqlite3
    migrations_paths: db/cable_migrate

And for defining the correct connection you would need to adapt your config/environments/development.rb by adding:
config.solid_queue.connects_to = { database: { writing: :queue, reading: :queue } }

Funcke · 2024-10-10T17:57:18Z

Also, quick edit:
in rails 8.0.0.beta1 you have to migrate the solid_queue migrations on the connection you have dedicated it for manually by running bin/rails db:migrate:<connection_name> in the case above it would be bin/rails db:migrate:queue.

kbachl · 2024-10-10T20:22:16Z

Hi,

I'm a bit puzzled at the moment. I indeed did forget to add
config.active_job.queue_adapter = :solid_queue
to development.rb.

The database.yml itself looks plain, as it was created by rails 8 beta 1:

default: &default
  adapter: sqlite3
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  timeout: 5000

development:
  <<: *default
  database: storage/development.sqlite3

My jobs also work and run when I call them. However, even I did a bin/rails db:migrate command, I now get an error saying:

ActiveRecord::StatementInvalid in MissionControl::Jobs::QueuesController#index
Could not find table 'solid_queue_jobs'

kbachl · 2024-10-10T20:29:01Z

PS: after adding config.active_job.queue_adapter = :solid_queue to development.rb my jobs also fail with same error

It seems that rails beta1 somehow makes solid-queue a first citizen but doesnt configure it enough to be ready to use even it seems so partially... need to look deeper into it

Funcke · 2024-10-11T00:47:35Z

Hi,
the second error indicates that the solid queue tables have not been created in the database.

Make sure you have set up a solid queue database for development that uses the data structure provided in db/queue_migrations.rb as mentioned in the example above.

Since the generated development configuration in a new rails 8.0.0.beta1 app uses the :async adapter for solid_queue by default, a corresponding database connection supporting the solid_queue schema has to be provided manually, as well as running the migrations.

mabras · 2024-10-11T10:35:25Z

My jobs also work and run when I call them. However, even I did a bin/rails db:migrate command, I now get an error saying:

ActiveRecord::StatementInvalid in MissionControl::Jobs::QueuesController#index
Could not find table 'solid_queue_jobs'

# config/environments/development.rb
config.solid_queue.connects_to = { database: { writing: :queue } }

Funcke · 2024-10-11T15:06:24Z

Hi @mabras,
What‘s your corresponding database configuration and how did you executw the expected job?

kbachl · 2024-10-17T09:51:37Z

Ok, so I did a "rails solid_queue:install" and it creates a "queu_schema.rb" that has much in it, e.g.:

ActiveRecord::Schema[7.1].define(version: 1) do
  create_table "solid_queue_blocked_executions", force: :cascade do |t|
    t.bigint "job_id", null: false
    t.string "queue_name", null: false
    t.integer "priority", default: 0, null: false
    t.string "concurrency_key", null: false
    t.datetime "expires_at", null: false
    t.datetime "created_at", null: false
    t.index [ "concurrency_key", "priority", "job_id" ], name: "index_solid_queue_blocked_executions_for_release"
    t.index [ "expires_at", "concurrency_key" ], name: "index_solid_queue_blocked_executions_for_maintenance"
    t.index [ "job_id" ], name: "index_solid_queue_blocked_executions_on_job_id", unique: true
  end

  create_table "solid_queue_claimed_executions", force: :cascade do |t|
    t.bigint "job_id", null: false
    t.bigint "process_id"
    t.datetime "created_at", null: false
    t.index [ "job_id" ], name: "index_solid_queue_claimed_executions_on_job_id", unique: true
    t.index [ "process_id", "job_id" ], name: "index_solid_queue_claimed_executions_on_process_id_and_job_id"
  end

...
  add_foreign_key "solid_queue_ready_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
  add_foreign_key "solid_queue_recurring_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
  add_foreign_key "solid_queue_scheduled_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
end

However, when I then run rails db:migrate:queue it gets replaced by this:

# This file is auto-generated from the current state of the database. Instead
# of editing this file, please use the migrations feature of Active Record to
# incrementally modify your database, and then regenerate this schema definition.
#
# This file is the source Rails uses to define your schema when running `bin/rails
# db:schema:load`. When creating a new database, `bin/rails db:schema:load` tends to
# be faster and is potentially less error prone than running all of your
# migrations from scratch. Old migrations may fail to apply correctly if those
# migrations use external dependencies or application code.
#
# It's strongly recommended that you check this file into your version control system.

ActiveRecord::Schema[8.0].define(version: 0) do
end

And nothing in the queue database changes at all. What exactly goes wrong here? Am I supposed to copy that over to a migration file?

kbachl · 2024-10-17T10:02:52Z

PS: i also tried a

rails db:schema:load:queue --trace

** Invoke db:schema:load:queue (first_time)
** Invoke db:test:purge:queue (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke db:check_protected_environments (first_time)
** Invoke db:load_config 
** Execute db:check_protected_environments
** Execute db:test:purge:queue
bin/rails aborted!
TypeError: Invalid type for configuration. Expected Symbol, String, or Hash. Got nil (TypeError)

        raise TypeError, "Invalid type for configuration. Expected Symbol, String, or Hash. Got #{config.inspect}"
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/user/.asdf/installs/ruby/3.3.5/lib/ruby/gems/3.3.0/gems/activerecord-8.0.0.beta1/lib/active_record/database_configurations.rb:183:in `resolve'

kbachl · 2024-10-17T10:18:55Z

OK! I got it working on rails 8 Beta1!

The step is simply: DONT USE rails solid_queue:install - but instead add the part of the production.rb to your development.rb:

config.active_job.queue_adapter = :solid_queue
config.solid_queue.connects_to = { database: { writing: :queue } }

Then delete all previous rails queue DBs and config it in your database.yml that way:

development:
  primary:
    <<: *default
    database: storage/development.sqlite3
  queue:
    <<: *default
    database: storage/development_queue.sqlite3
    migrations_paths: db/queue_migrate

after this make sure you got the original "queue_schema.rb" file from rails 8!

Then do a rails db:prepare , NOT A rails db:migrate:queue as this will create an empty schema.rb file instead!
Then after the db prepare has run you can use the db:migrate as it then will recreate it from the datase itself.

Quite an annoying path to take to have this working in dev...

rosa · 2024-10-17T11:14:19Z

@kbachl, this behaviour is caused by a bug in Rails that has already been fixed: rails/rails#52829. In general you don't need to go through all that trouble to get Solid Queue working in dev.

As for the error with the async adapter, yes, that's pending: #32

kbachl · 2024-10-17T15:50:03Z

@rosa thanks for the info!
I cant believe I hit that bug...

igrigorik · 2024-10-23T05:56:34Z

Hit the same issue. @kbachl's fix worked — ty.

@rosa what's the best to unwind and reset? Would prefer not to carry this forward in existing project.

rameerez · 2024-10-25T00:58:09Z

I got the same error running Rails 8.0.0.rc1 – @kbachl's fix worked, thanks a lot!

I agree this is quite cumbersome and counterintuitive to set up to have it working on dev, I'd expect Rails to come with good defaults so that solid_queue would just work out of the box in dev.

kbachl · 2024-10-25T06:14:09Z

@rosa Sorry to bother you again, but even after I upgraded to Rails 8.0.0.rc1 I had the bug that I descriped above, meaning doing a db:migrate killed somehow my solid queue database?

rosa · 2024-10-29T17:26:41Z

what's the best to unwind and reset? Would prefer not to carry this forward in existing project.

@kbachl, @rameerez, @igrigorik: have you try running bin/rails db:prepare rather than bin/rails db:migrate:queue, after running bin/rails solid_queue:install and configuring Solid Queue in development (like here)? That should work just fine, and it follows Solid Queue's installation instructions.

but even after I upgraded to Rails 8.0.0.rc1 I had the bug that I descriped above, meaning doing a db:migrate killed somehow my solid queue database?

It seems that's a regression or a different version of rails/rails#52829. But it sounds like a bug in Rails in any case.

rosa · 2024-11-05T21:10:56Z

Going to close this one as we have #32 for the async adapter issue.

PedroAugustoRamalhoDuarte · 2024-11-23T11:51:09Z

OK! I got it working on rails 8 Beta1!

The step is simply: DONT USE rails solid_queue:install - but instead add the part of the production.rb to your development.rb:
config.active_job.queue_adapter = :solid_queue
config.solid_queue.connects_to = { database: { writing: :queue } }
Then delete all previous rails queue DBs and config it in your database.yml that way:
development:
  primary:
    <<: *default
    database: storage/development.sqlite3
  queue:
    <<: *default
    database: storage/development_queue.sqlite3
    migrations_paths: db/queue_migrate
after this make sure you got the original "queue_schema.rb" file from rails 8!

Then do a rails db:prepare , NOT A rails db:migrate:queue as this will create an empty schema.rb file instead! Then after the db prepare has run you can use the db:migrate as it then will recreate it from the datase itself.

Quite an annoying path to take to have this working in dev...

Thanks @kbachl , should we add this instructions on README.md?

aquaflamingo · 2025-01-05T01:40:55Z

Sorry to revive an old thread, I'm trying with a fresh project on Rails 8.0.1 and these steps don't seem work anymore sadly.

rails db:prepare

Only generate the development database, and doesn't actually generate the solid_queue database.

It seems, moreover, that rails db:schema:queue might have been removed from task list?

What Else I Tried

I tried to manually create a db/queue_migrate/queue_schema.rb and run migrations, no luck
I renamed db/queue_migrate/schema.rb and tried - still no luck
Running ./bin/jobs start will produce

bin/jobs start

.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/database_configurations.rb:229:in `resolve_symbol_connection': The `queue` database is not configured for the `development` environment. (ActiveRecord::AdapterNotSpecified)

  Available database configurations are:

  default
development
test
production: primary, cache, queue, cable

from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/database_configurations.rb:179:in `resolve'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:391:in `resolve_config_for_connection'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:100:in `block (2 levels) in connects_to'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:99:in `each'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:99:in `block in connects_to'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:98:in `each'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activerecord-8.0.1/lib/active_record/connection_handling.rb:98:in `connects_to'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/solid_queue-1.1.2/app/models/solid_queue/record.rb:7:in `<class:Record>'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/solid_queue-1.1.2/app/models/solid_queue/record.rb:4:in `<module:SolidQueue>'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/solid_queue-1.1.2/app/models/solid_queue/record.rb:3:in `<main>'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/3.3.0/bundled_gems.rb:74:in `require'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/3.3.0/bundled_gems.rb:74:in `block (2 levels) in replace_require'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/zeitwerk-2.7.1/lib/zeitwerk/core_ext/kernel.rb:26:in `require'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/solid_queue-1.1.2/lib/solid_queue/configuration.rb:79:in `ensure_correctly_sized_thread_pool'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:361:in `block in make_lambda'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:178:in `block in call'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:667:in `block (2 levels) in default_terminator'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:666:in `catch'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:666:in `block in default_terminator'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:179:in `call'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:558:in `block in invoke_before'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:558:in `each'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:558:in `invoke_before'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:108:in `run_callbacks'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:912:in `_run_validate_callbacks'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activemodel-8.0.1/lib/active_model/validations.rb:474:in `run_validations!'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/activemodel-8.0.1/lib/active_model/validations.rb:365:in `valid?'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/solid_queue-1.1.2/lib/solid_queue/supervisor.rb:13:in `start'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/solid_queue-1.1.2/lib/solid_queue/cli.rb:26:in `start'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/command.rb:28:in `run'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/invocation.rb:127:in `invoke_command'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor.rb:538:in `dispatch'
        from /Users/robert/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/thor-1.3.2/lib/thor/base.rb:584:in `start'
        from bin/jobs:6:in `<main>'

Seems the error arises here in solid_queue, and later in Active Record.

rosa · 2025-01-05T10:02:54Z

Hey @aquaflamingo, have you followed the steps to configure Solid Queue in development?

aquaflamingo · 2025-01-05T22:49:04Z

Hi @rosa, yes performed all those steps, the only step I wasn't clear about was:

# You can either set the env var, or check for development
plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"] || Rails.env.development?

Since it says you can use the Puma plugin or "You can also just use bin/jobs".

Trying bin/jobs failed, and I couldn't get the database created for queue at all.

aquaflamingo · 2025-01-05T22:55:16Z

🤦 Ah I finally got it.

The issue is that in the default rails boilerplate database.yml there is NO primary key, I missed that in the README configuration.

development:
  <<: *default
  database: storage/development.sqlite3

You must make sure that you add the primary key, if you don't it looks like Rails just ignores everything else:

development:
  primary:
    <<: *default
    database: storage/development.sqlite3
+  queue:
+    <<: *default
+    database: storage/development_queue.sqlite3
+    migrations_paths: db/queue_migrate

Would it make sense for me to update the docs to add another + for "primary"? Example:

development:
+ primary:
    <<: *default
    database: storage/development.sqlite3
+  queue:
+    <<: *default
+    database: storage/development_queue.sqlite3
+    migrations_paths: db/queue_migrate

For posterity, here's a fresh setup for folks: https://www.youtube.com/watch?v=KSh-dcnTvUg

PR here if you want to merge or decline Rosa: rails/solid_queue#477

rosa closed this as completed Nov 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mission_control-jobs fails with new Rails 8.0.0.beta1 #173

mission_control-jobs fails with new Rails 8.0.0.beta1 #173

kbachl commented Oct 9, 2024

Roupiye commented Oct 10, 2024

Roupiye commented Oct 10, 2024

Funcke commented Oct 10, 2024 •

edited

Loading

Funcke commented Oct 10, 2024

kbachl commented Oct 10, 2024

kbachl commented Oct 10, 2024

Funcke commented Oct 11, 2024

mabras commented Oct 11, 2024

Funcke commented Oct 11, 2024

kbachl commented Oct 17, 2024

kbachl commented Oct 17, 2024

kbachl commented Oct 17, 2024

rosa commented Oct 17, 2024

kbachl commented Oct 17, 2024

igrigorik commented Oct 23, 2024

rameerez commented Oct 25, 2024

kbachl commented Oct 25, 2024

rosa commented Oct 29, 2024 •

edited

Loading

rosa commented Nov 5, 2024

PedroAugustoRamalhoDuarte commented Nov 23, 2024

aquaflamingo commented Jan 5, 2025

rosa commented Jan 5, 2025

aquaflamingo commented Jan 5, 2025

aquaflamingo commented Jan 5, 2025 •

edited

Loading

mission_control-jobs fails with new Rails 8.0.0.beta1 #173

mission_control-jobs fails with new Rails 8.0.0.beta1 #173

Comments

kbachl commented Oct 9, 2024

Roupiye commented Oct 10, 2024

Roupiye commented Oct 10, 2024

Funcke commented Oct 10, 2024 • edited Loading

Funcke commented Oct 10, 2024

kbachl commented Oct 10, 2024

kbachl commented Oct 10, 2024

Funcke commented Oct 11, 2024

mabras commented Oct 11, 2024

Funcke commented Oct 11, 2024

kbachl commented Oct 17, 2024

kbachl commented Oct 17, 2024

kbachl commented Oct 17, 2024

rosa commented Oct 17, 2024

kbachl commented Oct 17, 2024

igrigorik commented Oct 23, 2024

rameerez commented Oct 25, 2024

kbachl commented Oct 25, 2024

rosa commented Oct 29, 2024 • edited Loading

rosa commented Nov 5, 2024

PedroAugustoRamalhoDuarte commented Nov 23, 2024

aquaflamingo commented Jan 5, 2025

What Else I Tried

rosa commented Jan 5, 2025

aquaflamingo commented Jan 5, 2025

aquaflamingo commented Jan 5, 2025 • edited Loading

Funcke commented Oct 10, 2024 •

edited

Loading

rosa commented Oct 29, 2024 •

edited

Loading

aquaflamingo commented Jan 5, 2025 •

edited

Loading