Redis::CannotConnectError: No sentinels available.

[palexvs]

Hello, I'm trying to setup sentinels, but i get error Redis::CannotConnectError: No sentinels available:

> redis = Redis.new(url: "redis://master.redis.com:6379/0", sentinels: [{host: "127.0.0.1", port: 26379}], master_name: :master)
> redis.zcount(1,0,'+inf')

Error:

Redis::CannotConnectError: No sentinels available.
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:551:in `sentinel_detect'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:555:in `resolve_master'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:519:in `resolve'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:317:in `establish_connection'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:94:in `block in connect'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:279:in `with_reconnect'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:93:in `connect'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:350:in `ensure_connected'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:207:in `block in process'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:292:in `logging'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:206:in `process'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:112:in `call'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/newrelic-redis-2.0.2/lib/newrelic_redis/instrumentation.rb:36:in `block in call_with_newrelic_trace'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/newrelic_rpm-3.12.0.288/lib/new_relic/agent/datastores.rb:111:in `block in wrap'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/newrelic_rpm-3.12.0.288/lib/new_relic/agent/method_tracer.rb:69:in `block in trace_execution_scoped'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/newrelic_rpm-3.12.0.288/lib/new_relic/agent/method_tracer_helpers.rb:82:in `trace_execution_scoped'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/newrelic_rpm-3.12.0.288/lib/new_relic/agent/method_tracer.rb:67:in `trace_execution_scoped'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/newrelic_rpm-3.12.0.288/lib/new_relic/agent/datastores.rb:108:in `wrap'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/newrelic-redis-2.0.2/lib/newrelic_redis/instrumentation.rb:35:in `call_with_newrelic_trace'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis.rb:1769:in `block in zcount'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis.rb:37:in `block in synchronize'
  from /opt/rbenv/versions/2.2.2/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis.rb:37:in `synchronize'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis.rb:1768:in `zcount'
  from (irb):66
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/railties-4.1.10/lib/rails/commands/console.rb:90:in `start'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/railties-4.1.10/lib/rails/commands/console.rb:9:in `start'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/railties-4.1.10/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/railties-4.1.10/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/rails/project/shared/bundle/ruby/2.2.0/gems/railties-4.1.10/lib/rails/commands.rb:17:in `<top (required)>'
  from ./bin/rails:8:in `require'
  from ./bin/rails:8:in `<main>'irb(main):067:0> 

But sentinel exists:

# redis-cli -p 26379
127.0.0.1:26379> info
# Server
redis_version:3.0.1
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:8114b371efa78b51
redis_mode:sentinel
os:Linux 3.13.0-24-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:1130
run_id:2f09b36efcc4c99f94f5b9ed6d06ff52b61b78bb
tcp_port:26379
uptime_in_seconds:3884
uptime_in_days:0
hz:12
lru_clock:6879962
config_file:/etc/redis/redis.conf

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=master,status=ok,address=10.0.0.1:6379,slaves=1,sentinels=1

[palexvs]

Renamed master name identifies a group of Redis instances from master to master.redis.com and it works not
Is it possible to add setting master_name for that?

[nguyenductung]

I think the line 496 in lib/redis/client.rb should be

@master = @options[:master_name]

The current assignment causes client.call(["sentinel", "get-master-addr-by-name", @master]) to return nil because master name is wrong.

def resolve_master
  sentinel_detect do |client|
    if reply = client.call(["sentinel", "get-master-addr-by-name", @master])
      {:host => reply[0], :port => reply[1]}
   end
  end
end

https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L556

[badboy]

@nguyenductung We should still use the hostname if master_name is not set.
@palexvs So according to that redis-cli output your monitored master is called master, not master.redis.com. redis-rb takes the host as the master name to query, why did you use master.redis.com in the first place?

[nguyenductung]

@badboy But if the master name is different from the host name, it'll result in Redis::CannotConnectError: No sentinels available.

What about

@master = @options[:master_name] || @options[:host]

?

[palexvs]

@badboy , yes it was my mistake.
I have used redis-sentinel gem before, where is possible to define a name identifies a group of Redis instances in configuration different from master hostname using param master_name. It is very useful.
Is it possible to add this functionality to redis-rb too?

Also I found that options hash should contain only symbol keys. I spent a lot of time to find this reason of my issue. I have loaded config from YML file and had a hash with string keys: "host" => '', .... 'port' => ...

[badboy]

@nguyenductung The good thing: With Sentinel the hostname from that "redis URL" is only used for the master name anyway, so why should it be different? But yes, I can still see having this as an option as useful. Care to submit a PR?

[nguyenductung]

@badboy I sent my PR but the tests failed due to Errno::EADDRINUSE: Address already in use. How should I fix that?

[badboy]

Restarted it, this is just a not-so-safe test setup right now…

[jperville]

Looking forward to seeing the associated PR merged.

[brodock]

why is this issue is closed? problem still not fixed

[colin-byrne-1]

shooting in the dark here... looking for a fix:

I have the generally the same setup and identical error output as @palexvs:

My initializer:

Sidekiq.configure_client do |config|
  config.redis = { 
    master_name: 'util-master',
    sentinels: redis_config['sentinels'].map { |sentinel|
      { :host => sentinel, :port => 26379 }
    },
  }
end

and I know the master_name is correct. Here is my output that shows that:

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=util-master,status=ok,address=10.1.2.20:6379,slaves=2,sentinels=5

It seems that no matter which combination of master_names and other configs, I get Redis::CannotConnectError: No sentinels with the same backtrace as above.

[brodock]

This is super confusing but here is what you need to do:

The Redis URL has to be defined as: redis://:<your-redis-password>@<master-name>/ and the list of sentinels.

@cobyrne09 So for your context, it will be something like:

Sidekiq.configure_client do |config|
  config.redis = { 
    url: 'redis://:yourpassword@util-master/',
    sentinels: redis_config['sentinels'].map { |sentinel|
      { :host => sentinel, :port => 26379 }
    },
  }
end

[colin-byrne-1]

Thank so much @brodock!
I do not have a redis password, so this is the syntax I got it to work with. I am not sure if the 'master_name' is necessary. Hopefully this helps someone else:

Sidekiq.configure_client do |config|
  config.redis = { 
    url: "redis://util-master/",
    master_name: :master,
    sentinels: redis_config['sentinels'].map { |sentinel|
      { :host => sentinel, :port => 26379 }
    }
  }
end

[brodock]

@cobyrne09 to actually run sentinel you need a password for redis versions >= 3.2, if you are connecting from a non-local ip (which both redis replicating and external clients communicating to sentinel will).

[colin-byrne-1]

thanks for the heads up @brodock. Still on 2.8.4, so I will have to deal with that soon enough...

[badboy]

Correction: you can totally run Sentinels without a password. Just make sure you don't expose them to the internet.

[badboy]

For reference: adding a master_name option is tracked in #640

[colin-byrne-1]

@brodock @badboy It seems like Sidekiq is picking up the processes on the slave server... is this the normal behavior? Because of this, it seems like it's sending jobs to the slave server, which is causing issues with call backs.