Conductor HA using redis sentinel/redis cluster without Dynomite

[guru1306]

Hi All

We are using Conductor as our workflow engine and are moving to an HA environment now. Dynomite HA seems to be dependent on Dynomite manager, which is available out of the box on only AWS. We are running in Azure environment. Our workflows have Decide type tasks (which in turn requires QueueDAO implementation).

Hence, we are looking to use Redis Sentinel/ Redis cluster as backend database for Conductor without Dynomite, but would like to know the valid HA configuration for this combination.

Is anyone using Conductor with Redis Sentinel/Redis Cluster in an HA configuration with multiple conductor instances and multiple nodes of Redis running?

Is it possible to use Redis without Dynomite for Conductor requiring HA ? Could you please share the configuration details which have worked for you ?

Thanks
Guru

[kishorebanala]

Hey @RuriRyan, Is this something you can help with? Thanks!

[RuriRyan]

Hiho,

we use Conductor with a redis sentinel deployment since a while now. We only run one replica of conductor-server though, because we haven't hit any performance issues yet. As we run conductor inside kubernetes HA isn't really a problem, because the pod gets recreated anyway if there is any problem.
For redis we use: https://github.com/DandyDeveloper/charts/tree/master/charts/redis-ha with next to no custom values.

Here's our config for conductor:

  config.properties: |
    # Servers.
    conductor.jetty.server.enabled=true
    conductor.grpc.server.enabled=false

    # Database persistence model.  Possible values are memory, redis, and dynomite.
    # If ommitted, the persistence used is memory
    #
    # memory : The data is stored in memory and lost when the server dies.  Useful for testing or demo
    # redis : non-Dynomite based redis instance
    # dynomite : Dynomite cluster.  Use this for HA configuration.
    db=redis_sentinel

    # Dynomite Cluster details.
    # format is host:port:rack separated by semicolon
    workflow.dynomite.cluster.hosts=$redisUrl:26379:de

    # Dynomite cluster name
    workflow.dynomite.cluster.name=$clusterName

    # Namespace for the keys stored in Dynomite/Redis
    workflow.namespace.prefix=conductor

    # Namespace prefix for the dyno queues
    workflow.namespace.queue.prefix=conductor_queues

    # No. of threads allocated to dyno-queues (optional)
    queues.dynomite.threads=10

    # Non-quorum port used to connect to local redis.  Used by dyno-queues.
    # When using redis directly, set this to the same port as redis server
    # For Dynomite, this is 22122 by default or the local redis-server port used by Dynomite.
    queues.dynomite.nonQuorum.port=22122

    # Elastic search instance type. Possible values are memory and external.
    # If not specified, the instance type will be embedded in memory
    #
    # memory: The instance is created in memory and lost when the server dies. Useful for development and testing.
    # external: Elastic search instance runs outside of the server. Data is persisted and does not get lost when
    #           the server dies. Useful for more stable environments like staging or production.
    workflow.elasticsearch.instanceType=external

    # Transport address to elasticsearch
    workflow.elasticsearch.url=http://$elasticUrl:9200

    # Name of the elasticsearch cluster
    workflow.elasticsearch.index.name=$indexName

    # We know who broke it, no need for additional workflowshaming
    workflow.owner.email.mandatory=false

    # Set the es version
    workflow.elasticsearch.version=6

    # Additional modules (optional)
    # conductor.additional.modules=class_extending_com.google.inject.AbstractModule

    # Load sample kitchen sink workflow
    loadSample=false

    EC2_AVAILABILITY_ZONE=de
    workflow.elasticsearch.cluster.health.color=yellow
    http.task.read.timeout=5000
    http.task.connect.timeout=5000
    conductor.additional.modules=com.netflix.conductor.contribs.ArchivingWorkflowModule
  log4j.properties: |
    log4j.rootLogger=INFO, A1
    log4j.appender.A1=org.apache.log4j.ConsoleAppender
    log4j.appender.A1.layout=org.apache.log4j.PatternLayout
    log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

[guru1306]

Thanks. The writes from conductor will go to master and the read will go to slaves. Is this correct? If so, do you face any issues due to Redis's eventual consistency as it takes time to sync the data?

What's the use of de in this line? workflow.dynomite.cluster.hosts=$redisUrl:26379:de