[QUESTION]: server log stops after starting in version 3.x when using docker image

[mkp-bartsch]

Hello,
we are using conductor for orchestrating microservices in our company and have some issues while setting up the latest version v.3.x.

Maybe you can help me:

Short description:
After building and starting the server and ui image (docker) from the latest tag, the server stops starting after "Using java options config:". This behaviour is independent of the configurations in the config-file.

Steps taken to reproduce the error:

1. get latest tag as branch to build latest images with

git pull
git checkout v3.3.2 -b v3.3.2

2. run docker build commands in the main conductor project folder:

docker build -f docker/server/Dockerfile -t conductor/server:v3.3.2 .
docker build -f docker/ui/Dockerfile -t conductor/ui:v3.3.2 .

3. execute with in-memory database for testing:

docker run -p 5000:5000 -d --name conductor_ui conductor/ui:v3.3.2
docker run -p 8080:8080 -d --name conductor_server conductor/server:v3.3.2

4. check logs and state of servies:

• The ui log seems to be okay. The service is reachable under localhost:5000
• The server starts and hangs up while starting. The log entry is short and stops after "Using java options config:":

Starting Conductor server
Property file:
Using an in-memory instance of conductor
Using java options config:

Expected behaviour:

• The server starts up and is reachable from the ui and accessible via http to post workflow definitions.

Things that I have already tried:

• Configure an elastic search and redis server and fitting the configurations to use these services for storing persistent data and queuing data. -> same error here.
• Reduce complexity in configurations for this git question.

Do any of you have a working local configuration where this error does not occur and the server starts properly?

Until now we used the version 2.3.x for our productive environment. We would like to upgrade to latest version because you fixed some bugs.

Thank you in advance!

[mkp-bartsch]

A short update. I got an environment up and running. So far everything seems to be stable.

Nevertheless, the log of the Conductor server stops after the first entries:

Starting Conductor server
Property file: config.properties
config.properties
Using 'config.properties'
Using java options config:

So in case of an error, you can only guess where the problem is.

Does anyone have an idea what the problem is? I think the problem exists with all current docker-based conductor server images.

[leynae]

I have the same issue. Server is up and running but nothing in the docker log

[gizmodude4]

Same issue here. Running from ./gradlew bootRun works, but docker-compose up is not

[buzz08]

The logs are not exposed to the console, it should be available in /app/logs/server.log inside the docker container instance.

Are you not able to see the server logs there?

By the way, for the whole thing to work - you will need Elasticsearch up and running as well as the option is now enabled by default. Check the config.properties files on the URL it tries to connect.

If you don't have ES, you will need to disable the indexing via a property. The downside to disabling that is the UI search functions won't work.

If you are running this in an M1 mac, you might run into trouble running ES6 docker container as the containers is based on x86 arch and will not work with arm64 smoothly.

[gizmodude4]

Quick update: I'll try looking for server logs there later tonight.

I couldn't get the server up and running with docker compose up -d in the docker directory and had the issue mentioned here #2540 (comment) where the server was seemingly up but wasn't responding to me trying to hit it via the API or the health check. Elasticsearch was up and responding to pings.

However, I did get things up and running using docker-compose -f docker-compose.yaml -f docker-compose-postgres.yaml up -d after a bit and it's still working now.

[mkp-bartsch]

The logs are not exposed to the console, it should be available in /app/logs/server.log inside the docker container instance.

Are you not able to see the server logs there?

By the way, for the whole thing to work - you will need Elasticsearch up and running as well as the option is now enabled by default. Check the config.properties files on the URL it tries to connect.

If you don't have ES, you will need to disable the indexing via a property. The downside to disabling that is the UI search functions won't work.

If you are running this in an M1 mac, you might run into trouble running ES6 docker container as the containers is based on x86 arch and will not work with arm64 smoothly.

Thank you for the explanation.
With the command docker-compose -f docker-compose.yaml -f docker-compose-postgres.yaml up the environment works. But the log remains in the console in any case.
I have checked the /app/logs/server.log file. In it is the complete log.
In the Conductor version 2.x this log was printed in the console. This was always very helpful in case of an error and it would be great if this also happens in version 3.x!

[boney9]

Hey - quick update - if you need the logs to show up on the console you can delete the logs redirect on this startup.sh command. Specifically this part:

> /app/logs/server.log

https://github.com/Netflix/conductor/blob/main/docker/serverAndUI/bin/startup.sh#L36

This is assuming you are using the serverAndUI docker.

Try this out locally on Docker desktop and let us know if its working

[javiercanillas]

This happened to me the second time I start the docker solution. Revisiting the server logs, I found that it calls to ES with a configured timeout of 30 seconds to wait on index health (turn to green). After that, the conductor server will fail if ES can't achieve a green state, logging out the issue.

In my case, I found that indexes are created using a configuration of 1 replica, since docker implementation only starts one ES, this statement cannot be fulfilled, so ES status will never be green (maybe yellow).

I will report this in another thread, still investigating where this can be changed.

[Jarga]

I ran into the same thing and found the following two config entries fix the issue:

conductor.elasticsearch.indexShardCount=1
conductor.elasticsearch.indexReplicasCount=0

Or optionally add these environment variables to the compose (config file entries have priority):

            - CONDUCTOR_ELASTICSEARCH_INDEX_SHARD_COUNT=1
            - CONDUCTOR_ELASTICSEARCH_INDEX_REPLICAS_COUNT=0

You will need to make sure to remove the existing indexes created with the old config and/or just remove the elasticsearch volume with the old indexes.

---

I also set the default shards/replicas for the elasticsearch instance indexes to be safe (There may be an easier way than this not 100% sure):

curl -X PUT http://localhost:9200/_template/default -H 'Content-Type: application/json' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}'

I don't think you need this part for conductor to consider things good but I do it to ensure any other indexes are created correctly and don't cause the instance to go into a yellow status.

[javiercanillas]

Yeap, found the same fix: #2590