[BUG] Docker compose does not interpret the .env inside the Spring Boot project.

[drakgoku]

Description

When everything finally works fine locally, what I do is generate the jars and run docker compose.

After compiling the spring boot .jars and doing

docker compose -f docker-compose-dev.yml up --build -d

Spring Boot and Docker seem to for some reason not correctly render the spring boot .env and the Spring Boot variables do not arrive correctly.

This is my mysql image in my docker-compose, for mysql and a microservice called profile.

...
  profile_service:
    build:
      context: ../../          /profile
      dockerfile: Dockerfile
    container_name: c_profile_service
    ports:
      - "8202:8202"
    depends_on:
      - db_mysql
...
db_mysql:
    image: mysql:8.0.39
    container_name: c_db_mysql
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    volumes:
      - mysql-data:/var/lib/mysql
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
...

The .env in my docker-compose-dev.yml is working fine, it's the Spring Boot one that's not working.
Let's go to the Spring Boot profile project.

application.properties:

# Microservice
spring.application.name=${SPRING_APPLICATION_NAME}

# SERVER PORT
server.port=${SERVER_PORT}

# DATABASE
spring.datasource.url=${SPRING_DATASOURCE_URL}
spring.datasource.username=${SPRING_DATASOURCE_USERNAME}
spring.datasource.password=${SPRING_DATASOURCE_PASSWORD}
spring.datasource.driver-class-name=${SPRING_DATASOURCE_DRIVER_CLASS_NAME}
spring.jpa.properties.hibernate.dialect=${SPRING_JPA_PROPERTIES_HIBERNATE_DIALECT}

# SCHEME DATABASE SQL
# spring.sql.init.schema-locations=
# WRITE SQL TO DATABASE
spring.sql.init.mode=${SPRING_SQL_INIT_MODE}

spring.config.import=optional:classpath:.env[.properties]

Spring Boot .env int /resources/

"SPRING_APPLICATION_NAME=...
SERVER_PORT=...
SPRING_DATASOURCE_URL=jdbc:mysql://c_db_mysql:3306/....
SPRING_DATASOURCE_USERNAME=...
SPRING_DATASOURCE_PASSWORD=...
SPRING_DATASOURCE_DRIVER_CLASS_NAME=...
SPRING_JPA_PROPERTIES_HIBERNATE_DIALECT=org.hibernate.dialect.MySQLDialect
SPRING_SQL_INIT_MODE=always

This works correctly if I run the project without being launched as an image:

SPRING_DATASOURCE_URL=jdbc:mysql://localhost:3306/.... WORKS

But if I launch it from the image with

docker compose -f docker- compose-dev.yml up --build -d: 

SPRING_DATASOURCE_URL=jdbc:mysql://c_db_mysql: 3306/.... NOT WORKS

I get this error from console docker container c_db_mysql:
java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org .hibernate.resource.transaction.backend.jdbc.internal.JdbcIsolationDelegate.sqlExceptionHelper()" is null

It seems that somehow docker is not correctly interpreting the .env that works locally. Why?

1 - If I run the image in docker and run the standalone project with: SPRING_DATASOURCE_URL=jdbc:mysql://localhost:3306/.... WORKS
2 - If I run the entire docker-compose-dev.yml, the .env variables don't seem to be read by docker.

Steps To Reproduce

1 - Run jars
2 - docker compose -f docker-compose-dev.yml up --build -d
3 - Error trying to access the database, variables arrive as null.

Compose Version

Docker Compose version v2.29.1-desktop.1

Docker Environment

Windows 11

Anything else?

No response

[drakgoku]

If I don't see a solution, I'll have to migrate to Kubernetes. What I think will be better than docker-compose.yml

[ndeloof]

Up to you to migrate to k8s and see if it is "better" (you'll discover a whole new universe of random issues :P)

About your issue, you should access your database using service name db_mysql , not c_db_mysql - I'm not sure where this prefix comes from ?

[drakgoku]

db_mysql = service name
c_db_mysql = container name

My understanding is that it is the name of the container, I tried both. Neither works.

This isn't about discovering a new world. If I have to pay to do some tests... I'm not happy about it. Having to go looking for hosts with free Kubernetes... instead of things working normally... What do you want me to say?

[drakgoku]

It seems that the solution is this:

spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false

In the docker console it appears:

2024-08-31 20:14:19 2024-08-31T18:14:19.162Z WARN 1 --- [8202] [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning

Could someone explain to me why? Is there some incompatibility with Docker and jdbc metadata?
ㅤ
ㅤ
ㅤ
ㅤ
In addition to the problem, I'm also thinking that the issue of putting
"depends_on" in docker-compose.yml and "jdbc:mysql://[container]:3306/.. " in spring boot is a bit annoying since it seems that spring only works based on the container. I think it creates a monolithic dependency.

I think it would be better without putting the depends_on and without the container for greater modularity... something like:

jdbc:mysql://domain:3306/.. 

I think it would be more appropriate without being tied to any container, rather you would be tied to the domain.

ㅤ
ㅤ
Note: I will leave this issue to your decision if you decide to close it or on the contrary if you want to contribute something extra and so if you want to keep it open and take it into account for a possible improvement or repair that can be done.

[ndeloof]

jdbc:mysql://[container]:3306/..

this is not container but service, which is actually a domain name container can be accessed (resolved by docker engine internal DNS)

[drakgoku]

My understanding is that when you define a service in Docker Compose, Docker creates a default network for your application and each container joins this network with the service name. Therefore, other containers can access this service using its service name / container as hostname.

https://docs.docker.com/compose/networking/

Docker networking

Each container can now look up the service name web or db and get back the appropriate container's IP address. For example, web's application code could connect to the URL postgres://db:5432 and start using the Postgres database._

It is important to note the distinction between HOST_PORT and CONTAINER_PORT. In the above example, for db, the HOST_PORT is 8001 and the container port is 5432 (postgres default). Networked service-to-service communication uses the CONTAINER_PORT. When HOST_PORT is defined, the service is accessible outside the swarm as well.

_Within the web container, your connection string to db would look like postgres://db:5432, and from the host machine, the connection string would look like postgres://{DOCKER_IP}:8001 for example postgres://localhost:8001 if your container is running locally.

ㅤ

In Docker Compose, I have defined the service as db_mysql, and within that service, you specify that the container will be called c_db_mysql. For inter-container communication, you would use the service name (db_mysql), but in my case, since I have specified a container name (c_db_mysql), I can use that name as well.

Both ways are correct. If you want an approximation, we can say that it is much better to use the service name.

ㅤ
ㅤㅤ
Even if you use the name of the service or container, it still seems monolithic to me, which was my question. I think I'll uncouple them and leave it like this: jdbc:mysql://domain:3306/.. I think it looks less tied to a docker image.
For example, something simple if you do local tests: jdbc:mysql://localhost:3306/ or if you're on a domain jdbc:mysql://www.blizzard.com:3306/

This last way, without being tied to an image, seems less monolithic to me.

By the way, do you know anything about what I told you about jdbc metadata and Docker that variables arrive empty and because of that having to disable metadata?

[ndeloof]

do you know anything about what I told you about jdbc metadata and Docker that variables arrive empty and because of that having to disable metadata?

Sorry, I don't get what you're asking for. Your application should access database relying on service name jdbc:mysql://db_mysql:3306/.. but I can't tell about your application internal structure and environment variable it requires to run. Compose only set container variables according to service's environment (and env_file) section. Double check result using docker compose exec profile_service env

[drakgoku]

You don't understand? I just read it now.

I'll explain it simply.
You have the project and the environment variables in a .env file.

Method A - If you run spring boot locally (the start button) everything works fine.
Method B - If you create a dockerfile for the project and deploy the image (creating a container) for some reason docker doesn't interpret the .env files, having to put in the config spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false

If you don't understand this, I don't know what else to say.

Water is a transparent, drinkable fluid. I couldn't explain it any more.

I switched from docker compose to Kubernetes.
If docker is not able to do something simple like reading environment variables...

[ndeloof]

Please stay cool here, I'm here to help you not to receive lessons.

docker doesn't interpret the .env files,

How do you tell docker to use those .env files? Are those copied inside container by your Dockerfile instructions? In you example compose.yaml there's no reference to any .env file that would help me understand your issue

[ndeloof]

[ndeloof]

Water is a transparent, drinkable fluid. I couldn't explain it any more

Your description also marche alcohol which isn't strictly the same. Explaining things is hard, please relax and try again

[drakgoku]

In spring boot, the .env files are usually placed in the resources folder.
\nameprojectbackend\user\src\main\resources (User is a service)

There is no need to tell it where the resources come from, you just have to add:
spring.config.import=optional:classpath:.env[.properties] in the application.properties file.
As I said above. That's what it does, it reads the .env properties.

That is, Spring Boot is smart enough to go and look up the properties on its own. You don't need to specify anything in the dockerfile.
Therefore, there is some docker syntax that conflicts with this.

That works fine if deployed locally with VSC, Intellij, Eclipse...
But if run from a Docker container it doesn't work fine. It gives an error if the image is run in Docker. So the solution is to set spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false. My question is, why do I have to set that to work in the Docker image?

A - Running it normally - IDE:
The project with the .env file in the resources folder and running it from any IDE works

B - Running it as a container from an image.
The project with the .env file in the resources folder and running it from a container does not work. It seems that Spring Boot does not take those variables. Solution: put spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false in the application.properties file.

Water is a transparent, drinkable fluid. I couldn't explain it any more

Your description also marche alcohol which isn't strictly the same. Explaining things is hard, please relax and try again

I said "water" not "alcohol".

[ndeloof]

It seems that Spring Boot does not take those variables

please double check this .env file is well present in the running container, using docker compose exec {app} to get a shell inside container and inspect filesystem. AFAICT this one should be set for local (non-dockerized) usage, and require some value override to configure execution inside a container, as database is not running on localhost but requires a service lookup to get actual sibling container IP (basically, jdbc:mysql://db_mysql:3306/.. as your database service name is db_mysql).

Also might be usefull to compare your application with https://medium.com/thefreshwrites/the-way-of-dockerize-a-spring-boot-and-mysql-application-with-docker-compose-de2fc03c6a42, especially use of environment to configure container environment variable
spring.datasource.url to use database service name in jdbc URL (or equivalent feature based on PropertySourcesPlaceholderConfigurer relying on a .env file)

[drakgoku]

Sorry. I don't understand anything you said.

1 - I don't have to include any file in docker. Can you imagine having to put the .java or .class? I can't. Spring boot is smart enough to go and fetch the .env files. Nothing needs to be included.
2 - How is it possible that it works using a simple parameter like the much mentioned spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false ?

Normally you can pass the .env from dockerfile or from the orchestrator itself, whether it's Kubernetes or Docker Compose. I don't need it since I pass them individually per project.

I have 20 images (microservices) and it's estimated that there will be more than 100, and if I have to put the .env of each project in a "docker compose" this is going to be too tedious.

I don't use docker compose, so if I just start the image in a container the same error occurs.

As I see that this will continue to be the same, remember that for the next people who have the problem, tell them to put: spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false. Tell them that there is some kind of collision. They will thank you and continue their work.

Remember this is a needle in a haystack. If you want to close the post, you can close it. But remember the solution at least.

[ndeloof]

1. I mean "by your Dockerfile". I have no details on the way you build your image an package application as a container image. Spring boot is smart enough to go and fetch the .env files. Nothing needs to be included. for sure, but when packaged in a container image, how does it find this .env file ? Seems it is loaded from classpath, so is bundled inside your jar. Can you please confirm ?

if I just start the image in a container the same error occurs

Seems to demonstrate a packaging issue or something wrong in runtime configuration. Your application container should be able to fully access the db container, the "network" setby composé is just a bridge network that doesn't bring any type of constraints

About the workaround you're using, this is obviously not the desired state. I can't see any reason you get such an issue.
I'd be happy to help you diagnose this, maybe you can reproduce with a minimal "hello world* spring boot app using the same configuration technique you adopted ?

[drakgoku]

I'm just passing by here to quickly note down this step of docker and kubernetes as a reminder.

I'll also quickly answer your question.

Nothing needs to be included. for sure, but when packaged in a container image, how does it find this .env file ? It seems it is loaded from classpath, so it is bundled inside your jar. Can you please confirm ?

With the property that I already showed in my first post.
spring.config.import=optional:classpath:.env[.properties]

Reminder:

• If you are going to use docker use:
  #ONLY FOR DOCKER as the metadata is not found at boot
  spring.jpa.properties.hibernate.dialect=${SPRING_JPA_PROPERTIES_HIBERNATE_DIALECT}
  spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false

Note: You will always see a correct "deprecation" message.

• If you are going to use kubernetes use:
  Comment out the two lines above as always seeing the deprecation message hurts your eyes.
  #pring.jpa.properties.hibernate.dialect=${SPRING_JPA_PROPERTIES_HIBERNATE_DIALECT}
  #spring.jpa.properties.hibernate.boot.allow_jdbc_metadata_access=false

  and you let Spring Boot itself determine the dialect thanks to the metadata.

[ndeloof]

I can't see any reason you get this behavior. Could you setup a minimal spring boot application to demonstrate this issue ?

[ndeloof]

Closing as reporter didn't provided a reproducible scenario so we can investigate