Cannot install mysql client onto 4.1.0rc2 (pkg-config not found)

[ilsaloving]

Bug description

I am trying to create a new docker container using 4.1.0rc2 as a base, and when I try to install the mysql client, I get pkg-config: not found. Stack trace is in Additional Content. Below is my docker file. Not sure if this should be pre-packaged with the superset container, or up to the user to install when extending the image?

FROM apache/superset:4.1.0rc2

USER root
RUN <<EOF
	apt-get update
	apt-get install -y curl links zip unzip basez
	apt-get purge -y --auto-remove
EOF
COPY superset_config.py superset-init.sh /app/
COPY --from=common get-secret.sh discover-roleid.sh /usr/local/bin/

USER superset

RUN <<EOF
	pip install --upgrade pip 
	pip install  \
	shillelagh[gsheetsapi] \
	gunicorn[gevent] \
	pymssql \
	elasticsearch-dbapi \
	elasticsearch-dbapi[opendistro] \
	mysqlclient \
	psycopg2 \
	trino \
	trino[sqlalchemy] \
	playwright \
	python-ldap \
	pip-system-certs \

EOF

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.10

Node version

Not applicable

Browser

Not applicable

Additional context

[superset stage-0 3/5] RUN <<EOF (pip install --upgrade pip...):
1.139 Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (24.2)
2.100 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
3.571 Collecting pymssql
3.786 Downloading pymssql-2.3.1-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (7.1 kB)
4.044 Collecting elasticsearch-dbapi
4.085 Downloading elasticsearch_dbapi-0.2.11-py3-none-any.whl.metadata (8.4 kB)
4.506 Collecting mysqlclient
4.546 Downloading mysqlclient-2.2.4.tar.gz (90 kB)
4.754 Installing build dependencies: started
8.710 Installing build dependencies: finished with status 'done'
8.713 Getting requirements to build wheel: started
8.932 Getting requirements to build wheel: finished with status 'error'
8.947 error: subprocess-exited-with-error
8.947
8.947 × Getting requirements to build wheel did not run successfully.
8.947 │ exit code: 1
8.947 ╰─> [27 lines of output]
8.947 /bin/sh: 1: pkg-config: not found
8.947 /bin/sh: 1: pkg-config: not found
8.947 /bin/sh: 1: pkg-config: not found
8.947 Trying pkg-config --exists mysqlclient
8.947 Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127.
8.947 Trying pkg-config --exists mariadb
8.947 Command 'pkg-config --exists mariadb' returned non-zero exit status 127.
8.947 Trying pkg-config --exists libmariadb
8.947 Command 'pkg-config --exists libmariadb' returned non-zero exit status 127.
8.947 Traceback (most recent call last):
8.947 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
8.947 main()
8.947 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
8.947 json_out['return_val'] = hook(**hook_input['kwargs'])
8.947 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
8.947 return hook(config_settings)
8.947 File "/tmp/pip-build-env-sncefshn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
8.947 return self._get_build_requires(config_settings, requirements=[])
8.947 File "/tmp/pip-build-env-sncefshn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
8.947 self.run_setup()
8.947 File "/tmp/pip-build-env-sncefshn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
8.947 exec(code, locals())
8.947 File "", line 155, in
8.947 File "", line 49, in get_config_posix
8.947 File "", line 28, in find_package_name
8.947 Exception: Can not find valid pkg-config name.
8.947 Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
8.947 [end of output]
8.947
8.947 note: This error originates from a subprocess, and is likely not a problem with pip.
9.005 error: subprocess-exited-with-error
9.005
9.005 × Getting requirements to build wheel did not run successfully.
9.005 │ exit code: 1
9.005 ╰─> See above for output.
9.005
9.005 note: This error originates from a subprocess, and is likely not a problem with pip.

---

failed to solve: process "/bin/sh -c \tpip install --upgrade pip\n\tpip install \\n\tshillelagh[gsheetsapi] \\n\tgunicorn[gevent] \\n\tpymssql \\n\telasticsearch-dbapi \\n\telasticsearch-dbapi[opendistro] \\n\tmysqlclient \\n\tpsycopg2 \\n\ttrino \\n\ttrino[sqlalchemy] \\n\tplaywright \\n\tpython-ldap \\n\tpip-system-certs \\n\n" did not complete successfully: exit code: 1

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

The error you're encountering is due to the absence of pkg-config in your Docker image. To resolve this, you need to install pkg-config before attempting to install mysqlclient. You can modify your Dockerfile as follows:

FROM apache/superset:4.1.0rc2

USER root
RUN apt-get update && apt-get install -y \
    pkg-config \
    default-libmysqlclient-dev \
    curl links zip unzip basez && \
    apt-get purge -y --auto-remove

RUN pip install --upgrade pip --root-user-action && \
    pip install --root-user-action \
    shillelagh[gsheetsapi] \
    gunicorn[gevent] \
    pymssql \
    elasticsearch-dbapi \
    elasticsearch-dbapi[opendistro] \
    mysqlclient \
    psycopg2 \
    trino \
    trino[sqlalchemy] \
    playwright \
    python-ldap \
    pip-system-certs

# Custom init
COPY superset_config.py superset-init.sh /app/

USER superset

This should resolve the pkg-config: not found error and allow you to install mysqlclient successfully.

To continue talking to Dosu, mention @dosu.

[ilsaloving]

I was able to solve it by installing pkg-config, but now I'm getting complaints about missing gcc. I thought this was a different issue from #29727 but apparently it's the same thing.

I'm wondering if there needs to be an easier mechanism or guidance for people to install needed database drivers. I can see this being a problem for less experienced people.

[ilsaloving]

Since the base image is running off Debian 12, I was able to make the following adjustments to my Dockerfile so everything would work without requiring a full compilation environment. I'm still fiddling with it, but this gets past the build errors.

Note: I also had to install the distro packages for ldap and psycopg2 as well, and then install psycopg2-binary via pip or superset wouldn't find it.

FROM apache/superset:4.1.0rc2

USER root
RUN <<EOF
	apt-get update
	apt-get upgrade -y
	apt-get install -y curl links zip unzip basez python3-mysqldb python3-ldap python3-psycopg2 python3-psycopg2cffi
	apt-get purge -y --auto-remove
EOF

COPY superset_config.py superset-init.sh /app/

RUN <<EOF
	pip --root-user-action install --upgrade pip 
	pip install  \
	psycopg2-binary \
	shillelagh[gsheetsapi] \
	gunicorn[gevent] \
	pymssql \
	elasticsearch-dbapi \
	elasticsearch-dbapi[opendistro] \
	trino \
	trino[sqlalchemy] \
	playwright \
	pip-system-certs \
	Pillow \

EOF

USER superset

[ilsaloving]

Final update... I will create a separate ticket for this but the distro-provided python3-ldap does not work, because it is installed for python 3.11, but superset uses 3.10. While I could alter the system to point to 3.11 instead, I'm hesitant to do that since it's not a supported configuration.