v1.5.10 memory leak regression when used with libcurl

[sbiscigl]

Hey from the aws cpp sdk, and we're trying to update our submodule dependencies and we noticed a regression/memory leak when using s2n in conjuction with libcurl and its blocking us from updating our dependencies.

Created a small reproduction

#include <curl/curl.h>
#include <s2n.h>

auto main() -> int {
    s2n_init();
    curl_global_init(CURL_GLOBAL_ALL);
    curl_global_cleanup();
    s2n_cleanup();
    s2n_cleanup_final();
    return 0;
}

Dockerfile that can reproduce it:

# Using offical Amazon Linux 2 image from public ECR
FROM public.ecr.aws/amazonlinux/amazonlinux:2

#Install g++
RUN yum groupinstall "Development Tools" -y

#Install required dependencies
RUN yum install -y curl-devel openssl-devel ninja-build cmake3 rsync gdb libasan

#Link cmake3 correctly
RUN ln -s /usr/bin/cmake3 /usr/bin/cmake

# v1.5.9 works
#RUN git clone -b v1.5.9 https://github.com/aws/s2n-tls.git &&\
#    cd s2n-tls &&\
#    mkdir build &&\
#    cd build &&\
#    cmake3 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-ggdb -fsanitize=address" -DCMAKE_INSTALL_PREFIX="/s2n-install" .. && \
#    cmake3 --build . &&\
#    cmake3 --install .

# v1.5.10/main has a memory leak
RUN git clone https://github.com/aws/s2n-tls.git &&\
    cd s2n-tls &&\
    mkdir build &&\
    cd build &&\
    cmake3 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-ggdb -fsanitize=address" -DCMAKE_INSTALL_PREFIX="/s2n-install" .. && \
    cmake3 --build . &&\
    cmake3 --install .

# Create test application
RUN mkdir test

# Create CMakeLists.txt
RUN touch /test/CMakeLists.txt &&\
    echo "cmake_minimum_required(VERSION 3.13)" >> /test/CMakeLists.txt &&\
    echo "project(test_s2n_init)" >> /test/CMakeLists.txt &&\
    echo "set(CMAKE_CXX_STANDARD 20)" >> /test/CMakeLists.txt &&\
    echo "find_package(CURL REQUIRED)" >> /test/CMakeLists.txt &&\
    echo "find_package(s2n REQUIRED)" >> /test/CMakeLists.txt &&\
    echo "add_executable(\${PROJECT_NAME} "main.cpp")" >> /test/CMakeLists.txt &&\
    echo "target_include_directories(\${PROJECT_NAME} PRIVATE \${CURL_INCLUDE_DIRS})" >> /test/CMakeLists.txt &&\
    echo "target_link_libraries(\${PROJECT_NAME} PRIVATE \${CURL_LIBRARIES} AWS::s2n)" >> /test/CMakeLists.txt

# Create simple test file
RUN touch /test/main.cpp &&\
    echo "#include <curl/curl.h>" >> /test/main.cpp &&\
    echo "#include <s2n.h>" >> /test/main.cpp &&\
    echo "" >> /test/main.cpp &&\
    echo "auto main() -> int {" >> /test/main.cpp &&\
    echo "    s2n_init();" >> /test/main.cpp &&\
    echo "    curl_global_init(CURL_GLOBAL_ALL);" >> /test/main.cpp &&\
    echo "    curl_global_cleanup();" >> /test/main.cpp &&\
    echo "    s2n_cleanup();" >> /test/main.cpp &&\
    echo "    s2n_cleanup_final();" >> /test/main.cpp &&\
    echo "    return 0;" >> /test/main.cpp &&\
    echo "}" >> /test/main.cpp

# Build test
RUN cd test &&\
    mkdir build &&\
    cd build &&\
    cmake3 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-ggdb -fsanitize=address" -DCMAKE_PREFIX_PATH="/s2n-install" .. &&\
    cmake3 --build .

can build and run this to replicate with

docker build -t test-image .
docker run --name test-image test-image /test/build/test_s2n_init

but you should see the stack trace

=================================================================
==1==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0xffff966dbdb3 in __interceptor_malloc (/lib64/libasan.so.4+0xdbdb3)
    #1 0xffff9626f35f in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6f35f)
    #2 0xffff962d6d8b in ENGINE_new (/lib64/libcrypto.so.10+0xd6d8b)
    #3 0x40c757 in s2n_rand_init /s2n-tls/utils/s2n_random.c:576
    #4 0x4063c3 in s2n_init /s2n-tls/utils/s2n_init.c:78
    #5 0x40606f in main /test/main.cpp:5
    #6 0xffff96033da3 in __libc_start_main (/lib64/libc.so.6+0x1fda3)
    #7 0x405f87  (/test/build/sdk_usage_workspace+0x405f87)

which is indicative of this change being the cultprit of it.

would guess this is a issue with static state being shared between s2n and libcurl in some way.

[boquan-fang]

Hello Sam,

Thanks for providing all those context. I have pull downed your Dockerfile and can confirm that the leak is caused by PR#4878. I did two git checkouts for the commit hash of PR#4878 and its parent commit. ASAN didn't complain for the parent commit, but did detect memory leaks for PR#4878.

[maddeleine]

So far I don't know why the leak started to occur. However it seems that the try-compile feature S2N_LIBCRYPTO_SUPPORTS_ENGINE evaluates to true in the repro. Which means that our custom randomness code is trying to be instantiated/cleaned up. Maybe it was turned off for this platform previously. I'm going to hazard a guess that the libcrypto being used here is Openssl 1.0.2.

[maddeleine]

Leak goes away if you re-order the init/cleanup code:

auto main() -> int {
    curl_global_init(CURL_GLOBAL_ALL);
    s2n_init();
    s2n_cleanup_final();
    curl_global_cleanup();
    return 0;
}

I suspect curl is wiping our randomness engine, maybe replacing it with their own. Therefore our engine pointer memory gets leaked since its being wiped.