Skip to content

Commit 13002cf

Browse files
authored
ODBC Standalone MSVC Build CI
Remove DataSet Fix cache key Use odbc-specific cache key Fix Lint and Add odbc registration step Link appropriate GitHub issues that blocks ODBC tests Disable test phase and add schedule Run everyday 7 am Vancouver time Enable ODBC build on MSVC CI Code Clean up and enable ODBC tests in CI Still need to modify ODBCUtilEnvironment if we decide to use it. Still need to add ODBC V2 support so a different env and conn is used for ODBC 2 tests. Draft enable ODBC global setup/teardown Run ODBC test once outside of test script If ODBC test is run using MinGW Shell, segfault occurs. I am not getting any seg fault on my local MSVC Windows when I run the tests without the bash script. But if Windows CI breaks from running the standalone exe then I will look into this Prepend vcpkg to search vcpkg before `<prefix>/lib/cmake` Since we are installing dependencies on vcpkg, if `lib/cmake` is searched first, then cmake will look into that directory and use the wrong paths. Convert VCPKG Windows path to MSYS path Set `VCPKG_ROOT` in test phase Fix ODBC dll name Use `arrow_flight_sql_odbc.dll` instead of `libarrow_flight_sql_odbc.dll` which is the naming convention used on MinGW Windows Link ODBC library on all platforms On my local MSVC Windows machine, Visual Studio is used to build without needing to link ` ${ODBCINST}` explicitly, its behavior might be different from Ninja which is what CI uses. `${ODBCINST}` is likely needed by Linux as well, so adding it for all platforms. Attempt to resolve `arrow-compute-grouper-benchmark` build issue I think `if(ARROW_FLIGHT_TEST_LINKAGE STREQUAL "static")` might be more appropriate here, as I am seeing build issues due to both dynamic and static linking occurring. I see in apache@59903d0, this check is used for `arrow-filesystem-s3fs-benchmark` Disable `UNITY_BUILD` for ODBC test due to conflict on `sqlite_sql_info.cc` On some workflows, unity build is set to ON to make the build faster. This is an attempt to resolve the build issue. Remove debug messages Fix lint Add `sql_info_undef.h` to sqlite_sql_info.cc Undefine duplicates in SQLGetInfo Add `#pragma once` to odbc_test_suite.h To avoid redefinition error Attempt to resolve conflict with sql/types.h Attempt to include Arrow headers before ODBC headers to avoid conflict with arrow/flight/sql/types.h [apacheGH-48084] Replace boost::optional with std::optional Addresses comment apache#40939 (comment) Replace boost::optional with std::optional in ODBC codebase apache#48084 Fix lint Remove `BOOST_SOURCE=BUNDLED` to use vcpkg's dynamic link to boost Since we need to use link to boost dynamically, we need to remove the bundled boost library flag. Add debug messages. Remove `ARROW_BOOST_USE_SHARED = OFF` Since we have a release build, can enable boost as shared. With `ARROW_BOOST_USE_SHARED = OFF`, I am getting error ``` D:\a\arrow\arrow\build\cpp\vcpkg_installed\x64-windows\include\boost/filesystem/config.hpp(96): fatal error C1189: #error: Must not define both BOOST_FILESYSTEM_DYN_LINK and BOOST_FILESYSTEM_STATIC_LINK ``` Also increased time limit, since 2 hours don't seem to be enough to finish the vcpkg build Change to release build Getting ``` orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in unity_2_cxx.cxx.obj orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MDd_DynamicDebug' in unity_2_cxx.cxx.obj ``` when building with debug and presumably ARROW_ORC Set ARROW_BOOST_USE_SHARED to OFF `ARROW_BOOST_USE_SHARED` is restored to `OFF`, which according to Windows doc should help with the debug build now. Retore value of `ARROW_BUILD_BENCHMARKS`, though noting it is set to `OFF` in GLib MSVC workflow. Add VCPKG set up Borrowed from the Glib workflow Add `/EHsc` flag * Add back `CMAKE_CXX_STANDARD: "17"` Remove `CMAKE_CXX_STANDARD` 17 * reason: gtest can't be used with C++17. Arrow project doesn't support C++ 17 yet. Extend run-time to 2hr windows-mingw also has timeout of 2hr Empty commit to trigger CI Specify `VCPKG_BINARY_SOURCES` and `VCPKG_DEFAULT_TRIPLET` Specify VCPKG_TARGET_TRIPLET as x64 windows Set ARROW_DEPENDENCY_SOURCE to VCPKG To make it easier to manage dependencies Enable static build on MSVC `ARROW_BUILD_BENCHMARKS` prevents Flight from being built Remove `ARROW_BOOST_USE_SHARED` Having static vs. shared issue Enable Flight & Flight SQL for ODBC Enable ODBC build on MSVC CI Enable regular ctest tests Remove undef items Add concurrency and permissions to odbc yml Create cpp_odbc.yml Fix architecture in CI * Add ODBC installer MSVC CI
1 parent fc3e8b1 commit 13002cf

File tree

9 files changed

+200
-10
lines changed

9 files changed

+200
-10
lines changed

.github/workflows/cpp.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -390,6 +390,10 @@ jobs:
390390
PIPX_BASE_PYTHON: ${{ steps.python-install.outputs.python-path }}
391391
run: |
392392
ci/scripts/install_gcs_testbench.sh default
393+
- name: Register Flight SQL ODBC Driver
394+
shell: cmd
395+
run: |
396+
call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\libarrow_flight_sql_odbc.dll
393397
- name: Test
394398
shell: msys2 {0}
395399
run: |

.github/workflows/cpp_odbc.yml

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: C++ ODBC
19+
20+
on:
21+
push:
22+
branches:
23+
- '**'
24+
- '!dependabot/**'
25+
tags:
26+
- '**'
27+
paths:
28+
- '.github/workflows/cpp_odbc.yml'
29+
- 'ci/scripts/cpp_*'
30+
- 'cpp/src/arrow/flight/sql/odbc/*'
31+
pull_request:
32+
paths:
33+
- '.github/workflows/cpp_odbc.yml'
34+
- 'ci/scripts/cpp_*'
35+
- 'cpp/src/arrow/flight/sql/odbc/*'
36+
schedule:
37+
- cron: '0 13 * * *'
38+
39+
concurrency:
40+
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
41+
cancel-in-progress: true
42+
43+
permissions:
44+
contents: read
45+
46+
jobs:
47+
windows:
48+
runs-on: windows-2022
49+
timeout-minutes: 240
50+
env:
51+
ARROW_BUILD_SHARED: ON
52+
ARROW_BUILD_STATIC: ON
53+
ARROW_BUILD_TESTS: ON
54+
ARROW_BUILD_TYPE: release
55+
ARROW_DEPENDENCY_SOURCE: VCPKG
56+
ARROW_FLIGHT: ON
57+
ARROW_FLIGHT_SQL: ON
58+
ARROW_FLIGHT_SQL_ODBC: ON
59+
ARROW_FLIGHT_SQL_ODBC_INSTALLER: ON
60+
ARROW_SIMD_LEVEL: AVX2
61+
CMAKE_CXX_STANDARD: "17"
62+
CMAKE_GENERATOR: Ninja
63+
CMAKE_INSTALL_PREFIX: /usr
64+
VCPKG_BINARY_SOURCES: 'clear;nuget,GitHub,readwrite'
65+
VCPKG_DEFAULT_TRIPLET: x64-windows
66+
steps:
67+
- name: Disable Crash Dialogs
68+
run: |
69+
reg add `
70+
"HKCU\SOFTWARE\Microsoft\Windows\Windows Error Reporting" `
71+
/v DontShowUI `
72+
/t REG_DWORD `
73+
/d 1 `
74+
/f
75+
- name: Checkout Arrow
76+
uses: actions/checkout@v5
77+
with:
78+
fetch-depth: 0
79+
submodules: recursive
80+
- name: Download Timezone Database
81+
shell: bash
82+
run: ci/scripts/download_tz_database.sh
83+
- name: Install msys2 (for tzdata for ORC tests)
84+
uses: msys2/setup-msys2@v2
85+
id: setup-msys2
86+
- name: Install cmake
87+
shell: bash
88+
run: |
89+
ci/scripts/install_cmake.sh 4.1.2 /usr
90+
- name: Install ccache
91+
shell: bash
92+
run: |
93+
ci/scripts/install_ccache.sh 4.12.1 /usr
94+
- name: Setup ccache
95+
shell: bash
96+
run: |
97+
ci/scripts/ccache_setup.sh
98+
- name: ccache info
99+
id: ccache-info
100+
shell: bash
101+
run: |
102+
echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT
103+
- name: Cache ccache
104+
uses: actions/cache@v4
105+
with:
106+
path: ${{ steps.ccache-info.outputs.cache-dir }}
107+
key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }}
108+
restore-keys: cpp-odbc-ccache-windows-x64-
109+
- name: Checkout vcpkg
110+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
111+
with:
112+
fetch-depth: 0
113+
path: vcpkg
114+
repository: microsoft/vcpkg
115+
- name: Bootstrap vcpkg
116+
run: |
117+
vcpkg\bootstrap-vcpkg.bat
118+
$VCPKG_ROOT = $(Resolve-Path -LiteralPath "vcpkg").ToString()
119+
Write-Output ${VCPKG_ROOT} | `
120+
Out-File -FilePath ${Env:GITHUB_PATH} -Encoding utf8 -Append
121+
Write-Output "VCPKG_ROOT=${VCPKG_ROOT}" | `
122+
Out-File -FilePath ${Env:GITHUB_ENV} -Encoding utf8 -Append
123+
- name: Setup NuGet credentials for vcpkg caching
124+
shell: bash
125+
run: |
126+
$(vcpkg fetch nuget | tail -n 1) \
127+
sources add \
128+
-source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" \
129+
-storepasswordincleartext \
130+
-name "GitHub" \
131+
-username "$GITHUB_REPOSITORY_OWNER" \
132+
-password "${{ secrets.GITHUB_TOKEN }}"
133+
$(vcpkg fetch nuget | tail -n 1) \
134+
setapikey "${{ secrets.GITHUB_TOKEN }}" \
135+
-source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json"
136+
- name: Build
137+
shell: cmd
138+
run: |
139+
set VCPKG_ROOT_KEEP=%VCPKG_ROOT%
140+
call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
141+
set VCPKG_ROOT=%VCPKG_ROOT_KEEP%
142+
bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"
143+
- name: Register Flight SQL ODBC Driver
144+
shell: cmd
145+
run: |
146+
call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\arrow_flight_sql_odbc.dll
147+
# GH-48270 TODO: Resolve segementation fault during Arrow library unload
148+
# GH-48269 TODO: Enable Flight & Flight SQL testing in MSVC CI
149+
# TODO: enable ODBC tests after GH-48270 and GH-48269 are resolved.
150+
# - name: Test
151+
# shell: cmd
152+
# run: |
153+
# set VCPKG_ROOT_KEEP=%VCPKG_ROOT%
154+
# call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
155+
# # For ORC
156+
# set TZDIR=${{ steps.setup-msys2.outputs.msys2-location }}\usr\share\zoneinfo
157+
158+
# # Convert VCPKG Windows path to MSYS path
159+
# for /f "usebackq delims=" %%I in (`bash -c "cygpath -u \"$VCPKG_ROOT_KEEP\""` ) do set VCPKG_ROOT=%%I
160+
161+
# bash -c "ci/scripts/cpp_test.sh $(pwd) $(pwd)/build"
162+
163+
- name: Install WiX Toolset
164+
shell: pwsh
165+
run: |
166+
Invoke-WebRequest -Uri https://github.com/wixtoolset/wix/releases/download/v6.0.0/wix-cli-x64.msi -OutFile wix-cli-x64.msi
167+
Start-Process -FilePath wix-cli-x64.msi -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait
168+
echo "C:\Program Files\WiX Toolset v6.0\bin\" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
169+
- name: Build MSI ODBC installer
170+
shell: pwsh
171+
run: |
172+
# Verify WiX version
173+
wix --version
174+
cd "${{ github.workspace }}\build\cpp"
175+
cpack
176+
- name: Upload the artifacts to the job
177+
uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
178+
with:
179+
name: flight-sql-odbc-msi-installer
180+
path: ${{ github.workspace }}\build\cpp\Apache Arrow Flight SQL ODBC-1.0.0-win64.msi

ci/scripts/cpp_test.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,8 +144,9 @@ if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && \
144144
CMAKE_PREFIX_PATH+="/lib/cmake/"
145145
;;
146146
esac
147+
# Search vcpkg before <prefix>/lib/cmake.
147148
if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_DEFAULT_TRIPLET}" ]; then
148-
CMAKE_PREFIX_PATH+=";${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET}"
149+
CMAKE_PREFIX_PATH="${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET};${CMAKE_PREFIX_PATH}"
149150
fi
150151
cmake \
151152
-S "${source_dir}/examples/minimal_build" \

cpp/cmake_modules/SetupCxxFlags.cmake

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,7 @@ if(WIN32)
186186
#
187187
# ARROW-2986: Without /EHsc we get C4530 warning
188188
set(CXX_COMMON_FLAGS "/W3 /EHsc")
189+
string(APPEND CMAKE_CXX_FLAGS " /EHsc")
189190
endif()
190191

191192
# Disable C5105 (macro expansion producing 'defined' has undefined

cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -129,13 +129,9 @@ if(WIN32)
129129
system_dsn.h)
130130
endif()
131131

132-
target_link_libraries(arrow_odbc_spi_impl PUBLIC arrow_flight_sql_shared
133-
arrow_compute_shared Boost::locale)
134-
135-
# Link libraries on MINGW64 and macOS
136-
if(MINGW OR APPLE)
137-
target_link_libraries(arrow_odbc_spi_impl PUBLIC ${ODBCINST})
138-
endif()
132+
target_link_libraries(arrow_odbc_spi_impl
133+
PUBLIC arrow_flight_sql_shared arrow_compute_shared Boost::locale
134+
${ODBCINST})
139135

140136
set_target_properties(arrow_odbc_spi_impl
141137
PROPERTIES ARCHIVE_OUTPUT_DIRECTORY

cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,6 @@
1919

2020
#include "arrow/flight/sql/odbc/odbc_impl/platform.h"
2121

22-
#include <sql.h>
23-
#include <sqlext.h>
2422
#include "arrow/array.h"
2523
#include "arrow/array/array_nested.h"
2624
#include "arrow/flight/sql/api.h"
@@ -32,6 +30,10 @@
3230
#include "arrow/flight/sql/odbc/odbc_impl/scalar_function_reporter.h"
3331
#include "arrow/flight/sql/odbc/odbc_impl/util.h"
3432

33+
// Include ODBC headers after arrow headers to avoid conflicts with sql_info_undef.h
34+
#include <sql.h>
35+
#include <sqlext.h>
36+
3537
// Aliases for entries in SqlInfoOptions::SqlInfo that are defined here
3638
// due to causing compilation errors conflicting with ODBC definitions.
3739
#define ARROW_SQL_IDENTIFIER_CASE 503

cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
#include "arrow/flight/sql/odbc/odbc_impl/spi/statement.h"
3131
#include "arrow/flight/sql/odbc/odbc_impl/util.h"
3232

33+
// Include ODBC headers after arrow headers to avoid conflicts with sql_info_undef.h
3334
#include <odbcinst.h>
3435
#include <sql.h>
3536
#include <sqlext.h>

cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,6 @@ add_arrow_test(flight_sql_odbc_test
5252
${ODBCINST}
5353
${SQLite3_LIBRARIES}
5454
arrow_odbc_spi_impl)
55+
56+
# Disable unity build due to sqlite_sql_info.cc conflict with sql.h and sqlext.h headers.
57+
set_target_properties(arrow-flight-sql-odbc-test PROPERTIES UNITY_BUILD OFF)

cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
#pragma once
19+
1820
#include "arrow/testing/gtest_util.h"
1921
#include "arrow/util/io_util.h"
2022
#include "arrow/util/utf8.h"

0 commit comments

Comments
 (0)