Skip to content

Commit 35c1bee

Browse files
alinaliBQvic-tsang
andcommitted
Add Arrow Flight SQL ODBC driver
Co-authored-by: alinalibq <[email protected]> Co-authored-by: rscales <[email protected]> Co-authored-by: justing-bq <[email protected]> Co-authored-by: vic-tsang <[email protected]> ODBC commits squashed into 1 to make rebasing to main easier. Add initial framework for odbc dll - Add ARROW_FLIGHT_SQL_ODBC option. If we set `ARROW_FLIGHT_SQL_ODBC=ON`, the flightsql odbc folder will be built - Add odbc api layer for entry_point.cc - builds odbc dll file, with ODBC APIs exported in odbc.def Address James' comments Fix `odbcabstraction` build errors and partially fix `flightsql_odbc` errors Fix boost-variant not found error - Adding dependencies from odbc/vcpkg.json to cpp/vcpkg.json - Fix whereami.cc and .h dependency; ported lates code Update whereami.cc - use `long` instead of `int64`. Fixed namespace issues. - PR CI fix: Add `parquet-testing` back Partial build fix for `flight_sql` folder - Replaced `namespace arrow` and `namespace odbcabstraction` with `using namespace ...` - fix flight_sql_connection.cc Fix `util::nullopt` to use `std::nullopt` - fix std::optional - fix BufferReader - Fix GetSchema - fix json_converter.cc - partial fix configuration.h - partial fix get_info_cache.cc - Fix winsock build error - Comment out `flight_sql` files that cannot build - Comment out configuration and unit tests - Comment out get info cache and system trust store Create initial odbc tests folder Implement SQLAllocEnv Fix cmake build Implement SQLFreeEnv Fix rest of build errors from `flightsql-odbc` - Fix get info errors - Fix for configuration window - added odbcinst library - Fix system trust store - unit test fixes - Add dependency of ARROW_COMPUTE. `arrow/compute/api.h` is used in `flight_sql`. Adding `ARROW_COMPUTE=ON` during build fixed run time unit tests failures. Implement SQLAllocConnect and SQLFreeConnect Fix build issue from static flight sql driver Lint and code style fixes Re-add deleted submodule parquet-testing clang-format lint fix cpplint lint fix Exclude whereami in rat exclude list C++/CLI lint fix Update parquet-testing to match commit from `main` Address Kou's comments ODBC directory lint fixes Catching the lint fixes outside of `flightsql-odbc` code Fix build warnings that get treated as error Implement SQLSetEnvAttr and SQLGetEnvAttr Implement use of ExecuteWithDiagnostics Doxygen Error Fixes and Address comments from Kou and James Address comments from Kou - Updates License.txt - Update cmake toolchain - Move whereami to `vendored` - Use string_view instead of NOLINT std::string Remove `whereami.cc` from arrow util build We are building whereami.cc as part of odbc Fix include headers to replace <> with "" Address comments from James Implement SQLGetDiagField SQLDriverConnect, SQLConnect and SQLDisconnect Implement stubs for SQLGetInfo, SQLGetDiagField and SQLGetDiagRec Separate RegisterDsn and UnregisterDsn from windows build Update code to save driver value from connection string Add ReadMes for ODBC and tests Fix test issues with string_view Address code reviews Update entry_points.cc to fix build issue Remove Dremio references Use emplace properly Address comment from Rob and add SQLDisconnect test case Add odbc.def and cmd file to rat_exclude Nit - remove duplicate lines Accidentally committed the change during git rebase Nit - remove usage of nullptr DSN window implementation Add licenses to `.cmd` and `.def` files Implement SQLGetDiagRec Build ODBC in Windows Workflow Tests are skipped for now. Updates for SQLGetDiagFieldW Enable Driver Logging Add todo to update logging system later Add logs Implement ODBC API with debug logging Enable mock test * Add todo for noauth validation * mock server with token auth Add tests * run same test with both modes * Enable ODBC tests in workflow * Switch current test cases to use FlightSQLODBCTestBase So the tests can be skipped when `TEST_CONNECT_STR` is not set. * Change tests to run on both mock and remote modes Wrap usage of TEST_CONNECT_STR where possible * Rename test fixtures and make connection string functions virtual * Fix lint issue * Attempt to enable ODBC build on Windows platforms * Attempt to fix clang64 and MinGW errors * Attempt to register ODBC * Address James' comments Use constant string for token * use ServerMiddleware to validate token Fix connection issues to DBT Labs PopulateCallOptions before making a connection Fix dsn window bug with advance properties Fix seg fault issue from empty string Implement SQLAllocStmt Follow-up DBT Labs connection fix Implement SQLGetDiag Rec and Field for statement Unicode support for DSN ODBC APIs * Let compiler append `W` to ODBC APIs where applicable. Initialize Kernel functions As per changes from #46261, we need to initialize Kernel library explicitly to get the functions registered Implement SQLSetConnectAttr and SQLGetConnectAttr ODBC Test Segfault Fix * Move `connection_test.cc` to last in `CMakeLists.txt`, this resolves the seg fault. The seg fault was not reproducible on my local environment, even when I use the msys2 shell. * Rename test executable to flight_sql_odbc_test SQLExecDirect implementation * SQLGetStmtAttr stub implementation * Fix missing break statement in SQLGetDiagRec * Run ShutdownProtobufLibrary as part of test environment * Add comment for GH-46889 * Pass call options to executed prepared statement `call_options_` contains the authentication token which is needed Implement SQLGetInfo SQLGetInfo Workflow Fixes SQLFetch & SQLGetData Implementation SQLGetStmtAttr stub implementation Stub call for SQLGetStmtAttr Enable statement handle allocation Implement SQLExecDirect - test hang issue Update odbc_api.cc Run `ShutdownProtobufLibrary` after all ODBC tests Resolves test hanging problem Fix missing break statement in SQLGetDiagRec Add tests Lint fixes Run ShutdownProtobufLibrary as part of test environment Add debug messages Draft code Remove drafts Add comment Add comment for GH-46889 Update statement test headers Pass call options to executed prepared statement `call_options_` contains the authentication token which is needed SQLFetch & SQLGetData initial impl EVERYTHING before this commit is for SQLExecDirect SQLFetch and SQLGetData TestSQLExecDirectSimpleQuery test Make GetData() return SQLRETURN `TestSQLExecDirectDataQuery` for remote and mock servers Remove unneeded test case Add more data types * Add SQL_GUID in getCTypeForSQLType * Add data types in query * Add checks for columns 1 to 25 * Update statement_test.cc Add varbinary test * Implement SQLNumResultCols Will leave tests for later * Implement SQLRowCount * Implement SQLMoreResults * Work on date retrieval fix The `TestSQLExecDirectDataQuery` checks for date is not working. * Fix date retrieval error by importing fix from Paul * imported fix from dremio/flightsql-odbc@d44d862, an Apache-Licensed repository * Add more checks for get data * Add more tests * Continue work on float truncation test * Disable float truncation test * Address comments from James - do not return errors for invalid rowCountPtr and columnCountPtr - add static cast for columnCount - Add tests with SQL_C_DEFAULT as target type - Add test checks for indicator - Fix SQL_NUMERIC, SQL_BIT, SQL_FLOAT target type * Fix build issues in CI * More fix for CI * Add test for invalid buffer length SQLPrepare & SQLExecute Implementation * Add tests * Address comments from Rob Code Style fixes * Move ifdefs outside of test cases * use `stmt` * Move logs to first line * fix test for `SQL_DRIVER_HSTMT` * Use camel case for tests Implement SQLGetStmtAttr and SQLSetStmtAttr SQLBindCol Implementation * SQL_UNBIND implementation + tests * Add tests for `SQL_ATTR_ROW_ARRAY_SIZE` * Add check for `SQL_ATTR_ROWS_FETCHED_PTR` Enable TestCloseConnectionWithOpenStatement test case SQLFetchScroll Implementation * Add checks for SQL_ATTR_ROWS_FETCHED_PTR * SQLFetchScroll is supposed to return the number of rows fetched using SQL_ATTR_ROWS_FETCHED_PTR SQLExtendedFetch Implementation * Implement rowCountPtr and rowStatusArray for SQLExtendedFetch * Add tests for SQLExtendedFetch * `SQLExtendedFetch` doesn't return `SQL_SUCCESS_WITH_INFO` for error state 22002. * Add tests for `SQL_ROWSET_SIZE` * Address comments from James GH-46584 Iterate endpoint * use suggestion from James for one-liner change to return `Status::OK` directly * fix for iterating endpoints, it is based on JDBC's fix for the same bug * save value of `FlightClientOptions` * Use `TestFlightServer` Add `arrow_flight_sql_shared` to enable usages for `TestFlightServer` * Fix build errors from missing `gmock` * Add checks for array data Update flight_sql_stream_chunk_buffer_test.cc Add `FlightStreamChunkBufferTest` unit test Draft test with `FlightSQLODBCMockEndpointTestBase` * Reference GH-47117 and Clean up code * Add driver and DSN to built-in properties to not pass driver/dsn as attributes to server * Allow `=` in values inside connection strings, fixes connectivity problems * Address comments from Rob SQLColumns Implementation * Initial impl for SQLColumns * Add test for SQLColumns * Add columns test with all supported column types - mock server doesn't support schema * Address comments from James - Test different data type return value for SQLColumns - Add todo comment for SQLDescribeCol tests --------- Co-authored-by: rscales <[email protected]> Fix bug of DSN not read properly * Fix reading DSN bug Use a simpler, more robust way to load the DSN SQLColAttribute implementation * SQLColAttribute initial impl * Switch to use `GetAttributeSQLWCHAR` to be unicode-compatible * Add SQLColAttribute tests and fix bugs in `flightsql-odbc` impl * Fixed bug with incorrect column count. It was returning column count + 1. * Fixed bug with incorrect numPrecRadix. It was returning SQL_NO_TOTAL (maps to -4) for non-numeric columns, but the correct value is 0. * Fixed bug with unsigned column to return true for unsigned columns (non-numeric columns or unsigned numeric columns) and false for signed columns (numeric columns) * Fixed bug with incorrect non-concise data type return value for date time type types. The correct return is SQL_DATETIME for all date time types * Address comments from James * use const for `ConvertToWString` * add nullptr check in odbc_api.cc layer * Add support for `SQL_COLUMN_LENGTH`, `SQL_COLUMN_PRECISION`, and `SQL_COLUMN_SCALE`. The driver will return the same values for ODBC2 and ODBC3. * Add tests for ODBC v2 SQLColAttributes values * I converted tests that can be run on remote servers where possible * Address comment from Rob Implement SQLTables Use C++ 20 standard for ODBC * Add support for building with C++20 * Upgrade spdlogs version as an attempt to fix build issues * Fix issue of missing unique_ptr definition Add tests for SQLMoreResults Implement SQLNativeSql Add diagnostic records * remove `SQLErrors`, as the driver manager is supposed to map `SQLErrors` to `SQLGetDiagRec` Implement tests for SQLNumResultCols SQLCloseCursor Implementation * fix error state to return the correct state 24000. * add tests for SQLCloseCursor and SQLFreeStmt(SQL_CLOSE) Add SQLColumns and SQLColAttribute tests with nullptr * Modify SQLTables test to run on both mock and remote * reorder error msg alphabetically Implement tests for SQLRowCount Fix merge conflicts Adjust searchable return value based on GH-46920 fix Fix `getCTypeForSQLType` return for interval types - Fix `getCTypeForSQLType` function to return the correct `C type` for interval SQL data types. Previously the function was checking for interval `C type` and returning interval `SQL type`, which was the opposite of the intended functionality. SQLGetTypeInfo implementation * Add tests for SQLGetTypeInfo * fix bug of createParams returning empty string instead of null * fix bug of non-concise data type being returned for datetime/interval * fix bug of longvarchar not converted to wlongvarchar when driver has unicode enabled * Address comments from James and add checks * test SQL_ALL_TYPES in ODBC 2.x. * test SQL_TYPE_* in ODBC 2.x, the driver manager reports invalid data type error. * test SQL_* datetime in ODBC 3.x for backwards compatibility. * add check for valid/invalid data type. Fix segfault issue from empty metadata * Use empty map in bug fix * Address comments from James Implement SQLDescribeCol Add SQLError Tests * add test to free null handles. Without handle value initialization, segfault error was seen Move `SQLGetDiagField` and `SQLGetDiagRec` tests to `errors_test.cc` * Address comments from James * Add ODBC Ver 2 tests * update test name to indicate if error handling is from driver manager. * add tests for warnings. * fix lint errors. * remove SQL_ATTR_APP_ROW_DESC that is not applicable to Environment Attribute. Add tests for SQLGetFunctions * Using `SQL_FUNC_EXISTS` macro fixed the issue of `api_exists` not read correctly * Fix SQLGetTypeInfo naming for ODBC ver 2 tests * Add more tests Add unsupported API checks. Add `TestSQLGetFunctionsODBCVer2`. Add `TestSQLGetFunctionsCheckSingleAPI`. * Update reset value * Add SQLDescribeCol function check Add additional SQLDescribeCol test cases Add Descriptor support in SQLAllocHandle and SQLFreeHandle * Descriptor allocation initial impl * Add descriptor handle tests * Add diagnostic error test for descriptor handle * the error is from driver manager as our driver doesn't have descriptor-specific APIs implemented yet * Add SQLGetDescField test from driver manager * Add doxygen doc comment Fix connection issues servers that require data in handshake * revert back to passing empty string in handshake Fix bug of system trust store not loaded properly * Bug: system trust store has default value of "true" regardless of "encryption" value. * But if encryption is set to false, system trust store cannot be used, so system trust store should be saved as false if user does not enable encryption. * It may confuse the user if they see encryption is set to false but system trust store is set to true. In this case, the driver will not do system trust store verification. The DSN window should reflect this accurately. CPack ODBC Windows msi installer Add ARROW_FLIGHT_SQL_ODBC_INSTALLER environment variable to enable creation of Windows installer. * Use CPack + WiX to create a `msi` installer. * run command `cpack` under the build directory to generate the installer. Register ODBC Driver on Windows in installer * In `wxs`, cannot use `package`, need to use `fragment` instead * Use component as feature will automatically be generated * Set Patch version + add installer instructions Use Arrow logo banner * replaces default banner with Arrow logo banner Indicate Driver Company and Version on Driver Manager * use versioninfo.rc.in template * use `1 VERSIONINFO` for it to work properly * Set version variables * Add `.rc` to gitignore * Add `@` variables to rc template Return nameLengthPtr value as length in characters Allow spaces while parsing Table Type mac changes added changes to build in MacOS Enable ODBC build on Windows Glib & Ruby workflows * Fix `boost::get` issue on GLib workflow * Fix GLib build issue with conflicts Compile entry_points.cc before odbc_api.cc due to conflict from sql.h and flight/types.h. Fix build warnings treated as error on Windows SQLDescribeCol expects `bufCharLen` to be of type `SQLSMALLINT`, so build warnings occur when `bufCharLen` is set to `SQLINTEGER`. Add more types for Array Conversion Set column attribute SQL_DESC_TYPE_NAME [Refactor] Remove reference of `Configuration` in odbc_connection.cc Remove reference of `Configuration` in odbc_connection.cc which is in library `odbcabstraction`. The reasoning is that `Configuration` is in `arrow_odbc_spi_impl` and this is a library that `odbcabstraction` depends on. I moved the logic for reading DSN values into `odbc_api.cc` instead, since that is a place that will result in no conflicts. The ODBC driver now does not save the DSN value in the connection string if the DSN value is put after Driver value. * Address comments from James Simply logic to check for first key value pair only Add TODO for non-UTC time zone support * Currently, the driver only supports UTC time zone Replace `spdlogs` with Arrow's Internal Logging * Add logging README * register kernel function conditionally * Resolves errors of kernel function already registered Logging files are not supported since GLOG is not enabled on Windows in Arrow repo. We can work + test on log file support on macOS/Linux phase Remove `RUN_ALL_TESTS` as they are not needed Previously, tests on Windows were not being run due to #47434. So we added `RUN_ALL_TESTS` as a workaround in early May, before #47434 was raised. Now that #47434 is resolved, we can remove `RUN_ALL_TESTS` as they are not needed. Other changes: * Renamed `arrow_odbc_spi_impl_test` to `odbc_spi_impl_test` so the test executable will be `arrow_odbc_spi_impl_test.exe` instead of `arrow_arrow_odbc_spi_impl_test.exe` Fix Function & Variable Naming Fix member variable naming Address code review comments for Arrow#40939 * Reword comment on null check Addresses comment #40939 (comment) * Use `std::memcpy` instead of `memcpy` Addresses comment #40939 (comment) * add include cstring * Remove warpdrive mentions * Fix lint * Address Justin's comments Combine utils and define utils namespace Use arrow::flight::sql::odbc namespace Update namespaces driver::flight_sql and driver::odbcabstraction to arrow::flight::sql::odbc. Fix seg fault from register log PR to keep apache-odbc in sync with arrow main branch. I found that, if the driver registers logs before compute kernel registration, segfault somehow occurs at when register kernel function is being used. I am not sure the root cause, but registering the logs after compute kernel registration seems to resolve the issue. Refactor odbcabstraction and odbc_impl flight_sql is renamed to odbc_impl. The files inside odbcabstraction have been moved to odbc_impl and odbcabstraction is removed. Replace boost::variant with std::variant * Remove `boost-optional` and `boost-variant` dependencies from vcpkg.json * Replace boost::variant with std::variant Fix EXPECT_EQ() to have expected on the left and actual on the right Code style naming convention Remove underscores in test names Sync arrow_flight_testing changes from Arrow::main branch Fix test failure due to typo Fix `FlightSQLODBCRemoteTestBase.TestSQLExecDirectDataQueryDefaultType` test failure due to typo The precision should be 38 in the original code. I think it got changed to 0 by accident somewhere. Disable tests with `DISABLED_` * Enable `TestSQLGetInfoDriverHdesc` We missed this test before, enabling the test now * Disable tests with `DISABLED_` Fix test assertions Define Test Fixtures * Connect/disconnect in SetUp/TearDown * Define Test Fixtures Resolve ODBC CI Build Errors * Fix `ConfigDSNW` not found error * Fix `TestFreeNullHandles` test `ConnectionTest` sets up the handles, so the `this->...` handles will not be null at beginning of test. Define New Base Test Fixtures Skip tests without connecting/disconnecting Address general test code review comments * Use RAII helper for allocating and freeing env/conn handles Avoids duplicated code Move test helper functions to anonymous namespace Add `[[nodiscard]]` for ODBC APIs Addresses community comment: https://github.com/apache/arrow/pull/47763/files#r2450014186 Remove `using List=` in test suite definitions Use static_cast for `SQLWCHAR` in type info test Use static_cast for `SQLWCHAR` in tables test * use mutable arrays for places where characters cannot be const Use static_cast for `SQLWCHAR` in columns test Update comment Use static_cast for `SQLWCHAR` in SQLGetInfo test * Address Justin's comments * Add fix for skipping the allocation of handles ^ Conflicts: ^ .github/workflows/cpp.yml ^ cpp/src/arrow/flight/sql/odbc/odbc_api.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/attribute_utils.h ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/config/configuration.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/config/configuration.h ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/encoding_utils.h ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/flight_sql_connection.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/flight_sql_connection_test.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/main.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.h ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/spi/connection.h ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/system_dsn.h ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/ui/dsn_configuration_window.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/util.cc ^ cpp/src/arrow/flight/sql/odbc/odbc_impl/util.h ^ cpp/src/arrow/flight/sql/odbc/tests/connection_test.cc ^ cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.cc Enable ODBC build in MacOS CI ODBC Standalone MSVC Build CI Remove DataSet Fix cache key Use odbc-specific cache key Fix Lint and Add odbc registration step Link appropriate GitHub issues that blocks ODBC tests Disable test phase and add schedule Run everyday 7 am Vancouver time Enable ODBC build on MSVC CI Code Clean up and enable ODBC tests in CI Still need to modify ODBCUtilEnvironment if we decide to use it. Still need to add ODBC V2 support so a different env and conn is used for ODBC 2 tests. Draft enable ODBC global setup/teardown Run ODBC test once outside of test script If ODBC test is run using MinGW Shell, segfault occurs. I am not getting any seg fault on my local MSVC Windows when I run the tests without the bash script. But if Windows CI breaks from running the standalone exe then I will look into this Prepend vcpkg to search vcpkg before `<prefix>/lib/cmake` Since we are installing dependencies on vcpkg, if `lib/cmake` is searched first, then cmake will look into that directory and use the wrong paths. Convert VCPKG Windows path to MSYS path Set `VCPKG_ROOT` in test phase Fix ODBC dll name Use `arrow_flight_sql_odbc.dll` instead of `libarrow_flight_sql_odbc.dll` which is the naming convention used on MinGW Windows Link ODBC library on all platforms On my local MSVC Windows machine, Visual Studio is used to build without needing to link ` ${ODBCINST}` explicitly, its behavior might be different from Ninja which is what CI uses. `${ODBCINST}` is likely needed by Linux as well, so adding it for all platforms. Attempt to resolve `arrow-compute-grouper-benchmark` build issue I think `if(ARROW_FLIGHT_TEST_LINKAGE STREQUAL "static")` might be more appropriate here, as I am seeing build issues due to both dynamic and static linking occurring. I see in 59903d0, this check is used for `arrow-filesystem-s3fs-benchmark` Disable `UNITY_BUILD` for ODBC test due to conflict on `sqlite_sql_info.cc` On some workflows, unity build is set to ON to make the build faster. This is an attempt to resolve the build issue. Remove debug messages Fix lint Add `sql_info_undef.h` to sqlite_sql_info.cc Undefine duplicates in SQLGetInfo Add `#pragma once` to odbc_test_suite.h To avoid redefinition error Attempt to resolve conflict with sql/types.h Attempt to include Arrow headers before ODBC headers to avoid conflict with arrow/flight/sql/types.h [GH-48084] Replace boost::optional with std::optional Addresses comment #40939 (comment) Replace boost::optional with std::optional in ODBC codebase #48084 Fix lint Remove `BOOST_SOURCE=BUNDLED` to use vcpkg's dynamic link to boost Since we need to use link to boost dynamically, we need to remove the bundled boost library flag. Add debug messages. Remove `ARROW_BOOST_USE_SHARED = OFF` Since we have a release build, can enable boost as shared. With `ARROW_BOOST_USE_SHARED = OFF`, I am getting error ``` D:\a\arrow\arrow\build\cpp\vcpkg_installed\x64-windows\include\boost/filesystem/config.hpp(96): fatal error C1189: #error: Must not define both BOOST_FILESYSTEM_DYN_LINK and BOOST_FILESYSTEM_STATIC_LINK ``` Also increased time limit, since 2 hours don't seem to be enough to finish the vcpkg build Change to release build Getting ``` orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in unity_2_cxx.cxx.obj orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MDd_DynamicDebug' in unity_2_cxx.cxx.obj ``` when building with debug and presumably ARROW_ORC Set ARROW_BOOST_USE_SHARED to OFF `ARROW_BOOST_USE_SHARED` is restored to `OFF`, which according to Windows doc should help with the debug build now. Retore value of `ARROW_BUILD_BENCHMARKS`, though noting it is set to `OFF` in GLib MSVC workflow. Add VCPKG set up Borrowed from the Glib workflow Add `/EHsc` flag * Add back `CMAKE_CXX_STANDARD: "17"` Remove `CMAKE_CXX_STANDARD` 17 * reason: gtest can't be used with C++17. Arrow project doesn't support C++ 17 yet. Extend run-time to 2hr windows-mingw also has timeout of 2hr Empty commit to trigger CI Specify `VCPKG_BINARY_SOURCES` and `VCPKG_DEFAULT_TRIPLET` Specify VCPKG_TARGET_TRIPLET as x64 windows Set ARROW_DEPENDENCY_SOURCE to VCPKG To make it easier to manage dependencies Enable static build on MSVC `ARROW_BUILD_BENCHMARKS` prevents Flight from being built Remove `ARROW_BOOST_USE_SHARED` Having static vs. shared issue Enable Flight & Flight SQL for ODBC Enable ODBC build on MSVC CI Enable regular ctest tests Remove undef items Add concurrency and permissions to odbc yml Create cpp_odbc.yml Fix architecture in CI * Add ODBC installer MSVC CI Mac Setup ODBC ini Script * added setup script for mac and platform folders updated read me windows file path updated windows folder related file paths added license Fix Lint issues Add install ODBC script Update install_odbc_ini.sh This script depends on the ODBC install script. Add TODO note and update DSN * Fix typo in README --------- Fix lint Fix lint Update flight_sql_stream_chunk_buffer.cc Update statement_test.cc Move MacOS ODBC workflows to cpp_odbc.yml Fix Lint Fix `ODBC::GetSqlWCharSize()` Add back missing checks for error handle tests Co-Authored-By: Victor Tsang <[email protected]> Co-Authored-By: Alina (Xi) Li <[email protected]> Co-Authored-By: vic-tsang <[email protected]>
1 parent f65ee2c commit 35c1bee

File tree

101 files changed

+6873
-361
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

101 files changed

+6873
-361
lines changed

.github/workflows/cpp.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -310,7 +310,8 @@ jobs:
310310
ARROW_DATASET: ON
311311
ARROW_FLIGHT: ON
312312
ARROW_FLIGHT_SQL: ON
313-
ARROW_FLIGHT_SQL_ODBC: OFF
313+
ARROW_FLIGHT_SQL_ODBC: ON
314+
ARROW_FLIGHT_SQL_ODBC_INSTALLER: ON
314315
ARROW_GANDIVA: ON
315316
ARROW_GCS: ON
316317
ARROW_HDFS: OFF
@@ -389,6 +390,10 @@ jobs:
389390
PIPX_BASE_PYTHON: ${{ steps.python-install.outputs.python-path }}
390391
run: |
391392
ci/scripts/install_gcs_testbench.sh default
393+
- name: Register Flight SQL ODBC Driver
394+
shell: cmd
395+
run: |
396+
call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\libarrow_flight_sql_odbc.dll
392397
- name: Test
393398
shell: msys2 {0}
394399
run: |

.github/workflows/cpp_odbc.yml

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: C++ ODBC
19+
20+
on:
21+
push:
22+
branches:
23+
- '**'
24+
- '!dependabot/**'
25+
tags:
26+
- '**'
27+
paths:
28+
- '.github/workflows/cpp_odbc.yml'
29+
- 'ci/scripts/cpp_*'
30+
- 'cpp/src/arrow/flight/sql/odbc/*'
31+
pull_request:
32+
paths:
33+
- '.github/workflows/cpp_odbc.yml'
34+
- 'ci/scripts/cpp_*'
35+
- 'cpp/src/arrow/flight/sql/odbc/*'
36+
schedule:
37+
- cron: '0 13 * * *'
38+
39+
concurrency:
40+
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
41+
cancel-in-progress: true
42+
43+
permissions:
44+
contents: read
45+
46+
jobs:
47+
windows:
48+
runs-on: windows-2022
49+
timeout-minutes: 240
50+
env:
51+
ARROW_BUILD_SHARED: ON
52+
ARROW_BUILD_STATIC: ON
53+
ARROW_BUILD_TESTS: ON
54+
ARROW_BUILD_TYPE: release
55+
ARROW_DEPENDENCY_SOURCE: VCPKG
56+
ARROW_FLIGHT: ON
57+
ARROW_FLIGHT_SQL: ON
58+
ARROW_FLIGHT_SQL_ODBC: ON
59+
ARROW_FLIGHT_SQL_ODBC_INSTALLER: ON
60+
ARROW_SIMD_LEVEL: AVX2
61+
CMAKE_CXX_STANDARD: "17"
62+
CMAKE_GENERATOR: Ninja
63+
CMAKE_INSTALL_PREFIX: /usr
64+
VCPKG_BINARY_SOURCES: 'clear;nuget,GitHub,readwrite'
65+
VCPKG_DEFAULT_TRIPLET: x64-windows
66+
steps:
67+
- name: Disable Crash Dialogs
68+
run: |
69+
reg add `
70+
"HKCU\SOFTWARE\Microsoft\Windows\Windows Error Reporting" `
71+
/v DontShowUI `
72+
/t REG_DWORD `
73+
/d 1 `
74+
/f
75+
- name: Checkout Arrow
76+
uses: actions/checkout@v5
77+
with:
78+
fetch-depth: 0
79+
submodules: recursive
80+
- name: Download Timezone Database
81+
shell: bash
82+
run: ci/scripts/download_tz_database.sh
83+
- name: Install msys2 (for tzdata for ORC tests)
84+
uses: msys2/setup-msys2@v2
85+
id: setup-msys2
86+
- name: Install cmake
87+
shell: bash
88+
run: |
89+
ci/scripts/install_cmake.sh 4.1.2 /usr
90+
- name: Install ccache
91+
shell: bash
92+
run: |
93+
ci/scripts/install_ccache.sh 4.12.1 /usr
94+
- name: Setup ccache
95+
shell: bash
96+
run: |
97+
ci/scripts/ccache_setup.sh
98+
- name: ccache info
99+
id: ccache-info
100+
shell: bash
101+
run: |
102+
echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT
103+
- name: Cache ccache
104+
uses: actions/cache@v4
105+
with:
106+
path: ${{ steps.ccache-info.outputs.cache-dir }}
107+
key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }}
108+
restore-keys: cpp-odbc-ccache-windows-x64-
109+
- name: Checkout vcpkg
110+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
111+
with:
112+
fetch-depth: 0
113+
path: vcpkg
114+
repository: microsoft/vcpkg
115+
- name: Bootstrap vcpkg
116+
run: |
117+
vcpkg\bootstrap-vcpkg.bat
118+
$VCPKG_ROOT = $(Resolve-Path -LiteralPath "vcpkg").ToString()
119+
Write-Output ${VCPKG_ROOT} | `
120+
Out-File -FilePath ${Env:GITHUB_PATH} -Encoding utf8 -Append
121+
Write-Output "VCPKG_ROOT=${VCPKG_ROOT}" | `
122+
Out-File -FilePath ${Env:GITHUB_ENV} -Encoding utf8 -Append
123+
- name: Setup NuGet credentials for vcpkg caching
124+
shell: bash
125+
run: |
126+
$(vcpkg fetch nuget | tail -n 1) \
127+
sources add \
128+
-source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" \
129+
-storepasswordincleartext \
130+
-name "GitHub" \
131+
-username "$GITHUB_REPOSITORY_OWNER" \
132+
-password "${{ secrets.GITHUB_TOKEN }}"
133+
$(vcpkg fetch nuget | tail -n 1) \
134+
setapikey "${{ secrets.GITHUB_TOKEN }}" \
135+
-source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json"
136+
- name: Build
137+
shell: cmd
138+
run: |
139+
set VCPKG_ROOT_KEEP=%VCPKG_ROOT%
140+
call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
141+
set VCPKG_ROOT=%VCPKG_ROOT_KEEP%
142+
bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"
143+
- name: Register Flight SQL ODBC Driver
144+
shell: cmd
145+
run: |
146+
call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\arrow_flight_sql_odbc.dll
147+
# GH-48270 TODO: Resolve segementation fault during Arrow library unload
148+
# GH-48269 TODO: Enable Flight & Flight SQL testing in MSVC CI
149+
# TODO: enable ODBC tests after GH-48270 and GH-48269 are resolved.
150+
# - name: Test
151+
# shell: cmd
152+
# run: |
153+
# set VCPKG_ROOT_KEEP=%VCPKG_ROOT%
154+
# call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
155+
# # For ORC
156+
# set TZDIR=${{ steps.setup-msys2.outputs.msys2-location }}\usr\share\zoneinfo
157+
158+
# # Convert VCPKG Windows path to MSYS path
159+
# for /f "usebackq delims=" %%I in (`bash -c "cygpath -u \"$VCPKG_ROOT_KEEP\""` ) do set VCPKG_ROOT=%%I
160+
161+
# bash -c "ci/scripts/cpp_test.sh $(pwd) $(pwd)/build"
162+
163+
- name: Install WiX Toolset
164+
shell: pwsh
165+
run: |
166+
Invoke-WebRequest -Uri https://github.com/wixtoolset/wix/releases/download/v6.0.0/wix-cli-x64.msi -OutFile wix-cli-x64.msi
167+
Start-Process -FilePath wix-cli-x64.msi -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait
168+
echo "C:\Program Files\WiX Toolset v6.0\bin\" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
169+
- name: Build MSI ODBC installer
170+
shell: pwsh
171+
run: |
172+
# Verify WiX version
173+
wix --version
174+
cd "${{ github.workspace }}\build\cpp"
175+
cpack
176+
- name: Upload the artifacts to the job
177+
uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
178+
with:
179+
name: flight-sql-odbc-msi-installer
180+
path: ${{ github.workspace }}\build\cpp\Apache Arrow Flight SQL ODBC-1.0.0-win64.msi
181+
macos:
182+
name: ${{ matrix.architecture }} macOS ${{ matrix.macos-version }} C++
183+
runs-on: macos-${{ matrix.macos-version }}
184+
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
185+
timeout-minutes: 75
186+
strategy:
187+
fail-fast: false
188+
matrix:
189+
include:
190+
- architecture: AMD64
191+
macos-version: "15-intel"
192+
- architecture: ARM64
193+
macos-version: "14"
194+
env:
195+
ARROW_BUILD_TESTS: ON
196+
ARROW_FLIGHT_SQL_ODBC: ON
197+
ARROW_HOME: /tmp/local
198+
steps:
199+
- name: Checkout Arrow
200+
uses: actions/[email protected]
201+
with:
202+
fetch-depth: 0
203+
submodules: recursive
204+
- name: Install Dependencies
205+
run: |
206+
brew bundle --file=cpp/Brewfile
207+
export LIBIODBC_DIR="$(brew --cellar libiodbc)/$(brew list --versions libiodbc | awk '{print $2}')"
208+
echo ODBC_INCLUDE_DIR="$LIBIODBC_DIR/include" >> $GITHUB_ENV
209+
echo ODBC_LIB_DIR="$LIBIODBC_DIR/lib" >> $GITHUB_ENV
210+
- name: Setup ccache
211+
run: |
212+
ci/scripts/ccache_setup.sh
213+
- name: ccache info
214+
id: ccache-info
215+
run: |
216+
echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT
217+
- name: Cache ccache
218+
uses: actions/cache@v4
219+
with:
220+
path: ${{ steps.ccache-info.outputs.cache-dir }}
221+
key: cpp-ccache-macos-${{ matrix.macos-version }}-${{ hashFiles('cpp/**') }}
222+
restore-keys: cpp-ccache-macos-${{ matrix.macos-version }}-
223+
- name: Build
224+
run: |
225+
# Homebrew uses /usr/local as prefix. So packages
226+
# installed by Homebrew also use /usr/local/include. We
227+
# want to include headers for packages installed by
228+
# Homebrew as system headers to ignore warnings in them.
229+
# But "-isystem /usr/local/include" isn't used by CMake
230+
# because /usr/local/include is marked as the default
231+
# include path. So we disable -Werror to avoid build error
232+
# by warnings from packages installed by Homebrew.
233+
export BUILD_WARNING_LEVEL=PRODUCTION
234+
ci/scripts/cpp_build.sh $(pwd) $(pwd)/build
235+
- name: Register Flight SQL ODBC Driver
236+
run: |
237+
chmod +x cpp/src/arrow/flight/sql/odbc/install/mac/install_odbc.sh
238+
sudo cpp/src/arrow/flight/sql/odbc/install/mac/install_odbc.sh $(pwd)/build/cpp/debug/libarrow_flight_sql_odbc.dylib
239+
- name: Test
240+
shell: bash
241+
run: |
242+
ci/scripts/cpp_test.sh $(pwd) $(pwd)/build

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ dependency-reduced-pom.xml
3232
MANIFEST
3333
compile_commands.json
3434
build.ninja
35+
build*/
3536

3637
# Generated Visual Studio files
3738
*.vcxproj
@@ -107,3 +108,6 @@ java/.mvn/.develocity/
107108
# rat
108109
filtered_rat.txt
109110
rat.txt
111+
112+
# rc
113+
*.rc

ci/scripts/cpp_build.sh

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,7 @@ else
213213
-DARROW_FLIGHT=${ARROW_FLIGHT:-OFF} \
214214
-DARROW_FLIGHT_SQL=${ARROW_FLIGHT_SQL:-OFF} \
215215
-DARROW_FLIGHT_SQL_ODBC=${ARROW_FLIGHT_SQL_ODBC:-OFF} \
216+
-DARROW_FLIGHT_SQL_ODBC_INSTALLER=${ARROW_FLIGHT_SQL_ODBC_INSTALLER:-OFF} \
216217
-DARROW_FUZZING=${ARROW_FUZZING:-OFF} \
217218
-DARROW_GANDIVA_PC_CXX_FLAGS=${ARROW_GANDIVA_PC_CXX_FLAGS:-} \
218219
-DARROW_GANDIVA=${ARROW_GANDIVA:-OFF} \
@@ -259,7 +260,7 @@ else
259260
-DCMAKE_BUILD_TYPE=${ARROW_BUILD_TYPE:-debug} \
260261
-DCMAKE_VERBOSE_MAKEFILE=${CMAKE_VERBOSE_MAKEFILE:-OFF} \
261262
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
262-
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
263+
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-} -I${ODBC_INCLUDE_DIR:-} -L${ODBC_LIB_DIR:-}" \
263264
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-17}" \
264265
-DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
265266
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
@@ -270,6 +271,7 @@ else
270271
-DgRPC_SOURCE=${gRPC_SOURCE:-} \
271272
-DGTest_SOURCE=${GTest_SOURCE:-} \
272273
-Dlz4_SOURCE=${lz4_SOURCE:-} \
274+
-DODBC_INCLUDE_DIR="${ODBC_INCLUDE_DIR:-}" \
273275
-Dopentelemetry-cpp_SOURCE=${opentelemetry_cpp_SOURCE:-} \
274276
-DORC_SOURCE=${ORC_SOURCE:-} \
275277
-DPARQUET_BUILD_EXAMPLES=${PARQUET_BUILD_EXAMPLES:-OFF} \

ci/scripts/cpp_test.sh

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ case "$(uname)" in
6161
n_jobs=$(sysctl -n hw.ncpu)
6262
# TODO: https://github.com/apache/arrow/issues/40410
6363
exclude_tests+=("arrow-s3fs-test")
64+
exclude_tests+=("arrow-flight-sql-odbc-test")
6465
;;
6566
MINGW*)
6667
n_jobs=${NUMBER_OF_PROCESSORS:-1}
@@ -144,8 +145,9 @@ if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && \
144145
CMAKE_PREFIX_PATH+="/lib/cmake/"
145146
;;
146147
esac
148+
# Search vcpkg before <prefix>/lib/cmake.
147149
if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_DEFAULT_TRIPLET}" ]; then
148-
CMAKE_PREFIX_PATH+=";${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET}"
150+
CMAKE_PREFIX_PATH="${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET};${CMAKE_PREFIX_PATH}"
149151
fi
150152
cmake \
151153
-S "${source_dir}/examples/minimal_build" \

cpp/Brewfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ brew "git"
2828
brew "glog"
2929
brew "googletest"
3030
brew "grpc"
31+
brew "libiodbc"
3132
brew "llvm"
3233
brew "lz4"
3334
brew "mimalloc"

cpp/CMakeLists.txt

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -720,9 +720,13 @@ endif()
720720

721721
install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/../LICENSE.txt
722722
${CMAKE_CURRENT_SOURCE_DIR}/../NOTICE.txt
723-
${CMAKE_CURRENT_SOURCE_DIR}/README.md DESTINATION "${ARROW_DOC_DIR}")
723+
${CMAKE_CURRENT_SOURCE_DIR}/README.md
724+
DESTINATION "${ARROW_DOC_DIR}"
725+
COMPONENT arrow_doc)
724726

725-
install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/gdb_arrow.py DESTINATION "${ARROW_GDB_DIR}")
727+
install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/gdb_arrow.py
728+
DESTINATION "${ARROW_GDB_DIR}"
729+
COMPONENT arrow_gdb)
726730

727731
#
728732
# Validate and print out Arrow configuration options

cpp/CMakePresets.json

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,7 @@
180180
"ARROW_BUILD_EXAMPLES": "ON",
181181
"ARROW_BUILD_UTILITIES": "ON",
182182
"ARROW_FLIGHT_SQL_ODBC": "ON",
183+
"ARROW_FLIGHT_SQL_ODBC_INSTALLER": "ON",
183184
"ARROW_TENSORFLOW": "ON",
184185
"PARQUET_BUILD_EXAMPLES": "ON",
185186
"PARQUET_BUILD_EXECUTABLES": "ON"
@@ -314,6 +315,17 @@
314315
"displayName": "Debug build with tests and Flight SQL",
315316
"cacheVariables": {}
316317
},
318+
{
319+
"name": "ninja-debug-flight-sql-odbc",
320+
"inherits": [
321+
"features-flight-sql",
322+
"base-debug"
323+
],
324+
"displayName": "Debug build with tests and Flight SQL ODBC",
325+
"cacheVariables": {
326+
"ARROW_FLIGHT_SQL_ODBC": "ON"
327+
}
328+
},
317329
{
318330
"name": "ninja-debug-gandiva",
319331
"inherits": [
@@ -510,6 +522,17 @@
510522
"displayName": "Release build with Flight SQL",
511523
"cacheVariables": {}
512524
},
525+
{
526+
"name": "ninja-release-flight-sql-odbc",
527+
"inherits": [
528+
"features-flight-sql",
529+
"base-release"
530+
],
531+
"displayName": "Release build with Flight SQL ODBC",
532+
"cacheVariables": {
533+
"ARROW_FLIGHT_SQL_ODBC": "ON"
534+
}
535+
},
513536
{
514537
"name": "ninja-release-gandiva",
515538
"inherits": [

0 commit comments

Comments
 (0)