fix(ut,orc): fix orc read ut under debian and adjust some options #37

SGZW · 2025-12-26T14:22:53Z

Purpose

Linked issue: close #xxx

src/paimon/format/orc/orc_file_batch_reader.cpp

refer: https://github.com/apache/arrow/pull/34591/files,  
// Orc timestamp type is error-prone since it serializes values in the writer timezone
// and reads them back in the reader timezone. To avoid this, both the Apache Orc C++
// writer and reader set the timezone to GMT by default to avoid any conversion.
// We follow the same practice here explicitly to make sure readers are aware of this.

rest of fix:

// refer: https://github.com/eggert/tz/blob/main/asia#L653
// When using the Asia/Shanghai timezone under Debian, timestamps prior to 1901 have an
// additional offset of 5 minutes and 43 seconds

The specific verification steps are as follows: I modified the code related to ORC format and added some debug logs, focusing primarily on the code for timestamp time zone conversion. The details are as follows:

  void TimestampColumnReader::next(ColumnVectorBatch& rowBatch, uint64_t numValues, char* notNull) {
    ColumnReader::next(rowBatch, numValues, notNull);
    notNull = rowBatch.hasNulls ? rowBatch.notNull.data() : nullptr;
    TimestampVectorBatch& timestampBatch = dynamic_cast<TimestampVectorBatch&>(rowBatch);
    int64_t* secsBuffer = timestampBatch.data.data();
    secondsRle_->next(secsBuffer, numValues, notNull);
    int64_t* nanoBuffer = timestampBatch.nanoseconds.data();
    nanoRle_->next(nanoBuffer, numValues, notNull);

    // Construct the values
    for (uint64_t i = 0; i < numValues; i++) {
      if (notNull == nullptr || notNull[i]) {
        uint64_t zeros = nanoBuffer[i] & 0x7;
        nanoBuffer[i] >>= 3;
        if (zeros != 0) {
          for (uint64_t j = 0; j <= zeros; ++j) {
            nanoBuffer[i] *= 10;
          }
        }

        // ORC-306: compensate -1s for JDK bug in java.sql.Timestamp
        int64_t writerTime = secsBuffer[i] + epochOffset_;
        if (writerTime < 0 && nanoBuffer[i] > 999999) {
            writerTime -= 1;
        }
        if (!sameTimezone_) {
          std::stringstream s1,s2;
          writerTimezone_->print(s1);
          readerTimezone_->print(s2);
          std::cerr << "### writer zone ### \n" << s1.str() << std::endl;
          std::cerr << "### reader zone ### \n" << s2.str() << std::endl;
          // adjust timestamp value to same wall clock time if writer and reader
          // time zones have different rules, which is required for Apache Orc.
          const auto& wv = writerTimezone_->getVariant(writerTime);
          const auto& rv = readerTimezone_->getVariant(writerTime);
          std::cerr << "wv: " << wv.toString() << ", rv: " << rv.toString() << std::endl;
          if (!wv.hasSameTzRule(rv)) {
            // If the timezone adjustment moves the millis across a DST boundary,
            // we need to reevaluate the offsets.
            int64_t adjustedTime = writerTime + wv.gmtOffset - rv.gmtOffset;
            const auto& adjustedReader = readerTimezone_->getVariant(adjustedTime);
            writerTime = writerTime + wv.gmtOffset - adjustedReader.gmtOffset;
          }
        }
        std::cerr << "epochOffset_: " << epochOffset_ << std::endl;
        std::cerr << "secsBuffer[i]: " << secsBuffer[i] << ", writerTime: " << writerTime << std::endl;
        secsBuffer[i] = writerTime;
      }
    }
  }

In the Debian environment, I used Ubuntu 24.04's tzdata (wget http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata_2025b-0ubuntu0.24.04.1_all.deb) and Debian's tzdata for the TZDIR environment variable separately, with the results as follows.

Debian:

### writer zone ###
Timezone file: /usr/share/zoneinfo/Asia/Shanghai
  Version: 2
  Future rule: CST-8
  standard CST 28800
  Variant 0: LMT 29143
  Variant 1: CDT 32400 (dst)
  Variant 2: CST 28800
  Transition: null (-576460752303423488) -> LMT
  Transition: 1900-12-31 15:54:17 (-2177481943) -> CST
  Transition: 1919-04-12 16:00:00 (-1600675200) -> CDT
  Transition: 1919-09-30 15:00:00 (-1585904400) -> CST
  Transition: 1940-05-31 16:00:00 (-933667200) -> CDT
  Transition: 1940-10-12 15:00:00 (-922093200) -> CST
  Transition: 1941-03-14 16:00:00 (-908870400) -> CDT
  Transition: 1941-11-01 15:00:00 (-888829200) -> CST
  Transition: 1942-01-30 16:00:00 (-881049600) -> CDT
  Transition: 1945-09-01 15:00:00 (-767869200) -> CST
  Transition: 1946-05-14 16:00:00 (-745833600) -> CDT
  Transition: 1946-09-30 15:00:00 (-733827600) -> CST
  Transition: 1947-04-14 16:00:00 (-716889600) -> CDT
  Transition: 1947-10-31 15:00:00 (-699613200) -> CST
  Transition: 1948-04-30 16:00:00 (-683884800) -> CDT
  Transition: 1948-09-30 15:00:00 (-670669200) -> CST
  Transition: 1949-04-30 16:00:00 (-652348800) -> CDT
  Transition: 1949-05-27 15:00:00 (-650019600) -> CST
  Transition: 1986-05-03 18:00:00 (515527200) -> CDT
  Transition: 1986-09-13 17:00:00 (527014800) -> CST
  Transition: 1987-04-11 18:00:00 (545162400) -> CDT
  Transition: 1987-09-12 17:00:00 (558464400) -> CST
  Transition: 1988-04-16 18:00:00 (577216800) -> CDT
  Transition: 1988-09-10 17:00:00 (589914000) -> CST
  Transition: 1989-04-15 18:00:00 (608666400) -> CDT
  Transition: 1989-09-16 17:00:00 (621968400) -> CST
  Transition: 1990-04-14 18:00:00 (640116000) -> CDT
  Transition: 1990-09-15 17:00:00 (653418000) -> CST
  Transition: 1991-04-13 18:00:00 (671565600) -> CDT
  Transition: 1991-09-14 17:00:00 (684867600) -> CST

### reader zone ###
Timezone file: /usr/share/zoneinfo/GMT
  Version: 2
  Future rule: GMT0
  standard GMT 0
  Variant 0: GMT 0
  Transition: null (-576460752303423488) -> GMT

wv: LMT 29143, rv: GMT 0
epochOffset_: 1420041600
secsBuffer[i]: -3660591639, writerTime: -2240520897

Ubuntu 24.04

### writer zone ###
Timezone file: /home/zhangwei.95/zoneinfo/usr/share/zoneinfo//Asia/Shanghai
  Version: 2
  Future rule: CST-8
  standard CST 28800
  Variant 0: LMT 29143
  Variant 1: CDT 32400 (dst)
  Variant 2: CST 28800
  Transition: 1900-12-31 15:54:17 (-2177481943) -> CST
  Transition: 1919-04-12 16:00:00 (-1600675200) -> CDT
  Transition: 1919-09-30 15:00:00 (-1585904400) -> CST
  Transition: 1940-05-31 16:00:00 (-933667200) -> CDT
  Transition: 1940-10-12 15:00:00 (-922093200) -> CST
  Transition: 1941-03-14 16:00:00 (-908870400) -> CDT
  Transition: 1941-11-01 15:00:00 (-888829200) -> CST
  Transition: 1942-01-30 16:00:00 (-881049600) -> CDT
  Transition: 1945-09-01 15:00:00 (-767869200) -> CST
  Transition: 1946-05-14 16:00:00 (-745833600) -> CDT
  Transition: 1946-09-30 15:00:00 (-733827600) -> CST
  Transition: 1947-04-14 16:00:00 (-716889600) -> CDT
  Transition: 1947-10-31 15:00:00 (-699613200) -> CST
  Transition: 1948-04-30 16:00:00 (-683884800) -> CDT
  Transition: 1948-09-30 15:00:00 (-670669200) -> CST
  Transition: 1949-04-30 16:00:00 (-652348800) -> CDT
  Transition: 1949-05-27 15:00:00 (-650019600) -> CST
  Transition: 1986-05-03 18:00:00 (515527200) -> CDT
  Transition: 1986-09-13 17:00:00 (527014800) -> CST
  Transition: 1987-04-11 18:00:00 (545162400) -> CDT
  Transition: 1987-09-12 17:00:00 (558464400) -> CST
  Transition: 1988-04-16 18:00:00 (577216800) -> CDT
  Transition: 1988-09-10 17:00:00 (589914000) -> CST
  Transition: 1989-04-15 18:00:00 (608666400) -> CDT
  Transition: 1989-09-16 17:00:00 (621968400) -> CST
  Transition: 1990-04-14 18:00:00 (640116000) -> CDT
  Transition: 1990-09-15 17:00:00 (653418000) -> CST
  Transition: 1991-04-13 18:00:00 (671565600) -> CDT
  Transition: 1991-09-14 17:00:00 (684867600) -> CST

### reader zone ###
Timezone file: /home/zhangwei.95/zoneinfo/usr/share/zoneinfo//GMT
  Version: 2
  Future rule: GMT0
  standard GMT 0
  Variant 0: GMT 0

wv: CST 28800, rv: GMT 0
epochOffset_: 1420041600
secsBuffer[i]: -3660591639, writerTime: -2240521240

Analysis

The result is self-evident: Debian’s tzdata contains an extra LMT Timezone entry, which ultimately leads to a 5 minutes and 30 seconds offset in the final result.

Root Cause of the Issue: ORC Timestamp conversion has bugs

First, it is essential to understand how ORC stores Timestamps:

During Writing

What is stored is the second offset relative to epoch_.

Calculation of epoch_

cpptime_t utcEpoch = timegm(&epochStruct);  // 2015-01-01 00:00:00 UTC
epoch_ = utcEpoch - getVariant(utcEpoch).gmtOffset;

A critical issue here is:epoch_ is calculated using the time zone variant offset at the moment of 2015-01-01.

Core Conflict

cppwriterTime = secsBuffer[i] + epochOffset_; This formula implies the following assumption:secsBuffer[i] is an offset relative to a "local time epoch",yet this "local time epoch" uses the time zone rules valid as of 2015-01-01.
Errors occur when the actual time point (epoch + secsBuffer[i]) applies different time zone rules.

Fix
For the time being, I have bypassed this issue by detecting the OS version. To resolve the problem at its root, we need to ensure that the writer zone of the ORC file is set to the UTC/GMT time zone for data writing, thus avoiding similar issues.

Tests

API and Format

Documentation

SGZW · 2025-12-26T14:39:41Z

@lucasfang @lxy-9602 @ChaomingZhangCN PTAL, thanks!

lucasfang · 2025-12-30T01:59:19Z

Nice and thorough work! We should check if such changes would introduce any compatibility issues, and whether the results are consistent with the behavior of Java Paimon. @lxy-9602 @zjw1111 @lszskye

SGZW · 2025-12-30T02:10:47Z

Nice and thorough work! We should check if such changes would introduce any compatibility issues, and whether the results are consistent with the behavior of Java Paimon. @lxy-9602 @zjw1111 @lszskye

CI report errors as follows:

  Error: error: failed to fetch some objects from 'https://github.com/alibaba/paimon-cpp.git/info/lfs'
  The process '/usr/bin/git' failed with exit code 2
  Waiting 16 seconds before trying again
  /usr/bin/git lfs fetch origin refs/remotes/pull/37/merge
  Fetching reference refs/remotes/pull/37/merge
  batch response: This repository exceeded its LFS budget. The account responsible for the budget should increase it to restore access.
  Error: error: failed to fetch some objects from 'https://github.com/alibaba/paimon-cpp.git/info/lfs'
  Error: The process '/usr/bin/git' failed with exit code 2

lucasfang · 2025-12-30T07:24:14Z

Nice and thorough work! We should check if such changes would introduce any compatibility issues, and whether the results are consistent with the behavior of Java Paimon. @lxy-9602 @zjw1111 @lszskye

CI report errors as follows:

  Error: error: failed to fetch some objects from 'https://github.com/alibaba/paimon-cpp.git/info/lfs'
  The process '/usr/bin/git' failed with exit code 2
  Waiting 16 seconds before trying again
  /usr/bin/git lfs fetch origin refs/remotes/pull/37/merge
  Fetching reference refs/remotes/pull/37/merge
  batch response: This repository exceeded its LFS budget. The account responsible for the budget should increase it to restore access.
  Error: error: failed to fetch some objects from 'https://github.com/alibaba/paimon-cpp.git/info/lfs'
  Error: The process '/usr/bin/git' failed with exit code 2

Problem solved.

Copilot

Pull request overview

This PR addresses ORC timestamp reading issues caused by timezone data differences between Debian and other Linux distributions, specifically for timestamps prior to 1901 when using the Asia/Shanghai timezone.

Key changes:

Sets the ORC reader timezone to GMT to avoid timezone conversion issues during reading
Adds OS detection utility to handle different expected test values on Debian vs other platforms
Updates test expectations to account for the 5 minutes and 43 seconds timezone offset present in Debian's timezone data

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/paimon/format/orc/orc_file_batch_reader.cpp	Sets ORC reader timezone to GMT to prevent timezone conversion errors
src/paimon/testing/utils/testharness.h	Adds OsReleaseDetector utility class to detect Debian OS
test/inte/read_inte_test.cpp	Adds conditional test expectations for Debian vs non-Debian platforms; includes unintentional whitespace change
src/paimon/format/orc/orc_file_batch_reader_test.cpp	Adds conditional test expectations for Debian vs non-Debian platforms
src/paimon/format/orc/complex_predicate_test.cpp	Adds conditional test expectations and removes unused field declarations from SetUp

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/paimon/testing/utils/testharness.h

test/inte/read_inte_test.cpp

src/paimon/format/orc/orc_file_batch_reader.cpp

src/paimon/testing/utils/testharness.h

SGZW · 2025-12-30T14:37:31Z

@lucasfang PTAL and merge it.

fix(ut,orc): fix orc read ut under debian and adjust some options

e0d4d2a

lucasfang requested a review from Copilot December 30, 2025 07:36

Copilot started reviewing on behalf of lucasfang December 30, 2025 07:39 View session

Copilot AI reviewed Dec 30, 2025

View reviewed changes

src/paimon/testing/utils/testharness.h Show resolved Hide resolved

test/inte/read_inte_test.cpp Outdated Show resolved Hide resolved

src/paimon/format/orc/orc_file_batch_reader.cpp Show resolved Hide resolved

src/paimon/testing/utils/testharness.h Show resolved Hide resolved

fix: check format params when run it

08d5ef8

SGZW force-pushed the fix_orc_ts branch from 18f8a54 to 08d5ef8 Compare December 30, 2025 08:14

SGZW added 2 commits December 30, 2025 16:15

Merge branch 'alibaba:main' into fix_orc_ts

c110859

add missing header

08212eb

SGZW force-pushed the fix_orc_ts branch from 581d298 to 08212eb Compare December 30, 2025 08:21

fix: refactor os_release version checker

2e40b82

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(ut,orc): fix orc read ut under debian and adjust some options #37

fix(ut,orc): fix orc read ut under debian and adjust some options #37

SGZW commented Dec 26, 2025 •

edited

Loading

Uh oh!

SGZW commented Dec 26, 2025

Uh oh!

lucasfang commented Dec 30, 2025

Uh oh!

SGZW commented Dec 30, 2025

Uh oh!

lucasfang commented Dec 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SGZW commented Dec 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fix(ut,orc): fix orc read ut under debian and adjust some options #37

Are you sure you want to change the base?

fix(ut,orc): fix orc read ut under debian and adjust some options #37

Conversation

SGZW commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Uh oh!

SGZW commented Dec 26, 2025

Uh oh!

lucasfang commented Dec 30, 2025

Uh oh!

SGZW commented Dec 30, 2025

Uh oh!

lucasfang commented Dec 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SGZW commented Dec 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SGZW commented Dec 26, 2025 •

edited

Loading