Timezone is ignored during I/O operations

[atsyplenkov]

I've just noticed some unexpected behaviour with POSIXct data types. The timezone argument is completely ignored by nanoparquet during both read_parquet and write_parquet operations. Below is a small reprex showing a comparison in behaviour between nanoparquet and arrow packages. As you can see arrow::write_parquet acknowledges the timezone, while nanoparquet::write_parquet drops it to defaults (UTC in my case).

Interestingly, the Parquet file, even when written with arrow::write_parquet, also cannot read the timezone when opened with nanoparquet::read_parquet, whereas arrow::read_parquet reads it as expected.

library(dplyr)
library(lubridate)
library(nanoparquet)
library(arrow)

set.seed(123)
df <-
storms |>
dplyr::slice_sample(n = 1000) |>
dplyr::mutate(
  name,
  datetime = lubridate::make_datetime(year,month,day,hour,tz = "UTC"),
  datetime_tz = lubridate::with_tz(datetime, "America/New_York"),
  .keep = "none"
)

# Compare timezones
waldo::compare(
  lubridate::tz(df$datetime),
  lubridate::tz(df$datetime_tz)
)
#> `old`: "UTC"             
#> `new`: "America/New_York"

# {nanoparquet} approach
tmp_nano <- tempfile(fileext = ".parquet")
nanoparquet::write_parquet(df, tmp_nano)

df_nano <- nanoparquet::read_parquet(tmp_nano)

waldo::compare(
  lubridate::tz(df_nano$datetime),
  lubridate::tz(df_nano$datetime_tz)
)
#> ✔ No differences

# {arrow} approach
tmp_arrow <- tempfile(fileext = ".parquet")
arrow::write_parquet(df, tmp_arrow)
df_arrow <- arrow::read_parquet(tmp_arrow)

waldo::compare(
  lubridate::tz(df_arrow$datetime),
  lubridate::tz(df_arrow$datetime_tz)
)
#> `old`: "UTC"             
#> `new`: "America/New_York"

# Read parquet file written with {nanoparquet} using {arrow}
arrow::read_parquet(tmp_nano) |>
  dplyr::pull(datetime_tz) |>
  lubridate::tz() # Should be America/New_York
#> [1] "UTC"

# Read parquet file written with {arrow} using {nanoparquet}
nanoparquet::read_parquet(tmp_arrow) |>
  dplyr::pull(datetime_tz) |>
  lubridate::tz() # Should be America/New_York
#> [1] "UTC"

^{Created on 2025-06-19 with reprex v2.1.1}

I am using nanoparquet_0.4.2 and arrow_20.0.0.2

[gaborcsardi]

The time zone is not part of a timestamp in Parquet. arrow stores the time zone in the Parquet metadata, that's how it is able to read it back from the file. Of course this means that other Parquet readers and writers will not be able to do the same, unless they also read and write the Arrow-specific metadata. E.g. this is reading the same file with DuckDB:

read_duckdb <- function(path) {
  duckdb:::sql(sprintf("FROM '%s'", path))
}
read_duckdb(tmp_arrow) |>
  dplyr::pull(datetime_tz) |>
  lubridate::tz() # Should be America/New_York
#> [1] "UTC"

And this is writing a file with DuckDB and reading it with Arrow:

write_duckdb <- function(x, path) {
  drv <- duckdb::duckdb()
  con <- DBI::dbConnect(drv)
  on.exit(DBI::dbDisconnect(con), add = TRUE)
  DBI::dbWriteTable(con, "tab", as.data.frame(x))
  DBI::dbExecute(con, DBI::sqlInterpolate(con,
    "COPY tab TO ?filename (FORMAT 'parquet', COMPRESSION 'snappy')",
    filename = path
  ))
}
tmp_duckdb <- tempfile(fileext = ".parquet")
write_duckdb(df, tmp_duckdb)

arrow::read_parquet(tmp_duckdb) |>
  dplyr::pull(datetime_tz) |>
  lubridate::tz() # Should be America/New_York
#> [1] ""

arrow::read_parquet(tmp_duckdb)$datetime_tz[1]
#> [1] "2021-09-16 14:00:00 CEST"

We could rely on the Arrow metadata for this, nanoparquet can read and write the Arrow metadata and we already support factors the same way.

But you if you work with non-UTC dates, you need to be aware that other Parquet tools won't be able to handle them.

[atsyplenkov]

Thanks, Gábor! I didn't know that. It would be nice to have a timezone stored as Arrow metadata. At least it reduces some headache when using parquet files with the {pins} package (and other R dependents), which rely on the {nanoparquet}.

I skimmed through the codebase and think I understand the way you suggested it can be implemented. I can prepare a PR if you are open to them.

[gaborcsardi]

A PR would be great, thanks!