Wrong number of bytes written for parquet column @ lib/ParquetOutFile.cpp:321

[YipengUva]

I have a function to convert large scale SAS data, e.g., over 5GB, to parquet format by chunks. nanoparquet::write_parquet is used to write the first chunk, and nanoparquet::append_parquet is used to append other chunks. I have this error message after appending about 10 chunks, "Wrong number of bytes written for parquet column @ lib/ParquetOutFile.cpp:321".

My sessionInfo is
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Red Hat Enterprise Linux 8.10 (Ootpa)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblasp-r0.3.15.so; LAPACK version 3.9.0

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8
[4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: America/Edmonton
tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices datasets utils methods base

other attached packages:
[1] dplyr_1.1.4 nanoparquet_0.4.2

[YipengUva]

#Minimal reproducible example

Load required libraries

library(nanoparquet)
library(dplyr) # For data manipulation

Define a consistent output file name

parquet_file <- "bug_repro_file.parquet"

--- Scenario 1: Appending a valid data frame (No Error) ---

cat("--- Scenario 1: Appending Valid Data ---\n")

Create an initial data frame with valid data

df_valid <- data.frame(
id = as.integer(1:5),
value = as.numeric(1:5),
label = c("A", "B", "C", "D", "E")
)

Create a Parquet file from the valid data frame

nanoparquet::write_parquet(df_valid, parquet_file)

Loop to append the same valid data frame multiple times

for (i in 1:15) {
cat("Appending valid chunk", i, "...\n")
tryCatch({
nanoparquet::append_parquet(df_valid, parquet_file)
}, error = function(e) {
cat("Unexpected ERROR:", conditionMessage(e), "\n")
stop("Scenario 1 failed unexpectedly. Bug might be different than suspected.")
})
}

cat("Scenario 1 completed successfully.\n\n")

--- Scenario 2: Appending a problematic data frame (Error) ---

cat("--- Scenario 2: Appending Problematic Data ---\n")

Create a problematic data frame with an all-NA column

Note: The 'id' column is of type 'integer', which is often the source of this bug

df_problematic <- data.frame(
id = as.integer(c(NA, NA, NA)),
value = c(NA_real_, NA_real_, NA_real_),
label = c(NA_character_, NA_character_, NA_character_)
)

Re-create the initial file with the first valid data frame for a clean test

nanoparquet::write_parquet(df_valid, parquet_file)

Loop to append the problematic data frame multiple times

for (i in 1:15) {
cat("Appending problematic chunk", i, "...\n")
tryCatch({
nanoparquet::append_parquet(df_problematic, parquet_file)
}, error = function(e) {
cat("Expected ERROR on chunk", i, ":", conditionMessage(e), "\n")
stop("Bug successfully reproduced. Exiting script.")
})
}

cat("Scenario 2 completed successfully. Bug was not reproduced.\n")

--- Cleanup ---

unlink(parquet_file)