Skip to content

Change default IO type from NETCDF4C to PNETCDF #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xylar merged 6 commits into from
Jan 29, 2026
Merged

Change default IO type from NETCDF4C to PNETCDF #325

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
200a89a
Change default IO type from NETCDF4C to PNETCDF
amametjanov Dec 8, 2025
dfcffe1
Update scorpio
amametjanov Dec 21, 2025
e68388f
Add end-define-mode calls while in data-mode netcdf writes
amametjanov Dec 21, 2025
c3d275c
Remove colon from ocn.hist.$SimTime for MPI-IO on Frontier
amametjanov Jan 24, 2026
b58353f
Update netcdf4->pnetcdf in default yaml and if no io-yaml exists
amametjanov Jan 27, 2026
a237a9d
fix a bug in reading IODefaultFormat from omega.yml
grnydawn Jan 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions components/omega/configs/Default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Omega:
IOStride: 1
IOBaseTask: 0
IORearranger: box
IODefaultFormat: NetCDF4
IODefaultFormat: pnetcdf
State:
NTimeLevels: 2
Advection:
Expand Down Expand Up @@ -114,7 +114,7 @@ Omega:
- Restart
History:
UsePointerFile: false
Filename: ocn.hist.$SimTime
Filename: ocn.hist.$Y-$M-$D_$h.$m.$s
Mode: write
IfExists: replace
Precision: double
Expand Down
13 changes: 7 additions & 6 deletions components/omega/src/base/IO.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,11 +189,11 @@ void init(const MPI_Comm &InComm // [in] MPI communicator to use
CHECK_ERROR_ABORT(Err, "IO: IO group not found in input Config");

// Read default file format
std::string InFileFmt = "netcdf4c"; // set default value
std::string InFileFmt = "pnetcdf"; // set default value
Err = IOConfig.get("IODefaultFormat", InFileFmt);
CHECK_ERROR_WARN(Err, "IO: DefaultFileFmt not found in Config - using {}",
InFileFmt);
FileFmt DefaultFileFmt = FileFmtFromString(InFileFmt);
DefaultFileFmt = FileFmtFromString(InFileFmt);

// Read parallel IO settings - default to single-task if config
// values do not exist
Expand Down Expand Up @@ -249,6 +249,7 @@ void openFile(

int PIOErr = 0; // internal SCORPIO/PIO return call
int Format = InFormat; // coerce to integer for PIO calls
int IsCDF5 = (InFormat == PIO_IOTYPE_PNETCDF) ? PIO_64BIT_DATA : 0;

switch (InMode) {

Expand All @@ -269,7 +270,7 @@ void openFile(
// file exists, we use create and fail with an error
case IfExists::Fail:
PIOErr = PIOc_createfile(SysID, &FileID, &Format, Filename.c_str(),
NC_NOCLOBBER | InMode);
NC_NOCLOBBER | IsCDF5 | InMode);
if (PIOErr != PIO_NOERR)
ABORT_ERROR("IO::openFile: PIO error opening file {} for writing",
Filename);
Expand All @@ -279,7 +280,7 @@ void openFile(
// we use create with the CLOBBER option
case IfExists::Replace:
PIOErr = PIOc_createfile(SysID, &FileID, &Format, Filename.c_str(),
NC_CLOBBER | InMode);
NC_CLOBBER | IsCDF5 | InMode);
if (PIOErr != PIO_NOERR)
ABORT_ERROR("IO::openFile: PIO error opening file {} for writing",
Filename);
Expand All @@ -291,10 +292,10 @@ void openFile(
case IfExists::Append:
if (std::filesystem::exists(Filename)) {
PIOErr = PIOc_openfile(SysID, &FileID, &Format, Filename.c_str(),
InMode);
IsCDF5 | InMode);
} else {
PIOErr = PIOc_createfile(SysID, &FileID, &Format, Filename.c_str(),
InMode);
IsCDF5 | InMode);
}

if (PIOErr != PIO_NOERR)
Expand Down
2 changes: 1 addition & 1 deletion components/omega/src/base/IO.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ enum FileFmt {
FmtHDF5 = PIO_IOTYPE_HDF5, ///< native HDF5 format
FmtADIOS = PIO_IOTYPE_ADIOS, ///< ADIOS format
FmtUnknown = -1, ///< Unknown or undefined
FmtDefault = PIO_IOTYPE_NETCDF4C, ///< NetCDF4 is default
FmtDefault = PIO_IOTYPE_PNETCDF, ///< PNETCDF is default
};

/// File operations
Expand Down
14 changes: 12 additions & 2 deletions components/omega/src/infra/IOStream.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2285,7 +2285,7 @@ Error IOStream::readStream(

// Open input file
int InFileID;
IO::openFile(InFileID, InFileName, Mode, IO::FmtDefault, ExistAction);
IO::openFile(InFileID, InFileName, Mode, IO::DefaultFileFmt, ExistAction);

// Read any requested global metadata
for (auto Iter = ReqMetadata.begin(); Iter != ReqMetadata.end(); ++Iter) {
Expand Down Expand Up @@ -2438,7 +2438,8 @@ void IOStream::writeStream(

// Open output file
int OutFileID;
IO::openFile(OutFileID, OutFileName, Mode, IO::FmtDefault, ExistAction);
IO::openFile(OutFileID, OutFileName, Mode, IO::DefaultFileFmt, ExistAction);
bool NeedEndDef = false; // check if we need to call end-define-mode

// For files with multiple frames or time slices, we need to determine the
// default Frame number for time-dependent fields. If the frame/time already
Expand Down Expand Up @@ -2491,6 +2492,7 @@ void IOStream::writeStream(
if (Frame < 1) {
writeFieldMeta(CodeMeta, OutFileID, IO::GlobalID);
writeFieldMeta(SimMeta, OutFileID, IO::GlobalID);
NeedEndDef = true;
}

// Create and write a field for any global data that is file or time
Expand All @@ -2506,6 +2508,10 @@ void IOStream::writeStream(
FileField->addMetadata(SimTimeName, SimTimeStr);
}
// Write and then destroy temporary field
if (!NeedEndDef) { // in data-mode, need to redef
PIOc_redef(OutFileID);
NeedEndDef = true;
}
writeFieldMeta("FileField", OutFileID, IO::GlobalID);
Field::destroy("FileField");

Expand Down Expand Up @@ -2557,8 +2563,12 @@ void IOStream::writeStream(
// Now we can write the field metadata
if (Frame < 1) { // only write if it's the first time
writeFieldMeta(FieldName, OutFileID, FieldID);
NeedEndDef = true;
}
}
if (NeedEndDef) {
IO::endDefinePhase(OutFileID);
}

// Now write data arrays for all fields in contents
for (auto IFld = Contents.begin(); IFld != Contents.end(); ++IFld) {
Expand Down
2 changes: 1 addition & 1 deletion externals/scorpio
Open in desktop
Submodule scorpio updated 794 files
Loading