Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to newer version of HighFive lib and implement new features #60

Open
LJoyL opened this issue Aug 7, 2023 · 3 comments · May be fixed by #61
Open

Update to newer version of HighFive lib and implement new features #60

LJoyL opened this issue Aug 7, 2023 · 3 comments · May be fixed by #61

Comments

@LJoyL
Copy link

LJoyL commented Aug 7, 2023

Current HighFive version : v2.4.1

Desired Update version : latest v2.7.1
mainly for these specific features :

see all release logs:
https://github.com/BlueBrain/HighFive/releases

@LJoyL
Copy link
Author

LJoyL commented Aug 7, 2023

Install new version of HighFive locally

  1. Installing library
git clone --recursive https://github.com/BlueBrain/HighFive.git HighFive

cmake -DHIGHFIVE_EXAMPLES=Off \
      -DHIGHFIVE_USE_BOOST=On \
      -DHIGHFIVE_PARALLEL_HDF5=Off \
      -DHIGHFIVE_UNIT_TESTS=Off \
      -DCMAKE_INSTALL_PREFIX=HighFive/build \
      -B HighFive/build \
      HighFive

cmake --build HighFive/build
cmake --install HighFive/build
  1. Turn OFF WITH_HIGHFIVE_AS_PACKAGE option in CMakeLists.txt
option(WITH_HIGHFIVE_AS_PACKAGE "HIGHFIVE externals as a dunedaq package" OFF)
  1. Adding export to env.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/lib/HighFive/build
export HIGHFIVE_INC=/path/to/lib/HighFive/include

@LJoyL
Copy link
Author

LJoyL commented Aug 7, 2023

Adding new functionalities

New splitting methods

in HDF5RawDataFile.cpp and HDF5RawDataFile.hpp, adding the following methods for HDF5RawDataFile class:

  • write_header_only : Write the header of a trigger record into file
    Need to initialize these objects before calling:
// the source_id_path map that we will build up as we write the TR header
HDF5SourceIDHandler::source_id_path_map_t source_id_path_map;

// the map of fragment types to SourceIDS
HDF5SourceIDHandler::fragment_type_source_id_map_t fragment_type_source_id_map;

// the map of subdetectors to SourceIDS
HDF5SourceIDHandler::subdetector_source_id_map_t subdetector_source_id_map;

// write the record header into the HDF5 file/group
HighFive::Group record_level_group;
  • write_fragment_only : write a full fragment at once into file
  • write_end_tr : terminate the write of a trigger record

Adding even more splitting with new methods :

  • write_fragment_begin : Write only the header and initialize the next fragment to write
  • write_fragment_part : Write a Chunk or buffer of the previously initialized fragment
  • write_fragment_end : Terminate the write of the previously initialized and wrote fragment

Adding allocation time parameter

Added for Trigger record writing and new methods listed before.

  • allocate_early : bool default false (incremental), decide to allocate the dataset at the creation time or at the writing time. dataset creation properties.
data_set_create_props.add(HighFive::AllocationTime(H5D_ALLOC_TIME_EARLY));

@LJoyL
Copy link
Author

LJoyL commented Aug 7, 2023

Adding new tests

HDF5lib tests

  • HDF5LIBS_write_fullframe_test.cpp : Test app using new methods writing a full frame at once.
  • HDF5LIBS_write_partframe_test.cpp : Test app using new methods writing chunked frames.
  • HDF5LIBS_TestWriter.py : New script to easily run benchmark tests on new and old writing tests.

HighFive tests

  • HighFive_full_bench.cpp : Test application to try new features implemented in last version of HighFive. This test only use HighFive functionalities and is not using HDF5libs structures.
  • run_HighFive_benchmark.sh : New script to benchmark and test new HighFive fonctionalities.

@LJoyL LJoyL linked a pull request Aug 7, 2023 that will close this issue
@LJoyL LJoyL linked a pull request Aug 7, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant