FCCAnalyses samples

[kjvbrt]

Hi,

I've tested the samples we use for the studies with the several tools which might be used to understand what is saved in those files. Most important one being podio-dump others are good to have. The test was done on CentOS7 machine and files are stored on EOS.

y - runs as expected
x - crash/problem

podio-dump

	stable	nightlies
spring2021	x1	x1
winter2023	y	y

edm4hep2json

	stable	nightlies
spring2021	x2	x4
winter2023	x3	y

collInfo

From FCC AuxTools.

	stable	nightlies
spring2021	y	y
winter2023	y	y

Test files

• spring2021
  /eos/experiment/fcc/ee/generation/DelphesEvents/spring2021/IDEA/p8_ee_ZH_ecm240/events_101027117.root
• winter2023
  /eos/experiment/fcc/ee/generation/DelphesEvents/winter2023/IDEA/p8_ee_ZZ_ecm240/events_092194859.root

Errors

1. RecoParticleRefData warning + segmentation violation

TClass::Init:0: RuntimeWarning: no dictionary for class edm4hep::RecoParticleRefData is available
PODIO: Reconstructing CollectionTypeInfo branch from other sources in file: '/eos/experiment/fcc/ee/generat
ion/DelphesEvents/spring2021/IDEA/p8_ee_ZH_ecm240/events_101027117.root'
 *** Break *** segmentation violation

.
.
.

#9  0x00007fa8cd20de16 in TClass::NewObject (this=this
entry=0x0, defConstructor=defConstructor
entry=TClass::kClassNew, quiet=quiet
entry=false) at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.10-i4xdfrlayuwec4fboqshcon324ybwn2e/sp
ack-src/core/meta/src/TClass.cxx:4982
#10 0x00007fa8cd210950 in TClass::New (this=this
entry=0x0, defConstructor=defConstructor
entry=TClass::kClassNew, quiet=quiet
entry=false) at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.10-i4xdfrlayuwec4fboqshcon324ybwn2e/sp
ack-src/core/meta/src/TClass.cxx:4966
#11 0x00007fa8b8c43fa3 in podio::ROOTLegacyReader::createCollectionBranches (this=this
entry=0x6f9dd40, collInfo=...) at /tmp/gitlab-runner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqcxq7err
ukecsdvy3bi2tr5/spack-src/src/ROOTLegacyReader.cc:194
#12 0x00007fa8b8c45172 in podio::ROOTLegacyReader::openFiles (this=0x6f9dd40, filenames=...) at /tmp/gitlab
-runner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqcxq7errukecsdvy3bi2tr5/spack-src/src/ROOTLegacyReade
r.cc:167

2. Different crashes depending on the requested collection. For ReconstructedParticles:

$ edm4hep2json /eos/experiment/fcc/ee/generation/DelphesEvents/spring2021/IDEA/p8_ee_ZH_ecm240/events_10102
7117.root -n 4 -l ReconstructedParticles
Warning in <TClass::Init>: no dictionary for class edm4hep::RecoParticleRefData is available
PODIO: Reconstructing CollectionTypeInfo branch from other sources in file: '/eos/experiment/fcc/ee/generat
ion/DelphesEvents/spring2021/IDEA/p8_ee_ZH_ecm240/events_101027117.root'
PODIO: Cannot create the collection type 'edm4hep::RecoParticleRefCollection' stored in branch 'Electron'. 
Contents of this branch cannot be read.
PODIO: Cannot create the collection type 'edm4hep::RecoParticleRefCollection' stored in branch 'Muon'. Cont
ents of this branch cannot be read.
PODIO: Cannot create the collection type 'edm4hep::RecoParticleRefCollection' stored in branch 'AllMuon'. C
ontents of this branch cannot be read.
PODIO: Cannot create the collection type 'edm4hep::RecoParticleRefCollection' stored in branch 'Photon'. Co
ntents of this branch cannot be read.

 *** Break *** segmentation violation

.
.
.

#6  0x00007fe4f0db3275 in podio::root_utils::setCollectionAddresses<podio::CollectionWriteBuffers> (branche
s=..., collBuffers=...) at /tmp/gitlab-runner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqcxq7errukecsdv
y3bi2tr5/spack-src/src/rootUtils.h:91
#7  podio::ROOTReader::getCollection (this=0x7ffe25bf8ac0, collInfo=...) at /tmp/gitlab-runner/spack-stage/
spack-stage-podio-0.16.2-cbwgcvkkkqcxq7errukecsdvy3bi2tr5/spack-src/src/ROOTReader.cc:112
#8  0x00007fe4f0db3685 in podio::ROOTReader::readCollection (this=0x7ffe25bf8ac0, name=...) at /tmp/gitlab-
runner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqcxq7errukecsdvy3bi2tr5/spack-src/src/ROOTReader.cc:63
#9  0x00007fe4efc54d29 in podio::EventStore::doGet (this=this
entry=0x7ffe25bf8b60, name=..., collection=
0x7ffe25bf7ff0: 0x0, setReferences=setReferences
entry=true) at /tmp/gitlab-runner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqcxq7errukecsdvy3bi2tr5/spa
ck-src/src/EventStore.cc:68
#10 0x00007fe4efc54fdd in podio::EventStore::get (this=0x7ffe25bf8b60, id=<optimized out>, collection=
0x7ffe25bf7ff0: 0x0) at /tmp/gitlab-runner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqcxq7errukecsdvy3b
i2tr5/spack-src/src/EventStore.cc:30
#11 0x00007fe4f0d1e811 in edm4hep::ReconstructedParticleCollectionData::setReferences(podio::ICollectionPro
vider const*, bool) () from /cvmfs/sw.hsf.org/spackages7/edm4hep/0.7.2/x86_64-centos7-gcc11.2.0-opt/nfdxp/l
ib64/libedm4hep.so
#12 0x00007fe4efc54d48 in podio::EventStore::doGet (this=0x7ffe25bf8b60, name=..., collection=
0x7ffe25bf8ef0: 0x0, setReferences=<optimized out>) at /tmp/gitlab-runner/spack-stage/spack-stage-podio-0.1
6.2-cbwgcvkkkqcxq7errukecsdvy3bi2tr5/spack-src/src/EventStore.cc:71
#13 0x0000000000411d8f in void read_events<podio::ROOTReader>(std::__cxx11::basic_string<char, std::char_tr
aits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::al
locator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > con
st&, int, bool) ()
#14 0x00000000004098c2 in main ()

3. Crash at SetBranchAddresspodio::CollectionIDTable

#6  0x00007f16588c8906 in TTree::SetBranchAddress<podio::CollectionIDTable> (ptr=0x0, add=0x7ffed9fb3050, b
name=0x7f16588d8138 "CollectionIDs", this=0x0) at /cvmfs/sw.hsf.org/spackages7/root/6.26.10/x86_64-centos7-
gcc11.2.0-opt/i4xdf/include/root/TTree.h:588
#7  podio::ROOTReader::openFiles (this=this
entry=0x7ffed9fb3b10, filenames=...) at /tmp/gitlab-runner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqc
xq7errukecsdvy3bi2tr5/spack-src/src/ROOTReader.cc:155
#8  0x00007f16588c8d8e in podio::ROOTReader::openFile (this=0x7ffed9fb3b10, filename=...) at /tmp/gitlab-ru
nner/spack-stage/spack-stage-podio-0.16.2-cbwgcvkkkqcxq7errukecsdvy3bi2tr5/spack-src/src/ROOTReader.cc:140
#9  0x0000000000411b1f in void read_events<podio::ROOTReader>(std::__cxx11::basic_string<char, std::char_tr
aits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::al
locator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > con
st&, int, bool) ()
#10 0x00000000004098c2 in main ()

4. Cannont find tree podio_metadata

Warning in <TClass::Init>: no dictionary for class edm4hep::RecoParticleRefData is available
Error in <TChain::AddFile>: cannot find tree with name podio_metadata in file /eos/experiment/fcc/ee/genera
tion/DelphesEvents/spring2021/IDEA/p8_ee_ZH_ecm240/events_101027117.root
terminate called after throwing an instance of 'std::runtime_error'
  what():  File /eos/experiment/fcc/ee/generation/DelphesEvents/spring2021/IDEA/p8_ee_ZH_ecm240/events_1010
27117.root couldn't be found
Aborted

In #227 the ROOTLegacyReader is used if legacy file is detected and
following crash occures:

$ edm4hep2json /eos/experiment/fcc/ee/generation/DelphesEvents/spr
ing2021/IDEA/p8_ee_ZH_ecm240/events_101027117.root -n 4
Warning in <TClass::Init>: no dictionary for class edm4hep::RecoParticleRefData is available
WARNING: Reading legacy file, some collections might not be recognized!
PODIO: Reconstructing CollectionTypeInfo branch from other sources in file: 'root://eospublic.cern.ch//eos/
experiment/fcc/ee/generation/DelphesEvents/spring2021/IDEA/p8_ee_ZH_ecm240/events_101027117.root'

 *** Break *** segmentation violation

.
.
.

#7  0x00007f1dbb0906b6 in TClass::NewObject (this=this
entry=0x0, defConstructor=TClass::kClassNew, quiet=false) at /tmp/root/spack-stage/spack-stage-root-6.28.04
-474lsx3d37cexm4vw5dk7twizjzkdeo4/spack-src/core/meta/src/TClass.cxx:4996
#8  0x00007f1dbb093340 in TClass::New (this=0x0, defConstructor=<optimized out>, quiet=<optimized out>) at 
/tmp/root/spack-stage/spack-stage-root-6.28.04-474lsx3d37cexm4vw5dk7twizjzkdeo4/spack-src/core/meta/src/TCl
ass.cxx:4980
#9  0x00007f1dbcf86142 in podio::ROOTLegacyReader::createCollectionBranches(std::vector<std::tuple<unsigned
 int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, unsigned int>,
 std::allocator<std::tuple<unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allo
cator<char> >, bool, unsigned int> > > const&) () from /cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09
-18/x86_64-centos7-gcc12.2.0-opt/podio/d402f747877239aaedefe993856e26749c91bea3=develop-ivgpec/lib64/libpod
ioRootIO.so
#10 0x00007f1dbcf874a0 in podio::ROOTLegacyReader::openFiles(std::vector<std::__cxx11::basic_string<char, s
td::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_tr
aits<char>, std::allocator<char> > > > const&) () from /cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09
-18/x86_64-centos7-gcc12.2.0-opt/podio/d402f747877239aaedefe993856e26749c91bea3=develop-ivgpec/lib64/libpod
ioRootIO.so
#11 0x00007f1dbcf878ea in podio::ROOTLegacyReader::openFile(std::__cxx11::basic_string<char, std::char_trai
ts<char>, std::allocator<char> > const&) () from /cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09-18/x8
6_64-centos7-gcc12.2.0-opt/podio/d402f747877239aaedefe993856e26749c91bea3=develop-ivgpec/lib64/libpodioRoot
IO.so

[tmadlener]

Thanks for the thorough report, let me give a few quick answers.

For 1) and 2) The RecoParticleRef datatype has been removed in #116. Since the root file layout didn't change as far as FCCAnalyses was concerned, we decided to go ahead with that without waiting for a schema evolution mechanism. Additionally, podio-dump was not around at that point. Long story short, for these files podio-dump is currently broken, and I am not sure how easy it is to fix it. One possibility would be to build the EDM4hep version at the time with a current podio version and then thing might work again. Another possibility could be to patch the meta data of these files such that podio-dump can handle things again.

4 might be related to this, but the posted error simply looks like a potentially broken or non-present file.

The failure in edm4hep2json is most likely also related to this, as ROOT simply doesn't know about the RecoParticleRef class, so it cannot create an instance of this either.

3 looks like a TTree is missing a branch or a TTree is entirely missing. This would need some further debugging to understand what is really going on.

The main question for me to understand is which problem needs to be solved, i.e. what do you want to achieve. If I understand the AuxTools correctly, there the main point is to get to the collectionID table, right? Is that enough or would you need the additional functionality that podio-dump offers as well?

For edm4hep2json I suppose you want to be able to use the spring2021 files also for the event display?

[kjvbrt]

The motivation was to have an understanding what is broken where.
I agree, that fixing all the issues is not feasible without nasty hacks, but why not fix the ones which can be done easily :)

Another possibility could be to patch the meta data of these files

How can this be done? Will this effectively disable those reference collections?

3 looks like a TTree is missing a branch or a TTree is entirely missing

I believe the problem here is the addition of dNdX to the edm4hep::Track.

If I understand the AuxTools correctly, there the main point is to get to the collectionID table, right?

We keep it around to be able to get this information in cases where podio-dump is broken. I don't want to add any further functionality to this script.

For edm4hep2json I suppose you want to be able to use the spring2021 files also for the event display?

That would be nice to have, but is not a priority.