v1.0

Initial release - this code supported the IDAP 200 Gbps testing of the IRIS-HEP analysis pipeline for the HSF meeting.

• IRIS-HEP Analysis Grand Challenge which this was based off of
• IRIS-HEP IDAP Meetings - indico category with various organizational meetings
• HSF Workshop Talk by Brian Summarizing this Work

What's Changed

• feat: add I/O pipeline without Dask by @alexander-held in #1
• feat: add data rate plot by @alexander-held in #2
• ServiceX Exploratory Notebook by @gordonwatts in #4
• ServiceX Exploratory Notebook (#4) by @gordonwatts in #7
• init version by @ivukotic in #10
• Simple ServiceX branch materialization script by @gordonwatts in #12
• feat: create list of containers by @alexander-held in #16
• feat: nicer format for container metadata by @alexander-held in #22
• feat: update list of branches by @alexander-held in #17
• Initial SX Performance Test by @gordonwatts in #19
• feat: include complete list of files by @alexander-held in #28
• feat: move single thread debugging and add fileset configuration options by @alexander-held in #35
• Migrate to using the PHYSLITE dataset by @gordonwatts in #38
• feat: optimize graph to reduce number of nodes and save pre-processing report by @alexander-held in #50
• Add a range of possible individual datasets to run by @gordonwatts in #51
• feat: update list of input files by @alexander-held in #56
• Use type parameterized getAttribute by @gordonwatts in #43
• Allow DASK read errors by @gordonwatts in #64
• Smarter way to count branches by @gordonwatts in #62
• Tune number of file splits by @gordonwatts in #66
• Count dask tasks after optimization by @gordonwatts in #67
• Move to ServiecX 3.0 Client by @gordonwatts in #72
• feat: simple distributed reading with plain uproot.open and worker tracking utility by @alexander-held in #58
• fix: error reporting with uproot.open tasks by @alexander-held in #75
• Add library of queries by @gordonwatts in #76
• Add the module name to the log formatter by @gordonwatts in #81
• Update AB-dev docs by @gordonwatts in #82
• Adjust logging for relavent info by @gordonwatts in #86
• docs: Remove GitHub serviceX client install requirement by @matthewfeickert in #88
• Adjust steps per file when we are dealing with a small file size. by @gordonwatts in #89
• Minor Updates by @gordonwatts in #91
• Improve DASK Running Speeds by @gordonwatts in #98
• Retry's to fsspec by @gordonwatts in #99
• Run on multiple datasets by @gordonwatts in #92
• feat: optimize branch selection for throughput by @alexander-held in #105
• feat: track worker count, save measurement information, optimize branches by @alexander-held in #106
• fix: container metadata for four-top sample by @alexander-held in #107
• feat: veto p5855 files by @alexander-held in #108
• Enable better testing by @gordonwatts in #111
• Dump Rate Info during SX Run by @gordonwatts in #113
• Supporting Multi-Dataset Run by @gordonwatts in #121

New Contributors

• @alexander-held made their first contribution in #1
• @gordonwatts made their first contribution in #4
• @ivukotic made their first contribution in #10

Full Changelog: https://github.com/iris-hep/idap-200gbps-atlas/commits/v1.0