make diff of time series to compare test productions+AliasDataFrame

[miranov25]

Adding - AliasDataFrame is a small utility that extends pandas.DataFrame functionality by enabling:

• Lazy evaluation of derived columns via named aliases
• Automatic dependency resolution across aliases
• Persistence via Parquet + JSON or ROOT TTree (via uproot + PyROOT)
• ROOT-compatible TTree export/import including alias metadata

[github-actions]

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

[miranov25]

✨ Add `AliasDataFrame Utilities for On-Demand Evaluation

This PR adds support for alias-based derived column computation, as used for example in TPC distortion error parameterization. It includes:

✅ Key Features

• Function Validation: Supports expressions using standard math, numpy, and previously defined aliases. Invalid aliases are warned during definition.

• Alias Dependency Resolution: Automatic topological sort of aliases with dependency tracking and detection of circular references.

• Output Type Specification: Each alias can specify its desired output dtype (e.g. np.float16, np.uint8). This can also be overridden during materialization.

  • Dtypes are preserved in .parquet exports.
  • TTree support can be extended to encode dtype metadata in a structured way.

• Alias Dependency Graph: Visualization of alias relationships using networkx and matplotlib.

🧪 Example Usage

The function below demonstrates how derived error estimates and quality flags can be defined in terms of other DataFrame columns and aliases:

def makeErrParamAlias(adf):
    adf.df["Beta2"] = np.minimum(50 / adf.df["dEdxTPC"], 1.0).astype(np.float16)
    adf.add_alias("errz0a0", "0.35*sqrt(1+(mP4**2)/Beta2)/(nClsTPC/150.)**1.5", dtype=np.float16)
    adf.add_alias("errz0b0", "0.006*sqrt(1+(mP4**2)/Beta2)/(nClsTPC/150.)**1.5", dtype=np.float16)
    adf.add_alias("errz0b1", "0.0015*sqrt(0.25+(mP4**2)/Beta2)/(nClsTPC/150.)**1.5", dtype=np.float16)
    adf.add_alias("erry0c1", "0.5*sqrt(0.25+(mP4**2)/Beta2)/(nClsTPC/150.)**2.5/150**2", dtype=np.float16)
    adf.add_alias("cutB6", "((abs(dz0_b0/errz0b0) > 6) * 1) + ((abs(dz0_b1/errz0b1) > 6) * 2) + ((abs(dy0_b0/errz0b0) > 6) * 4) + ((abs(dy0_b1/errz0b1) > 6) * 8)", dtype=np.uint8)
    adf.add_alias("cutC6", "((abs(dy0_c1/erry0c1) > 6) * 1) + ((abs(dy0_c0/erry0c1) > 6) * 2)", dtype=np.uint8)
    adf.add_alias("cutA6", "((abs(dy0_a0/errz0a0) > 6) * 1) + ((abs(dz0_a0/errz0a0) > 6) * 2)", dtype=np.uint8)
    adf.add_alias("cutT", "((cutB6 + cutC6 + cutA6) > 0)", dtype=np.uint8)
    return adf

📊 Alias Dependency Graph

Visual representation of dependencies:

[miranov25]

🧾 Output Storage for AliasDataFrame: TTree or Parquet + JSON/Metadata

This update improves how aliases and derived columns are saved and loaded across different formats.

✅ Key Features

• Selective Column Export:

  • save(..., dropAliasColumns=True) stores only non-alias columns in .parquet (default), matching export_tree() behavior.

• Alias Dtype Persistence:

  • Output dtypes (e.g. np.float16, np.uint8) are now stored as type names (e.g. "float16").
  • Correctly reloaded using getattr(np, ...) to ensure .astype(...) works.

• Dual Metadata Storage:

  • Aliases and dtypes are stored both:
    • in .parquet file metadata (pyarrow)
    • and in a .aliases.json file for inspection and fallback

🔍 Example Outputs (for exame above #2014 (comment))

ROOT TTree alias list:

TNamed	cutB6	((abs(dz0_b0/errz0b0) > 6) * 1) + ((abs(dz0_b1/errz0b1) > 6) * 2) + ...
TNamed	cutT	((cutB6 + cutC6 + cutA6) > 0)

Parquet + JSON metadata:

{
  "aliases": {
    "cutB6": "((abs(dz0_b0/errz0b0) > 6) * 1) + ((abs(dz0_b1/errz0b1) > 6) * 2) + ...",
    "cutT": "((cutB6 + cutC6 + cutA6) > 0)"
  },
  "dtypes": {
    "cutB6": "uint8",
    "cutT": "uint8"
  }
}

• The .parquet file contains embedded metadata.
• The .aliases.json provides a transparent sidecar view.
• If metadata is missing or outdated, the loader will fall back to the JSON.

Example usage of tree with aliases (later RDataFrame) - in TTree query:

root [20] tree->GetListOfAliases()->ls()
OBJ: TList	TList	Doubly linked list : 0
 OBJ: TNamed	errz0a0	0.35*sqrt(1+(mP4**2)/Beta2)/(nClsTPC/150.)**1.5 : 0 at: 0x445d4e0
 OBJ: TNamed	errz0b0	0.006*sqrt(1+(mP4**2)/Beta2)/(nClsTPC/150.)**1.5 : 0 at: 0x4465bb0
 OBJ: TNamed	errz0b1	(0.0015*sqrt(0.25+(mP4**2)/Beta2)/(nClsTPC/150.)**1.5) : 0 at: 0x4465cb0
 OBJ: TNamed	erry0c1	(0.5*sqrt(0.25+(mP4**2)/Beta2)/(nClsTPC/150.)**2.5)/(150**2) : 0 at: 0x4465db0
 OBJ: TNamed	cutB6	((abs(dz0_b0/errz0b0) > 6) * 1) + ((abs(dz0_b1/errz0b1) > 6) * 2) + ((abs(dy0_b0/errz0b0) > 6) * 4) + ((abs(dy0_b1/errz0b1) > 6) * 8) : 0 at: 0x4465eb0
 OBJ: TNamed	cutC6	((abs(dy0_c1/erry0c1) > 6) * 1) + ((abs(dy0_c0/erry0c1) > 6) * 2) : 0 at: 0x4465f60
 OBJ: TNamed	cutA6	((abs(dy0_a0/errz0a0) > 6) * 1) + ((abs(dz0_a0/errz0a0) > 6) * 2) : 0 at: 0x4466070
 OBJ: TNamed	cutT	((cutB6 + cutC6 + cutA6) > 0) : 0 at: 0x4466180

root [17] tree->Draw("(dz0_a0/errz0a0):mP4>>his(200,-5,5,100,-5,5)","cutT==0","colz")
(long long) 3682828
root [18] tree->Draw("abs(dz0_a0/errz0a0):mP4>>rofdz(200,-5,5,100,0,20)","cutT==0&&abs(dy0_a0/errz0a0)<20","profsame")

[miranov25]

🔄 Update Changes Summary

✅ Constants Support

• New parameter is_constant=True in add_alias marks aliases as constants (e.g. countsFV0_median = 2096.0).
• Constants are evaluated once and injected directly during alias materialization.
• Constants are not materialized as DataFrame columns unless explicitly requested.

🧠 Smart Dependency Handling

• During materialize_alias or materialize_all, constants are evaluated and injected before dependency resolution.
• Dependency graphs and topological sorting skip constants to avoid recomputation.

💾 Parquet and ROOT I/O Support

• Metadata (aliases, dtypes, constants) is stored inside .parquet file as Arrow schema metadata.
• Export to ROOT TTree uses SetAlias, with workaround for ROOT interpreter bug by forcing numeric constants to use val + 0.

🧪 Unit Tests

• Added robust pytest unit tests with:

  • Standard aliases
  • Aliases with custom dtype
  • Constants and mixed expressions
  • Dependency resolution
  • Reloading and validating from saved Parquet files

[miranov25]

Add ROOT SetAlias export and Python-to-ROOT AST translation for aliases

Extended commit description:

• Introduced convert_expr_to_root() static method using ast to translate Python expressions into ROOT-compatible syntax, including function mapping (mod → fmod, arctan2 → atan2, etc.).

• Patched export_tree() to:

  • Apply ROOT-compatible expression conversion.
  • Handle ROOT’s TTree::SetAlias limitations (e.g. constants) using (<value> + 0) workaround.
  • Save full Python alias metadata (aliases, dtypes, constants) as JSON in TTree::GetUserInfo().

• Patched read_tree() to:

  • Restore alias expressions and metadata from UserInfo JSON.
  • Maintain full alias context including constants and types.

• Preserved full compatibility with the existing parquet export/load code.

• Ensured Python remains the canonical representation; conversion is only needed for ROOT alias usage.