Mixer validator

[mariia-iureva]

This validator ensures that the Mixer job configuration is correct, data is properly aligned, and filters are valid before starting the actual Mixer job.

• Here's the ticket
• Here's a sample output

Configuration Validation

• Loads and validates the configuration file structure
• Checks for required fields and correct data types in the config

S3 Path and Permission Validation

• Validates S3 paths for documents, attributes, and output
• Checks permissions (read/write) for the specified S3 paths
• Verifies the existence of parent directories for output paths

Stream Filter Validation

• Validates filter expressions in the configuration
• Checks for syntax errors and provides warnings for potential issues

Document and Attribute Alignment

• Samples a specified number of documents and their corresponding attribute files
• Validates the alignment between document and attribute files:
  - Checks that document and attribute files have the same number of lines
  - Verifies that both document and attribute files are valid JSONL
  - Ensures required fields are present in both document and attribute files

Filter Execution Simulation

• Executes filter commands on a sample of attribute files
• Reports on the number of lines that pass or are excluded by the filters
• Identifies any errors encountered during filter execution

Attribute Name Validation

• Cross-references attribute names used in filters with those found in attribute files
• Identifies missing attributes and potential typos in attribute names
• Provides a comprehensive list of all attributes found in the files

File Sampling and Analysis

• Downloads and analyzes a sample of files from the specified S3 paths
• Counts lines in both document and attribute files
• Validates the structure and content of sampled files

Reporting and Logging

• Provides detailed console output for each validation step
• Warns about potential issues or misconfigurations
• Summarizes the results of each validation phase

Error Handling and Cleanup

• Handles errors gracefully and provides informative error messages
• Cleans up temporary files and directories after validation

[Whattabatt]

You're going to need to add this file's dependencies to pyproject.toml in order for it to run in a clean environment

[mariia-iureva]

You're going to need to add this file's dependencies to pyproject.toml in order for it to run in a clean environment

Addressed this one and added dependencies

[Whattabatt]

The warnings produce a lot of noise, it'd be good to accept a 'verbose' flag and only log the warnings if it's set to true.

[Whattabatt]

!= ? Any of the other weirder operators https://www.w3schools.com/python/python_operators.asp ?

[mariia-iureva]

The warnings produce a lot of noise, it'd be good to accept a 'verbose' flag and only log the warnings if it's set to true.

Added --verbose flag and hid most of the print statements in it

[Whattabatt]

You should delete the old validate_mixer.py script but otherwise looks good!