A high-performance file format for data storage, built with Rust and Python. LyFile provides efficient storage and retrieval of tabular data and high-dimensional vectors, with optimized SIMD operations for nearest neighbor search.
- Fast read/write performance with compression support
- Efficient storage for both tabular data and high-dimensional vectors
- SIMD-optimized nearest neighbor search (AVX2, SSE4.1, NEON)
- Support for multiple data input formats:
- Pandas DataFrame
- Python dictionary
- PyArrow Table
- Memory-mapped vector reading for large datasets
- Multi-threaded processing for improved performance
TODO
TODO