Skip to content

Latest commit

 

History

History
127 lines (80 loc) · 5.44 KB

README.rst

File metadata and controls

127 lines (80 loc) · 5.44 KB
Raw
Github Action workflow status and link. PyPI release badge and link. Documentation status badge and link. Journal of Open Source Software publication badge and link.

🔍 ER-Evaluation: An End-to-End Evaluation Framework for Entity Resolution Systems

ER-Evaluation is a Python package for the evaluation of entity resolution (ER) systems.

It provides an entity-centric approach to evaluation. Given a sample of resolved entities, it provides:

  • summary statistics, such as average cluster size, matching rate, homonymy rate, and name variation rate.
  • comparison statistics between entity resolutions, such as proportion of links from one which is also in the other, and vice-versa.
  • performance estimates with uncertainty quantification, such as precision, recall, and F1 score estimates, as well as B-cubed and cluster metric estimates.
  • error analysis, such as cluster-level error metrics and analysis tools to find root cause of errors.
  • convenience visualization tools.

For more information on how to resolve a sample of entities for evaluation and model training, please refer to our data labeling guide.

Installation

Install the released version from PyPI using:

pip install er-evaluation

Or install the development version using: .. code:: bash

pip install git+https://github.com/Valires/er-evaluation.git

Documentation

Please refer to the documentation website er-evaluation.readthedocs.io.

Usage Examples

Please refer to the User Guide or our Visualization Examples for a complete usage guide.

In summary, here's how you might use the package.

  1. Import your predicted disambiguations and reference benchmark dataset. The benchmark dataset should contain a sample of disambiguated entities.
import er_evaluation as ee

predictions, reference = ee.load_pv_disambiguations()
  1. Plot summary statistics and compare disambiguations.
ee.plot_summaries(predictions)

media/plot_summaries.png

ee.plot_comparison(predictions)

media/plot_comparison.png

  1. Define sampling weights and estimate performance metrics.
ee.plot_estimates(predictions, {"sample":reference, "weights":"cluster_size"})

media/plot_estimates.png

  1. Perform error analysis using cluster-level explanatory features and cluster error metrics.
ee.make_dt_regressor_plot(
        y,
        weights,
        features_df,
        numerical_features,
        categorical_features,
        max_depth=3,
        type="sunburst"
)

media/plot_decisiontree.png

Development Philosophy

ER-Evaluation is designed to be a unified source of evaluation tools for entity resolution systems, adhering to the Unix philosophy of simplicity, modularity, and composability. The package contains Python functions that take standard data structures such as pandas Series and DataFrames as input, making it easy to integrate into existing workflows. By importing the necessary functions and calling them on your data, you can easily use ER-Evaluation to evaluate your entity resolution system without worrying about custom data structures or complex architectures.

Citation

Please acknowledge the publications below if you use ER-Evaluation:

  • Binette, Olivier. (2022). ER-Evaluation: An End-to-End Evaluation Framework for Entity Resolution Systems. Available online at github.com/Valires/ER-Evaluation
  • Binette, Olivier, Sokhna A York, Emma Hickerson, Youngsoo Baek, Sarvo Madhavan, Christina Jones. (2022). Estimating the Performance of Entity Resolution Algorithms: Lessons Learned Through PatentsView.org. arXiv e-prints: arxiv:2210.01230
  • Upcoming: "An End-to-End Framework for the Evaluation of Entity Resolution Systems With Application to Inventor Name Disambiguation"

Public License