Apache Tika Jaccard Based File Similarity
This project demonstrates using the Tika-Python package (Python port of Apache Tika) to compute file similarity based on Metadata features.
The script can iterate over all files in the current directory or given files by command line and derives their metadata features, then computes the union of all features. The union of all features become the "golden feature set" that all document features are compared to via intersect. The length of that intersect per file divided by the length of the unioned set becomes the similarity score.
Scores are sorted in reverse (descending) order which can be shown in three different Data-Driven document visualizaions.
- Install Tika-Python
git clone https://github.com/chrismattmann/tika-img-similarity
You can also check out ETLlib
This compares metadata feature names as a golden feature set
#!/usr/bin/env python2.7
python similarity.py -f [directory of files] or
python similarity.py -c [file1 file2 file3 ...]
This compares metadata feature names together with its value as a golden feature set
#!/usr/bin/env python2.7
python value-similarity.py -f [directory of files] or
python value-similarity.py -c [file1 file2 file3 ...]
* python cluster-scores.py (for generating cluster viz)
* open cluster-d3.html(or dynamic-cluster.html for interactive viz) in your browser
You can set threshold (default threshold = 0.01) in cluster-scores.py
###circlepacking viz
* python circle-packing.py (for generating circlepacking viz)
* open circlepacking.html(or dynamic-circlepacking.html for interactive viz) in your browser
###composite viz This is a combination of cluster viz and circle packing viz. The deeper color, the more the same attributes in the cluster.
* open compositeViz.html in your browser
###Big data way if you are dealing with big data, you can use it this way:
* python generateLevelCluster.py (for generating level cluster viz)
* open levelCluster-d3.html in your browser
You can set max number for each node _maxNumNode(default _maxNumNode = 10) in generateLevelCluster.py
Send them to Chris A. Mattmann.
- Chris A. Mattmann, JPL
- Dongni Zhao, USC
This project is licensed under the Apache License, version 2.0.