better docs

Author: conjuncts

.gitattributes

@@ -0,0 +1 @@
+*.ipynb linguist-detectable=false

README.md

@@ -7,23 +7,23 @@
 
 **tables**!
 
-There are many pdfs out there, and many of those pdfs have tables. But despite a plethora of table extraction options, there is still no consensus on a definitive extraction method. 
+There are many pdfs out there, and many of those pdfs have tables. But despite a plethora of table extraction options, there is still no definitive extraction method. 
 
 # About gmft
 
-gmft is a high-throughput toolkit for converting pdf tables to many formats, including cropped image, text + positions, plaintext, csv, and pandas dataframes.
+gmft is a lightweight, performant, high-throughput toolkit for converting pdf tables to many formats, including cropped image, text + positions, plaintext, csv, and pandas dataframes.
 
 gmft aims to "just work", offering strong performance with the default settings. 
 
-gmft relies on microsoft's [Table Transformers](https://github.com/microsoft/table-transformer), which qualitatively is the most performant and reliable of many tested alternatives. See the comparison here.
+gmft relies on microsoft's [Table Transformers](https://github.com/microsoft/table-transformer), which qualitatively is the most performant and reliable of many tested alternatives. See the comparison [here](https://docs.google.com/spreadsheets/d/e/2PACX-1vSpMUb4oV7d3UwRrThKbjjfmoorjWhTm620BcX5dhQqo7MRaXmK04y8mH_hImw7JZs-NDzHui7jhAvN/pubhtml?gid=0&single=true).
 
 Install: `pip install gmft`
 
-Quickstart: [demo notebook](https;//github.com/conjuncts/gmft/blob/main/notebooks/demo.ipynb)
+Quickstart: [demo notebook](https;//github.com/conjuncts/gmft/blob/main/notebooks/demo.ipynb), [bulk extract](https://github.com/conjuncts/gmft/blob/main/notebooks/bulk_extract.ipynb).
 
 # Why use gmft?
 
-**TL;DR:** gmft is convenient, fast, lightweight, configurable, and gives great results. Check out the [demo notebook](https;//github.com/conjuncts/gmft/blob/main/notebooks/demo.ipynb) for the approximate extraction quality.
+**TL;DR:** gmft is convenient, fast, lightweight, configurable, and gives great results. Check out the [bulk extract](https://github.com/conjuncts/gmft/blob/main/notebooks/bulk_extract.ipynb) notebook for approximate extraction quality. 
 
 ## Many Formats
 
@@ -47,13 +47,14 @@ Because of the relatively few dependencies and high throughput, gmft is very lig
 
 ### High throughput
 
-In most cases, OCR is not necessary; pdfs already contain text positional data. Using this existing data drastically speeds up inference. With that being said, gmft can still extract tables from images and scanned pdfs through the image output.
+Benchmark using Colab's **cpu** indicates an approximate rate of ~1.381 s/page; converting to df takes ~1.168 s/table. See the comparison here. This makes gmft about **10x faster** than alternatives like unstructured, nougat, and open-parse/unitable on cpu. ([src](https://docs.google.com/spreadsheets/d/e/2PACX-1vSpMUb4oV7d3UwRrThKbjjfmoorjWhTm620BcX5dhQqo7MRaXmK04y8mH_hImw7JZs-NDzHui7jhAvN/pubhtml?gid=0&single=true), [calculations](https://docs.google.com/spreadsheets/d/e/2PACX-1vSpMUb4oV7d3UwRrThKbjjfmoorjWhTm620BcX5dhQqo7MRaXmK04y8mH_hImw7JZs-NDzHui7jhAvN/pubhtml?gid=39227585&single=true)) How? 
 
-Benchmark using Colab's **cpu** indicates an approximate rate of ~1.381 s/page; converting to df takes ~0.945 s/table. See the comparison here.
+- gmft focuses on table extraction, so figures, titles, sections, etc. are not extracted. 
+- In most cases, OCR is not necessary; pdfs already contain text positional data. Using this existing data drastically speeds up inference. With that being said, gmft can still extract tables from images and scanned pdfs through the image output. 
+- PyPDFium2 is chosen for its [high throughput](https://github.com/py-pdf/benchmarks) and permissive license.
+- The base model, tatr is blazing fast.
 
-Gmft focuses on table extraction, so figures, titles, sections, etc. are not extracted.
 
-PyPDFium2 is chosen for its [high throughput](https://github.com/py-pdf/benchmarks) and permissive license.
 
 ### Few dependencies
 
@@ -75,7 +76,9 @@ gmft uses Microsoft's TATR, which is trained on a diverse dataset, PubTables-1M.
 
 The authors are confident that the extraction quality is unmatched. When the model fails, it is usually an OCR issue, merged cell, or false positive. Even in these cases, the text is still highly useable. **Alignment of a value to its row/column header tends to be very accurate** because of the underlying maximization algorithm.
 
-We acknowledge UniTable, a newer model which achieves SOTA results in many datasets like PubLayNet and FinTabNet. Though we plan to support Unitable in the future, Unitable is much larger (~1.5 GB), taking almost 2 orders of magnitude (about x90) longer to run on cpu. Therefore, TATR is still used for its higher throughput. In addition, experimentation does not necessarily show a strict improvement in quality. Contrary to gmft, Unitable may fail first through misalignment because of misplaced html tags (see example.) This may impact use cases where alignment is critical.
+We acknowledge UniTable, a newer model which achieves SOTA results in many datasets like PubLayNet and FinTabNet. Though we plan to support Unitable in the future, Unitable is much larger (~1.5 GB), taking almost 2 orders of magnitude (about x90) longer to run on cpu. Therefore, TATR is still used for its higher throughput. In addition, experimentation does not necessarily show a strict improvement in quality. Contrary to gmft, Unitable may fail first through misalignment because of misplaced html tags. This may impact use cases where alignment is critical.
+
+We invite the reader to explore the [comparison notebooks](https://drive.google.com/drive/u/0/folders/114bWRj5H4aE-BA5UKH9S5ol8LC6vhqfR) to survey your own use cases and compare results.
 
 # Limitations
 
@@ -85,8 +88,6 @@ Multi-indices (multiple column headers) are not yet supported.
 
 Slightly rotated tables will probably fail, especially large tables that are not perfectly level.
 
-
-
 # Acknowledgements
 
 A tremendous thank you to the TATR authors: Brandon Smock, Rohith Pesala, and Robin Abraham, for making gmft possible. The image->csv step is highly inspired by TATR's inference.py code, but has been rewritten for performance.
@@ -97,6 +98,8 @@ Thank you to Niels Rogge for porting TATR to huggingface and writing the [visual
 
 Gmft focuses highly on pdf tables. For more general document understanding, I recommend checking out [open-parse](https://github.com/Filimoa/open-parse), [unstructured](https://github.com/Unstructured-IO/unstructured), [surya](https://github.com/VikParuchuri/surya), [deepdoctection](https://github.com/deepdoctection/deepdoctection), and [DocTR](https://github.com/mindee/doctr).
 
-In particular, open-parse and unstructured also do quite well on the same example pdfs in terms of extraction quality. Open-parse offers Unitable, a larger model which may achieve higher quality but runs much slower on cpu (see [reliability section](#Reliable) for more discussion.) Importantly, open-parse allows extraction of auxiliary information paragraphs, etc., (not just tables) useful for RAG.
+Nougat is excellent in outputting full mathpix markdown (.mmd), which includes latex formulas, bold/italics, and fully latex-typeset tables.
+
+Open-parse and unstructured also do quite well on the same example pdfs in terms of extraction quality. Open-parse offers Unitable, a larger model which may achieve higher quality but runs much slower on cpu (see [reliability section](#Reliable) for more discussion.) Importantly, open-parse allows extraction of auxiliary information paragraphs, etc., (not just tables) useful for RAG.
 
 gmft is released under MIT. 

notebooks/quickstart.ipynb

@@ -33,4 +33,4 @@ requires = ["flit_core >=3.4,<4"]
 
 [tool.flit.sdist]
 include = ["gmft/**/*.py", "README.md", "LICENSE"]
-exclude = ["gmft/pdf_bindings_mu.py", "dist/**/*", "notebooks/**/*", "samples/**/*", "reading_list.md", "*.zip"]
+exclude = ["gmft/pdf_bindings_mu.py", "dist/**/*", "notebooks/**/*", "samples/**/*", "plans/**/*", "reading_list.md", "*.zip"]