add docling parser

Signed-off-by: Khaled Sulayman <[email protected]> Co-authored-by: Aakanksha Duggal <[email protected]>
instructlab · Sep 27, 2024 · 2471ba9 · 2471ba9
1 parent 2ccf5ab
commit 2471ba9
Show file tree

Hide file tree

Showing 4 changed files with 269 additions and 129 deletions.
diff --git a/requirements.txt b/requirements.txt
@@ -10,6 +10,8 @@ openai>=1.13.3,<2.0.0
 #       removed once that one is removed.
 # do not use 8.4.0 due to a bug in the library
 # https://github.com/instructlab/instructlab/issues/1389
+pypdf>=5.0.0
 tabulate>=0.9.0
 tenacity>=8.3.0,!=8.4.0
+transformers>=4.44.2
 xdg-base-dirs>=6.0.1
diff --git a/src/instructlab/sdg/utils/chunking.py b/src/instructlab/sdg/utils/chunking.py