Integrate Context-Aware Chunking and PDF Support

[khaledsulayman]

Add context-aware document chunking for PDFs.

Checking by filetype to decide whether to use the context-aware chunker (pdf) or text-splitter (md).
Currently uses docking v1 json format.

Resolves: #271

[bbrowning]

I'm keeping track of the work here, but withholding any feedback until you get to a point where you think it's ready. If you need any help tracking down CI or other test failures while working on this, let me know and I'll make time.

[bbrowning]

I did an initial once-over - may not have caught everything, and some of the things I caught I think will be obvious once we can run tests (including end-to-end CI tests) on this.

We'll want to add a pdf file and data generation from it to our existing end-to-end tests to exercise that code-path, as there's a lot of new logic here that we'll want to ensure works in the CI environment.

I tried to focus mostly on the issues that will either cause packaging headaches (keeping our dependencies as slim as possible), user-impacting issues running this across multiple systems (file path handling), or breakage to things like existing markdown functionality. We can clean up other bits later once we get clean end-to-end tests working for markdown and PDFs.

[bbrowning]

Thanks for iterating on this and getting things to a better place! I think this is good enough to get in, knowing that we already have a number of follow-up issues (#333, #334, #335) that we want to tackle after this merges. Some of those follow-up issues will reduce the extensive chunking code here, so while I agree with the comments from @jwm4 about the chunking logic itself being hard to follow, I think we can quickly reduce the complexity of that with the expected move to docling v2.

I'd also like to see more unit, functional, and/or e2e tests here and will contribute to those myself once this merges so that we can divide and conquer the remaining work to be done versus everything needed to serialize behind this PR.

[jwm4]

I still don't like this chunker. It is confusing, idiosyncratic, and there are hardly any comments. However, my easily-fixable comments have been addressed and I understand it is time to move on for now. So I am approving this as-is for now with the understanding that it will undergo a major rework very soon. There is a discussion at DS4SD/docling#191 about better approaches for chunking using Docling (v2) outputs, and I am hoping that discussion will influence the direction this goes in the future.