Update crawler with google docs support #32

nattvara · 2024-05-02T21:17:23Z

Description

This PR will add support for downloading and indexing the content of google docs. It will also fix a few bugs in the crawler.

Remove some new kth urls from being indexed
Add support for downloading google docs, sheets and slides
Add support for specifying an extra url in a course that will always be crawled even if there isn't a link to it in the course
fix issue where secure cookies couldn't be added
fix issues where large files would crash the crawler

N/A

…ts to be used for queries

…d urls

…dsheets to pdfs

…le them gracefully and log the error

…rogram to crash. this introduces a filesize cap

nattvara added 9 commits May 1, 2024 10:19

fix issue where urls waiting to be indexed would cause broken snapsho…

c47397b

…ts to be used for queries

add https://www.kth.se/en and https://www.kth.se/en/student to ignore…

588a025

…d urls

add google docs downloader that exports docs, presentations and sprea…

80a5210

…dsheets to pdfs

add extra_urls field to Course model

ca092a4

update crawler to add extra urls to snapshot

fab593f

update playwright to include cookies on kth github in addition to canvas

aeaa910

increase distance of extra urls to start from 1

6d012ea

fix issue where crawler would crash on crawler errors instead of hand…

d92a9be

…le them gracefully and log the error

fix issue where large binary files hosted on canvas would cause the p…

967981b

…rogram to crash. this introduces a filesize cap

nattvara merged commit 5516574 into main May 12, 2024
12 checks passed

nattvara deleted the feature/fix-crawling-bug branch May 12, 2024 15:44