Code

feat(tools/web): general web scraper#99

Code

Merged

idiotWu merged 46 commits intomainnpi-ai/npi:mainfrom feat-scrapernpi-ai/npi:feat-scraperCopy head branch name to clipboardOct 23, 2024

+1,797-159

Commits

Commits on Sep 27, 2024

• feat(tools/web): scraper

  idiotWu

  committed
  8b3d51a

  Copy full SHA for 8b3d51a

Commits on Sep 30, 2024

• refactor(scraper): use mutation observer to detect node additions

  idiotWu

  committed
  b3b8ad8

  Copy full SHA for b3b8ad8

• refactor(scraper): remove base64 images from markdown

  idiotWu

  committed
  545045b

  Copy full SHA for 545045b

Commits on Oct 1, 2024

• refactor(scraper): convert div to blocks

  idiotWu

  committed
  41f924c

  Copy full SHA for 41f924c

• fix(scaper): response matching regex

  idiotWu

  committed
  5897ac4

  Copy full SHA for 5897ac4

• refactor(scraper): trim whitespaces

  idiotWu

  committed
  2de496d

  Copy full SHA for 2de496d

• fix(scraper): return all results if limit is -1

  idiotWu

  committed
  6bbba11

  Copy full SHA for 6bbba11

• fix(scaper): response regex

  idiotWu

  committed
  d330389

  Copy full SHA for d330389

• fix(navigator): stops when no action parsed

  idiotWu

  committed
  06f44e8

  Copy full SHA for 06f44e8

• fix(navigator): system prompt

  idiotWu

  committed
  05915aa

  Copy full SHA for 05915aa

• refactor(scraper): optimize navigator instruction

  idiotWu

  committed
  5b2b620

  Copy full SHA for 5b2b620

• refactor(scraper): try calling navigator if no more items loaded after

  Show description for 968ecbf
  idiotWu

  committed
  968ecbf

  Copy full SHA for 968ecbf

• refactor(navigator): add result to history

  idiotWu

  committed
  e069ec7

  Copy full SHA for e069ec7

• refactor(scraper): optimize navigator instruction

  idiotWu

  committed
  335efe0

  Copy full SHA for 335efe0

Commits on Oct 11, 2024

• feat(browser_tool): support full-page mode

  idiotWu

  committed
  924146d

  Copy full SHA for 924146d

• feat(utils): add llm_tool_call

  idiotWu

  committed
  769f304

  Copy full SHA for 769f304

• feat(analyzer): add web page analyzer

  idiotWu

  committed
  dfc1df7

  Copy full SHA for dfc1df7

• feat(tools/page_analyzer): support infinite scrolling detection

  idiotWu

  committed
  4adba42

  Copy full SHA for 4adba42

• feat(tools/page_analyzer): support scraping type detection

  idiotWu

  committed
  362bac9

  Copy full SHA for 362bac9

• feat(page_analyzer): validate pagination button

  idiotWu

  committed
  5c51ab1

  Copy full SHA for 5c51ab1

• refactor(page_analyzer): optimize pagination validation

  idiotWu

  committed
  7e2d2ac

  Copy full SHA for 7e2d2ac

• chore(page_analyzer): update test links

  idiotWu

  committed
  9f8e944

  Copy full SHA for 9f8e944

Commits on Oct 16, 2024

• refactor(scraper): support user-specified pagination type

  idiotWu

  committed
  1e52183

  Copy full SHA for 1e52183

• feat(scraper): support batch_size

  idiotWu

  committed
  2caadda

  Copy full SHA for 2caadda

• feat(scraper): support specifying output file

  idiotWu

  committed
  661498a

  Copy full SHA for 661498a

• fix(scraper): open the page before inferring columns

  idiotWu

  committed
  dac9ea5

  Copy full SHA for dac9ea5

• fix(scrape/infer_column): force structured output

  idiotWu

  committed
  9759093

  Copy full SHA for 9759093

• refactor(scraper): support structured column definitions

  idiotWu

  committed
  f51070f

  Copy full SHA for f51070f

• fix(scraper/summarize): enclose column values in double quotes

  idiotWu

  committed
  868f927

  Copy full SHA for 868f927

• refactor(scraper): remove limits when no item selector specified

  idiotWu

  committed
  bb6ff49

  Copy full SHA for bb6ff49

• refactor(scraper): stream output in csv format

  idiotWu

  committed
  7972055

  Copy full SHA for 7972055

• feat(scraper): convert relative links to absolute links

  idiotWu

  committed
  6ac5027

  Copy full SHA for 6ac5027

• feat(scraper): support streaming

  idiotWu

  committed
  c97c7d3

  Copy full SHA for c97c7d3

• feat(page_analyzer): infer selectors

  idiotWu

  committed
  6b431c5

  Copy full SHA for 6b431c5

Commits on Oct 18, 2024

• refactor(page_analyzer): use most contentful elements to infer selectors

  idiotWu

  committed
  c268857

  Copy full SHA for c268857

• refactor(browser_tools): add load_page method

  idiotWu

  committed
  cdc6cc3

  Copy full SHA for cdc6cc3

• refactor(scraper): wait for items to be attached

  idiotWu

  committed
  eeef547

  Copy full SHA for eeef547

• test(scraper): full-auto twitter scraping demo

  idiotWu

  committed
  8985ca1

  Copy full SHA for 8985ca1

• refactor(html_to_md): use div's aria-label when possible

  idiotWu

  committed
  53a6620

  Copy full SHA for 53a6620

• refactor(scraper): include link to details when inferring columns

  idiotWu

  committed
  5260d73

  Copy full SHA for 5260d73

Commits on Oct 21, 2024

• test(scraper): add single entry test

  idiotWu

  committed
  cca9d1b

  Copy full SHA for cca9d1b

• refactor(scraper): optimize single entry column inference

  idiotWu

  committed
  59684a6

  Copy full SHA for 59684a6

• refactor(scraper): separate single/multi processing

  idiotWu

  committed
  b5ef7e9

  Copy full SHA for b5ef7e9

• refactor(scraper): support column types

  idiotWu

  committed
  09eb62f

  Copy full SHA for 09eb62f

• refactor(page_analyzer): optimize pagination button inference

  idiotWu

  committed
  5600f7a

  Copy full SHA for 5600f7a

• feat(analyzer): support items_selector in infinite scroll detection

  idiotWu

  committed
  2a1a2ea

  Copy full SHA for 2a1a2ea