Code

Processing pipeline support#1004

Code

Draft

janhoy wants to merge 18 commits intodadoonet:masterdadoonet/fscrawler:masterfrom cominvent:pipelinecominvent/fscrawler:pipelineCopy head branch name to clipboard

+716-28

Commits

Commits on Sep 4, 2020

• Processing pipeline support (dadoonet#1003).

  janhoy

  committed
  7c53731

  Copy full SHA for 7c53731

• Move config into fs and actually use it

  Show description for 72aa0b7
  janhoy

  committed
  72aa0b7

  Copy full SHA for 72aa0b7

• Move config into fs and actually use it

  Show description for a451ce1
  janhoy

  committed
  a451ce1

  Copy full SHA for a451ce1

• Address some automated PR review feedback

  janhoy

  committed
  0a37d83

  Copy full SHA for 0a37d83

• Fix Codacy issues

  janhoy

  committed
  225f348

  Copy full SHA for 225f348

• More fixes

  janhoy

  committed
  8a57fb7

  Copy full SHA for 8a57fb7

• More fixes

  janhoy

  committed
  0b54155

  Copy full SHA for 0b54155

Commits on Sep 5, 2020

• Avoid nullpointer when pipeline not defined

  janhoy

  committed
  1dc0912

  Copy full SHA for 1dc0912

Commits on Sep 6, 2020

• Introduce a init() method that takes a config with settings like fsSettings and esClient

  Show description for 430de8b
  janhoy

  committed
  430de8b

  Copy full SHA for 430de8b

• Remove empty constructor, add debug msg when a file enters the pieline

  janhoy

  committed
  a7f8bf5

  Copy full SHA for a7f8bf5

• Tikaprocessor should not close stream, it is done elsewhere

  janhoy

  committed
  601112b

  Copy full SHA for 601112b

• Extract Tika and indexing into separate methods, to make it even easier to subclass the DefaultProcessingPipeline, calling these two methods plus your own in addition.

  Show description for 4b779d6
  janhoy

  committed
  4b779d6

  Copy full SHA for 4b779d6

• Revert some unnecessary changes, try to please codacy

  janhoy

  committed
  c2d5e31

  Copy full SHA for c2d5e31

• Remove unused import

  janhoy

  committed
  d3c0272

  Copy full SHA for d3c0272

• Remove unused import

  janhoy

  committed
  a4ab68a

  Copy full SHA for a4ab68a

Commits on Sep 7, 2020

• TikaProcessor configurable for what metadata fields to extract

  janhoy

  committed
  ca37bd3

  Copy full SHA for ca37bd3

• Merge branch 'master' into pipeline

  janhoy

  committed
  ef93e99

  Copy full SHA for ef93e99

• Throw exception if pipeline is not found

  janhoy

  committed
  fe03ca8

  Copy full SHA for fe03ca8