Memory error

[goodmami]

Something is using up too much memory:

  File ".../run_freki.py", line 58, in analyze
    for i, line in enumerate(respace(blk, xoffset=(l_margin * -1))):
  File ".../run_freki.py", line 153, in respace
    cur_line.append(' ' * dx)
MemoryError

[goodmami]

This still happens. I can add that it often happens when the page number is very high (e.g. >90) or if a zone is very large (and possibly contains many tokens that couldn't be broken into lines). Either way, too much is being stored in memory at once: the original XML document, the numpy arrays, the FrekiLines and FrekiBlocks (which aren't serialized until the whole document has been collected)... We should probably put more attention on processing a page at a time (or a window of N pages), serializing the results, and discarding the page data.