New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

index における文章の保持の仕方を省メモリなものに変える #3

Open

p-baleine opened this issue Mar 18, 2021 · 0 comments

Contributor

p-baleine commented Mar 18, 2021

今は、index で文章そのもののリストを値として保持するようになっている:

naivesearch/naivesearch/indexer/inverted_index.py

Lines 27 to 30 in 43125c1

    
           for d in reader: 
        
               for chunker in chunkers: 
        
                   for chunk in chunker(d): 
        
                       self.index[chunk].append(d)

これを ID などに変えて、省メモリ化を図る。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment