Create a Collaborative Regex Engine

[teolemon]

Why

• Managing the variability of formulations is no longer possible at our scale (languages, variations) without detonating our current detectors.
  • 16.000 labels (out of 2000 explicitly known)
  • 8.800 origins of ingredients (150 countries and their cities / regions)
  • 33.000 categories (4000 explicitly known)
  • 3000 types, recyclability and packaging materials
  • …
• Need to distribute rule making to more people

What

Solution: Create an extraction rules engine

• Store all the OCRs in an ElasticSearch engine
• Create a system to store “IF this THEN that" rules in a database (with CRUD operations)
  • Triggers
    • OCR keyword
    • A combination of several triggers (OCR keyword 1 + 2, OCR Keyword + existing value…)
  • Output: key/value: “Origins”: “France” (2 rules if several key/values, or the possibility to have several key/values for a single trigger)
  • Status: Enabled/Disabled
  • User: Creator of the rule
• Create a system to apply the rules in the future on incoming photos
• Create a system to apply the rule to all or part of the existing OCR / Images (list with checkboxes)
• Create an authentication system with differentiated privileges
  • lambda users: propose rules (not activated)
  • moderators: create or activate proposed rules
  • superadmin
• Create a table with the history of the modifications applied via the rules (optional, we can use the existing insight system for this)
• Create a system to suggest values to search from an already available ngrams file
• Correlate recurring textual patterns with (non validated predictions AND/OR values already in the Open Food Facts database)

Docs

• Mockups & Design doc
• Tasklist

Examples

• Extract Origins in the EU Organic logo
  Agriculture Espagne or UK agriculture is sometimes mentioned, and means the ingredients come from the given country.
  https://world.openfoodfacts.org/images/products/356/007/117/4446/10.jpg
  https://world.openfoodfacts.org/images/products/356/007/117/4446/10.json
  ES-ECO-035-RI\nAGRICULTURE\nBIOLOGIQUE AGRICULTURE ESPAGNE\n
  https://user-images.githubusercontent.com/1689815/88476160-94366d80-cf36-11ea-947f-d3c39cdda03e.png
• Importé du Pérou

Part of

• 🎯 What can I work on ? #374