Price tags input from picture

[monsieurtanuki]

Problem

Entering price tags is a bit tedious.

Proposed solution

What if, in "Add price tags", we were able to extract as much data as possible from the proof picture.
The easy part would be automatically detecting the barcodes, and we may have already something coded for that.
The cherry on top would be finding the shop from the GPS, and reading the prices from the picture.
cc @raphodn @raphael0202

Intended typical proof picture

cf. #4588

[raphodn]

It depends if you want the intelligence to be in the mobile or in the backend :)
It also depends if you want a single contributor to do the whole proof + price addition workflow (at least for "Price tag" proofs), or seperate into sub-actions.

Following discussions here - openfoodfacts/open-prices#526 (comment) - and quite a lot of work by @raphael0202 , we're very close to having actual price predictions from proof images stored in the backend.
The next logical step will be a hunger-games-like UI where users will be able to validate the price predictions and actually create the prices.

So uploading good quality proofs (with location & date) is key. AI & crowdsourcing can do the rest for the price extraction, no need to ask the initial contributor :)

[monsieurtanuki]

It depends if you want the intelligence to be in the mobile or in the backend :)

@raphodn I get your point, but if you go this way scanning is AI and could be done in the backend too.
It depends on where you put the cursor.
As a user I may feel more involved if I do a little work, with instant visible results, rather than taking pictures for the cloud. I already know that I'll get issues like "background tasks don't work" if we do something with limited visible results.
Of course "your" AI can work too, it's not an opposition.

That said, my first move would be to see how many barcodes our scanner would detect with a picture like in the OP.

[raphael0202]

We won't do full price addition without human validation (yet), so someone will check the extracted value anyway. It can be a random user, or the user who uploaded the proof in a subsequent screen.
@monsieurtanuki you can test the contribution assistant (https://prices.openfoodfacts.org/experiments/contribution-assistant) to see how automatic extraction from price tags behaves currently.

[monsieurtanuki]

@raphael0202 I've just tested the contribution assistant:

• this is definitely an online tool, which forces you to stay in the shop in front of the shelf a bit longer than usual. And to have a decent connection.
• I understand but didn't like the fact that the user had to check if the picture was "good enough", then if barcodes were "visible enough" and had to crop each price tag. An evolution (on device) would be something like "Hmm, I see barcode 1234 and barcode 6789, please confirm or take another picture". Assuming that the most painful part is scanning the barcodes.
• there was a little bug: the prices were correct but displayed as "per kg" instead of "per unit" (it was something like 2 yogurts)
• the second barcode didn't match an existing product, which could mean either a wrong scan or a new product. In both cases that didn't look good, and I was somehow discouraged from going on
• therefore I didn't go on. Later I went back to the page to go on, but the page was reinitialized and I lost all my data.

I may sound negative, but here I'm talking as a user. I believe that the most pain-in-the-neck step is scanning barcodes, so this is where I'd expect some improvements. And if the picture is good enough for prices, I assume that prices are readable too.
Btw I recently PR'ed in Smoothie a multi-product scanner: you scan several products in a row without going back and forth to another page. It's more natural (and more fun!).

[raphael0202]

this is definitely an online tool, which forces you to stay in the shop in front of the shelf a bit longer than usual. And to have a decent connection.

You can select an existing proof (either from your device gallery or from the already uploaded proof), so it doesn't have to be an online tool! Personally I upload bunches of pictures using the multiple image uploader (https://prices.openfoodfacts.org/prices/add/multiple), then process the prices later using the contribution assistant.

there was a little bug: the prices were correct but displayed as "per kg" instead of "per unit" (it was something like 2 yogurts)

Was it a product with barcode? Gemini should detects whether it's a raw product (without barcode), and select the "category" or "barcode" type accordingly, but it can fails of course.

I may sound negative, but here I'm talking as a user. I believe that the most pain-in-the-neck step is scanning barcodes, so this is where I'd expect some improvements. And if the picture is good enough for prices, I assume that prices are readable too.

Well, if the barcode can be read from the price tag, you don't have to scan barcodes. That's the magic of this tool: if you take pictures that are qualitative enough, you can add 100 prices in 5 minutes in the shop, ~20 minutes on your desktop. And we're working on automating the extraction flow further.

[monsieurtanuki]

You can select an existing proof (either from your device gallery or from the already uploaded proof), so it doesn't have to be an online tool!

Fair enough, but in that case you must be confident that the images are "good enough" a priori, right?
My main concern here is about the initial 2-step bottleneck: having to confirm that the picture is "good", and having to crop the price tags. A local tool that would confirm the barcodes (and therefore the picture) on the fly would be great, wouldn't it?
And the bulk mode does sound great!

[monsieurtanuki]

@raphael0202 About the "price per kg" bug:

[raphael0202]

Fair enough, but in that case you must be confident that the images are "good enough" a priori, right?

Indeed, it depends on the quality of the input image: the contributor should not take a picture too far away from the shelf, and should not shake.

My main concern here is about the initial 2-step bottleneck: having to confirm that the picture is "good", and having to crop the price tags. A local tool that would confirm the barcodes (and therefore the picture) on the fly would be great, wouldn't it?

Price tag cropping will be done automatically, we already integrated a ML model to do this, that will be integrated in the contribution assistant soon :)
And indeed, it would be useful!

[monsieurtanuki]

For the record I've just coded a small flutter app with ML Kit (barcode scanner and text recognition):

• most texts are recognized (but in non trivial order), including barcodes as numbers
• barcodes seems to be detected when there's only one barcode in the picture

As a suggestion, a typical process for the Smoothie user would be to:

• point the camera towards a single price tag
• have the barcode recognized by Smoothie
• save the "good enough" picture
• send the picture as a proof, possibly in a background task
• let the Prices server predict the price value - as well as the barcode
  • either let the user confirm the server prediction in real time
  • or let any user "hunger game" the predicted prices, later, in a different session