Create Program to Process Images Locally from ML Model

[keenanjohnson]

Now that we have seen promising results from training our own image processing machine learning model, we should create a local application that let's anyone process images easily. All the code is currently open source, but requires quite a bit of setup to still use which isn't ideal.

https://github.com/keenanjohnson/underwater-auto-image-encoder

The major processing steps:

1. Convert from gpr to tiff file (currently I use the official go pro tool to do this)
2. Process the image with the inference.py script in (https://github.com/keenanjohnson/underwater-auto-image-encoder)

Item 2 above is the most difficult part, but I am hoping it's easy to package both steps into one little program as well.

[keenanjohnson]

I have the basic outline and functionality of an application that looks like this in this branch: https://github.com/keenanjohnson/underwater-auto-image-encoder/tree/basic-gui

Everything seems to be working, I just need to package it up as an executable program.

[keenanjohnson]

Hey @zhrandell, @reidtosa, @m-h-williams, @clydemcqueen, and @tonioteran .

I think I have finished the development for the GUI tool that lets you point to a directory of .gpr files, select a model (like one of the ones from here: https://drive.google.com/drive/u/0/folders/1Vdctr52LTxoS6eecFiGS5LROZYSqJ3vl) and automatically process files into the enhanced .tiff or jpegs at the standard size.

The program is available for download on the release page here: https://github.com/keenanjohnson/underwater-auto-image-encoder/releases/tag/v0.1.0

or generally from this repo: https://github.com/keenanjohnson/underwater-auto-image-encoder

It took a bit longer than expected as I had to fix some bugs in the windows version of the official gpr_tools program, but I was able to fork that and get everything working eventually. gopro/gpr#51

It seems to work on my windows machine and macbook, but I'm sure there will be ideas or bugs as we go, but curious to see if you all find it useful in your workflow.

[zhrandell]

@keenanjohnson, this is awesome, thank you so much for all the work you've put into this!!

I was able to download the program without any issues, and it worked flawlessly when pointed at a folder of .GPR photos . . . it took 12min to process 7 photos, and all the JPEGs were in the output folder. Nicely done! (and bonus points for the "dark mode" option!)

We'll want to get this installed on our Mapping Station computer in Market Square - I'll be curious to see the run-time, as that machine is beefier than our work laptops.

[keenanjohnson]

That's great to hear! I'm sure we can improve the performance a bit as well if we find that this type of solution is what's really useful for the team.

[keenanjohnson]

I'm really curious to see how things go with you and the team when you combine this preprocessing with your second stage processing in coralnet-toolbox to tell us if this model is good enough to start being a time savings for you all or if we need to explore other model types or continue training.

[zhrandell]

I'm really curious to see how things go with you and the team when you combine this preprocessing with your second stage processing in coralnet-toolbox to tell us if this model is good enough to start being a time savings for you all or if we need to explore other model types or continue training.

Totally! I've started to think this through, and I'll create a Git Issue relatively shortly to organize next-steps to specifically compare the performance of the trained Toolbox classification model on hand-edited vs UIE processed images. One caveat is that the Toolbox classification model was trained on the "smoother" hand-edited 224 x 224 image patches, so I'll be interested to see how that model does with the slightly increased noise associated with the UIE output . . . there's a lot we can do experimentally to dig-in!

[zhrandell]

One related question, Keenan - is the UIE program you created well suited to also eventually provide the ability for us to train additional models via input image pairs (e.g., from a new survey area), or do you envision that happening via a different method? Or is that still TBD? I imagine model training is likely more complex than applying a trained model . . .

Some day it might be worth testing to see how well the Mapping Station (the big gaming computer in our Market Square area) works to train a model . . . that would allow us to see if we want to do that training locally, or instead want to explore a cloud solution.

[keenanjohnson]

Good question! The user processing application is a bit stripped down to not need all of the libraries / tools needed for training (to try and reduce the download size / memory requirements), so it doesn't currently have all the things needed for training, but we could extend this program or add a second that allows that. I think we would have to weigh whether we want to pursue the cloud option for training though as that would be an entirely different technical path with a web UI login etc for the future.

[keenanjohnson]

The initial version of this seems complete! We will add issues for new features as we add them.