Mapbox improvements (#1)

* parse settings from variables.txt

* avoid parsing same dict multiple times

* use data instead of saving to thousands of files

* add mapbox service with basic needed functions

* update requirements and add main script file

* update readme

* adding better variables

* fixing var names

* Update README.md

* restructure and simplify code

* improve how settings are set

* Update variables.txt

* removing old files

* Update .gitignore

* fix some problems

* updating mapbox permissions

* adding some more mapbox permissions

* Update README.md

* fix image zoom level

* Update README.md

* adding some test cases

* adding tests

* more tests

* fix get data by key in nested dict

* allow maki in marker and ensure hex color format

* change marker label default in description

* always str colours

* uncomment geojson generation line

* ensure marker label doesn't have svg extension

* make label optional

* add line width to multiline style

* Update variables.txt

Co-authored-by: David G <[email protected]>

.gitignore

@@ -158,8 +158,6 @@ cython_debug/
 #.idea/
 
 # End of https://www.toptal.com/developers/gitignore/api/python
-
-/geojson-files
 /mapbox-images
-*.json
 .DS_Store
+*.geojson

README.md

@@ -4,40 +4,69 @@ Use GoPro telemetry to generate a picture-in-picture map.
 
 ## Usage
 
-0. Install required packages `pip3 install -r requirements.txt`
-1. First extract the telemetry file (as json) from your GoPro video using [gopro-telemetry](https://github.com/JuanIrache/gopro-telemetry/). [Detailed instructions about how to do this can be found in this post](https://www.trekview.org/blog/2022/gopro-telemetry-exporter-getting-started/).
-2. Run the `python3 generategeojson.py -f TELEMETRY.json` to create the `.geojson` files from the data set (making sure to replace `TELEMETRY.json` with the telemetry file created at step 1). This will create of sequentially numbered `.geojson` files for each GPS time/lat/lon in the telemetry file in the directory `geojson-files/`.
-3. The `multiline.geojson` file generated contains the multiline for the entire path in the dataset, and [should be uploaded as a layer for a custom style on MapBox here](https://studio.mapbox.com/styles).
-4. Open the `geojsonrequest.py` and:
-  * paste in your MapBox API key into the `access_key` variable. 
-  * paste in your username and layer replacing the default `trekview/cl1nbz22z002614pn626zhm7u`.
-5. Run `python3 geojsonrequest.py`. This will generate .jpg map images in the directory `mapbox-images/`.
-6. Image overlay on video TODO
+0. First extract the telemetry file (as json) from your GoPro video using [gopro-telemetry](https://github.com/JuanIrache/gopro-telemetry/). [Detailed instructions about how to do this can be found in this post](https://www.trekview.org/blog/2022/gopro-telemetry-exporter-getting-started/).
+1. Install required packages `pip3 install -r requirements.txt`
+2. Fill in the `variables.txt` with your Mapbox API Key and username, and other variables if needed.
+3. Run `python3 main.py -f TELEMETRY.json`, replacing `TELEMETRY.json` with the file created at step 0. This will generate a `multiline.geojson` file, upload it to a new style in your Mapbox accountg and generate .jpg map images in the directory `mapbox-images/`. This process will happen at `services/geojson_service/generate_images()`.
+
+### `variables.txt`
+
+To run the script you need to set the required variables (and optional variables if you don't want to use defaults):
+
+* `mapbox_key` (required): your MapBox API key. [You can get a MapBox API key here](https://account.mapbox.com/) that will allow you [50,000 free static image lookups each month](https://www.mapbox.com/pricing/#glstatic). Ensure your key has the following permissions when you create it; `Styles:Tiles`, `Styles:Read`, `Styles:Write`, `Styles:List`, `Styles:Download`, `Styles:Protect`, `Fonts:Read`, `Datasets:Read`, `Vision:Read`, `Uploads:Read`, `Uploads:List`, `Uploads:Write`, `Tilesets:Read`, `Tilesets:List` and `Tilesets:Write`
+    * Default: null
+* `mapbox_username` (required): you MapBox username/account name. [You can see this under your account settings in MapBox](https://account.mapbox.com/)
+    * Default: null
+* `mapbox_base_style`: [Set your basemap style](https://docs.mapbox.com/api/maps/styles/). Options include (but not limited to): `mapbox/streets-v11`, `mapbox/outdoors-v11`, `mapbox/satellite-v9`, and `mapbox/satellite-streets-v11`
+    * Default: `mapbox/outdoors-v11`
+* `mapbox_user_style`: If you want to reuse a previous style you can pass it here like so `trekview/cl20cn42p009i15o97k316e8u` (this variable is not needed 99% of the time).
+    * Default: null
+* `mapbox_marker_label`: You can add a label inside the marker. It can be a digit (0-9), a letter (a-z) or a Maki icon (without svg extension, i.e: circle): https://labs.mapbox.com/maki-icons/`.
+    * Default: ''
+* `mapbox_img_w`: defines the width of the image for overlay. Recommended is 20% of video input width. Image height will be generated automatically based of 4:3 resolution.
+    * Default: `500`
+* `mapbox_zoom_level`: the zoom level for the map (recommended between 8-10). [See MapBox docs for more](https://docs.mapbox.com/help/glossary/zoom-level/). In short, the higher the zoom number, the closer to the ground the zoom is.
+    * Default: 10
+* `mapbox_line_colour_hex`: the colour you want for the line passed as a 6 digit hex code
+        * Default: `E48241`
+* `mapbox_marker_colour_hex`: the colour you want for the map point, passed as a 6 digit hex code (e.g. `000000` for black)
+    * Default: `000000`
+* `video_overlay_l_offset`: Left pixel padding
+    * Default: 40% of video width
+* `video_overlay_t_offset`: Top pixel padding
+    * Default: 60% of video width
 
-## Variable
+## Sample varible for common GoPro video sizes
 
-TODO
+### HERO video @ 5.2k
 
-To run the script you need to set the following variables:
+```
+mapbox_key: YOUR_KEY
+mapbox_username: YOUR_USER
+mapbox_base_style: outdoors
+mapbox_zoom_level: 10
+mapbox_img_w: 500
+mapbox_line_colour_hex: E48241
+mapbox_marker_colour_hex: 000000
+video_overlay_l_offset_pc:
+video_overlay_t_offset_pc:
+```
+
+`python3 main.py -f GS018422-gps-only.json`
+
+### 360 video @ 5.6k
 
-* `mapbox_key`: your MapBox API key. [You can get a MapBox API key here](https://account.mapbox.com/) that will allow you [50,000 free static image lookups each month](https://www.mapbox.com/pricing/#glstatic).
-* `mapbox_base_style`: [Set your basemap style](https://studio.mapbox.com/styles). Options are: `streets`, `outdoors`, `satellite`, `satellite_streets`, `light`, `dark`
-* `mapbox_img_w`: defines the width of the image for overlay. Recommended is 20% of video input width. Image height will be generated automatically based of 4:3 resolution.
-* `mapbox_zoom_level`: the zoom level for the map (recommended between 8-10). [See MapBox docs for more](https://docs.mapbox.com/help/glossary/zoom-level/).
-* `mapbox_marker_colour_hex`: the colour you want for the map point, passed as a 6 digit hex code (e.g. `000000` for black)
-* `mapbox_line_colour_hex`: the colour you want for the line passed as a 6 digit hex code
-* `video_overlay_l_offset`: Left pixel padding
-* `video_overlay_t_offset`:
 
+`python3 main.py -f GS018422.json`
 
 
 ## Todo 
 
 1. Automate step 3 by creating a mapbox style programatically
 2. Merge step 2 and 3 into a single command
-3. Add variables to MapBox Scripts (mapbox_img_w - as shown 
-2. Add the final logic to overlay mapbox images on video
-3. Bundle everything into a single script, taking `variables.txt` and an input json via the command line.
+3. Add variables to MapBox Scripts
+4. Add the final logic to overlay mapbox images on video (to not be done yet)
+5. Bundle everything into a single script, taking `variables.txt` and an input json via the command line.
 
 ## Overlay Scripts
 

main.py

@@ -0,0 +1,28 @@
+import argparse
+from time import sleep
+
+from services.geojson_service import generate_images, get_data
+from services.mapbox_service import create_base_style
+from settings import MAPBOX_USER_STYLE
+
+parser = argparse.ArgumentParser()
+files_dir = "geojson-files"
+
+parser.add_argument("-f", "--file", help="Telemetry Filename Including .json")
+args = parser.parse_args()
+
+if not args.file:
+    print(
+        "Please provide the telemetry filename (in base directory) using the -f flag, e.g. python3 main.py -f telemetry.json"
+    )
+    exit()
+
+
+images_data = get_data(args.file)
+# override style recreation (to reuse one created previously, for example)
+if not MAPBOX_USER_STYLE:
+    MAPBOX_USER_STYLE = create_base_style()
+    # we wait some seconds just to allow the style to be available (sometimes fails for first few images otherwise)
+    sleep(5)
+# call the image generator method
+generate_images(MAPBOX_USER_STYLE, images_data)

requirements.txt

@@ -1,3 +1,4 @@
 geojson==2.5.0
 requests==2.27.1
-urllib3==1.26.9
+urllib3==1.26.9
+mapbox==0.18.0

services/geojson_service.py

@@ -0,0 +1,70 @@
+import json
+import os
+import urllib.parse
+from multiprocessing.sharedctypes import Value
+
+import requests
+from settings import MAPBOX_KEY, MAPBOX_USERNAME
+
+from services.mapbox_service import generate_image
+
+files_dir = "geojson-files"
+images_dir = "mapbox-images"
+
+
+def generate_images(style, images_data):
+    try:
+        os.mkdir(f"./{images_dir}")
+    except OSError as error:
+        print(error)
+
+    for index, data in enumerate(images_data):
+        image = generate_image(data, style)
+
+        if image:
+            with open(f"./{images_dir}/{index:06}.png", "wb") as f:
+                f.write(image)
+                f.close()
+                print(f"Fetched image: {index}")
+
+
+# Find the samples value in dict
+def find_by_key(data, target):
+    if target in data.keys():
+        return data[target]
+    for key, value in data.items():
+        if isinstance(value, dict):
+            return find_by_key(value, target)
+
+
+def generate_multiline_geojson(data):
+    multiline = []
+    filedata = ""
+
+    for index, x in enumerate(data[1::2]):
+        multiline.append(
+            [[data[index][0], data[index][1]], [data[index + 1][0], data[index + 1][1]]]
+        )
+
+    filedata = f'{{"type": "FeatureCollection","features": [{{"type": "Feature","geometry": {{"type": "MultiLineString","coordinates": {multiline}}},"properties": {{"prop0": "value0"}}}}]}}'
+
+    f = open(f"./multiline.geojson", "w")
+    f.write(filedata)
+    f.close()
+
+
+def get_data(file: str):
+    with open(f"./{file}") as json_file:
+        data = json.load(json_file)
+        linestring = []
+
+        data = find_by_key(data, "GPS5").get("samples", [])
+
+        for x in data:
+            if "GPS (Lat.) [deg]" in x:
+                linestring.append([x["GPS (Lat.) [deg]"], x["GPS (Long.) [deg]"]])
+            else:
+                linestring.append([x["value"][1], x["value"][0]])
+
+        generate_multiline_geojson(linestring)
+        return linestring