Merge pull request #45 from plaintextpackets/speedtest

Speedtest

Author: plaintextpackets

.env

@@ -39,13 +39,17 @@ threshold_latency = "100" # 100ms latency threshold as max
 threshold_jitter = "30" # 30ms jitter threshold as max
 threshold_dns_latency = "100" # 100ms dns latency threshold as max
 
+# Speetest configuration (be very careful when running on a metered connection!)
+# - This configuration is for setting up a "speed test" or rather a test of your internet bandwidth.
+# - In order to test your upload and download bandwidth we use speedtest.net as source. So your client will connect there and upload and download some data.
+# - That also means that a random server is selected for the test (usually the nearest one)
+# - Setting the SPEEDTEST_INTERVAL too agressively will cause speedtest.net to block your requests, recommend 15 minutes (900 seconds) and above
+SPEEDTEST_ENABLED="True" # set this to "True" to enable speed test function
+SPEEDTEST_INTERVAL="937" # interval on which the speedtest will run, in seconds - note using a prime number helps reduce the number of collisions between netprobe and speed tests
 
 
 # SYSTEM VARIABLES - DO NOT TOUCH
 
-DEVICE_ID="44c42e36-f2eb-44d0-babc-1a749498e016"
-SITE_ID="f0ac2f33-1a20-471d-b430-d905900e0e71"
-
 PRESENTATION_PORT = "5000"
 PRESENTATION_INTERFACE = "0.0.0.0"
 

README.md

@@ -1,6 +1,6 @@
-# Netprobe Lite
+# Netprobe
 
-Simple and effective tool for measuring ISP performance at home. The tool measures several performance metrics including packet loss, latency, jitter, and DNS performance. It also aggregates these metrics into a common score, which you can use to monitor overall health of your internet connection.
+Simple and effective tool for measuring ISP performance at home. The tool measures several performance metrics including packet loss, latency, jitter, and DNS performance. It also has an optional speed test to measure bandwidth. Netprobe aggregates these metrics into a common score, which you can use to monitor overall health of your internet connection.
 
 ## Support the Project
 
@@ -17,14 +17,16 @@ https://youtu.be/Wn31husi6tc
 
 ## Requirements and Setup
 
-To run Netprobe Lite, you'll need a PC running Docker connected directly to your ISP router. Specifically:
+To run Netprobe, you'll need a PC running Docker connected directly to your ISP router. Specifically:
 
-1. Netprobe Lite requires the latest version of Docker. For instructions on installing Docker, see YouTube, it's super easy.
+1. Netprobe requires the latest version of Docker. For instructions on installing Docker, see YouTube, it's super easy.
 
-2. Netprobe Lite should be installed on a machine (the 'probe') which has a wired Ethernet connection to your primary ISP router. This ensures the tests are accurately measuring your ISP performance and excluding and interference from your home network. An old PC with Linux installed is a great option for this.
+2. Netprobe should be installed on a machine (the 'probe') which has a wired Ethernet connection to your primary ISP router. This ensures the tests are accurately measuring your ISP performance and excluding and interference from your home network. An old PC with Linux installed is a great option for this.
 
 ## Installation
 
+### First-time Install
+
 1. Clone the repo locally to the probe machine:
 
 ```
@@ -43,12 +45,46 @@ docker compose up
 docker compose down
 ```
 
+### Upgrading Between Versions
+
+When upgrading between versions, it is best to delete the deployment altogether and restart with the new code. The process is described below.
+
+1. Stop Netprobe in Docker and use the -v flag to delete all volumes (warning this deletes old data):
+
+```
+docker compose down -v
+```
+
+2. Clone the latest code (or download manually from Github and replace the current files):
+
+```
+git clone https://github.com/plaintextpackets/netprobe_lite.git
+```
+
+3. Re-start Netprobe:
+
+```
+docker compose up
+```
+
 ## How to use
 
 1. Navigate to: http://x.x.x.x:3001/d/app/netprobe where x.x.x.x = IP of the probe machine running Docker.
 
 2. Default user / pass is 'admin/admin'. Login to Grafana and set a custom password.
 
+## How to customize
+
+### Enable Speedtest
+
+By default the speed test feature is disabled as many users pay for bandwidth usage (e.g. cellular connections). To enable it, edit the .env file to set the option to 'True':
+
+```
+SPEEDTEST_ENABLED="True"
+```
+
+Note: speedtest.net has a limit on how frequently you can connection and run the test. If you set the test to run too frequently, you will receive errors. Recommend leaving the 'SPEEEDTEST_INTERVAL' unchanged.
+
 ### Change Netprobe port
 
 To change the port that Netprobe Lite is running on, edit the 'compose.yml' file, under the 'grafana' section:
@@ -72,23 +108,41 @@ DNS_NAMESERVER_4_IP="8.8.8.8" # Replace this IP with the DNS server you use at h
 
 Change 8.8.8.8 to the IP of the DNS server you use, then restart the application (docker compose down / docker compose up)
 
+### Use external Grafana
+
+Some users have their own Grafana instance running and would like to ingest Netprobe statistics there rather than running Grafana in Docker. To do this:
+
+1. In the compose.yaml file, add a port mapping to the Prometheus deployment config:
+
+```
+  prometheus:
+    ...
+    ports:
+      - 'XXXX:9090'    
+```
+... where XXXX is the port you wish to expose Prometheus on your host machine
+
+2. Remove all of the Grafana configuration from the compose.yaml file
+
+3. Run Netprobe and then add a datasource to your existing Grafana as http://x.x.x.x:XXXX where x.x.x.x = IP of the probe machine running Docker
+
 ### Data storage - default method
 
 By default, Docker will store the data collected in several Docker volumes, which will persist between restarts.
 
 They are:
 
 ```
-netprobe_lite_grafana_data (used to store Grafana user / pw)
-netprobe_lite_prometheus_data (used to store time series data)
+netprobe_grafana_data (used to store Grafana user / pw)
+netprobe_prometheus_data (used to store time series data)
 ```
 
 To clear out old data, you need to stop the app and remove these volumes:
 
 ```
 docker compose down
-docker volume rm netprobe_lite_grafana_data
-docker volume rm netprobe_lite_prometheus_data
+docker volume rm netprobe_grafana_data
+docker volume rm netprobe_prometheus_data
 ```
 
 When started again the old data should be wiped out.
@@ -144,13 +198,22 @@ mkdir -p data/grafana data/prometheus
 
 ### Run on startup
 
-To configure the tool to work as a daemon (run on startup, keep running), edit 'compose.yml' and add the following to each service:
+Netprobe will automatically restart itself after the host system is rebooted, provided that Docker is also launched on startup. If you want to disable this behavior, modify the 'restart' variables in the compose.yaml file to this: 
+
+```
+restart: never
+```
+
+### Wipe all stored data
+
+To wipe all stored data and remove the Docker volumes, use this command:
 
 ```
-restart: always
+docker compose down -v
 ```
+This will delete all containers and volumes related to Netprobe.
+
 
-More information can be found in the Docker documentation.
 
 ## FAQ & Troubleshooting
 

compose.yml

@@ -14,28 +14,51 @@ services:
       - ./config/redis/redis.conf:/etc/redis/redis.conf
     networks:
       - netprobe-net
+    dns:
+      - 8.8.8.8
+      - 8.8.4.4      
 
   netprobe:
     restart: always
     container_name: netprobe-probe
-    image: "plaintextpackets/netprobe:latest"
+    image: "plaintextpackets/netprobe:speedtest"
     volumes:
       - .:/netprobe_lite
     environment:
       MODULE: "NETPROBE"
     networks:
       - netprobe-net
+    dns:
+      - 8.8.8.8
+      - 8.8.4.4      
+
+  speedtest:
+    restart: always
+    container_name: netprobe-speedtest
+    image: "plaintextpackets/netprobe:speedtest"
+    volumes:
+      - .:/netprobe_lite
+    environment:
+      MODULE: "SPEEDTEST"
+    networks:
+      - netprobe-net
+    dns:
+      - 8.8.8.8
+      - 8.8.4.4      
 
   presentation:
     restart: always
     container_name: netprobe-presentation
-    image: "plaintextpackets/netprobe:latest"
+    image: "plaintextpackets/netprobe:speedtest"
     volumes:
       - .:/netprobe_lite
     environment:
       MODULE: "PRESENTATION"
     networks:
       - netprobe-net
+    dns:
+      - 8.8.8.8
+      - 8.8.4.4      
 
   prometheus:
     restart: always
@@ -49,6 +72,9 @@ services:
       - '--storage.tsdb.path=/prometheus'
     networks:
       - netprobe-net
+    dns:
+      - 8.8.8.8
+      - 8.8.4.4      
 
   grafana:
     restart: always
@@ -63,6 +89,9 @@ services:
       - '3001:3000'
     networks:
       - netprobe-net
+    dns:
+      - 8.8.8.8
+      - 8.8.4.4      
 
 volumes:
   prometheus_data:

config/__init__.py

@@ -15,9 +15,9 @@ class Config_Netprobe():
     probe_interval = int(os.getenv('PROBE_INTERVAL'))
     probe_count = int(os.getenv('PROBE_COUNT'))
     sites = os.getenv('SITES').split(',')
-    device_id = os.getenv('DEVICE_ID')
-    site_id = os.getenv('SITE_ID')
     dns_test_site = os.getenv('DNS_TEST_SITE')
+    speedtest_enabled = os.getenv("SPEEDTEST_ENABLED", 'False').lower() in ('true', '1', 't')
+    speedtest_interval = int(os.getenv('SPEEDTEST_INTERVAL'))
 
     DNS_NAMESERVER_1 = os.getenv('DNS_NAMESERVER_1')
     DNS_NAMESERVER_1_IP = os.getenv('DNS_NAMESERVER_1_IP')