Merge pull request #1 from LunarWatcher/dashboard-i-guess

Initial dashboard stuff

CMakeLists.txt

@@ -89,11 +89,13 @@ else()
 endif()
 add_definitions(-DHAZEL_VERSION="${HAZEL_VERSION}")
 # }}}
+add_definitions(-DCROW_ENFORCE_WS_SPEC=1)
 
 add_subdirectory(src)
+add_subdirectory(tests EXCLUDE_FROM_ALL)
 
-#add_custom_target(test
-    #COMMAND tests
-    #DEPENDS tests
-    #COMMENT "Test hazel")
+add_custom_target(test
+    COMMAND tests
+    DEPENDS tests
+    COMMENT "Test hazel")
 # vim:ft=cmake

README.md

@@ -5,8 +5,9 @@ Internal metaserver for my homelab. It's going to be a while before it's general
 
 ## Requirements
 
- * A Linux-based server
- * C++20 compiler
- * CMake 3.28 (Pro tip: [CMake is available via pip](https://pypi.org/project/cmake/))
- * libpq-dev, uuid-dev, libasio-dev (Debian-based names; look up your distro's package manager for the applicable packages). Additional dependencies are sourced automagically during the build process
- * Postgresql
+* A Linux-based server
+* C++20 compiler
+* CMake 3.28 (Pro tip: [CMake is available via pip](https://pypi.org/project/cmake/))
+* libpq-dev, uuid-dev, libasio-dev (Debian-based names; look up your distro's package manager for the applicable packages). Additional dependencies are sourced automagically during the build process
+* Postgresql
+

docs/features/dashboard/Index.md

@@ -0,0 +1,83 @@
+# Dashboard
+
+The dashboard is one of Hazel's built-in features. 
+
+## Modules
+
+The dashboard is built around different modules. Note that the modules are separated into distinct types, so they can't be easily mixed in the final output. This may become an option in the future, but is not  currently a goal.
+
+### Link module (`links` config key)
+
+The link module can be used as a way to, shock, link to other stuff. 
+
+Some sites support additional features in the display. These are called "dynamic apps" internally, because I wasn't creative when I named them. These can show, for example, aggregate statistics directly on the dashboard sourced from the service linked to. 
+
+Note that these are optional, including for enabled services, and have to be manually configured.
+
+#### Config format
+
+```json
+"dashboard": {
+    "links": {
+        "name": "Display name in the dashboard",
+        "type": "app type; optional, see 'Supported dynamic apps' for the list of allowed values",
+        "url": "https://example.com",
+        "config": {
+            "Contains a set of key-value pairs used for configuring dynamic apps. Unsupported values are quietly ignored. See the docs for each dynamic app for allowed values"
+        }
+    }
+}
+```
+
+#### Dynamic app caveats
+
+Dynamic app updates are only done on a single thread. This means that if one update takes a significant amount of time, other updates may be delayed.
+
+It's therefore strongly recommended to not set the `update_frequency` interval too low, as this may result in update cycles taking longer than planned. Additionally, due to implementation reasons, this will slow down clients that want to load at the same time as the update. 
+
+This is a consequence of how the dynamic apps are set up. They're not queried by the client when the page is loaded, but handled fully on the backend. This is done to allow the dashboard to be displayed by other users without leaking API keys and similar.
+
+In the event of future multi-user support, some dynamic apps may be handled etnirely client-sided. 
+
+#### Supported dynamic apps
+
+* Pi-hole (`pihole`)
+* Miniflux (`miniflux`)
+* Uptime-kuma (`uptime-kuma`)
+
+##### Pi-hole
+| Config key name | Description | Type | Required |
+| --------------- | ----------- | ---- | -------- |
+| `api_key`       | The API key for pihole, as found in pihole's settings | String | Yes |
+| `update_frequency` | The number of seconds between updates | integer, >= 5 | No; defaults to 600 |
+| `verify_ssl` | Whether or not to verify certificates. This may need to be set to `true` for self-signed certificates | bool | No; defaults to false. |
+
+##### Uptime-kuma
+
+| Config key name | Description | Type | Required |
+| --------------- | ----------- | ---- | -------- |
+| `dashboard`     | The dashboard to gather data from. Note that a public dashboard must be configured in uptime-kuma | string | Yes |
+| `update_frequency` | The number of seconds between updates | integer, >= 5 | No; defaults to 60 |
+| `verify_ssl` | Whether or not to verify certificates. This may need to be set to `true` for self-signed certificates | bool | No; defaults to false. |
+
+##### Miniflux
+
+| Config key name | Description | Type | Required |
+| --------------- | ----------- | ---- | -------- |
+| `api_key`[^1]     | An API key for Miniflux, generated under settings. | string | Yes |
+| `update_frequency` | The number of seconds between updates | integer, >= 5 | No; defaults to 1800 |
+| `verify_ssl` | Whether or not to verify certificates. This may need to be set to `true` for self-signed certificates | bool | No; defaults to false. |
+
+#### Self-signed certificates
+
+**NOTE:** This only applies if you haven't installed the CA certificate to the system. If you can `curl https://your-domain` without it complaining about an SSL cert problem, it's probably fine. You'll also get an equivalent error in Hazel's logs (`journalctl -r -u hazel`) if the requests fail.
+
+If you can't get this to work, the rest of the section is for you. 
+
+---
+
+For now, self-signed certificates are supported by disabling SSL verification. You can do this by setting `verify_ssl` in dynamic apps to `false`. Note that this can have security implications if  the domains are hijacked.
+
+libcpr, the underlying request library, [supports custom CA verification](https://docs.libcpr.org/advanced-usage.html#certificate-authority-ca-bundle), but this has not been set up because it's tedious and I just don't want to. Pull requests adding support for self-signed certificates are welcome.
+
+[^1]: Miniflux's API does support username and password authentication, but this is not supported by Hazel to minimise username and password hard-coding in config

etc/hazel/hazel.conf.template

@@ -31,6 +31,19 @@
             }
         }
     },
+    "dashboard": {
+        "links": [
+            {
+                "name": "Pihole",
+                "type": "pihole",
+                "url": "https://pihole.lan",
+                "config": {
+                    "api_key": "abcd69420",
+                    "update_frequency": 600
+                }
+            }
+        ]
+    },
     "server": {
         "port": 6906
     }

src/CMakeLists.txt

@@ -5,6 +5,7 @@ set (SOURCE_FILES
     # Endpoints
     hazel/features/miniflux/MinifluxProxy.cpp
     hazel/features/WebCore.cpp
+    hazel/features/webcore/Dashboard.cpp
 
     # Adapters
     hazel/automation/adapters/DiscordAdapter.cpp
@@ -13,6 +14,10 @@ set (SOURCE_FILES
     # Utils
     hazel/json/JsonImpl.cpp
 
+    # Dashboard stuff
+    hazel/data/DashboardDataProvider.cpp
+    hazel/data/dashboard/DashboardUpdaters.cpp
+
     # Server (just the core shit)
     hazel/server/Hazel.cpp
 )
@@ -39,6 +44,18 @@ target_link_libraries(hazelsrc
 )
 target_link_libraries(hazel hazelsrc)
 
+# Copying web content
+add_custom_target(webcontent
+    COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/www
+    COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/www ${CMAKE_BINARY_DIR}/www
+    COMMENT "Copying web to output dir"
+)
+add_custom_target(webcontentdebug
+    COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_SOURCE_DIR}/www ${CMAKE_BINARY_DIR}/www-debug
+    COMMENT "Symlink debug symlink repo"
+)
+add_dependencies(hazelsrc webcontent webcontentdebug)
+
 # Doesn't work because no sudo
 # Maybe it doesn't make sense to use /etc then? maybe use /opt/hazel/conf/?
 # Won't help with the systemd service, but there's an install script for that
@@ -52,10 +69,9 @@ install(
         PATTERN *.so*
         PATTERN *.a EXCLUDE
 )
-# For when there's actual web content
-#install(
-    #DIRECTORY www
-    #DESTINATION .
-#)
+install(
+    DIRECTORY ${CMAKE_BINARY_DIR}/www
+    DESTINATION .
+)
 
 # vim:ft=cmake

src/hazel/Main.cpp

@@ -1,5 +1,13 @@
 #include "hazel/server/Hazel.hpp"
 
+#include "spdlog/spdlog.h"
+#include "spdlog/cfg/env.h"
+
 int main() {
+#ifdef HAZEL_DEBUG
+    spdlog::set_level(spdlog::level::debug);
+#else
+    spdlog::cfg::load_env_levels();
+#endif
     hazel::HazelCore::getInstance().init();
 }

src/hazel/data/DashboardDataProvider.cpp

@@ -0,0 +1,93 @@
+#include "DashboardDataProvider.hpp"
+#include "hazel/data/DashboardStructs.hpp"
+#include "hazel/data/dashboard/DashboardUpdaters.hpp"
+#include "hazel/server/Config.hpp"
+#include "spdlog/spdlog.h"
+#include <limits>
+
+namespace hazel {
+
+DashboardDataProvider::DashboardDataProvider(DashboardConfig& conf) {
+    if (conf.links.size()) {
+        std::vector<DashboardLinkModule> links;
+        for (auto& link : conf.links) {
+            long long updateFreq = 60;
+            if (link.config.has_value()
+                && link.config->contains("update_frequency")) {
+                updateFreq = link.config->at("update_frequency").get<long long>();
+                if (updateFreq < 5) {
+                    updateFreq = 5;
+                }
+            } else {
+                switch (link.type) {
+                case LinkDynamicApp::Pihole:
+                    updateFreq = 600;
+                    break;
+                case LinkDynamicApp::Miniflux:
+                    updateFreq = 1800;
+                    break;
+                default:;
+                }
+            }
+            links.push_back(DashboardLinkModule {
+                .title = link.name,
+                .url = link.url,
+                .icon = "TODO",
+                .colour = "#000",
+                .fields = {},
+                .lastUpdate = Clock::time_point(std::chrono::seconds(0)),
+                .updateFrequency = std::chrono::seconds(updateFreq),
+                .appType = link.type,
+                .extras = link.config.value_or(nlohmann::json::object()),
+            });
+
+        }
+        this->dataArchive.links = std::move(links);
+    }
+
+    this->updateProcessor = std::thread(std::bind(&DashboardDataProvider::update, this));
+}
+
+void DashboardDataProvider::update() {
+    while (true) {
+        // Dirty is set to cache.empty() to make sure the string is initialised properly at least once.
+        // This can be necessary if, after the initialisation, none of the dashboard components 
+        // contain dynamic data. This would result in the update function returning false for all
+        // components, resulting in an empty string.
+        //
+        // Also note that this raw access is safe. The update function is the only function that does a write,
+        // and locking read-only is unnecessary here.
+        // Even with a write on another thread, length should never dip back to zero even in a race
+        // condition, so it should be fine. 
+        // Plus, if multi-thread writes become a thing, the data archive also needs to be
+        // wrapped in a thread-safe wrapper, which I foresee could be Fun:tm:. This should
+        // never work with enough data for that to be applicable though, so I'm choosing to
+        // completely ignore it :p
+        long long sleepTime;
+        bool dirty = 
+            Updaters::updateDashboardData(dataArchive, sleepTime)
+            || jsonCache.raw().empty();
+
+        // In some cases, the next update for a Thing:tm: can be scheduled immediately after the last update.
+        // This max statement is to make sure it isn't immediately or almost immediately executed to reduce cache writes.
+        sleepTime = std::max(sleepTime, 5ll);
+
+        if (dirty) {
+            this->jsonCache.write([&](std::string& cache) {
+                cache = nlohmann::json(dataArchive).dump();
+            });
+        }
+        if (sleepTime == std::numeric_limits<long long>::max()) {
+            spdlog::warn("sleepTime is the max long value. The dashboard probably doesn't have anything in it that needs updates. Stopping update thread. If this is wrong, please open a bug report: https://github.com/LunarWatcher/hazel/issues/");
+            break;
+        }
+        spdlog::debug("Dashboard data refreshed (dirty = {}). Sleeping for {}", dirty, sleepTime);
+        std::this_thread::sleep_for(std::chrono::seconds(sleepTime));
+    } 
+}
+
+std::string DashboardDataProvider::getJsonData() {
+    return this->jsonCache.copy();
+}
+
+}

src/hazel/data/DashboardDataProvider.hpp

@@ -0,0 +1,36 @@
+#pragma once
+
+#include "hazel/data/DashboardStructs.hpp"
+#include "hazel/server/Config.hpp"
+#include "hazel/sync/RWContainer.hpp"
+#include <optional>
+#include <thread>
+#include <shared_mutex>
+#include <nlohmann/json.hpp>
+
+namespace hazel {
+
+class DashboardDataProvider {
+private:
+    /**
+     * Underlying struct for the data.
+     * This is used to update and manage the data 
+     * in a non-JSON format.
+     */
+    DashboardData dataArchive;
+
+    RWContainer<std::string> jsonCache;
+
+    /**
+     * The thread in charge of running the update() function
+     */
+    std::thread updateProcessor;
+
+    void update();
+public:
+    DashboardDataProvider(DashboardConfig& conf);
+
+    std::string getJsonData();
+};
+
+}