Skip to content

Commit

Permalink
Remove redundancy in pieChart instances - JavaScript
Browse files Browse the repository at this point in the history
Remove redundancy in dockerStats - JavaScript
Auto calculate current process metrics by default
Reuse disk usage metrics to avoid redundant calls
Update runbook and README.md
  • Loading branch information
dormant-user committed Sep 29, 2024
1 parent 9c54523 commit 70c4fcb
Show file tree
Hide file tree
Showing 15 changed files with 181 additions and 147 deletions.
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,19 +53,24 @@ pyninja start
> _By default, `PyNinja` will look for a `.env` file in the current working directory._
</details>
- **APIKEY** - API Key for authentication.
- **NINJA_HOST** - Hostname for the API server.
- **NINJA_PORT** - Port number for the API server.
- **WORKERS** - Number of workers for the uvicorn server.
- **REMOTE_EXECUTION** - Boolean flag to enable remote execution.
- **API_SECRET** - Secret access key for running commands on server remotely.
- **MONITOR_USERNAME** - Username to authenticate the monitoring page.
- **MONITOR_PASSWORD** - Password to authenticate the monitoring page.
- **MONITOR_SESSION** - Session timeout for the monitoring page.
- **MAX_CONNECTIONS** - Maximum number of monitoring sessions allowed in parallel.
- **SERVICE_MANAGER** - Service manager filepath to handle the service status requests.
- **PROCESSES** - List of process names to include in the monitor page.
- **SERVICES** - List of service names to include in the monitor page.
- **GPU_LIB** - GPU library filepath to use for monitoring.
- **DISK_LIB** - Disk library filepath to use for monitoring.
- **SERVICE_LIB** - Memory library filepath to use for monitoring.
- **PROCESSOR_LIB** - Processor library filepath to use for monitoring.
- **DATABASE** - FilePath to store the auth database that handles the authentication errors.
- **RATE_LIMIT** - List of dictionaries with `max_requests` and `seconds` to apply as rate limit.
- **APIKEY** - API Key for authentication.
- **LOG_CONFIG** - Logging configuration file path.

⚠️ Enabling remote execution can be extremely risky and a major security threat.
So use **caution** and set the **API_SECRET** to a strong value.
Expand Down
11 changes: 8 additions & 3 deletions docs/README.html
Original file line number Diff line number Diff line change
Expand Up @@ -90,19 +90,24 @@ <h2>Environment Variables<a class="headerlink" href="#environment-variables" tit
<div><p><em>By default, <code class="docutils literal notranslate"><span class="pre">PyNinja</span></code> will look for a <code class="docutils literal notranslate"><span class="pre">.env</span></code> file in the current working directory.</em></p>
</div></blockquote>
</details><ul class="simple">
<li><p><strong>APIKEY</strong> - API Key for authentication.</p></li>
<li><p><strong>NINJA_HOST</strong> - Hostname for the API server.</p></li>
<li><p><strong>NINJA_PORT</strong> - Port number for the API server.</p></li>
<li><p><strong>WORKERS</strong> - Number of workers for the uvicorn server.</p></li>
<li><p><strong>REMOTE_EXECUTION</strong> - Boolean flag to enable remote execution.</p></li>
<li><p><strong>API_SECRET</strong> - Secret access key for running commands on server remotely.</p></li>
<li><p><strong>MONITOR_USERNAME</strong> - Username to authenticate the monitoring page.</p></li>
<li><p><strong>MONITOR_PASSWORD</strong> - Password to authenticate the monitoring page.</p></li>
<li><p><strong>MONITOR_SESSION</strong> - Session timeout for the monitoring page.</p></li>
<li><p><strong>MAX_CONNECTIONS</strong> - Maximum number of monitoring sessions allowed in parallel.</p></li>
<li><p><strong>SERVICE_MANAGER</strong> - Service manager filepath to handle the service status requests.</p></li>
<li><p><strong>PROCESSES</strong> - List of process names to include in the monitor page.</p></li>
<li><p><strong>SERVICES</strong> - List of service names to include in the monitor page.</p></li>
<li><p><strong>GPU_LIB</strong> - GPU library filepath to use for monitoring.</p></li>
<li><p><strong>DISK_LIB</strong> - Disk library filepath to use for monitoring.</p></li>
<li><p><strong>SERVICE_LIB</strong> - Memory library filepath to use for monitoring.</p></li>
<li><p><strong>PROCESSOR_LIB</strong> - Processor library filepath to use for monitoring.</p></li>
<li><p><strong>DATABASE</strong> - FilePath to store the auth database that handles the authentication errors.</p></li>
<li><p><strong>RATE_LIMIT</strong> - List of dictionaries with <code class="docutils literal notranslate"><span class="pre">max_requests</span></code> and <code class="docutils literal notranslate"><span class="pre">seconds</span></code> to apply as rate limit.</p></li>
<li><p><strong>APIKEY</strong> - API Key for authentication.</p></li>
<li><p><strong>LOG_CONFIG</strong> - Logging configuration file path.</p></li>
</ul>
<p>⚠️ Enabling remote execution can be extremely risky and a major security threat.
So use <strong>caution</strong> and set the <strong>API_SECRET</strong> to a strong value.</p>
Expand Down
11 changes: 8 additions & 3 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,19 +53,24 @@ pyninja start
> _By default, `PyNinja` will look for a `.env` file in the current working directory._
</details>
- **APIKEY** - API Key for authentication.
- **NINJA_HOST** - Hostname for the API server.
- **NINJA_PORT** - Port number for the API server.
- **WORKERS** - Number of workers for the uvicorn server.
- **REMOTE_EXECUTION** - Boolean flag to enable remote execution.
- **API_SECRET** - Secret access key for running commands on server remotely.
- **MONITOR_USERNAME** - Username to authenticate the monitoring page.
- **MONITOR_PASSWORD** - Password to authenticate the monitoring page.
- **MONITOR_SESSION** - Session timeout for the monitoring page.
- **MAX_CONNECTIONS** - Maximum number of monitoring sessions allowed in parallel.
- **SERVICE_MANAGER** - Service manager filepath to handle the service status requests.
- **PROCESSES** - List of process names to include in the monitor page.
- **SERVICES** - List of service names to include in the monitor page.
- **GPU_LIB** - GPU library filepath to use for monitoring.
- **DISK_LIB** - Disk library filepath to use for monitoring.
- **SERVICE_LIB** - Memory library filepath to use for monitoring.
- **PROCESSOR_LIB** - Processor library filepath to use for monitoring.
- **DATABASE** - FilePath to store the auth database that handles the authentication errors.
- **RATE_LIMIT** - List of dictionaries with `max_requests` and `seconds` to apply as rate limit.
- **APIKEY** - API Key for authentication.
- **LOG_CONFIG** - Logging configuration file path.

⚠️ Enabling remote execution can be extremely risky and a major security threat.
So use **caution** and set the **API_SECRET** to a strong value.
Expand Down
11 changes: 8 additions & 3 deletions docs/_sources/README.md.txt
Original file line number Diff line number Diff line change
Expand Up @@ -53,19 +53,24 @@ pyninja start
> _By default, `PyNinja` will look for a `.env` file in the current working directory._
</details>

- **APIKEY** - API Key for authentication.
- **NINJA_HOST** - Hostname for the API server.
- **NINJA_PORT** - Port number for the API server.
- **WORKERS** - Number of workers for the uvicorn server.
- **REMOTE_EXECUTION** - Boolean flag to enable remote execution.
- **API_SECRET** - Secret access key for running commands on server remotely.
- **MONITOR_USERNAME** - Username to authenticate the monitoring page.
- **MONITOR_PASSWORD** - Password to authenticate the monitoring page.
- **MONITOR_SESSION** - Session timeout for the monitoring page.
- **MAX_CONNECTIONS** - Maximum number of monitoring sessions allowed in parallel.
- **SERVICE_MANAGER** - Service manager filepath to handle the service status requests.
- **PROCESSES** - List of process names to include in the monitor page.
- **SERVICES** - List of service names to include in the monitor page.
- **GPU_LIB** - GPU library filepath to use for monitoring.
- **DISK_LIB** - Disk library filepath to use for monitoring.
- **SERVICE_LIB** - Memory library filepath to use for monitoring.
- **PROCESSOR_LIB** - Processor library filepath to use for monitoring.
- **DATABASE** - FilePath to store the auth database that handles the authentication errors.
- **RATE_LIMIT** - List of dictionaries with `max_requests` and `seconds` to apply as rate limit.
- **APIKEY** - API Key for authentication.
- **LOG_CONFIG** - Logging configuration file path.

⚠️ Enabling remote execution can be extremely risky and a major security threat.
So use **caution** and set the **API_SECRET** to a strong value.
Expand Down
2 changes: 2 additions & 0 deletions docs/genindex.html
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,8 @@ <h2 id="L">L</h2>
<h2 id="M">M</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="index.html#pyninja.monitor.resources.map_docker_stats">map_docker_stats() (in module pyninja.monitor.resources)</a>
</li>
<li><a href="index.html#pyninja.models.EnvConfig.max_connections">max_connections (pyninja.models.EnvConfig attribute)</a>
</li>
<li><a href="index.html#pyninja.models.RateLimit.max_requests">max_requests (pyninja.models.RateLimit attribute)</a>
Expand Down
17 changes: 17 additions & 0 deletions docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -1884,6 +1884,23 @@ <h2>Authenticator<a class="headerlink" href="#id1" title="Permalink to this head
</section>
<section id="module-pyninja.monitor.resources">
<span id="resources"></span><h2>Resources<a class="headerlink" href="#module-pyninja.monitor.resources" title="Permalink to this heading"></a></h2>
<dl class="py function">
<dt class="sig sig-object py" id="pyninja.monitor.resources.map_docker_stats">
<span class="sig-prename descclassname"><span class="pre">pyninja.monitor.resources.</span></span><span class="sig-name descname"><span class="pre">map_docker_stats</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">json_data</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#pyninja.monitor.resources.map_docker_stats" title="Permalink to this definition"></a></dt>
<dd><p>Map the JSON data to a dictionary.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>json_data</strong> – JSON data from the docker stats command.</p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>Returns a dictionary with container stats.</p>
</dd>
<dt class="field-odd">Return type<span class="colon">:</span></dt>
<dd class="field-odd"><p>Dict[str, str]</p>
</dd>
</dl>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="pyninja.monitor.resources.get_cpu_percent">
<span class="sig-prename descclassname"><span class="pre">pyninja.monitor.resources.</span></span><span class="sig-name descname"><span class="pre">get_cpu_percent</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">cpu_interval</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#pyninja.monitor.resources.get_cpu_percent" title="Permalink to this definition"></a></dt>
Expand Down
Binary file modified docs/objects.inv
Binary file not shown.
2 changes: 1 addition & 1 deletion docs/searchindex.js

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions pyninja/main.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import logging
import os
import pathlib

import uvicorn
Expand Down Expand Up @@ -150,6 +151,7 @@ def start(**kwargs) -> None:
BASE_LOGGER.warning("Remote execution disabled")
# Conditional endpoint based on monitor_username and monitor_password
if all((models.env.monitor_username, models.env.monitor_password)):
models.env.processes.append(str(os.getpid()))
PyNinjaAPI.routes.extend(get_all_monitor_routes(dependencies))
PyNinjaAPI.add_exception_handler(
exc_class_or_status_code=exceptions.RedirectException,
Expand Down
25 changes: 22 additions & 3 deletions pyninja/monitor/resources.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
import json
import logging
import os
import shutil
from concurrent.futures import ThreadPoolExecutor
from typing import Dict, List

Expand All @@ -15,6 +14,27 @@
LOGGER = logging.getLogger("uvicorn.default")


def map_docker_stats(json_data: Dict[str, str]) -> Dict[str, str]:
"""Map the JSON data to a dictionary.
Args:
json_data: JSON data from the docker stats command.
Returns:
Dict[str, str]:
Returns a dictionary with container stats.
"""
return {
"Container ID": json_data.get("ID"),
"Container Name": json_data.get("Name"),
"CPU": json_data.get("CPUPerc"),
"Memory": json_data.get("MemPerc"),
"Memory Usage": json_data.get("MemUsage"),
"Block I/O": json_data.get("BlockIO"),
"Network I/O": json_data.get("NetIO"),
}


def get_cpu_percent(cpu_interval: int) -> List[float]:
"""Get CPU usage percentage.
Expand Down Expand Up @@ -45,7 +65,7 @@ async def get_docker_stats() -> List[Dict[str, str]]:
LOGGER.debug(stderr.decode().strip())
return []
return [
{key: value for key, value in json.loads(line).items() if key != "PIDs"}
map_docker_stats(json.loads(line))
for line in stdout.decode().strip().splitlines()
]

Expand All @@ -62,7 +82,6 @@ async def get_system_metrics() -> Dict[str, dict]:
return dict(
memory_info=psutil.virtual_memory()._asdict(),
swap_info=psutil.swap_memory()._asdict(),
disk_info=shutil.disk_usage("/")._asdict(),
load_averages=dict(m1=m1, m5=m5, m15=m15),
)

Expand Down
4 changes: 4 additions & 0 deletions pyninja/monitor/routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,10 @@ async def websocket_endpoint(websocket: WebSocket, session_token: str = Cookie(N
session_timestamp = models.ws_session.client_auth.get(websocket.client.host).get(
"timestamp"
)
# Base task with a placeholder asyncio sleep to start the task loop
task = asyncio.create_task(asyncio.sleep(0.1))
# Store disk usage information (during startup) to avoid repeated calls
disk_info = shutil.disk_usage("/")._asdict()
while True:
# Validate session asynchronously (non-blocking)
# This way of handling session validation is more efficient than using a blocking call
Expand Down Expand Up @@ -225,6 +228,7 @@ async def websocket_endpoint(websocket: WebSocket, session_token: str = Cookie(N
await websocket.close()
break
data = await resources.system_resources()
data["disk_info"] = disk_info
try:
await websocket.send_json(data)
except WebSocketDisconnect:
Expand Down
Loading

0 comments on commit 70c4fcb

Please sign in to comment.