Release v0.1.1 (#253)

* audio upload extension with gdrive credentials

* FIX: API parameters

* UPDATE: tmp files cleanup n code refactored as per review

* REFACTOR: minor refactor as per review

* REFACTOR: minor update as per review

* UPDATE: gdrive sync logic

* REFACTOR: code update as per gdrive and update credential client

* REFACTOR: validation updated - as per review from CR

* UPDATE: code has been refactore for UUID for diffrent audio upload sources

* REFACTOR: updated code as per review

* Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs.

* Update test script to use docker-compose-test.yml for all test-related operations

* Added standard MIT license

* Fix/cleanup model (#219)

* refactor memory

* add config

* docstring

* more cleanup

* code quality

* code quality

* unused return

* DOTTED GET

* Refactor Docker and CI configurations

- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.

* Update configuration files for model providers and Docker setup

- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.

* Update transcription job handling to format speaker IDs

- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.

* Remove loading of backend .env file in test environment setup

- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.

* Enhance configuration management and setup wizard

- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.

* Enhance HTTPS configuration in setup wizard

- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.

* Add source parameter to audio file writing in websocket controller

- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.

---------

Co-authored-by: 0xrushi <[email protected]>

* fix/broken-tests (#230)

* refactor memory

* add config

* docstring

* more cleanup

* code quality

* code quality

* unused return

* DOTTED GET

* Refactor Docker and CI configurations

- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.

* Update configuration files for model providers and Docker setup

- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.

* Update transcription job handling to format speaker IDs

- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.

* Remove loading of backend .env file in test environment setup

- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.

* Enhance configuration management and setup wizard

- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.

* Enhance HTTPS configuration in setup wizard

- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.

* Add source parameter to audio file writing in websocket controller

- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.

* Refactor error handling in system controller and update memory config routes

- Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions.
- Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes.
- Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation.

---------

Co-authored-by: 0xrushi <[email protected]>

* Feat/add obsidian 3 (#233)

* obsidian support

* neo4j comment

* cleanup code

* unused line

* unused line

* Fix MemoryEntry object usage in chat service

* comment

* feat(obsidian): add obsidian memory search integration to chat

* unit test

* use rq

* neo4j service

* typefix

* test fix

* cleanup

* cleanup

* version changes

* profile

* remove unused imports

* Refactor memory configuration validation endpoints

- Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation.
- Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input.
- Adjusted the API call in the frontend to point to the new validation endpoint.

* Refactor health check model configuration loading

- Updated the health check function to load model configuration from the models registry instead of the root config.
- Improved error handling by logging warnings when model configuration loading fails.

---------

Co-authored-by: 0xrushi <[email protected]>

* Update .gitignore to exclude all files in app/ios and app/android directories (#238)

* fix: Copy full source code in speaker-recognition Dockerfile (#243)

Adds COPY src/ src/ step after dependency installation to ensure
all source files are available in the Docker image. This improves
build caching while ensuring complete source code is present.

* Enhance configuration management and add new setup scripts (#235)

* Enhance configuration management and add new setup scripts

- Updated .gitignore to include config.yml and its template.
- Added config.yml.template for default configuration settings.
- Introduced restart.sh script for service management.
- Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration.
- Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist.

* Refactor transcription providers and enhance configuration management

- Updated Docker Compose files to include the new Neo4j service configuration.
- Added support for Obsidian/Neo4j integration in the setup process.
- Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet.
- Enhanced error handling and logging in transcription processes.
- Improved environment variable management in test scripts to prioritize command-line overrides.
- Removed deprecated Parakeet provider implementation and streamlined audio stream workers.

* Update configuration management and enhance file structure, add test-matrix (#237)

* Update configuration management and enhance file structure

- Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template.
- Modified service scripts to load the new configuration path for `config.yml`.
- Enhanced `.gitignore` to include the new configuration files and templates.
- Updated documentation to reflect changes in configuration file locations and usage.
- Improved setup scripts to ensure proper creation and management of configuration files.
- Added new test configurations for various provider combinations to streamline testing processes.

* Add test requirements and clean up imports in wizard.py

- Introduced a new `test-requirements.txt` file to manage testing dependencies.
- Removed redundant import of `shutil` in `wizard.py` to improve code clarity.

* Add ConfigManager for unified configuration management

- Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility.
- Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process.
- Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`.
- Enhanced user feedback during configuration updates, including success messages for changes made to configuration files.

* Refactor transcription provider configuration and enhance setup process

- Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral.
- Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys.
- Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`.
- Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables.
- Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`.
- Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance.

* Enhance ConfigManager with deep merge functionality

- Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly.
- Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities.

* Refactor run-test.sh and enhance memory extraction tests

- Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process.
- Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility.
- Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling.
- Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling.
- Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management.

* unify tests to robot test, add some more clean up

* Update health check configuration in docker-compose-test.yml (#241)

- Increased the number of retries from 5 to 10 for improved resilience during service readiness checks.
- Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence.

* Add step to create test configuration file in robot-tests.yml

- Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml.
- Added logging to confirm the creation of the test config file, improving visibility during the test setup process.

* remove cache step since not required

* coderabbit comments

* Refactor ConfigManager error handling for configuration file loading

- Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance.
- Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files.

* Refactor _find_repo_root method in ConfigManager

- Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability.
- Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root.

* Enhance speaker recognition service integration and error handling (#245)

* Enhance speaker recognition service integration and error handling

- Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes.
- Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root.
- Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues.
- Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively.
- Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image.

* Remove integration tests workflow and enhance robot tests with HF_TOKEN verification

- Deleted the `integration-tests.yml` workflow file to streamline CI processes.
- Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests.

* Fix key access in system admin tests to use string indexing for speakers data

* Refactor Robot Framework tests and enhance error handling in memory services

- Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup.
- Updated the Robot Framework tests to utilize a unified test script for improved consistency.
- Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers.
- Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding.
- Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity.

* Refactor test container cleanup in run-robot-tests.sh

- Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values.
- Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable.

* Enhance run-robot-tests.sh for improved logging and cleanup

- Set absolute paths for consistent directory references to simplify navigation.
- Capture container logs, status, and resource usage for better debugging.
- Refactor cleanup process to utilize dynamic backend directory references, improving maintainability.
- Ensure proper navigation back to the tests directory after operations.

* Add speaker recognition configuration and update test script defaults

- Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration.
- Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance.
- Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed.

* Refactor speaker recognition configuration management

- Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance.
- Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling.
- Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration.

* Add minimum worker count verification to infrastructure tests

- Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks.
- Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability.
- Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers.

* Update configuration management and enhance model handling

- Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling.
- Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading.
- Refactored imports in deepgram.py to improve clarity and reduce redundancy.

* Refactor configuration management in wizard and ChronicleSetup (#246)

* Refactor configuration management in wizard and ChronicleSetup

- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.

* Fix string formatting for error message in ChronicleSetup

* added JWT issuers for audience auth for service interop and shared us… (#250)

* added JWT issuers for audience auth for service interop and shared user accounts

* amended default value in line wioth code

---------

Co-authored-by: 01PrathamS <[email protected]>
Co-authored-by: Stu Alexandere <[email protected]>
Co-authored-by: Stuart Alexander <[email protected]>
Co-authored-by: 0xrushi <[email protected]>

Author: 4 people

.env.template

@@ -55,6 +55,10 @@ SPEAKER_SERVICE_URL=http://${DOMAIN}:${SPEAKER_PORT}
 # JWT secret key - make this random and long
 AUTH_SECRET_KEY=your-super-secret-jwt-key-here-make-it-random-and-long
 
+# JWT-token issuer ACCEPTED_ISSUERS can be a comma-separated list of accepted issuers
+# defaults to 'chronicle,ushadow' if not set
+# ACCEPTED_ISSUERS=chronicle,ushadow
+
 # Admin account
 ADMIN_EMAIL=[email protected]
 ADMIN_PASSWORD=secure-admin-password

backends/advanced/src/advanced_omi_backend/auth.py

@@ -50,6 +50,14 @@ def _verify_configured(var_name: str, *, optional: bool = False) -> Optional[str
 ADMIN_PASSWORD = _verify_configured("ADMIN_PASSWORD")
 ADMIN_EMAIL = _verify_configured("ADMIN_EMAIL", optional=True) or "[email protected]"
 
+# Accepted token issuers - comma-separated list of services whose tokens we accept
+# Default: "chronicle,ushadow" (accept tokens from both chronicle and ushadow)
+ACCEPTED_ISSUERS = [
+    iss.strip() 
+    for iss in os.getenv("ACCEPTED_TOKEN_ISSUERS", "chronicle,ushadow").split(",") 
+    if iss.strip()
+]
+logger.info(f"Accepting tokens from issuers: {ACCEPTED_ISSUERS}")
 
 class UserManager(BaseUserManager[User, PydanticObjectId]):
     """User manager with minimal customization for fastapi-users."""
@@ -100,7 +108,8 @@ async def get_user_manager(user_db=Depends(get_user_db)):
 def get_jwt_strategy() -> JWTStrategy:
     """Get JWT strategy for token generation and validation."""
     return JWTStrategy(
-        secret=SECRET_KEY, lifetime_seconds=JWT_LIFETIME_SECONDS
+        secret=SECRET_KEY, lifetime_seconds=JWT_LIFETIME_SECONDS,
+        token_audience=["fastapi-users:auth"] + ACCEPTED_ISSUERS
     )