Scopes and shared data sources

[dadrus]

Regarding the title, I have a few questions:

I may be mistaken, but it seems to me that the implementation of Scope-based functionality is inconsistent and even flawed. While it does support tenant-specific repositories with policies - which is the primary reason I use Scopes - there are several inconsistencies. Let me explain this with an example configuration message used to set up a scope (called documents) via the /scopes endpoint of the OPAL Server:

{
  "scope_id": "documents",
  "policy": {
      # removed as it’s not relevant to the question  
  },
  "data": {
    "entries": [
      {
        "periodic_update_interval": 30,
        "url":"http://caddy:8080/users.json",
        "topics": ["documents:data:users"], # <- this is relevant 
        "dst_path":"/users"
      },
      {
        "periodic_update_interval": 30,
        "url":"http://caddy:8080/organizations.json",
        "topics": ["documents:data:organizations"], # <- this is relevant 
        "dst_path":"/organizations"
      },
      {
        "periodic_update_interval": 30,
        "url":"http://caddy:8080/departments.json",
        "topics": ["documents:data:departments"], # <- this is relevant 
        "dst_path":"/departments"
      },
      {
        "url": "postgresql://documents@db:5432/documents",
        "config": {
            "fetcher": "PostgresFetchProvider",
            "query": "select id::text, concat('ip:', ip) as parent_id from document;",
            "connection_params": {
                "password": "documents"
            },
            "fetch_key": "id"
        },
        "topics": ["documents:data:documents"], # <- this is relevant 
        "dst_path": "documents"
      }
    ]
  }
}

If I omit the scope name from the topics, it will be ignored by the Agent configured for the documents scope, even though the configuration is delivered. I would expect each topic to include only the name of the corresponding data source, like users, organizations, departments, and documents, without the documents:data: prefix.

However, when delivering a DataUpdate message via the server’s /scope/{scope_id}/data/update endpoint - in my case, scope_id would be documents- the DataUpdate message must include only the actual topic name (There is btw a bug related to that), excluding the documents:data: prefix. Here's an example:

{
  "id": "58bb77af-22a4-4327-be83-d44197f2c724",
  "entries": [
    {
      "url": "",
      "config": {},
      "topics": [ "documents" ], # <- this is relevant
      "dst_path": "documents",
      "save_method": "PATCH",
      "data": [
        {
          "op": "add",
          "path": "0cde79f8-7903-4ec0-931e-b7adf9fe3e4c",
          "value": {
            "parent_id": "department:1"
          }
        }
      ]
    }
  ],
  "reason": "New document created",
  "callback": {
    "callbacks": []
  }
}

Internally, the implementation updates this message by adding the documents:data: prefix to the topics names, making it documents:data:documents.

There is also an option to send the same message via the broadcast channel, wrapped in the BroadcastNotification message
defined by FastAPI. The envelope would look like this:

{
  "notifier_id": "someid",
  "topics": [ "documents:data:documents" ], # <- this is relevant
  "data": {
      # the document from above
  }
}

In both cases—whether using the /scope/{scope_id}/data/update endpoint or the BroadcastNotification - the sender must know the scope ID, as without it the message won’t be delivered. Alternatively, I could use the /data/update endpoint, but in that case, I would need to specify all the "scoped" topics in the configuration of each entry (Not tried that now, but according to the code, it should work).

All of this creates challenges when trying to use the same data sources across different Scope configurations, especially when updates are required. These updates must include the scopes for all relevant Agents, which makes the data sources dependent on the knowledge of which Agents exist and which scopes are configured for them. Ideally, the setup should function more like a topic for each data source, where the Agent’s configuration references the topic, rather than the other way around.

To provide more context about what I'm trying to achieve, here’s my use case: My tenants are the different teams within the same organization. They should be able to write and test their own policies, which may rely on services from other teams as data sources. Each of these services may emit update events, like the one I shared above. I want each team to be able to manage their policies within their own repositories, without needing a centralized one (as required without Scopes).

Currently, it seems that Scopes are intended to create fully isolated tenant setups. Is this the correct understanding? If so, is there a way to achieve my goal with OPAL, and if so, how?

Theoretically, if the /data/update endpoint can be used for the purposes mentioned above, I could create a custom component that listens for incoming update messages, uses the /scopes endpoint to iterate through the configured scopes, and updates the topics entries of the update messages accordingly. However, I’m curious if OPAL has a native solution already implemented to achieve this.

Thank you in advance.

[soumithbasina]

Hi, is there any progress regarding this? Thanks!