Brainstorming / design discussion: introducing a File Access layer between application logic and storage backendsSuggestions for New Features

[6leonardo]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I searched existing Suggestions and Feature Requests and did not find any discussion about centralizing file access with a context-aware boundary or architectural refactor of file handling. The current suggestions focus on specific app features or improvements, but not on core file access abstractions.

While digging into the codebase to understand how file access is currently handled, I did a focused analysis of the main upload/download flows, controllers, services, and pipelines.

This analysis confirmed that file access today is mostly performed directly by application services and controllers, without a dedicated intermediate layer between business logic and the storage backend.

I’m opening this as a design/brainstorming discussion rather than a feature request.

---

What the current code looks like

From the analysis, file access is currently performed in several different ways:

• Controllers and services rely directly on FileService, ToolFileManager, and DatasourceFileManager
• In some places, storage is accessed directly (e.g. storage.load(...) in file_preview.py)
• File metadata tables such as UploadFile and ToolFile are queried directly by application code to retrieve storage keys
• Ownership and tenant assumptions are enforced implicitly and inconsistently

For example:

• files.py delegates most logic to FileService, assuming contextual checks have already happened
• In file_preview.py, user/app/tenant checks are performed locally in the controller, rather than being centralized
• Upload and download endpoints span multiple entry points (console, web, service API), as well as RAG pipelines and async tasks

This makes it hard to have a single place where file access can be audited, validated, or reasoned about.

---

Observed issues

Based on the current structure:

• File access logic is spread across heterogeneous components
• Tenant and ownership assumptions are enforced in different places
• There is no single “file access boundary” above the storage backend
• Introducing cross-cutting concerns (auditing, validation, future policy) would require touching many call sites

This is not a functional bug, but a structural limitation.

---

Proposed direction (for discussion)

The idea is to introduce a thin File Access layer between application logic and the existing storage backends:

Application layer
  (controllers, services, pipelines, tasks)
        ↓
File Access layer
  - loads file metadata (file ↔ storage mapping)
  - receives explicit access context
    (tenant, user, purpose)
  - enforces tenant ownership
  - delegates I/O to the existing storage backend
        ↓
Storage backends
  (local FS, S3, remote providers)

Important clarifications:

• This layer would sit on top of the existing storage backend abstraction, not replace it
• Existing metadata tables (UploadFile, ToolFile, etc.) would remain unchanged
• The goal is to centralize how files are accessed, not to redesign storage

---

Access context and security

From the current code, tenant isolation already appears as a clear and consistent boundary, so enforcing tenant ownership in the File Access layer from the beginning seems reasonable.

User- and role-level decisions, instead, are currently deeply tied to application logic and specific use cases (apps, tools, workflows), and there is no dedicated file-level security table today.

For this reason, an incremental approach would make sense:

• First iteration

  • wrap read/write operations
  • enforce tenant ownership centrally
  • propagate user / purpose as contextual metadata only
  • keep existing role/app logic unchanged

• Later iterations (optional)

  • auditing
  • more explicit file-level policies
  • tighter integration with security concerns if needed

---

Incremental implementation approach

This does not need to be a big-bang refactor.

A possible path could be:

1. Define a minimal File Access interface (plus adapters)
2. Apply it to a restricted flow (e.g. console upload/download) as an MVP
3. Gradually expand coverage once the pattern is validated

One important aspect to consider early is how access context can be propagated in areas where it is not immediately available (async tasks, plugins, background jobs), possibly falling back to a “best effort” or system context to avoid regressions.

2. Additional context or comments

This is meant as a discussion to validate whether:

• a clearer file access boundary would be useful for the project
• the proposed direction aligns with long-term maintainability
• an incremental refactor of this kind would be acceptable upstream

If the direction makes sense, I’d be happy to help with a small initial PR as a proof of concept.

3. Can you help us with this feature?

• I am interested in contributing to this feature.