Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Unify the async query request metadata model #3140

Open
ykmr1224 opened this issue Oct 29, 2024 · 0 comments
Open

[FEATURE] Unify the async query request metadata model #3140

ykmr1224 opened this issue Oct 29, 2024 · 0 comments
Labels
enhancement New feature or request untriaged

Comments

@ykmr1224
Copy link
Collaborator

(moved from opensearch-spark#602 since this issue is for SQL repository)

Is your feature request related to a problem?
Currently in async-query-core, we have three models which store the async query request and status: AsyncQueryJobMetadata, StatementModel, and IndexDMLResult. And those are stored in the same request index (.query_request_index_{datasourceName}) with different documentId (qid{requestId}, {requestId}, index{requestId}).
These three models overwraps each other and redundant. It also causes confusion.

What solution would you like?
Unify them to single model, and store single record in request index.

I propose following unified model to be used:

AsyncQueryMetadata

  • queryId (PK)
  • query // from StatementModel
  • langType // from StatementModel
  • datasourceName
  • accountId
  • jobType
  • submitTime
  • sessionId
  • indexName
  • state // from StatementModel/IndexDMLResult
  • error // from StatementModel/IndexDMLResult
  • queryRunTime
  • applicationId
  • jobId
  • version (= "2.0")
  • resultIndex
  • statementId (= queryId) // from StatementModel

How to make it backward compatible?

In new version, we will use single interface to query/save the query metadata.
The documentId will be queryId (without prefix).
Since current StatementModel uses queryId as a key, use case for StatementModel would be fine (whether it hits older record or newer record, it contains required fields for statement use case).
For the use case for AsyncQueryMetadata (It is only when GetAsyncQueryResults API is called), we can first query with queryId (without prefix), and if the version field is not "2.0", fallback to query with "qid" prefix, which would retrieve AsyncQueryJobMetadata(older) record.
IndexDMLResult does not have read use case. (does not have concern on backward compatibility)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request untriaged
Projects
None yet
Development

No branches or pull requests

1 participant