Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cat-gateway): Add simple signed_docs select query, update insert query. #1393

Merged
merged 49 commits into from
Jan 3, 2025
Merged
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
bc762dc
add upsert signed docs query
Mr-Leshiy Dec 12, 2024
8686038
add integration tests for signed docs query
Mr-Leshiy Dec 13, 2024
c966f61
wip
Mr-Leshiy Dec 13, 2024
40d039e
fix
Mr-Leshiy Dec 13, 2024
7e11ced
wip
Mr-Leshiy Dec 13, 2024
dfee564
wip
Mr-Leshiy Dec 13, 2024
0d2dc80
wip
Mr-Leshiy Dec 13, 2024
d14c428
remake query from upsert to insert
Mr-Leshiy Dec 13, 2024
c872de4
add simple select signed docs query
Mr-Leshiy Dec 13, 2024
ea95f13
Merge branch 'main' into feat/select-signed-docs
Mr-Leshiy Dec 13, 2024
facdcb5
cleanup jinja, add jinit initialization
Mr-Leshiy Dec 13, 2024
0c491b0
Merge branch 'main' into feat/select-signed-docs
Mr-Leshiy Dec 13, 2024
4844246
update tests
Mr-Leshiy Dec 13, 2024
7a4c3c2
fix
Mr-Leshiy Dec 13, 2024
2f83960
fix spelling
Mr-Leshiy Dec 13, 2024
a7587c7
add advanced select query
Mr-Leshiy Dec 13, 2024
6bdb60b
fix docs
Mr-Leshiy Dec 14, 2024
d0fd295
add QueryLimits
Mr-Leshiy Dec 16, 2024
c52828e
wip
Mr-Leshiy Dec 16, 2024
d866f71
wip
Mr-Leshiy Dec 16, 2024
9fc43b4
update tests
Mr-Leshiy Dec 16, 2024
b088b06
fix docs
Mr-Leshiy Dec 16, 2024
3f3eccf
fix cardano-chain-follower dep, remove init of jinja
Mr-Leshiy Dec 18, 2024
e1181dd
refactor QueryLimits struct
Mr-Leshiy Dec 18, 2024
c6bb226
revert select_signed_docs query
Mr-Leshiy Dec 18, 2024
d26b843
Merge branch 'main' into feat/select-signed-docs
Mr-Leshiy Dec 18, 2024
5315e59
refactor QueryLimits
Mr-Leshiy Dec 18, 2024
eaa0e36
wip
Mr-Leshiy Dec 18, 2024
e14fe96
rename QueryLimits methods
Mr-Leshiy Dec 19, 2024
b46043a
replace `new` with TryFrom impl
Mr-Leshiy Dec 19, 2024
1dbdd50
wip
Mr-Leshiy Dec 19, 2024
7e09a23
wip
Mr-Leshiy Dec 19, 2024
780c7da
wip
Mr-Leshiy Dec 19, 2024
e0e78ac
refactor
Mr-Leshiy Dec 19, 2024
6904aae
fix fmt
Mr-Leshiy Dec 19, 2024
0b15dd0
wip
Mr-Leshiy Dec 19, 2024
38d1606
move docs query filters into different mod
Mr-Leshiy Dec 19, 2024
b787ae0
refactor `filtered_select_signed_docs` function
Mr-Leshiy Dec 20, 2024
eb1441d
wip
Mr-Leshiy Dec 20, 2024
5e40d91
Merge branch 'main' into feat/select-signed-docs
Mr-Leshiy Dec 20, 2024
8ce23c7
replace query_stmt with trait Display impl
Mr-Leshiy Jan 2, 2025
8c076e7
rename methods
Mr-Leshiy Jan 2, 2025
d97601e
add query_stream fn
Mr-Leshiy Jan 2, 2025
74e2962
fix clippy
Mr-Leshiy Jan 2, 2025
f35f850
Merge branch 'main' into feat/select-signed-docs
Mr-Leshiy Jan 2, 2025
1074e01
fix test
Mr-Leshiy Jan 2, 2025
8af1cef
get rid of usage of `pin!`
Mr-Leshiy Jan 2, 2025
febece0
Merge branch 'main' into feat/select-signed-docs
stevenj Jan 3, 2025
246f4f1
Merge branch 'main' into feat/select-signed-docs
stevenj Jan 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .config/dictionaries/project.dic
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ metamap
mgrybyk
miniaturizable
minicbor
minijinja
mithril
mitigations
mocktail
Expand Down
3 changes: 2 additions & 1 deletion catalyst-gateway/bin/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ repository.workspace = true
workspace = true

[dependencies]
cardano-chain-follower = { version = "0.0.5", git = "https://github.com/input-output-hk/catalyst-libs.git", tag = "v0.0.9" }
cardano-chain-follower = { version = "0.0.6", git = "https://github.com/input-output-hk/catalyst-libs.git", tag = "v0.0.10" }
c509-certificate = { version = "0.0.3", git = "https://github.com/input-output-hk/catalyst-libs.git", tag = "v0.0.3" }
rbac-registration = { version = "0.0.2", git = "https://github.com/input-output-hk/catalyst-libs.git", tag = "v0.0.8" }

Expand Down Expand Up @@ -95,6 +95,7 @@ jsonschema = "0.26.1"
bech32 = "0.11.0"
const_format = "0.2.33"
regex = "1.11.1"
minijinja = "2.5.0"

[dev-dependencies]
proptest = "1.5.0"
Expand Down
3 changes: 3 additions & 0 deletions catalyst-gateway/bin/src/db/event/common/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
//! Reusable common database objects

pub(crate) mod query_limits;
57 changes: 57 additions & 0 deletions catalyst-gateway/bin/src/db/event/common/query_limits.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
//! `QueryLimits` query argument object.

#![allow(dead_code)]

use crate::service::common::types::generic::query::pagination::{Limit, Page};

/// A query limits struct.
pub(crate) struct QueryLimits(QueryLimitsInner);

/// `QueryLimits` inner enum representation.
enum QueryLimitsInner {
/// Return all entries without any `LIMIT` and `OFFSET` parameters
All,
/// Specifies `LIMIT` parameter
Limit(u64),
/// Specifies `LIMIT` and `OFFSET` parameters
LimitAndOffset(u64, u64),
}

impl QueryLimits {
/// Create a `QueryLimits` object without the any limits.
pub(crate) const ALL: QueryLimits = Self(QueryLimitsInner::All);
/// Create a `QueryLimits` object with the limit equals to `1`.
pub(crate) const ONE: QueryLimits = Self(QueryLimitsInner::Limit(1));

/// Create a `QueryLimits` object from the service `Limit` and `Page` values.
///
/// # Errors
/// - Invalid `limit` value, must be more than `0`.
/// - Invalid arguments, `limit` must be provided when `page` is not None.
pub(crate) fn new(limit: Option<Limit>, page: Option<Page>) -> anyhow::Result<Self> {
match (limit, page) {
(Some(limit), Some(page)) => {
Ok(Self(QueryLimitsInner::LimitAndOffset(
limit.into(),
page.into(),
)))
},
(Some(limit), None) => Ok(Self(QueryLimitsInner::Limit(limit.into()))),
(None, None) => Ok(Self(QueryLimitsInner::All)),
(None, Some(_)) => {
anyhow::bail!("Invalid arguments, `limit` must be provided when `page` is not None")
},
}
}

/// Returns a string with the corresponding query limit statement
pub(crate) fn query_stmt(&self) -> String {
Mr-Leshiy marked this conversation as resolved.
Show resolved Hide resolved
match self.0 {
QueryLimitsInner::All => String::new(),
QueryLimitsInner::Limit(limit) => format!("LIMIT {limit}"),
QueryLimitsInner::LimitAndOffset(limit, offset) => {
format!("LIMIT {limit} OFFSET {offset}")
},
}
}
}
7 changes: 6 additions & 1 deletion catalyst-gateway/bin/src/db/event/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,13 @@ use std::{

use bb8::Pool;
use bb8_postgres::PostgresConnectionManager;
use error::NotFoundError;
use tokio_postgres::{types::ToSql, NoTls, Row};
use tracing::{debug, debug_span, error, Instrument};

use crate::settings::Settings;

pub(crate) mod common;
pub(crate) mod config;
pub(crate) mod error;
pub(crate) mod legacy;
Expand Down Expand Up @@ -104,7 +106,10 @@ impl EventDB {
}
let pool = EVENT_DB_POOL.get().ok_or(Error::DbPoolUninitialized)?;
let conn = pool.get().await?;
let row = conn.query_one(stmt, params).await?;
let row = conn
.query_opt(stmt, params)
.await?
.ok_or::<anyhow::Error>(NotFoundError.into())?;
Ok(row)
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
//! `FullSignedDoc` struct implementation.

use super::SignedDocBody;

/// Full signed doc event db struct
#[derive(Debug, Clone, PartialEq)]
pub(crate) struct FullSignedDoc {
/// Signed doc body
body: SignedDocBody,
/// `signed_doc` table `payload` field
payload: Option<serde_json::Value>,
/// `signed_doc` table `raw` field
raw: Vec<u8>,
}

impl FullSignedDoc {
/// Creates a `FullSignedDoc` instance.
#[allow(dead_code)]
pub(crate) fn new(
body: SignedDocBody, payload: Option<serde_json::Value>, raw: Vec<u8>,
) -> Self {
Self { body, payload, raw }
}

/// Returns the document id.
pub(crate) fn id(&self) -> &uuid::Uuid {
self.body.id()
}

/// Returns the document version.
pub(crate) fn ver(&self) -> &uuid::Uuid {
self.body.ver()
}

/// Returns the document author.
#[allow(dead_code)]
pub(crate) fn author(&self) -> &String {
self.body.author()
}

/// Returns the `SignedDocBody`.
#[allow(dead_code)]
pub(crate) fn body(&self) -> &SignedDocBody {
&self.body
}

/// Returns all signed document fields for the event db queries
pub(crate) fn postgres_db_fields(&self) -> [&(dyn tokio_postgres::types::ToSql + Sync); 7] {
let body_fields = self.body.postgres_db_fields();
[
body_fields[0],
body_fields[1],
body_fields[2],
body_fields[3],
body_fields[4],
&self.payload,
&self.raw,
]
}

/// Creates a `FullSignedDoc` from postgresql row object.
pub(crate) fn from_row(
id: &uuid::Uuid, ver: Option<&uuid::Uuid>, row: &tokio_postgres::Row,
) -> anyhow::Result<Self> {
let ver = if let Some(ver) = ver {
*ver
} else {
row.try_get("ver")?
};

Ok(FullSignedDoc {
body: SignedDocBody::new(
*id,
ver,
row.try_get("type")?,
row.try_get("author")?,
row.try_get("metadata")?,
),
payload: row.try_get("payload")?,
raw: row.try_get("raw")?,
})
}
}
118 changes: 108 additions & 10 deletions catalyst-gateway/bin/src/db/event/signed_docs/mod.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,32 @@
//! Signed docs queries

mod full_signed_doc;
mod signed_doc_body;
#[cfg(test)]
mod tests;

use super::EventDB;
pub(crate) use full_signed_doc::FullSignedDoc;
pub(crate) use signed_doc_body::SignedDocBody;

use super::{common::query_limits::QueryLimits, EventDB, NotFoundError};
use crate::jinja::{get_template, JinjaTemplateSource};

/// Insert sql query
const INSERT_SIGNED_DOCS: &str = include_str!("./sql/insert_signed_documents.sql");

/// Make an insert query into the `event-db` by adding data into the `signed_docs` table
/// Select sql query jinja template
pub(crate) const SELECT_SIGNED_DOCS_TEMPLATE: JinjaTemplateSource = JinjaTemplateSource {
stevenj marked this conversation as resolved.
Show resolved Hide resolved
name: "select_signed_documents.jinja.template",
source: include_str!("./sql/select_signed_documents.sql.jinja"),
};

/// Filtered select sql query jinja template
pub(crate) const FILTERED_SELECT_SIGNED_DOCS_TEMPLATE: JinjaTemplateSource = JinjaTemplateSource {
name: "filtered_select_signed_documents.jinja.template",
source: include_str!("./sql/filtered_select_signed_documents.sql.jinja"),
};

/// Make an insert query into the `event-db` by adding data into the `signed_docs` table.
///
/// * IF the record primary key (id,ver) does not exist, then add the new record. Return
/// success.
Expand All @@ -20,13 +38,93 @@ const INSERT_SIGNED_DOCS: &str = include_str!("./sql/insert_signed_documents.sql
/// - `ver` is a UUID v7
/// - `doc_type` is a UUID v4
#[allow(dead_code)]
pub(crate) async fn insert_signed_docs(
id: &uuid::Uuid, ver: &uuid::Uuid, doc_type: &uuid::Uuid, author: &String,
metadata: &Option<serde_json::Value>, payload: &Option<serde_json::Value>, raw: &Vec<u8>,
) -> anyhow::Result<()> {
EventDB::modify(INSERT_SIGNED_DOCS, &[
id, ver, doc_type, author, metadata, payload, raw,
])
.await?;
pub(crate) async fn insert_signed_docs(doc: &FullSignedDoc) -> anyhow::Result<()> {
Mr-Leshiy marked this conversation as resolved.
Show resolved Hide resolved
match select_signed_docs(doc.id(), Some(doc.ver())).await {
Ok(res_doc) => {
anyhow::ensure!(
&res_doc == doc,
"Document with the same `id` and `ver` already exists"
);
return Ok(());
},
Err(err) if err.is::<NotFoundError>() => {},
Err(err) => return Err(err),
}

EventDB::modify(INSERT_SIGNED_DOCS, &doc.postgres_db_fields()).await?;
Ok(())
}

/// Make a select query into the `event-db` by getting data from the `signed_docs` table.
///
/// * This returns a single document. All data from the document is returned, including
/// the `payload` and `raw` fields.
/// * `ver` should be able to be optional, in which case get the latest ver of the given
/// `id`.
///
/// # Arguments:
/// - `id` is a UUID v7
/// - `ver` is a UUID v7
pub(crate) async fn select_signed_docs(
Mr-Leshiy marked this conversation as resolved.
Show resolved Hide resolved
id: &uuid::Uuid, ver: Option<&uuid::Uuid>,
) -> anyhow::Result<FullSignedDoc> {
let query_template = get_template(&SELECT_SIGNED_DOCS_TEMPLATE)?;
let query = query_template.render(serde_json::json!({
"id": id,
"ver": ver,
}))?;
let row = EventDB::query_one(&query, &[]).await?;

FullSignedDoc::from_row(id, ver, &row)
}

/// A `select_signed_docs` query filtering argument.
Mr-Leshiy marked this conversation as resolved.
Show resolved Hide resolved
#[allow(dead_code)]
pub(crate) enum DocsQueryFilter {
/// All entries
All,
/// Select docs with the specific `type` field
DocType(uuid::Uuid),
/// Select docs with the specific `id` field
DocId(uuid::Uuid),
/// Select docs with the specific `id` and `ver` field
DocVer(uuid::Uuid, uuid::Uuid),
/// Select docs with the specific `author` field
Author(String),
}

impl DocsQueryFilter {
/// Returns a string with the corresponding query docs filter statement
pub(crate) fn query_stmt(&self) -> String {
match self {
Self::All => "TRUE".to_string(),
Self::DocType(doc_type) => format!("signed_docs.type = '{doc_type}'"),
Self::DocId(id) => format!("signed_docs.id = '{id}'"),
Self::DocVer(id, ver) => {
format!("signed_docs.id = '{id}' AND signed_docs.ver = '{ver}'")
},
Self::Author(author) => format!("signed_docs.author = '{author}'"),
}
}
}

/// Make an filtered select query into the `event-db` by getting data from the
/// `signed_docs` table.
#[allow(dead_code)]
pub(crate) async fn filtered_select_signed_docs(
Mr-Leshiy marked this conversation as resolved.
Show resolved Hide resolved
conditions: &DocsQueryFilter, query_limits: &QueryLimits,
) -> anyhow::Result<Vec<SignedDocBody>> {
let query_template = get_template(&FILTERED_SELECT_SIGNED_DOCS_TEMPLATE)?;
let query = query_template.render(serde_json::json!({
"conditions": conditions.query_stmt(),
"query_limits": query_limits.query_stmt(),
}))?;
let rows = EventDB::query(&query, &[]).await?;

let docs = rows
.iter()
.map(SignedDocBody::from_row)
.collect::<anyhow::Result<_>>()?;

Ok(docs)
}
Loading
Loading