feat(eventdb): Update the event DB to use the latest postgres builder…

…s from CAT-CI and integrate DB docs. (#189)

* refactor(gateway): Rename catalyst-voices-backend to the catalyst-gateway

* docs(gateway): Update readme

* refactor(gateway): Move the code into the new directory trees, cleanup inside tree still todo.

* refactor(gateway): Basic refactor of the event-db config/data and the db interface logic

* docs(gateway): Initial docs for the crates directory inside catalyst gateway

* refactor(gateway): Move code out of cat-data-service and put it directly in bin

* feat(gateway): first build seems to work, but fails due to WIP refactor

* refactor(gateway): Refactor builds properly to a static executable.

* refactor(gateway): autofixed lints

* refactor(gateway): WIP remove lint errors

* refactor(gateway): WIP refactoring work to remove lint errors

* refactor(gateway): Don't panic, return the error, and generalize the error itself.

* refactor(gateway): generalize error response

* docs(gateway): indexing ok here, slice already checked to have values

* refactor(gateway): Indexing is safe here, so note it.

* refactor(gateway): don't panic, return an error.  reduce size of function by splitting into multiple logical units.

* refactor(gateway): Purge metrics from legacy service.  Unused and can panic.

* refactor(gateway): remove unused rust files from EventDB directory

* refactor(gateway): remove unused dependencies

* docs(gateway): Allow intentional panic for test code

* refactor(gateway): eliminate or notate why unwraps/expect are safe

* refactor(gateway): All lints are clean and restrictively applied

* refactor(gateway): remove unused dependencies from workspace cargo file

* refactor(gateway): restrict visibility of event_db to pub(crate) as its fully internalized now.

* docs(gateway): root level fully documented

* docs(gateway): state fully documented

* docs(gateway): Don't worry about full docs for legacy service.

* docs(gateway): Service top level fully documented

* docs(gateway): middleware docs complete

* docs(gateway): Fully documented the docs part of the service

* docs(gateway): common responses and module root fully documented

* docs(gateway): Common objects now fully documented

* docs(gateway): Api endpoint docs are complete

* docs(gateway): DB Types fully documented

* docs(gateway): Queries partially documented

* docs(gateway): Everything that needs docs is documented

* feat(gateway): Initial CI and Cat-Gateway build ready for PR

* feat(rust): Add recommended rust extensions for vscode, and rust settings.

* docs(gateway): Fix spelling

* ci(gateway): Use the proper versioned CI tooling from catalyst-ci

* feat(docs): Docs WIP setup

* docs(docs): Fix the docs so they use the latests docs-ci and are able to properly include data from build targets

* docs(docs): fix spelling and markdown lint issues

* docs(docs): Update to use latest Cat-CI Docs and reduce replication

* docs(docs): Fix the top level parts of the docs and remove test values

* docs(docs): try and build and deploy docs to gh-pages

* docs(docs): fix spelling issue

* docs(docs): Use proper docs action, and also add branch cleanup for branch docs

* feat(backend): Start work redefining the event db for catalyst voices.

* feat(gateway): WIP catalyst-voices database respec.

* feat(gateway): Gateway DB defined up to `event` tables. WIP

* feat(gateway): Event DB respec WIP.  Registration tables work.

* feat(gateway): CIP15/36 Registration and staked ADA DB schemas Done but untested.

* fix(gateway): add back objectives tales for now.

* feat(gateway): Event DB passes CI lints, but still WIP.

* refactor(event-db): WIP alignment with updated cat-ci postgres builders

* fix(event-db): lint checks now pass

* feat(event-db): WIP building with latest postgresql builder

* docs(event-db): Update DB Docs and Registration Schema

* fix(event-db): Fix spelling and missing sub-dir issue for ci

* fix(event-db): remove obsolete unit test for event-db

.config/dictionaries/project.dic

@@ -77,6 +77,7 @@ rxdart
 saibatizoku
 seckey
 slotno
+sqlfluff
 stevenj
 subosito
 tacho

.sqlfluff

@@ -0,0 +1,24 @@
+# cspell: words capitalisation
+
+[sqlfluff]
+dialect = postgres
+large_file_skip_char_limit = 0
+max_line_length = 120
+
+[sqlfluff:indentation]
+tab_space_size = 2
+
+[sqlfluff:rules:layout.long_lines]
+ignore_comment_lines = True
+ignore_comment_clauses = True
+
+[sqlfluff:rules:capitalisation.keywords]
+capitalisation_policy = upper
+[sqlfluff:rules:capitalisation.identifiers]
+extended_capitalisation_policy = lower
+[sqlfluff:rules:capitalisation.functions]
+extended_capitalisation_policy = upper
+[sqlfluff:rules:capitalisation.literals]
+extended_capitalisation_policy = upper
+[sqlfluff:rules:capitalisation.types]
+extended_capitalisation_policy = upper

Earthfile

@@ -31,3 +31,12 @@ repo-docs:
     COPY --dir *.md LICENSE-APACHE LICENSE-MIT .
 
     SAVE ARTIFACT /repo repo
+
+repo-config-2:
+    # Create artifacts of config file we need to refer to in builders.
+    FROM scratch
+
+    WORKDIR /repo
+    COPY --dir .sqlfluff .
+
+    SAVE ARTIFACT /repo repo

catalyst-gateway/event-db/.sqlfluffignore

@@ -0,0 +1,4 @@
+# Ignore generated historic data files
+
+fund_*.sql
+setup-db.sql

catalyst-gateway/event-db/Earthfile

@@ -6,87 +6,56 @@ VERSION 0.7
 
 # cspell: words
 
-# Event db builder target. Prepares all necessary artifacts.
-# Arguments:
-#  * with_historic_data: add historic data from the `historic_data` dir (default `true`).
-#  * with_test_data: add test data from the `test_data` dir (default `true`).
+# Internal: builder is our Event db builder target. Prepares all necessary artifacts.
+#   CI target : dependency
 builder:
-    FROM github.com/input-output-hk/catalyst-ci/earthly/postgresql:v2.0.7+postgres-base
+    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:fix/postgres-builders+BUILDER \
+        --sqlfluff_cfg=./../../+repo-config/repo/.sqlfluff
 
-    ARG with_historic_data=true
-    ARG with_test_data=true
-
-    WORKDIR /build
-
-    COPY refinery.toml .
-    COPY --dir migrations .
-    RUN mkdir data
-
-    IF [ $with_historic_data = true ]
-        COPY --dir historic_data .
-
-        RUN python3 historic_data/fund_2/mk_fund2_sql.py historic_data/fund_2/fund2_database_encrypted.sqlite3 >> historic_data/fund_2.sql
-        RUN python3 historic_data/fund_3/mk_fund3_sql.py historic_data/fund_3/fund3_database_encrypted.sqlite3 >> historic_data/fund_3.sql
-        RUN python3 historic_data/fund_4/mk_fund4_sql.py historic_data/fund_4/fund4_database_encrypted.sqlite3 >> historic_data/fund_4.sql
-        RUN python3 historic_data/fund_5/mk_fund5_sql.py historic_data/fund_5/fund5_database_encrypted.sqlite3 >> historic_data/fund_5.sql
-        RUN python3 historic_data/fund_6/mk_fund6_sql.py historic_data/fund_6/fund6_database_encrypted.sqlite3 >> historic_data/fund_6.sql
-        RUN python3 historic_data/fund_7/mk_fund7_sql.py historic_data/fund_7/fund7_database_encrypted.sqlite3 >> historic_data/fund_7.sql
-        RUN python3 historic_data/fund_8/mk_fund8_sql.py historic_data/fund_8/fund8_database_encrypted.sqlite3 >> historic_data/fund_8.sql
-        RUN python3 historic_data/fund_9/mk_fund9_sql.py historic_data/fund_9/fund9_database_encrypted.sqlite3 >> historic_data/fund_9.sql
-        RUN python3 historic_data/fund_9/mk_fund9_voteplan_sql.py historic_data/fund_9/fund9_database_encrypted.sqlite3 >> historic_data/fund_9.sql
-
-        RUN mv ./historic_data/*.sql ./data
-    END
-
-    IF [ $with_test_data = true ]
-        COPY --dir test_data .
-
-        RUN mv ./test_data/*.sql ./data
-    END
-
-    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:v2.0.7+BUILDER
+    COPY ./../../+repo-config-2/repo/.sqlfluff .
+    COPY setup-db.sql /sql/
+    COPY --dir json_schemas .
 
+# check if the sql files are properly formatted and pass lint quality checks.
+#   CI target : true
 check:
     FROM +builder
 
-    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:v2.0.7+CHECK
+    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:fix/postgres-builders+CHECK
+
 
+# format all SQL files in the current project.  Local developers tool.
+#   CI target : false
 format:
     LOCALLY
 
-    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:v2.0.7+FORMAT --src=$(echo ${PWD})   
+    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:fix/postgres-builders+FORMAT --src=$(echo ${PWD}/../../)   
 
-# Build event db docker image.
-# Arguments:
-#  * tag: docker image `tag`.
-#  * registry: docker image `registry`.
-#  * with_historic_data: add historic data from the `historic_data` dir (default `true`).
-#  * with_test_data: add test data from the `test_data` dir (default `true`).
+# build - an event db docker image.
+#   CI target : true
 build:
-    ARG with_historic_data=true
-    ARG with_test_data=true
-    ARG tag="latest"
-    ARG registry
-
-    FROM +builder --with_historic_data=$with_historic_data --with_test_data=$with_test_data
-
-    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:v2.0.7+BUILD --tag=$tag --registry=$registry --image_name=event-db
-
-test:
-    FROM github.com/input-output-hk/catalyst-ci/earthly/postgresql:v2.0.7+postgres-base
-
-    COPY github.com/input-output-hk/catalyst-ci/earthly/utils:v2.0.7+shell-assert/assert.sh .
-
-    COPY ./docker-compose.yml .
-    WITH DOCKER \
-        --compose docker-compose.yml \
-        --load event-db:latest=(+build --with_historic_data=false) \
-        --service event-db \
-        --allow-privileged
-        RUN sleep 5;\
-            res=$(psql postgresql://catalyst-event-dev:[email protected]:5432/CatalystEventDev -c "SELECT COUNT(*) FROM event");\
+    FROM +builder
 
-            source assert.sh;\
-            expected=$(printf " count \n-------\n     5\n(1 row)");\
-            assert_eq "$expected" "$res"
-    END
+    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:fix/postgres-builders+BUILD --image_name=event-db
+    DO github.com/input-output-hk/catalyst-ci/earthly/postgresql:fix/postgres-builders+DOCS --image_name=event-db
+
+# test the event db database schema
+#   CI target : true
+#test:
+#    FROM github.com/input-output-hk/catalyst-ci/earthly/postgresql:fix/postgres-builders+postgres-base
+
+#    COPY github.com/input-output-hk/catalyst-ci/earthly/utils:fix/postgres-builders+shell-assert/assert.sh .
+
+#    COPY ./docker-compose.yml .
+#    WITH DOCKER \
+#        --compose docker-compose.yml \
+#        --load event-db:latest=(+build --with_historic_data=false) \
+#        --service event-db \
+#        --allow-privileged
+#        RUN sleep 65;\
+#            res=$(psql postgresql://catalyst-event-dev:[email protected]:5432/CatalystEventDev -c "SELECT COUNT(*) FROM event");\
+
+#            source assert.sh;\
+#            expected=$(printf " count \n-------\n     5\n(1 row)");\
+#            assert_eq "$expected" "$res"
+#    END

catalyst-gateway/event-db/diagrams.json

@@ -0,0 +1,16 @@
+{
+    "name": "Catalyst Gateway - Event DB",
+    "all_schema": {
+        "comments": true,
+        "included_tables": [],
+        "excluded_tables": [
+            "refinery_schema_history"
+        ],
+        "column_description_wrap": 50,
+        "table_description_wrap": 120
+    },
+    "full_schema": {
+        "title": "Full Schema",
+        "comments": false
+    }
+}

catalyst-gateway/event-db/json_schemas/.cspell.json

@@ -0,0 +1,19 @@
+{
+    "$schema": "https://raw.githubusercontent.com/streetsidesoftware/cspell/main/cspell.schema.json",
+    "version": "0.2",
+    "enabled": true,
+    "language": "en,en-US",
+    "overrides": [
+        {
+            "filename": "**/*.json",
+            "allowCompoundWords": true,
+            "words": [
+                "sshuser",
+                "cexplorer",
+                "Cardano",
+                "jormungandr",
+                "dbsync"
+            ]
+        }
+    ]
+}

catalyst-gateway/event-db/json_schemas/config/dbsync.json

@@ -0,0 +1,98 @@
+{
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "$comment": "Custom URI schema: catalyst_schema://<uuid>/<group>/<name>",
+    "$id": "catalyst_schema://d899cd44-3513-487b-ab46-fdca662a724d/config/dbsync",
+    "title": "DBSync Connection",
+    "description": "DBSync connection configuration",
+    "type": "object",
+    "properties": {
+        "common": {
+            "$comment": "This provides the base settings for all networks. Anything thats common to all networks should be defined here.",
+            "$ref": "#/definitions/dbsync_conn"
+        },
+        "networks": {
+            "$comment": "Individual networks defined here.  Only need to define settings which differ from the common settings. Must define at least 1 network.",
+            "type": "object",
+            "properties": {
+                "mainnet": {
+                    "$ref": "#/definitions/dbsync_conn"
+                },
+                "preprod": {
+                    "$ref": "#/definitions/dbsync_conn"
+                },
+                "preview": {
+                    "$ref": "#/definitions/dbsync_conn"
+                },
+                "local": {
+                    "$ref": "#/definitions/dbsync_conn"
+                }
+            },
+            "anyOf": [
+                {
+                    "required": [
+                        "mainnet"
+                    ]
+                },
+                {
+                    "required": [
+                        "preprod"
+                    ]
+                },
+                {
+                    "required": [
+                        "preview"
+                    ]
+                },
+                {
+                    "required": [
+                        "local"
+                    ]
+                }
+            ]
+        }
+    },
+    "required": [
+        "common",
+        "networks"
+    ],
+    "additionalProperties": false,
+    "definitions": {
+        "dbsync_conn": {
+            "$comment": "DBSync connection",
+            "type": "object",
+            "properties": {
+                "host": {
+                    "type": "string",
+                    "description": "The hostname or IP address of the PostgreSQL server.",
+                    "example": "localhost",
+                    "default": "localhost"
+                },
+                "port": {
+                    "type": "integer",
+                    "description": "The port number of the PostgreSQL server. Note, the default will be the ssh localPort if the connection is tunneled over SSH.",
+                    "example": 5432,
+                    "default": 5432
+                },
+                "database": {
+                    "type": "string",
+                    "description": "The name of the PostgreSQL database to connect to.",
+                    "example": "cexplorer",
+                    "default": "cexplorer"
+                },
+                "username": {
+                    "type": "string",
+                    "description": "The username for authentication.",
+                    "example": "cexplorer",
+                    "default": "cexplorer"
+                },
+                "password": {
+                    "$comment": "If the password is not defined, it will be read from an env var called either DBSYNC_PWD_<NETWORK> and if thats not defined DBSYNC_PWD.",
+                    "type": "string",
+                    "description": "The password for authentication.",
+                    "example": "password"
+                }
+            },
+            "additionalProperties": false
+        }
+    }
+}