🏛️ [EPIC] : Catalyst Gateway Voting

[stevenj]

Summary

First phase of an improved vote storage solution for catalyst voices

JIRA Epic

https://input-output.atlassian.net/browse/NPG-8295

Description

Vote storage solution for Catalyst Voices.

The Catalyst gateway will become an improved vote storage mechanism for Catalyst and the staging ground for work on fully decentralizing vote storage which will only be fully realized in the Hermes/Athena projects.

Todo

Phase 0 - Prepare Voting Libraries and Algorithms

• 🛠️ [TASK] : Design Immutable Storage Ledger which will be used for vote storage and realtime publishing of votes. #693
• 🛠️ [TASK] : Voting transaction design docs #749
• 🛠️ [EPIC] : Voting crate catalyst-libs#32
• 🏛️ [EPIC] : Catalyst voting v2 transaction catalyst-libs#63

Phase 1 - API Setup

In this phase, API endpoints will be created, with no functionality, that answer the same API calls as the existing jormungandr node.
These APIs are interim and allow us to have backwards compatibility, so even though we are implementing them new, they are to be marked as deprecated to signal our intent to obsolete them.

This Phase does not create any implementation beyond setting up the endpoints themselves, and setting up the data structures associated with the requests and responses. The OpenAPI documentation must be fully complete for each endpoint to be Done.

We ONLY need to implement endpoints that are used by:

1. The Mobile voting app
2. The Web based voting app GVC (Which can be assumed to be identical to the Mobile voting app for this process).
3. Produce results from tally, or are involved in the tally.
4. Other endpoints inside jormungandr are NOT to be implemented.
5. The behavior of any and all CLI tools regarding these APIs is out of scope and backward compatibility with them is also out of scope. No work will be done to support backwards compatibility with CLI tools, either for Voting or operational purposes.

Even if an endpoint is listed in these tasks, it must be validated if it fits the above criteria before work commenced.
Any endpoints that fit the above criteria, but are not listed, needs to be raised with engineering management BEFORE being added to the list.
Any endpoint to do with the operation of voting, such as fragment logs are out of scope and that data will be reported by alternative means.
Endpoints also only need to be implemented as far as the response is used by an in-scope application. This needs to be validated.
For example, if an endpoint return 35 fields, but the app only uses 2 of them and ignores the rest, then response needs to be reduced to the set of data the app needs to operate correctly and all other data excluded.

Do not rely solely on the existing API documentation validate against both:

1. Jormungandr code
2. Norns voting app

• 🛠️ [TASK] : Compatibility Voting Endpoint /api/v0/vote/active/plans #53
• 🛠️ [TASK] : Compatibility Voting Endpoint /api/v0/message #54
• 🛠️ [TASK] : Compatibility Voting Endpoint /api/v1/fragments #55
• 🛠️ [TASK] : Compatibility Voting Endpoint /api/v1/fragments/statuses #56
• 🛠️ [TASK] : Compatibility Voting Endpoint /api/v1/votes/plan/account-votes/{account_id} #57
• GET /api/v0/settings
• GET /api/v0/fund/
• GET /api/v0/proposals
• POST /api/v0/proposals
• GET /api//v0/proposals/${id}
• GET /api/v0/reviews/${proposalId}
• 🛠️ [TASK] : Add API Endpoints to Catalyst Gateway Legacy endpoint #1081
  • POST /api/v0/search
  • POST /api/v0/search_count
  • GET /api/v0/challenges/
  • GET /api/v0/challenges/${id}
  • GET /api/v0/node/stats
  • GET /api/v0/account/${id}

See: https://github.com/input-output-hk/catalyst-core/blob/f1553974f6bd531508107e1f973c734080b386e6/book/src/07_web_api/openapi/core_ledger_v0.yaml#L720 for OpenAPI Schema of these endpoints, for reference.

Phase 2 - DB Setup

Votes will be stored into the cassandra type db, pending archival into an indelible voting ledger.
All valid votes logged to the DB will be assumed to be also in the voting ledger for this phase.

Unlike Jormungandr, we should be able to pre-configure all the fund information including cryptographic keys and vote plans well in advance of voting commencing, and even in advance of the final snapshot.

We do not need to create a block0 as it is known in Jormungandr just prior to voting, as we can directly query the registration data as a source of truth, both during voting and during tally as a validation.

• 🛠️ [TASK] : DB Schema design for Vote Buffering #116
• 🛠️ [TASK] : EventDB vote store query #117
• 🛠️ [TASK] : Query to get votes cast by voter #118
• 🛠️ [TASK] : Query to iterate all votes from all voters in a challenge #119
• 🛠️ [TASK] : Ensure shcema has appropriate indexes for efficient query operation #120
• 🛠️ [TASK] : Ensure voteplan queries are sufficent for voting endpoints #121
• 🛠️ [TASK] : Integration test data setup for voting #122 -> First subtask. Other subtasks can be prepared but not tested until Phase 3 is implemented.

Phase 3 - Endpoint Implementation

This phase adds the implementation to the endpoints created in Phase 1 with the data.

At this stage there should be no caching of votes, all vote storage and retrieval is directly on the DB. There is no mempool of any description. The endpoints which cast votes should be done before the endpoints which return statuses.

• 🛠️ [TASK] : Add implementation for /api/v0/vote/active/plans #123
• 🛠️ [TASK] : Implement /api/v0/message #124
• 🛠️ [TASK] : Implement /api/v1/fragments #125
• 🛠️ [TASK] : Implement /api/v1/fragment/statuses #126
• 🛠️ [TASK] : Implement /api/v1/votes/plan/account-votes/{account_id} #127
• 🛠️ [TASK] : Integration test data setup for voting #122 All other subtasks.

Phase 4 - Tally and Results

• Product and Engineering to collaborate on defining the necessary functional requirements for tallying and results generation.

Phase 5 - Persistent tamper proof voting ledger storage.

• Design a tamper proof and persistent voting ledger storage mechanism.

Phase 6 - Initial Decentralization of voting ledger.

• Design a decentralized real-time distribution mechanism for the tamper proof voting ledger storage.