Skip to content

Latest commit

 

History

History
98 lines (61 loc) · 5.42 KB

RFP：venus-market - system development.md

File metadata and controls

98 lines (61 loc) · 5.42 KB

RFP：venus-market system development

project description: “venus-market system development”

Rfp type: "dev-core"

Initiator: venus-team

Project Description

To maximize data storage and retrieval is one of the key function of Filecoin network. Currently, venus has finished the development of the basic components. The next step is to finish the development of venus-market as the last component of venus. Different from lotus, venus distributed mining pool is an assembly of multiple miners. Venus-market provides a unified entry for data storage and data retrieval for venus distributed mining pool. And data distribution in different location can also help explore the function of venus distributed mining pool as the entry of ipfs gateway.

The goal of venus-market is to finish the development of a highly available independent program with full functions and improved APIs and commands. The system is designed to support a full process of data storage and data retrieval, a variety of important data storage and data transmission protocol, multiple miners to distribute orders, multiple locations to select nodes, and cluster horizontal scaling.

Deliverables

An open-sourced, dual-licensed (under MIT and APACHE2) implementation that:

Track 1:

  • data storage and data retrieval, fully compatible with the client side of lotus storage and retrieval.

    Storage and retrieval of venus can smoothly access the existing lotus storage retrieval client, and for specific miners, those nodes currently using lotus can connect to venus-market to provide targeted services.

Track 2:

  • a client-side program that minimizes storage and retrieval. In addition to the existing functions of lotus, it also needs to support miner list query and miner selection and query who and where the data belongs to.

    Because venus-market will do multi-point backup storage based on multiple miners, and multi-location nodes to accelerate retrieval, it will also support more data transmission protocols which cannot be implemented only by the lotus clients, but also needs venus-market to provide the related client sides.

Track 3:

  • Structured data storage

    In addition to the badger local storage, the support to storage critical data (state machine, asset-related, order, piece) on MySQL in order to better ensure the security of core data.

Track 4:

  • Multiple data transmission protocols

    Disk and memory are consumed in the go-data-transfer transmission process. Here we hope to better abstract the data transmission process and to support a variety of different protocol transmission methods. In addition to the basic go-data-transfer transmission method, it is currently considered to availably support http fragmented transmission.

Track 5:

  • Asset storage localization

    The asset data generated by fundmgr and paychanel in lotus is stored in the node through the API, but in venus-market, it is hoped that it can be stored in mysql instead of the storage of the node.

Track 6:

  • piece data storage, to support multiple storage types

    venus-market manages the received piece data, which is temporarily stored in a piece storage, waiting for the sealer system to package these pieces. It is hoped that this piece storage can support multiple storage methods and is easy to implement.

Recommended Team

  • Have system development experience for influential program
  • Familiar with Filecoin's current mining and storage and retrieval mechanism
  • Those who are willing to jointly promote the venus open source community are preferred

Milestones & Funding

Total Funding Amount: TBD. Please propose a budget in your proposal.

Suggested Milestones:

Milestone No. Milestone Description Funding Estimated Timeframe
1 Finalize scope, workflow design, architecture design TBD 3-4 weeks
2 All project deliverables have been completed (website, documentation, code base) TBD 2 weeks
3 Maintenance and upgrade during Testnet and Mainnet TBD 2-3weeks

Acceptance Criteria

Acceptance criteria for Finalize scope, workflow design, architecture & Design mockups:

  • This milestone's deliverables must represent the final design of the service.
  • The workflow design should address the core goals proposed in the description of this brief.

Compatible with lotus client-side storage and retrieval

  • The lotus client can smoothly access any miner in the venus distributed mining pool for storage retrieval services
  • Improved API and verifiable examples

venus-market client side

  • Implement all requirements in Track 2
  • Improved API and verifiable examples

Structured Data Storage

  • The listed core data needs to be stored in mysql, and to add additional data is also involved
  • Improved API and verifiable examples

Multiple data transfer protocols

  • Implement go-data-transfer and http fragmented transmission, and ensure related security
  • Improved API and verifiable examples

Asset storage localization

  • The method of data storage of asset information is shifted from being saved from the API, to being saved locally.The original API is supported but also needs to be modified, for example, miners need to extract the asset information through the API.
  • Improved API and verifiable examples

Piece data storage, supports multiple storage types

  • Implement two piece storage types of file system and object storage. And the corresponding save read interface
  • Improved API and verifiable examples