Dylan's Original Codr Ideas

[DylanBulmer]

Conceptualizing

• Open-sourced/core features are completely free to use
• Needs to be self-hosted
• Developed as a PaaS (platform as a service) or BaaS (backend as a service)
• Built as a mircoservice architecture.

Importing data:

• Config files
  • Abstract/customizable dataset configuration via YAML, JSON, or TOML files for loading data into the annotation tool
• Validation of file import
  • Make sure all dataset items match the expected configuration
  • Display a loading bar of data being sent to the server for processing
    • Stream data to the server, one dataset item per chunk.
    • Use secure WebSockets (wss://)???
  • Display a message for rejected results
    • Allow exporting the rejected items.

Exporting data:

• No-code implementation for customizing export files???
  • A canvas UI element that has the annotator output data on the left and a list of user-defined names on the right with the ability to click and drag lines to connect the data to the desired customized name for the export file.
• Allow different export file types???
  • CSV
  • JSON
  • YAML
  • Plain Text (tab indented or something)???

Implementation Ideas

Separate client and server containers

• Client: React Remix or NextJS app using Tailwind UI
• Server: Express RESTful API with @dylanbulmer/openapi for documentation.

Needs a better thought-out authentication and authorization method.

• ABAC (ACL) using CASL — https://casl.js.org/v6/en/
• PassportJS for authentication — https://www.passportjs.org/
• JWT authentication — https://www.passportjs.org/packages/passport-jwt/
• OAuth2 for third-party use of open-source framework???
  • Longer-term goal… probably.
  • This would be for the paid BaaS (backend as a service).
  • Looks easy to implement — https://github.com/oauthjs/express-oauth-server/blob/master/examples/mongodb/model.js

Database

• MongoDB using mongoose
• Is TypeORM needed???
• Hosted using MongoDB Atlas or self-hosted???

Framework Design:

• Singleton (codr)
• Services:
  • Authentication
  • Configuration
  • Administration (sysadmin permission required)
  • Dataset management
  • Annotator
  • Analytics
• Build it like Firebase 9.0
  • They have an “app” singleton that’s passed into a function to receive the desired service… 
(view code example below)
  • Better for scalability.
    • This allows us to easily “host” multiple Codr instances side-by-side without them talking to each other.
    • In the product, we can easily build a wrapper controller that can manage the instances
      • The controller would automatically:
        • Create DNS records (in cloudflare?)
        • Store server configuration (in separate database hosted by us)
        • Orchestrate instances (generate the k8s files for the instance and deploy it to EKS or similar)
  • Great for optimization
    • Only imports the desired services of Codr without needing the entire codebase.

const app = initCodr(/** Settings to initialize the app here */);

const auth = getAuth(app);


const annotator = getAnnotator(app);

Sepideh’s notes:

• Analytics
  • Analysis of similarities
• Live Collaboration of annotating

TODO:

• Create user stories
• Create system requirements
• Create an SRS??? Do I really want to?… kind of
• Design ALL data structures
• Determine the development stack