README

Brain crawler is a project to build a so called “second brain”. As of today, the world yields us an overhelming amount of information. It has become necessary to have a systematic way to store it in order to keep up. Not only storing it is the issue, but also storing it in a way for easy access later.

This is why Brain Crawler is built up onto the following principles:

• plain text storage. This allows for long term storage, on a longer time scale that the lifetime of this project. For the same reason, even though it only supports org-mode for now, it is built in a format agnostic way. Markdown support could be added in the future. More importantly, it permits the version control of the files, i.e. putting them in a git repository.
• client / server architecture as well as local usage. A second brain should be accessible any time : this is why it is necessary to make a smartphone client. However, it should not become unavailable when internet goes down : this is why a local copy should be able to run by itself (only computer support planned for now).
• graph based. Based on my own knowledge of second brain like systems, the hard problem is scalability. A system based on nodes and links between nodes is more resilient to evolution and proliferation of nodes than a rigid hierarchy.

Implementation details

Client

Android/iOS

A smartphone client is currently being developped using react-native.

Web client

The nodes will also be accessible on the web, using a web client developped alongside with the server using django.

The client is responsible for the parsing of the org-mode files.

Server

The server maintains a database of the nodes and the links between them. This database does not store the content of the nodes ; they are read on the fly for each request. The files are written in org-mode. The server does not have to parse the org mode files ; it just need to look for links inside to list backlinks.

The nodes and links can be queried using a standard REST API. Note that it does not implement the edition the nodes : as the files could be put inside a git repository, they should not be modified improperly.

The server will be written in Python using Django.

Roadmap

Server

API

• [X] node list no details
• [X] node list add the link to detail
• [X] node detail add content of file

Permission model

• [X] readonly everywhere
• [X] only owner can see nodes

Parser

Indexing of files

Other

Web client

Smartphone client

Create a splash screen with Expo

All the views

• graph, neurons-like display
  • node preview ?
• search
• node display
• settings
• log in screen

How the views are laid out

• Tab controller
  • Graph
    • When the user touches a node, a bottom screen appears with the details
    • If scrolled up, it turns into the full view page
  • Settings
    • Server
      • URL input
      • custom port switch
  • port input if on
    • Login
      • Text logged in as …
      • Log out button with confirmation
• Inside the graph view
  • Search
    • Search suggestions

Misc notes

Django with AWS Lightsail