Port the tool to React

[rorybyrne]

It looks like interactivity is a key part of this concept, so vanilla Javascript will quickly become a painful experience. We should instead use tools that are built for that purpose - like React. It will be much easier to extend and scale.

[rorybyrne]

Current plan:

• Write a vanilla-Typescript module for converting a LaTeX file into some intermediary format, separate from the React code
• Define what the intermediary format looks like, using Typescript types
• Try to isolate this section of the code, so that it could be an independent package, and easily-portable to other languages.
• Write a React component which takes an abstract paper as input, and renders it for reading
• Write a React component which takes an abstract paper as input, and renders just the "resources" (citations, figures, etc.)
• Think about how to achieve parity with functionality like hovering in the main panel. Ideally this would be generic enough to support many types of panels, which interact with the main panel in some way.

[rorybyrne]

I think it's quite likely that the LaTeX -> AST conversion will happen on a server in the medium-long term, so that code could be written in any language. The React code will just send a raw LaTeX file to the API and receive the AST as a response from the server.

[rorybyrne]

@thesamovar:

My React implementation is close to parity in terms of functionality (plus some extras!), but I haven't done anything with the CSS yet.You can demo it here, and the code is here. Heads up, the paper's contents are only half-there because it was a lot of effort to manually construct the data structure.

If you make a react branch on this repo, then I can open a PR into that branch? That way we can keep the main branch for the vanilla-javascript implementation, until we decide to switch over.

I'll write a few bullet points in the PR explaining the main points about React (and Redux, the state-management library), and where in the codebase different things can be found. I'll also point one or two commits which are good examples of how to add functionality to the codebase (e.g. adding the pin icon to the citations in the related panel).

[thesamovar]

I've invited you to the repo. Feel free to make a branch but please don't merge yet.

I think you're probably right that we need to move to a more sustainable method, but I have to say looking at the react code makes my heart sink a bit. It's such a step up in complexity.

Is there a way to run such code statically without having to have an active server like in your example? I'd like to be able to offer people a way to locally host them on their own static html websites, etc.

[rorybyrne]

Thanks, I'll make sure it stays in a separate branch.

I think you're probably right that we need to move to a more sustainable method, but I have to say looking at the react code makes my heart sink a bit. It's such a step up in complexity.

Here's an idea: to avoid losing your momentum, you could just continue prototyping ideas in the vanilla-js version for as long as you want to, and I can continue re-implementing them with more polish in the React branch. I'll make sure to keep commits clean and self-contained so that I can point you towards specific commits which demonstrate how a specific feature is implemented in React (e.g. this commit which adds the pin icon and click-to-expand functionality). Then if you ever find the time to run through a couple of React tutorials you should be in a good position to make the switch over to the React codebase. What do you think?

Is there a way to run such code statically without having to have an active server like in your example? I'd like to be able to offer people a way to locally host them on their own static html websites, etc.

Yeah, I think so. We could build this as an Electron app, which is a runtime for making desktop apps in Javascript (e.g. VS Code is built using Electron). That would allow us to build this as a desktop app which consumes a local folder of documents. We can't do that with Javascript running in the browser though, it's running in a sandbox and can't access the OS or filesystem.

We'd need to choose one path or the other though, either an app which consumes local documents, or a website which is backed by a "global" database of documents.

[thesamovar]

This all sounds good to me. I'd like to learn React and Electron, it's just hard to find the time. Would you be available to go through the setup with me by video call at some point? That might help get everything installed and running nicely at least. Maybe not necessary - there are probably good tutorials online I can follow.

I'd like there to be a common code base that would allow us to do web-driven and local if possible, since there should be a lot of shared / overlapping functionality I assume?

How about static HTML? It reads in a latex/docx/pdf file and it dumps out an html/css/js like the way it works at the moment? Then people can just upload to their own web servers. Might be the best way of getting traction quickly, especially if we can host that converter in a website so nobody needs to download anything.

I like your parallel development plan too.

[rorybyrne]

I think a call makes sense. Wednesday or Friday morning (pre-noon) works for me, or else any time this weekend.

I'd like there to be a common code base that would allow us to do web-driven and local if possible, since there should be a lot of shared / overlapping functionality I assume?

In principle that should be possible. We could extract a good chunk of the functionality out into a library which can be hosted and installed from npm (Javascript's version of pip). But, I think we should start by picking one approach and doing it right. Trying to support both simultaneously would just confuse things.

How about static HTML? It reads in a latex/docx/pdf file and it dumps out an html/css/js like the way it works at the moment? Then people can just upload to their own web servers. Might be the best way of getting traction quickly, especially if we can host that converter in a website so nobody needs to download anything.

I agree, that does sound like a great way to drive early traction (you'd make a good startup founder!). I think it would be even easier to adopt if people could just visit https://notpaper.com and upload a file, then put a link to https://notpaper.com/some-paper on their website (instead of serving the HTML themselves). It wouldn't take long for me to spin up a server to back this.

That gives us flexibility to start by converting the uploaded files to static HTML, and then later converting them to our intermediary format which can be rendered via React. We could even do both at the same time (in theory) at slightly different web addresses, e.g. upload a file on the homepage and then view it at https://notpaper.com/static/mypaper or https://notpaper.com/react/mypaper.

Does the current converter work reliably on new papers, without manual intervention? If not, how close is it?

[thesamovar]

Emailed you about call.

The problem with server is that it has costs, which if this takes off could become substantial. If it's possible to run everything locally (pure js in the browser) that would be much better I think. But, in the short term having it hosted on a server is fine.

Current converter works OK but not brilliantly. Problem is that latex has a lot of fiddly bits. It's not really the ideal format, but you have to work with what you've got. All papers pretty much written in Word or latex so...

[rorybyrne]

If it's possible to run everything locally (pure js in the browser) that would be much better I think

The spanner in the works here is that Javascript in the browser can't run (e.g.) pandoc, because the JS is in a sandbox and cannot run a shell command like what's happening here. So if we wanted to host a converter website which would take LaTeX as input and return a zipfile of HTML/CSS/JS, we would need to have a server doing the actual conversion via pandoc behind the scenes.

I think we should just host a server though, and not let worries about costs get in the way. We can get a free server and database from AWS, and we end up getting enough traction to cost 10-20 per month then I don't mind covering that. If the project really takes off then that's a very good problem to have, and we could definitely find a way to get it funded.

[thesamovar]

Fair points!

I'd still like to be able to create local versions so that people can view offline or put on their own website, but the main thing can be a server indeed.

[rorybyrne]

For offline viewing, I think an Electron-based desktop app sounds like a good fit. You launch it and it reads files from your local drive and converts them on the fly using pandoc under the hood. We could even add a button in the app to convert a specific paper to HTML for self-hosting, and if we decided that we want to host it ourselves then we could do something like Obsidian Publish which does exactly that, with Markdown files.

Either way, it looks like there's a lot of ways to skin this cat. I don't have particularly strong opinions beyond using scalable technology (i.e. React) and avoiding common architectural pitfalls. I'm happy to follow your intuition for the functionality since you know more than I do about what tools researchers want.

It will be helpful for me if we could come up with a list of "things this tool should be, and should not be" at some point. E.g. it should be usable offline for viewing local files, and it should not be a text editor. Then I can help mapping that to a roadmap of some sort.