Project update 2

[chuanqisun]

Hi all,

A lot has happened since the last update. In 2021, my life was impacted by COVID-19 and I'm dealing with some additional difficulties in life. As a result, the time and energy I had for coding was less than I expected. I just want to thank you for your patience and understanding.

On a positive note, osmosfeed project (and its parent osmoscraft) is not going away. I still managed to build several prototypes in the last year, exploring ways to solve problems in the current version. I'm happy to share some exciting progress.

Learning

After several rounds of prototyping, I've landed on a somewhat satisfying design:

1. The feed parser needs to be built from scratch, using a low level xml utils, instead of using off-the-shelf RSS parser. This gives us the flexibility to deal with all the inconsistencies across feed formats, e.g. off-the-shelf parser had difficulty distinguishing:
   • full content in text
   • full content in html
   • summary in text
   • summary in html

The parser I wrote from scratch lets us track all four of them independently, so we can decide later which one needs to be rendered based on conditions that is unknown during parsing. The parser design was a trial-and-error process where I learned a lot:

• using cheerio + htmlparser2 (adopted, good perf, nice query syntax, robust community)
• using cheerio + stock parser (too slow)
• using the browser built-in xml parser
  • this one actually out-performs the adopted solution but requires feed parsing on the fly in browser. The delay is awful but I think we can revisit this option in the future if we want to perform live update of feeds without rebuilding on the server.
• using the browser built-in xml parser in a headless chrome on server side (too complex, and the github workflow needs to download a huge binary, slowing down the build)

2. The cli needs to have separated runtime and plugins. The runtime is responsible for scheduling tasks and handle concurrency. The plugins are responsible for data transformation. The plugins are ideally pure functions while the states are contained in the runtime. This design allows for easy testing of plugins as well as future extensibility.

3. The crawling (previously known as enrichment) needs to be decoupled from the core feed data. Some websites have huge payload that once included into the core feed data will hurt the performance (e.g. when diffing updating). The new design still downloads and saves html files, but render them only when requested from the UI, using JavaScript fetch API.

Prototype sneak peek

• 2-D layout that prioritizes new content to the top left corner of the screen.
• Reading pane that renders full content (from crawler).
• Video and images are rendered naturally.

You can try the live prototype yourself. But just to call out, the arrow buttons in the reading pane are not wired up yet. The intention is to navigate the 2-D layout without closing the pane. Vim and arrow keybindings will be supported too.

Here is the starter-kit (previously known as template) behind the prototype. It is NOT ready for use, but is fully function as a prototype.

• No longer requires messing with gh-page branch. The build and deploy happen in main/master.
• No longer requires cache url setup. The git repo itself serves as the cache during build.

Road ahead

I've created a project to keep track of the v2 release. I believe most of the hard problems are figured out during prototyping and what's left is just lots of hours of coding. I'll post more updates are new features become ready in the v2 branch.

Meanwhile, I'll continue maintaining the v1. It's unlikely to receive new features but bug fixes are still coming. As for v1->v2 migration, there will be documentation. There is some breaking change in the config file and the v1 cache is not valid for v2. However, v2 is designed for the long run, so hopefully future migrations can be avoided.

Lastly, thanks for staying with this community. Your ideas, bug reports, and blog articles in different languages have made me a very happy hacker. Just to name a few:

• Japanese: https://paiza.hatenablog.com/entry/2021/07/07/150000
• Chinese: https://shansan.top/2021/06/29/osmosfeed-rss-reader/

I hope that the v2 release will be my best way to say "thank you" ❤️