chrome-based http proxy for better fetching of original page content

[Ivan8or]

I have recently gotten into RSS and have settled on miniflux as my aggregator.

My experience with miniflux itself has been positive so far, but my experience with the sites i am collecting articles from less so...

Specifically, i have run into several websites which enforce javascript challenges on their articles, i suppose to prevent creatures like us from avoiding their ads and tracking.

After looking around online and being disappointed with the existing options, I have decided to take it upon myself to create a web scraper curlk which bypasses simple fingerprinting / js challenges and, very importantly, takes the form of an http proxy.

This is a single docker application which can be run in conjunction with miniflux and configured to be either miniflux's global HTTP_CLIENT_PROXY or as an individual feed's prox(ies).

this is how I currently use curlk in my miniflux deployment:

services:
  ... <omitted for brevity>
  miniflux:
    image: miniflux/miniflux:2.2.14
    environment:
      ... <omitted for brevity>
      - HTTP_CLIENT_PROXY=http://curlk:3800

  curlk:
    image: ivan8or/curlk:latest
    cap_add:
      - SYS_ADMIN
    init: true
    shm_size: "4gb"
    restart: unless-stopped

NOTE: you must enable 'Allow self-signed or invalid certificates' for all feeds which use curlk to request https urls

you can find the project at https://github.com/Ivan8or/curlk and the latest docker images at https://hub.docker.com/r/ivan8or/curlk