[Feature request] Allow-/denypattern filter for articles

[svoop]

Read and tick the following checkbox after you have created the issue or place an x inside the brackets ;)

• I have read the CONTRIBUTING.md and followed the provided tips

Is your feature request related to a problem? Please describe.
Some feeds feature articles which are not interesting to the subscriber. For instance: I subscribe to a tech news feed which features articles on Android and iOS. However, since I don't have an iPhone, I'm not interested in the latter and would like to ignore such articles.

Describe the solution you'd like
A cascade of two regexp patterns might help: If the article subject matches the denypattern, it is ignored unless it also matches the allowpattern. Example:

1. Denypattern: /iOS|iPhone/i
2. Allowpattern: /Android/i

Subject	Denypattern	Allowpattern	Result
New Apps in Oct	no match	skipped	allow
New Apps for Android	no match	skipped	allow
New Apps for iPhone	match	no match	deny
New Apps for iPhone and Android	match	match	allow

[alistair-marshall]

I came here looking for exactly this feature. In my case, I want to follow a local news source, but the majority of the articles are about sport. I would like to be able to filter out anything with '/sport/' in the url.

[mejo-]

This would be a very valuable feature for journalists!

[ei8fdb]

@svoop How many regexps do you think would be common for a feed? This could get rather long rather quickly!

@Grotax @SMillerDev could you add a design label to this ticket?

[svoop]

@ei8fdb From a UI perspective, multiple regexps are quite a bit more difficult and I don't see much benefit from it.

[ei8fdb]

@ei8fdb From a UI perspective, multiple regexps are quite a bit more difficult and I don't see much benefit from it.

Agreed, hence why I asked. Where do you suggest the user create those filters?

[svoop]

I guess there are two possible places:

Two additional inputs (denypattern, allowpattern) when adding a new subscription. However, I'd recommend not to, because regexp are a power user feature and it would only confuse the average user – specially since there is little to no space for explanations.

An additional context menu point "Filter" which opens an overlay which contains explanations and the inputs to add/edit the denypattern and allowpattern regexp.

The explanation could be something along the lines:

You can filter articles with regular expressions: An articles whose title matches the deny expression 
is ignored unless it matches the allow expression as well.

Deny:  [/iOS|iPhone/i                    ]
Allow: [/Android/i                       ]

[Save]

The expressions here are examples best added as <input placeholder="/iOS|iPhone/i" ...>.

The term "regular expression" could be linked to something like https://www.petefreitag.com/cheatsheets/regex/ – there might be better, it's just what popped up on a quick Google seach.

[stefan-niedermann]

Maybe this can get be implemented in a few iterations. The minimal viable feature set doesn't need regex, or allow/block states. I think just setting a filter to block everything which matches %term% was a easy first step which can be extended in the future.
Also the furst iteration doesn't necessarily need a UI in my opinion - a config php file is also a good starting point.

Such kind of filter already was for example powerful enough for example to block ads like "Heise+" or "Advertisment: " and so on.

[svoop]

@stefan-niedermann Will the filtering rather be done in SQL on the database (after reading/parsing the RSS feed) or in PHP while reading/parsing the RSS feed?

[stefan-niedermann]

@svoop i am no developer or maintainer of this app, i am a plain user and suggested a more minimal approach for the sake of a quick implementation 😉 😇

[Grotax]

Is there any commercial or open source feed reader that offers something likes this?

I think filtering during parsing might be a bit too much for feed parser especially since the feed parsing is a heavy task and there is no parallel job we want to keep it as slim as possible.

So filtering could be either done on the API level or by the clients.

[Grotax]

Is the web-interface a client as well? Do the filter get applied here as well?

Yea it is also a client, so same behavior regarding this compared to the android app.

And at least for me I would like to get rid of those filtered items entirely.. I don't just want to "hide" them temporarily.

Not sure if we can do that as we probably don't want to put more logic in the feed parser than needed, cron an php are not the perfect match for feed parsing.

IMHO I believe the filtering should probably happen before items are stored on the server. I'm not exactly sure how many feeds an average user has but my list is growing quite fast (every day by around 300 items). If those items are never marked as read (because they're filtered) they will be stored forever, is that correct? I'm not sure if that will be a problem later down the road.

The issue with that is the way news parses feeds and puts them in the DB, for your wish we would need to filter the feeds before they get stored to the DB every time, so let's say you have a feed with 100 items in total, the old stuff the author deletes from the feed.
OK so every time your client fetches the feed we need to run a regex over every item that is not already in the DB.
Now we do that for all feeds and for every user.

I'm not sure how big the impact of that regex is but with the current job model of nextcloud (blocking non concurrent) I don't think we want this, that is why the python updater was developed.

[svoop]

I'm not sure how big the impact of that regex is but with the current job model of nextcloud (blocking non concurrent) I don't think we want this, that is why the python updater was developed.

Regex are generally very fast, but it's possible to produce expressions which cause a lot of backtracking which will degrade performance exponentially. So yes, your concern is very valid.

Would it help to store the filter result to the db as well e.g. as a bool filtered:

• NULL => article has not been filtered yet
• true => article has been filtered and should not be provided to the clients
• false => article has been filtered and should be provided to the clients

This might prevent having to filter on every sync. (However, it's quite likely I'm missing some things due to my lack knowledge concerning the actual implementation.)

[David-Development]

OK so every time your client fetches the feed we need to run a regex over every item that is not already in the DB.
Now we do that for all feeds and for every user.

I see where you are going with this.. It's a valid point 👍 I was just concerned about storage space and database speed if we have thousands of filtered items in there. But I guess the server is not running on a smartphone using SQLite so this shouldn't be an issue.
I agree with @svoop that it sounds like a good idea to have a filtered attribute in the database so that you only fetch articles once (and maybe mark them as read..)? I'm not sure how the server side is handling things.. Do you ever clean up old articles? (Read articles that are not starred?) Or are unread articles stored forever on the server?

[anoymouserver]

The simplest implementation would mark the filtered feed items as read. This means that the user still has the option of viewing the articles and it also remains compatible with all existing clients.

Old items are purged if they are read and not starred. You could define 'Maximum read count per feed' in the admin settings, this defines how many items are kept (there are at least the number of items of the last fetch kept to avoid refetching).

[Grotax]

@anoymouserver cool idea we could call that "Mark automatically as read"

I basically do the same with e-mails sometimes.

[LivInTheLookingGlass]

@Grotax AntennaPod has a similar mechanism on a per-feed basis. If an episode matches some pattern, you can use that to include or exclude an episode from autodownload

[Grotax]

Yea true, but Antennapod then has the attribute downloaded, which is either true or false. That can be easily visualised for the user and if the user wants to overrule the decision the download button is right there.

So it doesn't really help in our case, automatically marking them as read seems to be the best option for me so far, but of course someone would needs to try and implement it.

[ThomasSteinbach]

It would be really useful having "filters" for feeds. Following example: The Gitlab Blog https://about.gitlab.com/blog/

For example I am just interested in Releases. On the website I simply can choose the category "releases" on the left. However the website only provides one feed stream: https://about.gitlab.com/atom.xml In this stream are all messages contained.

It would be really helpful if in the Nextcloud News app it would be possible to filter either title or content by search words.

A simple "contains string" filter would suffice for the beginning. A regex filter would be more flexible - and maybe not harder to implement.

[David-Development]

I think it's an important point that those rules would also apply to the content of the article. A customer reached out to me requesting this feature (filtering for news in his area). So he'd probably write some regexp that matches either the url, title or content for villages and cities around him. And as @mejo- already mentioned early on - this would also be very helpful for journalists if they can filter for keyword in the articles itself.

[temporarium]

Just wanted to keep this idea going.
I think post-processing a feed by marking as read items that match a rule would be the easiest solution. Simple rules for title and body, so as not to complicate matters.

[chylex]

The Cloudflare blog includes tags in the form of multiple <category> elements. It would be useful to be able to exclude (or include only) a selection of categories.

[mathisdt]

Some of the above requirements are handled by a small Python script which uses the News API. It can apply regular expressions to body and title and check for the age of a post, and if all defined filters match, a post is marked as read.

More information is in the readme file in the repository: https://github.com/mathisdt/nextcloud-news-filter

[codegax]

Maybe for the UI instead of having a regex string could be something like:

Ignore articles that have the following words in their title: [Football, iOS, Elon Musk]

[svoop]

You can just as easily have both:

• Football, iOS, Elon Musk -> simple keywords (split at ,\s* and match case insensitively)
• /macOS (10|11|12)/i -> match against this regexp (easily identified by the leading /)

Both require nothing but a text input, the latter looks quirkier, but hey, it's aimed at power users.

[dredshep]

Is there any good way of doing this nowadays from the UI? Is there any plug-in for this? Is the python script above something that has to be run server-side every time I want to add a new filter?

[mathisdt]

I can only speak regarding the Python script - it has to be run anywhere (but with network access to the Nextcloud server) each time the filter should be applied, not only when the filters should be changed. I have a small server at home which runs the script every 5 minutes. It can also be run directly on the Nextcloud server if the infrastructure supports Python and Cron.