Search and lists: add exclude BT Strict

[JimKillock]

I'm slightly worried this is inflating our block lists.

A feature to exclude BT Strict from search would be helpful

[dantheta]

If it's useful, I can remove URLs that are only blocked on BT-Strict from existing lists.

[alexhaydock]

I'll let @JimKillock reply on whether that's a good idea, but if you do that are you able to leave the two lists relating to School websites intact?

https://www.blocked.org.uk/list/Schools
https://www.blocked.org.uk/list/sch.uk

As BT-Strict and TalkTalk Kidsafe are intended for young children I don't think it necessarily invalidates our conclusion if it's mostly these "more advanced" filters that are filtering school websites.

[JimKillock]

Excluding existing results sounds reasonable to me @dantheta

[JimKillock]

Did we do this?

[dantheta]

Not yet, but it's a subset of the sort by networks in search ticket.

…

On 18 March 2019 19:46:12 GMT, JimKillock ***@***.***> wrote: Did we do this? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #239 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[dantheta]

I've created a list of the sites blocked only on BT-Strict.

https://docs.google.com/spreadsheets/d/1LHLVQj7ozERmqbrf7ggORZ0t99_LyLnYdmy2JO1q9SA/

Let me know if you're happy for me to go ahead with the removal (apart from the lists Alex specified).

[JimKillock]

@edjw We will need to think about this carefully given where we are at now. We should do it, but it would change the report figures quite a bit.

[JimKillock]

Thanks, I think we're likely to want to make the calculations before we do anything. If we can make the calculations non-destructively via the filtering, that's the best option.

Note, the original request was for a filter rather than a sweep! Maybe this can be covered in the other ticket?

[edjw]

@dantheta - If possible, could you set up this filtering functionality today so we can look at the numbers by tomorrow morning (Friday)?

[dantheta]

Exclude BT-Strict filter is now available on the saved lists summary page. Filtering to just one ISP is available on the lists summary page, the list contents page and the search engine.

The bt-strict exclude is cached, because the query takes quite a while to run.

[dantheta]

Exclude filter is also now available on the list items page. Not sure it makes quite as much sense there, as items which are not blocked on BT-Strict and items which are blocked on networks other than BT-Strict are still shown.

[dantheta]

This one got a little bit complicated - please let me know if there's anything conspicuously missing.

The search network exclusion is a bit trickier; if a negative critera ('BT-Strict' in blocking networks) is applied, it will filter out any URLs which are blocked on BT-Strict. What we'd really want it to do is include all URLs except those which are blocked only on BT-Strict. In the meantime, the search page supports the exclusive option: show results which are blocked on the selected network.

[JimKillock]

Yes I see what you mean. I think the answer would be to have a positive filter, for the same result, eg: Virgin + BT Light filter + Sky + TT MNOs Virgin + BT Light filter + Sky + TT + MNOs

[JimKillock]

The filters we have are great: could we have a final filter, for both fixed and mobile, but not BT Strict? This would just save time selecting the options one by one.

[dantheta]

I had one more idea too - "invert", to invert the selected filter. Using exclude was performance poison (due to not using an index and the creation of a massive temporary table), but having the frontend manipulate the list works really quickly.

[JimKillock]

I don't mind how it is done, but the logic has to reflect that we only exclude sites where BT Strict is the only block :)

[edjw]

@dantheta – What's the status of this issue Dan?

[dantheta]

Excluding BT-Strict is possible on each of the lists pages (like https://www.blocked.org.uk/list/2019_report_all_lqbtq?status=&exclude=1&network=BT-Strict).

Excluding ISPs in keyword search isn't currently exposed in the frontend, but it is pretty much the same piece of work as openrightsgroup/blocked-org-uk#383.

[dantheta]

The search index has been cleared of entries which are blocked on bt-strict only, and is now updating to the new format which includes TLD.

next-site suggestions based on category should be clear of BT-Strict only as well, but we'll have to see how well that's working with a few tests.