Check if URL exists and if it does not, turn off forwarding.

[AramZS]

No description provided.

[boonebgorges]

A good idea in the abstract. But when would the check take place? At the time of item creation + promotion, it's highly likely that the URL will resolve properly. If the source page goes away, it's likely to happen in the future. So we'd need some mechanism for verifying URL resolution at a time other than initial creation/promotion. Two ideas come to mind:

1. A regularly-scheduled "health check". Once every week or month or whatever, PF would ping the URL of each item with forwarding turned on.
2. Just-in-time checking. Each time a forward is performed, do a parallel check to ensure that the URL still resolves. See

   pressforward/includes/functions.php

   Line 1037 in f0d4731

   $wait = get_option( 'pf_link_to_source', 0 );

Regarding option 1, we couldn't do it all in a single cron job, because it would time out with just a couple 404s. A single cron job for each URL is probably easier to build than a batch process, though it could theoretically result in hundreds of separate cron jobs. The real downside here is that the frequency is all wrong: for PF items that receive little to no traffic, you're checking too frequently; while for items that are visited frequently, if the source URL breaks, you might have days or weeks between the breakage and PF's health check.

So, option 2 seems a bit better. To ensure 100% coverage, we would have to do a check with every single forward event. For busy PF items, this is too much: it slows down the forwarding process, and could get PF sites banned for creating unwanted traffic. So we probably need a reasonable throttle, say once per 6 or 12 or 24 hours. This still creates the possibility of a gap between the URL going down and PF recognizing the fact, but it'll be fairly minimal.

Other considerations:

• We would presumably need some sort of admin-facing messaging. At minimum, something at Dashboard > Posts that indicates that PF turned off forwarding for a given item. I could imagine an admin even wanting email notifications for such an event, though this would probably need some configuration options. In any case, PF admins who have turned on forward_to_origin don't intend the single-item template for that given item to be visible, so when it becomes visible without any admin intervention, there needs to be some indication of the fact.
• Source sites can go down temporarily. If a source site experiences an outage, and PF happens to check the URL during the outage, forwarding will be turned off for that PF item. Should PF continue to check the URL periodically to see if it becomes available? My sense is that we probably should, though it could be very occasional - say, once per week or once per month. This observation also indicates to me that we need to be saving a couple different pieces of info when we note that a URL doesn't resolve: the date on which forwarding was disabled, and the date when the URL was last checked; if we are re-checking non-resolving URLs, then these dates will be different, which is information we'll probably want to expose to the admin.