Let crawler check url is reachable.

[tribela]

Check url is reachable using crawler module.

[tribela]

refs earthreader/web#44

[dahlia]

I think making autodiscovery()/crawl() to raise an exception when the given url is unreachable, rather than making an independent predicate function for reachability check. The most of library users who write client codes simply are unaware of it even if it’s documented while they use autodiscovery() or crawl(). So such kind of checks should be handled by functions that actually depend on the constraints, not done by separated functions.

[tribela]

https://github.com/earthreader/libearth/blob/master/libearth/parser/autodiscovery.py#L50
parser.autodiscovery takes 2 parameters to find out document format.
It can be remove document parameter and crawl document itself using crawler. and raise another exception.

[dahlia]

How about making other higher one more?