Log when paths are rejected because of robots.txt #27
Description
Currently, if a path is not fetched because it is excluded by means of a robots.txt file, the user is not notified of that fact. That makes it non-obvious why APE doesn't spider the entire site.
It would be useful if rejected paths would be logged, either at the default level or with a single --verbose option. Maybe log each rejected path with --verbose and only log a count at the default level: that would at least hint the user that there is something going on that --verbose would provide more details about.