robots.txt helper

Build or test robots.txt rules against a path. Reminder: robots.txt is not access control and does not replace noindex.