Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If a bot ignores robots.txt that's a paddlin'. Right to the blacklist.


The linked article explains what happens when you block their IP.


For reference:

> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).

It's really absurd that they seem to think this is acceptable.


Block the whole ASN in that case.


What about adding fake sleeps?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: