Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> 403 is generally a bad way to get crawlers to go away

Hardly... the article links says that a 403 will cause Google to stop crawling and remove content... that's the desired outcome.

I'm not trying to rate limit, I'm telling them to go away.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: