Yes, this was my first thought too - Facebook is essentially denial of servicing people's sites and it sounds like they're aware of it. Move fast and break your things. If this were your average Joe I'm fairly certain something like the DMCA could be used.

What? How does the DCMA apply here at all?

The DMCA has a couple major parts. The part most people are familiar with deals with safe harbor provisions and "DMCA takedowns". The relevant part for this discussion makes it illegal to circumvent digital access-control technology. So you could argue that if you have reasonable measures in place to stop automated access of your content such as robots.txt then Facebook continuing to send automated requests would be a circumvention of your access controls.

Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it?

(I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated requests, for things like "link unfurling" in things like Slack. I am guessing the crawler ignores robots.txt because it is doing the request on behalf of a human user, not to create some sort of index. Google is attempting to standardize this widely-understood convention: https://developers.google.com/search/reference/robots_txt)

There is a cost to apply throttling, you still have to receive those requests and filter/block them, plus incoming bandwidth.

The post mentions setting `og:ttl` and replying with HTTP 429 - Too Many Requests, and both being ignored.

Plenty of crawlers, including Facebook's, are operating out of huge pools of IPs usually smeared across multiple blocks. If you have an idea on how to ratelimit a crawler doing 2 req/s from 300+ distinct IPs with random spoofed UAs let me know when your startup launches.

There are many features that you can pick up on. TCP fingerprints, header ordering, etc. You build a reputation for each IP address or block, then block the anomalous requests.

If you don't want to do this yourself and you're already using Cloudflare... congratulations, this is exactly why they exist. Write them a check every month for less than one hour of engineering time, and their knowledge is your knowledge. Your startup will launch on time!

The author specified that the crawler ignores 429 status code. So they do have some rate limit.

I noticed that, but I get the feeling that sending that status code didn't reduce the amount of work the program did, so it's kind of a pointless. You can say "go away", but at some point, you probably have to make them go away. (All RFC6585 says about 429 is that it MUST NOT be cached, not that the user agent can't try the request again immediately. It is very poorly specified.)

The author mentioned that they were using Cloudflare, and I see at least two ways to implement rate limiting. One is to have your application recognize the pattern of malicious behavior, and instruct Cloudflare via its API to block the IP address. Another is to use their rate-limiting that detects and blocks this stuff automatically (so they claim).

Like, there are many problems here. One is a rate limit that doesn't reduce load enough to serve legitimate requests. Another is wanting a caching service to cache a response that "MUST NOT" be cached. And the last is expecting people on the Internet to behave. There are always going to be broken clients, and you probably want the infrastructure in place to flat-out stop replying to TCP SYN packets from broken/malicious networks for a period of time. If the SYN packets themselves are overwhelming, welp, that is exactly why Cloudflare exists ;)

The author is right to be mad at Facebook. Their program is clearly broken. But it's up to you to mitigate your own stuff. This time it's a big company with lots of money that you can sue. The next time it will be some script kiddie in some faraway country with no laws. Your site will be broken for your users in either case, and so it falls on you to mitigate it.

Rate limiting websites is a lot of work. You have to figure out what kind of limits you should set, you have to figure out if that should be on an IP or a /24 or an ASN, if it's an ASN, you have to figure out how to turn an IP into an ASN.

If your website is pretty fast, you probably won't necessarily care about a lot of hits from one source, but if it's supposed to be a small website on inexpensive hosting, and all those hits add up to real transfer numbers, maybe it's an issue.

In the ToS, add a charge of 2 guineas, 5 shilling, a sixpence and a peppercorn per request.

Nah make it a strand of saffron per request and get your moneys' worth out of them.

I saw also of 300reqs per second.