I created the same thing a year ago. I underestimated the CPU load and number of pages on many sites (especially e-commerce sites). It doesn't take many crawls to quickly fill a db.
I created the same thing a year ago. I underestimated the CPU load and number of pages on many sites (especially e-commerce sites). It doesn't take many crawls to quickly fill a db.