Supercrawler was a Node.js web crawler. It was designed to be highly configurable and easy to use.
When Supercrawler successfully crawled a page (which could have been an image, a text document or any other file), it would fire your custom content-type handlers. It defined your own custom handlers to parse pages, save data and do anything else you need.
On December 17th 2020 SuperCrawler.com merged with Worio.com