A simple crawler made in JavaScript for Node.
Generates a sitemap for a site.
A really lightweight and simple robots-txt parser.