Techcrunch Incremental Scrapy Spider With MongoDB

This project is the support for this blog post : Incremental crawler with Scrapy and MongoDB

Local setup

pip3 install -r requirements.txt

Setup a local MongoDB server. On Mac OS X :

brew install mongodb
brew services start mongodb

scrapy crawl techcrunch -a limit_pages=2

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
tc_scraper		tc_scraper
.gitignore		.gitignore
README.md		README.md
requirements.txt		requirements.txt
scrapy.cfg		scrapy.cfg