Crawling Local Files With Scrapy Without An Active Project?
Is it possible to crawl local files with Scrapy 0.18.4 without having an active project? I've seen this answer and it looks promising, but to use the crawl command you need a proje
Solution 1:
As an option, you can run Scrapy from a script, here is a self-contained example script and the overview of the approach used.
This doesn't mean you have to put everything in one file. You can still have spider.py
, items.py
, pipelines.py
- just import them correctly in the script you start crawling from.
Post a Comment for "Crawling Local Files With Scrapy Without An Active Project?"