Scraping Css Values Using Scrapy Framework

Is there a way to scrap css values while scraping using python scrapy framework or by using php scraping. any help will be appreaciated

Solution 1:

scrapy.Selector allows you to use xpath to extract properties of HTML elements including CSS.

e.g. https://github.com/okfde/odm-datenerfassung/blob/master/crawl/dirbot/spiders/data.py#L83

(look around that code for how it fits into an entire scrapy spider)

If you don't need web crawling and just html parsing, you can use xpath directly from lxml in python. Another example:

https://github.com/codeformunich/feinstaubbot/blob/master/feinstaubbot.py

Finally, to get at the css from xpath I only know how to do it via css=element.attrib['style'] - this gives you everything inside of the style attribute which you further split by e.g. css.split(';') and then each of those by ':'.

It wouldn't surprise me if someone has a better suggestion. A little knowledge is enough to do a lot of scraping and that's how I would approach it based on previous projects.

Solution 2:

Yes, please check the documentation for selectors basically you've two methods response.xpath() for xpath and response.css() for css selectors. For example, to get a title's text you could do any of the following:

response.xpath('//title/text()').extract_first()
response.css('title::text').extract_first()