Scrapy: grabbing sibling elements of a regex match
Scrapy: grabbing sibling elements of a regex match I'm using Scrapy to scrape college essay topics from college websites. I know how to match a keyword using a regular expression, but the information that I really want is the other elements in the same div as the match. The Response.css(...).re(...) function in Scrapy returns a string. Is there any way to navigate to the parent div of the regex match? Example: https://admissions.utexas.edu/apply/freshman-admission#fndtn-freshman-admission-essay-topics. On the above page, I can match the essay topics h1 using: response.css("*::text").re("Essay Topics"). However, I can't figure out a way to grab the 2 actual essay topics in the same div under Topic A and Topic N. 1 Answer 1 That's not the right way to do it. You should use something like below response.xpath("//div[@id='freshman-admission-essay-topics']//h5...