Scrapy: grabbing sibling elements of a regex match
Scrapy: grabbing sibling elements of a regex match
I'm using Scrapy to scrape college essay topics from college websites. I know how to match a keyword using a regular expression, but the information that I really want is the other elements in the same div as the match. The Response.css(...).re(...) function in Scrapy returns a string. Is there any way to navigate to the parent div of the regex match?
Example: https://admissions.utexas.edu/apply/freshman-admission#fndtn-freshman-admission-essay-topics. On the above page, I can match the essay topics h1 using: response.css("*::text").re("Essay Topics"). However, I can't figure out a way to grab the 2 actual essay topics in the same div under Topic A and Topic N.
1 Answer
1
That's not the right way to do it. You should use something like below
response.xpath("//div[@id='freshman-admission-essay-topics']//h5//text()").extract()
In case you just want css then you can use
In [7]: response.css("#freshman-admission-essay-topics h5::text, #freshman-admission-essay-topics h5 span::text").extract()
Out[7]: ['Topic A xa0xa0', 'Topic N']
How can it be generic and extract something specific at the same time?
– Tarun Lalwani
2 days ago
Let’s say I want to extract all Li’s in the same div as the regex “Essay Topics”. I could do that if re.compile didn’t return a string but the html element itself. Is there no way to do that?
– Eric L
yesterday
You will still need to use xpath as such. Something like
response.xpath("//li[contains(text(), 'Essay Topics')]/..//text()").extract()– Tarun Lalwani
20 hours ago
response.xpath("//li[contains(text(), 'Essay Topics')]/..//text()").extract()
Great! Thank you so much. Would I be able to do this in CSS as well since that's what I'm more comfortable with?
– Eric L
1 hour ago
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
Thanks for your point. I understand the way you laid out, but I was hoping for a more generalizable parsing approach so I could scrape multiple sites using the same scheme. Is there no way to do something like soup.find_all(href=re.compile("elsie")) # [<a class="sister" href="example.com/elsie" id="link1">Elsie</a>] in beautiful soup where you get the tag instead of the string match?
– Eric L
Jun 30 at 17:13