Scraping With Selenium Web Driver

I am scraping urls of this page https://www.betexplorer.com/soccer/russia/premier-league-2019-2020/results/ For each urls like this https://www.betexplorer.com/soccer/russia/premie

Solution 1:

You should fix your XPath expressions. Use findElement for the first 3. findElements for the last.

To get the home odd :

//td[a[.="bet365"]]/following-sibling::td[span][1]/span

To get the draw odd :

//td[a[.="bet365"]]/following-sibling::td[span][2]/span

To get the away odd :

//td[a[.="bet365"]]/following-sibling::td[span][3]/span

To get them all :

//td[a[.="bet365"]]/following-sibling::td[span]/span

Getting them all is probably better since you call driver.find_elements_by_xpath 1 time. For each match, store each element of the result (a list) in your specific lists (append the first in the home odd list, the second in the draw odd list, the third in the away odd list).

EDIT : To get the opening odds, extract the value from the attribute data-opening-odd. XPath :

//td[a[.="bet365"]]/following-sibling::td[span][1]/@data-opening-odd
//td[a[.="bet365"]]/following-sibling::td[span][2]/@data-opening-odd
//td[a[.="bet365"]]/following-sibling::td[span][3]/@data-opening-odd
//td[a[.="bet365"]]/following-sibling::td[span]/@data-opening-odd

Selenium syntax :

awayodd = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//td[a[.="bet365"]]/following-sibling::td[span][3]'))).get_attribute("data-opening-odd")