Selenium Printing Same Information Repeatedly

Hello I am trying to scrape some data from a website that has data in its 'dl' tag here is how the website structure looks

&l

Solution 1:

driver.find_element_by_tag_name('dl') will always return the first matching element. You need to use element to locate the <dl>s

for element in driver.find_elements_by_class_name('ThatsThem-record-overview'):
    dl = element.find_element_by_tag_name('dl') # scraping data under dl tag print(dl.text)

Or just locate those elements directly

for element in driver.find_elements_by_css_selector('.ThatsThem-record-overview dl'):
    print(element.text)

Solution 2:

Seems you were close. Using the classrecord-overview should have fetched you all the required data. However it would be better to target the individual name and email by traversing to the child tags. Additionally inducing WebDriverWait will optimize your program performance.

So, ideally you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR:

  names[] = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.record-overview>h2>span")))]
  emails[] = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.record-overview dl.row dd")))]
  for name, email in zip(names, emails):
      print("{} Email is {}".format(name, email))
  

• Using XPATH:

  names[] = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class, 'record-overview')]/h2/span")))]
  emails[] = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class, 'record-overview')]//dl[@class='row']//dd")))]
  for name, email in zip(names, emails):
      print("{} Email is {}".format(name, email))
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.uiimportWebDriverWaitfrom selenium.webdriver.common.byimportByfrom selenium.webdriver.supportimport expected_conditions asEC

Share