Python Click 'more' Button Is Not Working

I tried to click 'More' button for each review so that I can expand these text reviews to the full contents and then I try to scrape those text reviews. Without clicking 'More' but

Solution 1:

Your the biggest mistake in all this code is except: pass. Without this you would resolve problem long time ago. Code raise error message with all information but you can't see it. You could at least use

except Exceptionas ex:
    print(ex)

---

Problem is that move_to_element() will not work with BeautifulSoup elements. I has to be Selenium's element - like

link = driver.find_element_by_link_text('More')

ActionChains(driver).move_to_element(link)

But after executing some functions Selenium needs some time to do it - and Python has to wait awaile.

I don't use BeautifulSoup to get data but if you want to use it then get driver.page_source after clicking all links. Or you will have to get again and again driver.page_source after every click.

Sometimes after clicking you may have to get again even Selenium elements - so I first get entry to click More and later I get partial_entry to get reviews.

I found that clicking More in first review it shows text for all reviews so it doesn't need to click on all More.

Tested with Firefox 69, Linux Mint 19.2, Python 3.7.5, Selenium 3.141

---

#from bs4 import BeautifulSoupfrom selenium import webdriver
from selenium.webdriver import ActionChains
import time

#Incognito Mode
option = webdriver.ChromeOptions()
option.add_argument("--incognito")

#Open Chrome#driver = webdriver.Chrome(executable_path="C:/Users/chromedriver.exe",chrome_options=option)

driver = webdriver.Firefox()

#url I want to visit.
lists = ['https://www.tripadvisor.com/VacationRentalReview-g30196-d6386734-Hot_51st_St_Walk_to_Mueller_2BDR_Modern_sleeps_7-Austin_Texas.html']

for url in lists:

    driver.get(url)
    time.sleep(3)

    link = driver.find_element_by_link_text('More')

    try:
        ActionChains(driver).move_to_element(link)
        time.sleep(1) # time to move to link

        link.click()
        time.sleep(1) # time to update HTMLexcept Exception as ex:
        print(ex)

    description = driver.find_element_by_class_name('vr-overview-Overview__propertyDescription--1lhgd')
    print('--- description ---')
    print(description.text)
    print('--- end ---')

    # first "More" shows text in all reviews - there is no need to search other "More"
    first_entry = driver.find_element_by_class_name('entry')
    more = first_entry.find_element_by_tag_name('span')

    try:
        ActionChains(driver).move_to_element(more)
        time.sleep(1) # time to move to link

        more.click()
        time.sleep(1) # time to update HTMLexcept Exception as ex:
        print(ex)

    all_reviews = driver.find_elements_by_class_name('partial_entry')
    print('all_reviews:', len(all_reviews))

    for i, review inenumerate(all_reviews, 1):
        print('--- review', i, '---')
        print(review.text)
        print('--- end ---')

---

EDIT:

To skip responses I search all class="wrap" and then inside every wrap I search class="partial_entry". I every wrap can be only one review and eventually one response. Review has alwasy index [0]. Some wraps don't keep review so they will gives empty list - and I have to check it before I can get element [0] from list.

all_reviews = driver.find_elements_by_class_name('wrap')
#print('all_reviews:', len(all_reviews))

for review in all_reviews:
    all_entries = review.find_elements_by_class_name('partial_entry')
    if all_entries:
        print('--- review ---')
        print(all_entries[0].text)
        print('--- end ---')