Nrthugu

I'm trying to scrape https://sherwin.com.ar/hagalo-usted-mismo/Donde-compro and i want to retrieve every paint store in every district from every state. That is a dynamic element on the website, and i can't get any data out of it.

You have to select a State from a dropdown list, and then a district from another dropdown list. Then you have to submit the options with the button down bellow. After all that the data is displayed in a list beside a "map" (that it doesn't apear, don't know why). I need to get that data. But i have no idea what to do. The code i have at this moment is:

from selenium import webdriver

from selenium.webdriver.support.ui import Select

from selenium.webdriver.support.ui import WebDriverWait



driver = webdriver.Chrome()

driver.get('https://sherwin.com.ar/hagalo-usted-mismo/Donde-compro')



provincias = driver.find_elements_by_xpath('//*[@id="ddlState"]/option')

provinciaSelect = Select(driver.find_element_by_xpath('//*[@id="ddlState"]'))

localidadSelect = Select(driver.find_element_by_xpath('//*[@id="ddlLocation"]'))

driver.implicitly_wait(2)



buscarPinturerias = driver.find_element_by_css_selector('#search_paint_shops')



for i in range(1,len(provincias)):

    provinciaSelect.select_by_index(i)

    driver.implicitly_wait(2)

    for j in range(1, len(localidadSelect.options)):

        localidadSelect.select_by_index(j)

        print(i,j)

I am getting this error

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

That's how far i could get. I know i have to Submit the result and wait until the data apear. I don't know how to do that

UPDATE

I've updated the code to this:

from selenium import webdriver

from selenium.webdriver.support.ui import Select

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.common.by import By

from selenium.common.exceptions import StaleElementReferenceException

from selenium.webdriver.support import expected_conditions as EC



driver = webdriver.Chrome('')

driver.get('https://sherwin.com.ar/hagalo-usted-mismo/Donde-compro')



# all options of the dropdown list

provincias = driver.find_elements_by_xpath('//*[@id="ddlState"]/option')

provinciaSelect = Select(driver.find_element_by_xpath('//*[@id="ddlState"]'))

localidadSelect = Select(driver.find_element_by_xpath('//*[@id="ddlLocation"]'))

driver.implicitly_wait(2)



buscarPinturerias = driver.find_element_by_css_selector('#search_paint_shops')



for i in range(1, len(provincias)):



    provinciaSelect.select_by_index(i)

    elemLoc = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="ddlLocation"]/option[2]' )))

    # to know in which state i am

    print(i)



    for j in range(1, len(localidadSelect.options)):



        localidadSelect.select_by_index(j)

        buscarPinturerias.submit()

        elemRes = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="map_info"]/div' )))

        resBlock = driver.find_element_by_xpath('//*[@id="map_info"]/div')

        dataBlock = resBlock.find_elements_by_tag_name('li')



        for l in dataBlock:

            # I have the error here now

            print(l.text.encode('utf-8'))

Now the problem is that if i make

print(l.text.encode('utf-8'))

it raises a

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

is it because i don't wait enough?