r/selenium

I have several functions to scrap classifieds web pages with Selenium. For some of them I noticed that at some point the function finds no ada but also doesn‘t detect the element corresponding to a search without results. I assume this means the web is blocking my scraper.

I tried implementing a loop-count with an increasing time wait up to 3 minutes between attempts with no success.

What methods do you use to avoid this?

I’m running headless, in case that matters.

weve been running our own selenium grid for like 4 years and it used to be fine

last sprint half the suite died because two nodes ran out of disk and nobody noticed until monday. spent most of tuesday just bringing the grid back up instead of actually testing anything

looking at what other qa teams moved to when they got tired of owning the browser infra. what stuck for you​