6/31 of our scrapers died unnoticed. How many of yours are dead?
I found out on Friday that 6 of our 31 scrapers have been writing zero rows for somewhere between two weeks and four months.
And they do not show any slight error nor do they give an alert. they run on schedule but come back with nothing and exit clean and i had three separate people look at a flat line on a dashboard and quietly decide that was just what the data looked like now.
So my python web scraping setup is roughly 60 percent functional and i have spent most of a year telling people it was fine. i felt sick for about an hour,
What I cannot work out is whether the fix is putting the fragile ones behind a web scraping api, or whether i would just be paying to move the silence somewhere I can see even less of it.
Give me two numbers. How many you run, and how many you would bet your own money are working right now. Be honest about the second one, I was not.