How do you avoid scraping the same page twice?
We mostly scrape job websites and crawl job postings every 3 hours. A volume of about 1M jobs per month.
My issue is that I need to fetch individual job pages only when they change. For example, once a job has been published, ATSs usually don't republish a new page when the job is closed. Instead, they simply update the expiration date, status, or other details. So instead of searching for new pages, I need to re-fetch the same ones from time to time, but it's very expensive, and I have no idea how to choose which ones are worth a second check.
The approach I'm using now is that we check, in order:
i) jobs with the closest expiration date,
ii) jobs that haven't been crawled for the longest time, and
iii) companies that historically update their job postings more frequently.
But still, we're not getting the right data at the right time. We usually detect those updates after 4–5 days, which is not sustainable. We need fresh jobs as soon as they change so applicants always see up-to-date listings.
Is there a way to monitor whether a webpage has changed without re-scraping it every time? Or is there a better approach I am ignoring?