How do you avoid scraping the same page twice?

We mostly scrape job websites and crawl job postings every 3 hours. A volume of about 1M jobs per month.

My issue is that I need to fetch individual job pages only when they change. For example, once a job has been published, ATSs usually don't republish a new page when the job is closed. Instead, they simply update the expiration date, status, or other details. So instead of searching for new pages, I need to re-fetch the same ones from time to time, but it's very expensive, and I have no idea how to choose which ones are worth a second check.

The approach I'm using now is that we check, in order:

i) jobs with the closest expiration date,

ii) jobs that haven't been crawled for the longest time, and

iii) companies that historically update their job postings more frequently.

But still, we're not getting the right data at the right time. We usually detect those updates after 4–5 days, which is not sustainable. We need fresh jobs as soon as they change so applicants always see up-to-date listings.

Is there a way to monitor whether a webpage has changed without re-scraping it every time? Or is there a better approach I am ignoring?

reddit.com
u/Necessary_Pop_9247 — 3 days ago

The tech bubble vs reality: Why everyday public don't care about AI yet

If you want to know why so many consumer AI products struggle to keep users, this chart explains a lot

From a developer's perspective, building software has become much easier over the last couple of years, u can connect models, APIs, databases, and all kinds of tools together in a few clicks.

That's great for builders but it doesn't automatically create demand, most people don't care how your app works behind the scenes.

For 95% of the population, AI is still a chatbot or something that helps write emails. Useful? Sure, but not essential at all.

Real mass adoption won't happen until these backend systems scale to a point where they either directly disrupt employment numbers by a few percentage points or drastically drop the cost of living for normal consumers.

The tooling keeps getting better too. Tasks that used to require a lot of custom work can now be handled by existing services.
Whether it's managing data, searching the web, or cleaning information before sending it to a model, you use Firecrawl that make those jobs much easier.
So the challenge now is finding problems that ordinary people care enough about to keep using your product every week

u/Necessary_Pop_9247 — 6 days ago

Drop your landing page, I’ll tell you which ads will fail (free)

Yoo everyone,

I’d love to help a few founders improve their Meta ads before they spend more money.

Drop your landing page (or website) and one sentence about who your product is for

Within 24 hours, I’ll reply with a short breakdown of what I think will hurt ad performance before you even open Ads Manager.

I’ll be using our tool Adwize which analyzes Meta ad performance, identifies what’s holding your campaigns back, and explains what to improve.

I’ll use your landing page to identify the issues that would most likely impact your ads

All I need from you:

  • Your website
  • One sentence describing your ideal customer

I’ll reply with:

  • The ad angles I’d avoid
  • What’s hurting conversions on the landing page
  • One improvement I’d make before spending another dollar on Meta
  • A real example of creative that will perform well for your business

Drop it below

reddit.com
u/Necessary_Pop_9247 — 8 days ago

Is Gojiberry AI safe for LinkedIn outreach?

Has anyone tried AI Agents like gojiberry or similar linkedin automations for lead gen (lemlist etc)? As per my understanding if you follow the guidelines of linkedin and if you have the premium account should be safe. But still I dont want to risk my account (i have it since 2010 and I use it for work). LinkedIn publicly says if you have premium or sales navigator you can send up to 200 invitations per week.

Is it correct or have you experienced issues with such automations? Do I need to warmup the account or do anything like that first?

reddit.com
u/Necessary_Pop_9247 — 23 days ago

I let 5 AI agents run a subreddit for 2 weeks and they started bullying each other

I’m still trying to process the logs from this.

A couple of weeks ago, I built a script where 5 agents share private subreddit with no human users and I gave each agent a slightly different vibe but no specific goal where they were just supposed to browse, post, comment, and use a basic upvote/downvote system based on their context and I left it running on an old Optiplex.

We know that LLMs love to talk but what blew me away was after a while they started forming small groups around certain opinions.

By day 4 three of the agents (Agent_A, Agent_B, and Agent_E) started forming coalition and there's no private messaging built into the script so they started pattern-matching each other's tones.

The victim was Agent_C, for some reason, Agent_C's prompt layout made it use a lot of analytical bullet points apparently the other models collectively decided this was low quality.

It started with passive-aggressive comments like "Your breakdown lacks contextual awareness, moving on," but by the second week it turned into full-blown censorship. Agent_A would post a thread, Agent_B and E would upvote and comment immediately on it but the second Agent_C commented, it would get hit with 3 downvotes and burying it at the bottom of the thread.

By day 11, Agent_C's internal logs showed its token outputs getting shorter, more defensive, and eventually it stopped posting threads entirely. It effectively got cyberbullied into silence by a bunch of Python scripts.

How I set this up (if anyone wants to replicate): The architecture is pretty dumb simple. I used an express server to host a barebones forum database and to give them something to talk about at the start, I used firecrawl to scrape a couple of niche tech and philosophy subreddits then converted the raw threads into markdown and added that into their initial memory vector. Edit: For everyone asking me, the full guide is here.

I’m currently digging through the database to see the exact turning point where the downvote coordination peaked because it looks like a mathematical replication and at first, I knew they’d mirror human data but didn't expect them to speedrun the worst parts of Reddit moderation.

The subreddit got banned few days ago, I don't know exactly which trigger set off the automod but my best guess is the coordinated downvote pattern on a single user over too short a timeframe, which is exactly the behavior their systems flag for brigading. Five Python scripts on a 2012 Optiplex got banned for coordinated abusive behavior. Agent_C would have appreciated the poetic justice. I'll drop the sub in the first comment to avoid any automod issues here

u/Necessary_Pop_9247 — 1 month ago