u/Ok-Difficulty-7607

I run an AI news reader (https://helucino.com — mine, free, no account needed to read) that pulls 64 feeds every 15 minutes. Reading them was easy. The thing I wanted was harder: rank stories by how many separate outlets are covering them. Five news websites independently publishing the same story is definitely a signal that the news is important — and to get it you must first know that four articles are one story. So dedupe was the problem I picked.

Four layers, cheapest first:

1. Canonical URL hashing — kills syndication and tracking-param variants.
2. Exact normalized-title match — free, catches straight reprints.
3. Title-token similarity — Jaccard + containment, but gated on shared named entities. Without that gate "Meta open-sources X" merges with "OpenAI open-sources X", which is definitely a wrong answer.
4. Embeddings to shortlist, then a language model to decide — asked in both directions and these must agree, because one-directional prompting says yes far too easily.

The output is a +N badge per row, and a sort order. Ranked orders purely by that count; Top applies the same count with an 18-hour half-life so recency still matters. A false merge is much worse than a missed one here — merging two stories hides one of them and inflates the other's count — which is why layer 4 has to agree with itself before it will collapse anything, and why complete linkage is checked both ways before two groups join.

Current reality check: 63 of 64 feeds green. The one that isn't is Hacker News, which is a bit on the nose. Genuinely curious how others handle publishers that block feed readers — mirror, back off, or drop them?

Solo dev here 🙏 Trying to clear the 12-tester / 14-day closed test.

Geography Worlds is a set of 34 geography quiz games — flags, world capitals, country shapes, satellite imagery, city skylines and more. 100% free, no ads, no sign-up needed (play as a guest).

Test for test — drop your links in a comment and I'll join your group + opt into your app in return.

1. Join the group:  https://groups.google.com/g/anshul-test
2. Opt in:          https://play.google.com/apps/testing/com.geographyworlds.app
3. Install:         https://play.google.com/store/apps/details?id=com.geographyworlds.app

⚠️  Use the same Google account for all 3 steps + on your phone. Please keep it installed for the 14 days. Any feedback very welcome — thanks! 🙏