r/MongoDB_Official

Ask any AI how to paginate a MongoDB collection and you get $skip. Every single time, unless you put the word keyset in the prompt.

I ran explain() on a million product documents to see what that actually costs. Page 500 examines 10,000 documents to return 20. Page 5000 examines 100,000. The keyset version examines 20 no matter which page you ask for.

The labels below are the prompts I gave the AI to generate each block.

How to paginate MongoDB query results using skip and limit by page number.

Bad:

const PAGE = 2;
const PAGE_SIZE = 20;
const RESULTS = await db.collection('products')
  .find({})
  .skip(PAGE * PAGE_SIZE)
  .limit(PAGE_SIZE)
  .toArray();

How to implement MongoDB keyset pagination using last seen document id with aggregation pipeline.

Good:

const PAGE_SIZE = 20;
let LAST_SEEN_ID = null;
try {
  const RESULTS = await db.collection('products').aggregate([
    ...(LAST_SEEN_ID ? [{$match:{_id:{$gt:LAST_SEEN_ID}}}] : []),
    {$sort:{_id:1}},
    {$limit:PAGE_SIZE},
    {$project:{name:1,sku:1,category:1,main_image:1}}
  ]).toArray();
  LAST_SEEN_ID = RESULTS.at(-1)?._id ?? LAST_SEEN_ID;
} catch (e) {
  console.error(e.message);
}

How to build MongoDB pagination with page cache supporting forward, backward, and direct page jumps.

Perfect:

const PAGE_SIZE = 20;
const PAGE_CACHE = new Map();

async function getPage(pageNum) {
  if (pageNum < 1) throw new Error('pageNum must be >= 1');

  const prevPage = PAGE_CACHE.get(pageNum - 1);
  const currPage = PAGE_CACHE.get(pageNum);

  const seek = currPage?.firstId ? {$match:{_id:{$gte:currPage.firstId}}}
              : prevPage?.lastSeenId ? {$match:{_id:{$gt:prevPage.lastSeenId}}}
              : pageNum>1 ? {$skip:(pageNum-1)*PAGE_SIZE}
              : null;

  const raw = await db.collection('products').aggregate([
    seek,
    {$sort:{_id:1}},
    {$limit:PAGE_SIZE + 1 },
    {$project:{name:1,sku:1,category:1,main_image:1}}
  ].filter(Boolean)).toArray();

  const hasNext = raw.length > PAGE_SIZE;
  const results = hasNext ? raw.slice(0, PAGE_SIZE) : raw;

  if (results.length > 0) {
    PAGE_CACHE.set(pageNum, {firstId:results[0]._id,lastSeenId:results.at(-1)._id});
  }

  return {results,hasPrev:pageNum > 1,hasNext};
}

try {
  const [TOTAL_DOCUMENTS, P1] = await Promise.all([
    db.collection('products').estimatedDocumentCount(),
    getPage(1)
  ]);
  const TOTAL_PAGES = Math.ceil(TOTAL_DOCUMENTS / PAGE_SIZE);
} catch (e) {
  console.error(e.message);
}

Bad is 0-indexed, so PAGE = 2 actually hands you the third page. Perfect counts from 1. AI flips between the two without ever telling you which one it picked.

$skip scans and discards every document before your page. It also breaks under concurrent writes. A new document inserted on page 2 shifts everything after it, so page 3 shows the same document twice or skips one entirely, and nothing errors.

Good is keyset. Constant time wherever you are, but forward only. Perfect adds a page cache so you can go forward, backward, and jump straight to any page number. It still falls back to $skip for a cold jump, then caches that position on the way through so it never pays for it twice.

The cache is the same LAST_SEEN_ID from the Good example, stored per page instead of in one variable. One variable only remembers where you stopped, which is the whole reason Good can't go backward. Remember the first and last _id of every page you've been to and you can land on any of them directly.

Keyset is only fast if the field you sort on is indexed. _id is indexed automatically and that index can't be dropped, so the examples above need no setup at all. Point the same pattern at created_at without adding an index and you get a COLLSCAN, 10,020 documents examined instead of 20. You moved the scan, you didn't remove it. A single field index works in both sort directions, while a compound index has to match the sort direction on every field or be its exact inverse.

getPage isn't just paginating, it's fetching what the screen renders. A product grid needs a name, a sku, a category and a thumbnail. It does not need the description, the variants array, the spec sheet or the eight other image URLs. A page of 20 full product documents came back at 45,324 bytes on my test catalog. Projected down to those four fields, 3,362. That is what crosses the network and sits in your app memory on every request, and AI hands you the whole document every time.

Put $project after $limit so you only shape the 20 documents you're keeping. _id comes back whether you list it or not, which matters because the keyset needs it.

Sorting by _id means sorting by creation time, because an ObjectId starts with a 4 byte timestamp. You can pull it back out with _id.getTimestamp(). Keep a created_at field anyway, querying by Date beats building an ObjectId every time you need a range. ObjectIds made in the same second on different servers have no guaranteed order between them either.

estimatedDocumentCount() is called estimated for a reason. It reads collection metadata instead of counting, which is why it came back in 1.3ms where countDocuments({}) took 202ms on the same million documents. It drifts after an unclean shutdown and it counts orphans on a sharded cluster. Fine for a page count, not for anything that has to be exact.

AI almost always reaches for toArray() before doing any work on your data. Most training examples are out-of-context snippets, so it doesn't know better. toArray() holds your entire result set in RAM before you can touch a single document. With a cursor, the driver fetches in batches. Processed documents get GC'd while the rest stream in.

How to fetch all active users from MongoDB and send emails using find and toArray.

Bad

            const users = await db.collection('users')
              .find({ active: true })
              .toArray();
            
            users.forEach(async (user) => {
              await sendEmail(user);
            });

How to stream MongoDB documents with a cursor using for await to process each document without loading all into memory.

Good:

            const cursor = db.collection('users').aggregate([
              { $match: { active: true } }
            ]);
    
            for await (const user of cursor) {
              await sendEmail(user);
            }

How to process MongoDB cursor results concurrently with a concurrency limit without blocking the async loop.

Perfect:

            function executor(limit) {
              let running = 0
              const queue = []
              const flush = () => {
                while (running < limit && queue.length) {
                  running++
                  queue.shift()().finally(() => { running--; flush() })
                }
              }
              return fn => { queue.push(fn); flush() }
            }
    
            const add = executor(10);
            const cursor = db.collection('users').aggregate([
              { $match: { active: true } }
            ]);
    
            for await (const user of cursor) {
              add(() => sendEmail(user))
            };

Bad loads everything into RAM then serializes. Good streams documents but still sends one email at a time. Perfect streams AND fires up to 10 emails concurrently without the loop ever waiting.

Bonus: A cursor with for await only makes sense when you're doing work per document. If you're just collecting into an array to send a response, use .toArray() directly. Wrapping .toArray() in a for await loop buys you nothing.

AI puts this in every route file it touches:

    const client = new MongoClient(uri);
    await client.connect();
    // query
    await client.close();

Looks clean. Problem is a MongoClient is a POOL, up to 100 connections. A free Atlas cluster takes 500 total. Five of these alive at once and your whole cluster budget is gone before one query runs. Then everyone blames Mongo for "randomly dying under load".

One client for the whole app. Create it once, import it everywhere. The driver does the pooling, that is literally its job.

Bonus AI also never gets right: Next.js dev SSR mode re-runs module scope on every hot reload. Your correct singleton turns into a new pool every time you hit save. Cache it on globalThis and it survives reloads:

    let client = globalThis._mongoClient;
    if (!client) {
      client = new MongoClient(process.env.MONGODB_URI);
      globalThis._mongoClient = client;
    }

Costs nothing in production, saves your dev cluster.

Grep your codebase for "new MongoClient". If you have more than one, you have a problem.

Make a module you import, and reuse so every script imports the same active connection.

    // db.js
    import { MongoClient } from 'mongodb';

    export const client = new MongoClient(process.env.MONGODB_URI, {
      appName: 'my-api',
    });

    export const db = client.db('app');

We have a pile. Upvote if you want it, and drop the worst Mongo query an AI ever handed you.

First of all. I want to thank Mongodb for just existing. If not for them I wouldn't be a business owner

I used Mongodb as my db to build multiple web apps since 2021 and one of them eventually took off & became profitable

That allowed me to quit my job & become a full time founder. Mongodb is super easy to setup if you are starting anything :)

Would love to know others in this community!

i've been using mongodb for years now and honestly, i've learned a lot of things the hard way 💀

when i first started, i thought mongodb was basically just:

"throw some json in there and you're good"

yeah... no lmao

over time i've screwed up schemas, made terrible queries, overused indexes, duplicated way too much data, and made collections that looked fine at first but became a pain later.

some of the stuff i wish i knew earlier

• when to embed vs reference
• how indexes can actually hurt you
• why "mongodb is schema-less" doesn't mean "no schema needed"
• how to structure collections without making future me hate present me
• why some queries are fine with 1k documents but absolutely awful with millions
• things i'd do differently if i started a new project today

figured i'd share what i've learned over the years since i'm probably not the only one who learned mongodb by breaking shit first 😂

Hey everyone, solo dev here. Just launched StudySprout after a few months of building.

The problem I kept running into: I'd have a folder full of PDFs and scattered notes with no real system for actually retaining what was in them. Note apps are good at storage, bad at retrieval. Anki's good at flashcards, bad at organizing source material. So I built something that tries to close that gap.

What it does:

• Upload a PDF → it gets auto-parsed into structured, topic-scoped pages (custom parser, not just raw text dumping)
• Or write notes directly in a block-based editor (Notion-style)
• Either way, it builds a concept graph across your files, detects prerequisite relationships (e.g. "learn Linear Algebra before Quantum Mechanics"), and generates AI flashcards with spaced repetition
• Real-time collaboration — multiple people can be in the same file, live-editing together (built on Yjs CRDTs + Socket.io)
• Shared workspaces so a study group can work off the same folders and flashcard sets

Tech stack: Next.js, TypeScript, MongoDB, Redis/BullMQ for background jobs, a separate Java rate-limiter service in front of the Gemini API, deployed across Vercel + Railway + Render.

Live demo: https://studysprouts.in/

Would genuinely love feedback — especially from anyone who's tried combining note-taking + spaced repetition before and hit walls with existing tools. What's broken, what's confusing, what's missing?

https://reddit.com/link/1vco7w0/video/lesryy0usrgh1/player

Just got a notification that MongoDB new community launch on reddit ...and I am here.

I am happy that they decided to launch this and will be very useful.

mongo db is one of the easiest db to integrate in your projects, in college projects or even my personal projects it is was my first choice..now when I am working with startups my first suggestion for their MVP & SaaS projects is MongoDB.

and I will be always grateful for the free plan they provide, for students for startups for launching projects its a savior..

thanks to the awesome team for building and improving it day by day..

for users who cant afford paid resources its hope..

When a new document is created it takes atleast 2-3 sec for that document to become searchable
Is there a way to fix it or decrease time from mongodb itself