Handling async PDF generation when clients expect the PDF URL to work immediately
Hello all -
We've been generating PDFs synchronously for the last 5 years, but recently moved to async generation.
Each PDF takes at most around 2 seconds to generate.
The problem is that I can't make a major architecture change right now, so the current flow looks like this:
A service pushes a job into a PDF queue.
Workers pick it up, generate the PDF, and write it to a predictable URL.
Meanwhile, the calling service has already saved that URL in its DB, sent webhooks, etc.
The URL is now being consumed by web, Android, iOS, iPad, external APIs, and webhook clients. Some external developers have built their own apps on top of our APIs, so I can't expect them to implement custom polling logic either.
Since I own the web codebase, I patched that side by opening a small HTML page instead of the PDF URL. It polls until the PDF is ready, then redirects to the actual PDF.
That works for web, but obviously doesn't solve the problem for native clients or third-party integrations.
What I'd ideally like is for the original PDF URL to be usable immediately, even while generation is in progress.
A legit way i found was serving a placeholder PDF that somehow refreshes itself until the actual PDF is ready. I looked into JavaScript inside PDFs and XFA, but haven't found a reliable way to make this work across different PDF viewers. Native clients might use entirely different rendering engines, and I can't assume they'll execute embedded JavaScript or allow network requests.
A hacky way i thought was configuring the CDN/S3 delivery path to return a placeholder PDF when the actual file isn't available yet, with caching disabled. But that introduces extra requests and potentially unnecessary cost, and I'm not convinced it's the right approach.
I'm particularly interested in solutions that don't require changes across every client, since I need to support third-party consumers too. While I'm aligned the ideal approach would have been DB insertion only on pdf generation but it's a 5 year old thing, which I can't patch right now :(