u/tropical_vortex

• Know your workload.
• Before building the feature, estimate input tokens, output tokens, expected concurrency, and whether the user needs an instant response or can tolerate asynchronous processing.
• Reduce tokens.
• Do not send full context because it is convenient. Compress, retrieve, summarize, and preserve provenance.
• Embrace parallelism.
• If the work is independent, split it. File scans, scan/offset based analysis, artifact classification, and output candidate often parallelize well.
• Microservices and queues add complexity, but they also let different stages scale, retry, and fail independently. Don't overoptimize.
• Expect failures.

LLM APIs fail. Providers rate-limit. Responses violate schema. Tool calls hang. Sandboxes break. Repos have bad tests. Treat every model call like a network call to a flaky dependency / data source, because that is what it is.

• Know your workload.
• Before building the feature, estimate input tokens, output tokens, expected concurrency, and whether the user needs an instant response or can tolerate asynchronous processing.
• Reduce tokens.
• Do not send full context because it is convenient. Compress, retrieve, summarize, and preserve provenance.
• Embrace parallelism.
• If the work is independent, split it. File scans, scan/offset based analysis, artifact classification, and output candidate often parallelize well.
• Microservices and queues add complexity, but they also let different stages scale, retry, and fail independently. Don't overoptimize.
• Expect failures.

LLM APIs fail. Providers rate-limit. Responses violate schema. Tool calls hang. Sandboxes break. Repos have bad tests. Treat every model call like a network call to a flaky dependency / data source, because that is what it is.