Downstream rate limiting and concurrency
I'm new to temporal so bear with me - I'm trying to orchestrate a workflow which fans out into highly concurrent activities. Each of these activities invokes a 3rd party API which completes asynchronously. The activity calls the API, returns and then the workflow waits for a signal from an SQS queue that the job is done and the workflow continues.
The API has rate limits and concurrency limits, so I'm using a task queue to abide by rate limits, but I'm struggling to know how to solve the concurrency in a "temporal native" way. Right now I'm essentially keeping a ledger in RDS of invocations and when I get the async result I mark that given invocation as complete. Before I push to the task queue I check how many in progress invocations there are and if theres no capacity the workflow waits. This feels like I'm doing something wrong but as far as I can tell there's no clean way to do this native to temporal? The other thing I'd considered was whether I could have the activity wait for completion before pulling from the queue but I don't believe this is supported.
Any help appreciated!