u/shalevbs

For the past 2 hours, our Karpenter setup hasn't been able to create new nodes in us-east-1.

We are consistently hitting AWS API rate limits and getting 503 errors. Other regions are working perfectly as intended. Is anyone else hitting this right now, or did AWS change something under the hood?

The errors we're seeing:

launching nodeclaim, creating instance, creating nodeclaim, getting launch template configs, getting launch templates, describing launch templates, operation error EC2: DescribeLaunchTemplates, https response error StatusCode: 503, RequestID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, api error RequestLimitExceeded: Request limit exceeded.

failed to get rate limit token, retry quota exceeded, 1 available, 5 requested

edit: they just updated their service health page:

Increased Launch Template API Error Rates
Jul 06 5:45 AM PDT We are investigating increased error rates when calling EC2 Launch Template APIs in US-EAST-1 Region. During this time, affected customers may experience errors when creating, modifying, or referencing launch templates. Other AWS services that rely on launch templates may also be impacted. We will provide another update by 6:30 AM PDT or sooner, if we have additional information to share.