u/One_Mention_5385

Infra engineer here, trying to figure out when self hosting LLMs actually makes sense vs just paying for the APIs. every blog post says "it depends" lol

If you've done it (or looked into it and bailed), would love to hear:

• how big was your API bill when you started thinking about it? what pushed you over the edge
• what did it really cost once you add up GPUs, idle time, and the engineering hours?
• what was the most painful part? cold starts, autoscaling, OOMs, model quality, getting paged at 2am
• if you went back to APIs, what made you switch back

ballpark numbers are totally fine. I'll put together a cost / decision writeup from the replies and post it back here

Infra engineer here, trying to figure out when self hosting LLMs actually makes sense vs just paying for the APIs. every blog post says "it depends" lol

If you've done it (or looked into it and bailed), would love to hear:

• how big was your API bill when you started thinking about it? what pushed you over the edge
• what did it really cost once you add up GPUs, idle time, and the engineering hours?
• what was the most painful part? cold starts, autoscaling, OOMs, model quality, getting paged at 2am
• if you went back to APIs, what made you switch back

ballpark numbers are totally fine. I'll put together a cost / decision writeup from the replies and post it back here

Infra engineer here, trying to figure out when self hosting LLMs actually makes sense vs just paying for the APIs. every blog post says "it depends" lol

If you've done it (or looked into it and bailed), would love to hear:

• how big was your API bill when you started thinking about it? what pushed you over the edge
• what did it really cost once you add up GPUs, idle time, and the engineering hours?
• what was the most painful part? cold starts, autoscaling, OOMs, model quality, getting paged at 2am
• if you went back to APIs, what made you switch back

ballpark numbers are totally fine. I'll put together a cost / decision writeup from the replies and post it back here

how is the vibes here? anyone been before? how bad is security and lines? any tips?