16h QLoRA, $0.25/hr P2P or $0.50/hr managed, what would you actually do
I am trying to settle a debate with myself about where to run a 16h QLoRA fine tune this weekend. 7B base, dataset fits 24GB, so a 4090 is plenty.
the choice keeps coming down to renting on a P2P marketplace (vast.ai etc) at like $0.20-0.30/hr vs spinning up something managed (RunPod, Lambda) at $0.50-0.60/hr. per attempt, that's $3-5 vs $8-10, and I'm doing multiple runs, so math favors P2P pretty hard if nothing goes wrong.
stuff that actually worries me
interruption handling. p2p hosts can yank the instant mid run. is resuming a QLoRA job from a checkpoint actually a few minutes of fuss, or a whole evening of debugging
checkpoint transfer. If I'm saving every 30 min , I bottlenecked by uploading to object storage, or is local-only + sync at end fine
setup time. managed is ~10 min and im training, p2p i sometimes burn an hour on image/SSH weirdness before anything actually starts
security. not a huge deal for a public model + my own data, but im still running code on someones box in their apartment
for anyone doing multi-hour LoRA jobs on cheap p2p 4090s, do the real failure rates match what gets posted or is it mostly survivorship bias? and for managed, is there a meaningful stability gap between RunPod / Lambda / others for this kind of workloa
trying to figure out if the cheap option is actually cheap once retries get factored in.