r/rclone

▲ 14 r/rclone+5 crossposts

Headless backups of Proton Drive kept dying from rclone throttling and I was pulling my hair out, so I built a read-only gateway on the official SDK (and now you can too)... (it works!!)

Like a lot of people here I back up my Proton Drive from a headless Linux server (restic, nightly) to whatever, Synology, pCloud, you name it.

The usual route is rclone's protondrive backend, and like others here I watched it get slower and slower until a flat folder with 785 photos took 160 seconds to list on a good day and never finished on a bad one. Six-hour hangs, dead mount points. The throttling is pattern-based: rclone walks the whole tree with thousands of metadata calls every night, and Proton's API understandably stops picking up. The part that surprised me: the official Drive SDK (github.com/ProtonDriveApps/sdk, MIT) is already good enough to fix this yourself. I built a small read-only gateway with it:

- metadata lives in a local SQLite cache, kept fresh through the SDK's event stream (the same mechanism the official apps use, polled once a minute)

- the tree is exposed as WebDAV on localhost, so restic reads it through a boring rclone webdav mount

- file contents are never stored, just streamed on demand, decrypted and integrity-verified by the SDK

Results on the same account that was being throttled: that 785-file folder lists in 0.106 seconds. First full snapshot (3,069 files, 42 GiB) took 22 minutes.

The nightly incremental run now takes 0.82 seconds and makes zero calls to Proton unless something actually changed. Restore came back bit-for-bit identical. It follows the SDK rules: identifies as external-drive-*, event-based, no recursive scans, personal use only (which the license explicitly allows). It can't write anything, so the worst it can do to your Drive is read it well 😊

Two things I ran into that might interest the Proton devs if they're reading: the incubating account package isn't on npm yet (vendoring works fine), and getSeekableStream() deadlocks under parallel load (32 of 70 concurrent ranged reads hung, reproducible; the regular download path is rock solid). Happy to file a proper issue with details.

Full write-up with all source code (free to reuse for personal setups): Medium (free) article

Not affiliated with Proton, just a subscriber who wanted his 3 a.m. backups back, and if the native Linux client lands before you build this, even better. That's the outcome I'm actually rooting for.

reddit.com
u/rfrancocantero — 2 days ago
▲ 5 r/rclone

How to hide cache files

Hello,

I love rclone but one problem I have is that I want to have all files store in my remotes hidden. The problem is that rclone stores the cached files unecrypted on the C drive. After reading a bit it seems like I could use --cache-dir to change the location from C drive to for example a mounted veracrypt volume which should hide the cached files when the veracrypt volume is unmounted.

Would this be a dumb idea?

I am not a very technical person so please don't judge if this is stupid.

reddit.com
u/Fast-Truck8566 — 11 days ago
▲ 41 r/rclone+2 crossposts

Sick of rclone 429 rate limits with Plex/Jellyfin? I built Warpbox: an open-source WebDAV proxy for TorBox.

Hey everyone,

I wanted to share a project I've been working on to solve a massive headache when running TorBox alongside media servers. Whenever I ran a library scan over a direct rclone mount to Plex or Jellyfin, the media server would probe file headers, hitting the API with a massive spike of concurrent requests all at once. This instantly breaches TorBox's 300 RPM rate limit, leading to immediate 429 errors and account lockouts.

Using --tpslimit=1 was a clumsy band-aid that killed my throughput, and generic WebDAV proxies don’t offer any protocol-specific protection.

It wasn't just that my streams were cutting out. During background scans, my setup would hit a wall of 429s, causing Plex to assume the files were missing entirely and automatically empty the trash... wiping out my library matching and watch history. Plus, constantly hammering the API limits is a fast track to getting a TorBox account flagged or banned under their automated abuse system.

To solve this for my own setup, I built Warpbox - a lightweight, high-performance WebDAV proxy written in Go specifically tuned for TorBox's API behaviour.

Repo: https://github.com/mainlink0435/warpbox

How it fixes the rate-limit issue:

  • Zero-API Browsing... It syncs your TorBox directory into a local SQLite database with fully configurable cache times. Media server lookups hit the local DB instantly. Cost: 0 API calls.
  • Smart Throttle Queue... If your media server fires off 200 concurrent requests, Warpbox queues them and trickles them to TorBox safely below the limit. rclone just sees a slow disk; nothing crashes.
  • Smart CDN Caching... Caches active download links with configurable TTLs so repeated file requests don't hammer the API when links change.
  • Direct Streaming Offload... For full-file downloads, Warpbox issues a 302 Redirect straight to the CDN, stepping out of the data path entirely to save local bandwidth.

Setup

It is fully containerised and ready to go. I've included a complete, copy-pasteable Docker Compose template in the repository README that spins up both Warpbox and an optimised rclone mount automatically.

Full transparency: I developed this project with AI assistance, but every architectural choice has been designed, guided and refined by me.

I've been using it for a while now for my own streaming library and it has completely eliminated my rate-limit issues. It's totally sorted out my setup, so I hope it saves you some headaches too. Feel free to give it a spin and enjoy.

reddit.com
u/downstairs_upstairs — 13 days ago