I built an open-source file sync engine using Kotlin Coroutines & Java NIO that beats rsync by 40x on massive directories.
Hi everyone,
I’ve been developing Janice, a high-performance asynchronous file synchronization tool built from the ground up to solve a very specific bottleneck: the massive overhead when syncing directories with hundreds of thousands of files.
The Problem & The Goal: Traditional tools like rsync are fantastic, but they often choke on file traversal during incremental syncs. My primary engineering goal for this project was to ensure it can process TB-level data at full disk rate without the CPU or thread management becoming the bottleneck.
The Architecture: To achieve this, I abandoned traditional threading models. Janice leverages Kotlin Coroutines alongside Java NIO's asynchronous file and networking capabilities. By utilizing purely async I/O, it pushes the limits of what the modern JVM can do with massive concurrency.
The Benchmarks (350,000 files, 13GB total): I ran tests comparing Janice to rsync (Windows Client -> Linux Server):
- Full Sync (Cold): Janice (201s) vs rsync (382s) -> Nearly 2x faster
- No Change Sync: Janice (3s) vs rsync (126s) -> 4100% faster
- 1 File Changed: Janice (2.7s) vs rsync (119s) -> 4300% faster
Why I'm posting here: It's fully open-source (MulanPSL-2.0), and I would love to get your feedback on the architecture, the coroutine dispatching strategies, or any edge cases I might have missed in the custom protocol.
(To respect the subreddit rules and avoid automated spam filters, I have not included any links here. I will drop the GitHub repository link and instructions on how to test it in the first comment below!)