https://reddit.com/link/1ulhi5p/video/d0i6b83hotah1/player

I wanted a true "OS-layer" dictation experience. No cloud, no $20/month SaaS wrappers, and zero latency. Just hold a hotkey, talk, release, and have the transcribed text instantly pasted into whatever input field currently has focus.

After 24 hrs of pure native Rust and Tauri v2 development, it is fully functional and running locally.

100% Offline & Air-Gapped by Design
This isn't "local" with a telemetry asterisk. It works entirely offline. Pull the Ethernet plug, go into a bunker, and it functions exactly the same. No background API pings, no usage tracking, and absolute data sovereignty. Your voice, your text, your machine.

The Stack:

• Core: Tauri v2 + Win32 APIs
• Frontend: Vue 3 + Tailwind v4 (transparent, click-through overlay HUD) as this transcript feature is not the only feature of upcoming app
• Audio: cpal (WASAPI), 16 kHz mono ring buffer
• Inference: whisper.cpp via whisper-rs 0.16 (ggml-small multilingual model)
• GPU Acceleration: Vulkan backend + flash-attention
• OS Integration: UIAutomation for focus tracking + enigo/arboard for clipboard paste and restore

The Latency:

• CPU Inference (ggml-small): ~2,500-5,700ms (completely unusable for an "OS layer" feel)
• GPU Vulkan Inference: ~160-300ms (feels instant, paste happens almost immediately after key release)

Engineering War Stories & Windows Hacks:

To make this feel like a native Windows feature rather than a sluggish Electron app, I had to solve a few low-level hurdles:

The "Click-Through" Focus Trap

If the Tauri HUD window takes focus when spawned, the target input field loses focus, making Ctrl+V paste impossible.

Fix: The window is kept hidden at boot, and when triggered, we apply set_ignore_cursor_events(true) and set_focusable(false) before bringing it to the front. The HUD follows the cursor, but Windows doesn't register it as a focusable element, keeping the target caret active.

Chrome Swallowing KeyUp Hooks

A global keyboard hook (using global-hotkey/tauri-plugin-global-shortcut) works fine until you're in Google Chrome. Chrome occasionally drops the key-up event, leaving the app in an infinite "listening" loop.

Fix: I bypassed the hook for release detection. Once the shortcut triggers the capture, a separate thread polls the hardware keyboard state directly via Win32 GetAsyncKeyState(VK_CONTROL) (or whichever modifier is held) at 30Hz to detect a guaranteed release.

The Windows 260-Character MAX_PATH Limit

Compiling whisper-rs with Vulkan support triggers deep shader generation paths under cargo's target directory, quickly exceeding Windows' 260-char path limit and failing the build.

Fix: Overrode the target directory to a short path (target-dir = "C:/ut") in .cargo/config.toml

Mixed-DPI Multi-Monitor Alignment

A transparent overlay window following the cursor behaves wildly when moving between monitors with different scale factors (e.g., a 4K 150% monitor next to a 1080p 100% monitor).

Fix: On every PTT press, we query the exact cursor position via GetCursorPos, find the active monitor using monitor_from_point, shift the Tauri window to that monitor, and dynamically calculate the correct coo rdinates based on that specific monitor's scale factor.

Non-Destructive Clipboard Restore

Using standard clipboard injection to paste the text would normally erase whatever image or text you had copied previously.

Fix: We query the clipboard format using arboard, cache the raw data (supports both text and image buffers), set the transcribed text, trigger the Ctrl+V simulation, wait 150ms for the OS to process the paste, and restore the original clipboard content.

Why Vulkan over CUDA?

Vulkan is universal. It runs out-of-the-box on NVIDIA, AMD, and Intel GPUs because vulkan-1.dll is already shipped in the display drivers. Users don't need to install any heavy SDKs or runtimes. If no compatible GPU is detected, the app gracefully falls back to CPU execution.

The project is currently configured as a single-crate monolith for performance, downloading the 466MB ggml-small model to the local app data directory on first launch.

What's Next?

This is only stage zero of a larger "invisible OS layer" I'm hacking together. The next sprints are dedicated to:

• Proof-test the latency on other devices (lower-spec configurations and Intel-based macOS systems)
• Local SLM Cleaning: Clean up raw phonetic typos and transcription anomalies locally in under 50ms. Adaptive Local Learning (Zero-Cloud Personalization): Dictation is dead on arrival if you have to manually correct acronyms or brand names (like "SpaceX" or "fnatic") every single time. To solve this 100% locally without slow retraining cycles, I’m building a three-tier pipeline:
  • Dynamic Whisper Prompting: Compiling a local text file of your custom vocabulary (tech jargon, project names, contacts) and injecting it directly into the whisper-rs context via set_initial_prompt (0ms execution overhead)
  • Context-Aware SLM Cleaning: Feeding your personal dictionary as hot-context into the local 0.5B SLM system prompt so it learns your phonetic patterns and habits over time

Happy to answer questions if you're battling similar local-first Tauri architectures!

Hey everyone,

I'm currently building "uni", a local-first, zero-chrome desktop orchestrator. Our obsession is latency: our target for the entire input-to-render loop is <50ms.

To achieve this, we just finished two technical spikes on Windows (using Rust + Tauri v2) to solve two core problems:

1. Shaving off every millisecond of Tauri window show/hide latency
2. Ingesting text/context under the cursor instantly without slow, resource-heavy OCR or screen-scraping

Here is the raw engineering data, the code strategies, and where we hit some brutal limitations

Part 1: Bypassing Tauri reveal latency with DWM cloaking (Spike A)

Standard window.show() / window.hide() in Tauri (and most web-view frameworks) has a noticeable composition lag. The OS has to allocate buffers, and the Desktop Window Manager (DWM) introduces frame delays (often 16-32ms) to composite the window

To fix this, we implemented a "cloaking" strategy using raw Win32 APIs. Instead of hiding the window, we keep it alive in the background and toggle its DWM cloaked state (DWMWA_CLOAK). This forces DWM to keep the Webview GPU buffers warm without rendering them to the screen

Our hotkey handler (RegisterHotKey) triggers this directly via Win32:

// In our dedicated MTA worker thread
unsafe {
  let is_cloaked: BOOL = if show { FALSE } else { TRUE };
  DwmSetWindowAttribute(
    hwnd,
    DWMWA_CLOAK,
    &is_cloaked as *const _ as *const _,
    std::mem::size_of::<BOOL>() as u32,
  );
}

The Metrics (reveal to painted latency, measured via high-res QueryPerformanceCounter + Webview rAF pingback):

• Standard Tauri Show/Hide (p50): 3.53 ms
• Win32 DWM Cloaking (p50): 1.61 ms

By cloaking the window, we cut the latency in half and completely eliminated the random frame-drops or white flashes during rapid toggles

Part 2: Extracting context under cursor via Windows UI Automation (Spike C)

We wanted to capture the text, element type, and bounding box of whatever is directly under the user's cursor when they hit Ctrl+Alt+U. OCR is too slow (>100ms) and inaccurate for code or UI hierarchies, so we turned to Windows UI Automation (UIA) using the windows crate

To hit our <15ms budget for context extraction, we designed a strict threading and caching model:

1. Thread Separation: COM calls are synchronous and cross-process. If the target app is busy, it blocks. We offloaded all UIA work to a dedicated Multi-Threaded Apartment (MTA) worker thread with a watchdog timer to discard calls that exceed our budget
2. Batching with Cache Requests: Doing raw properties lookups (CurrentName, CurrentControlType) requires separate cross-process round-trips. We used IUIAutomationCacheRequest to batch everything into exactly 2 round-trips:

&#8203;

// Batching properties in Rust to avoid cross-process round-trip hell
let cache_request = uia.CreateCacheRequest()?;
cache_reque
st.AddProperty(UIA_NamePropertyId)?;
cache_request.AddProperty(UIA_ControlTypePropertyId)?;
cache_request.AddProperty(UIA_BoundingRectanglePropertyId)?;

// Single round-trip hit-test + properties retrieval
let element = uia.ElementFromPoint(point)?;
let cached_element = element.BuildUpdatedCache(&cache_request)?;

The Brutal Benchmarks (Warm p95)

We ran 500+ iterations per target app. Here is where UIA shines, and where it failed miserably

1. VS Code (Electron) -> GO (p95: ~2.5 ms): Once the accessibility tree is warm (cold hit takes ~110ms to build the lazy tree), retrieving text under the cursor is incredibly fast. Monaco editor exposes the active text block beautifully via the Name property and TextPattern
2. Google Chrome / Web -> NO via Point (p95: 60 - 180 ms). This was our first major roadblock. While the cache retrieval is fast (~40ms), the raw ElementFromPoint call is heavily queued in Chrome's renderer process. If the page is rendering heavy JS, the hit-test gets blocked. Verdict: Doing active coordinate-based hit-testing on browsers is non-viable for instant HUDs. We are pivoting to using GetFocusedElement / focus-changed events, or fallback to our local WebSocket browser bridge
3. Figma Desktop -> Total Black Box (p95: ~2.7 ms). We hypothesized that Figma's canvas WebGL was a single opaque element, but the reality is worse: Figma Desktop renders its entire UI, including layers, sidebars, and panels, inside the WebGL canvas. UIA sees nothing but a full-screen empty window. No children, no text, no nodes. Verdict: A native OS accessibility layer is useless for Figma. A local WebSocket plugin/bridge is mandatory to stream the JSON layer tree to our Rust core

Next Steps

For Spike D, we are building a lightweight Rust-side WebSocket server to ingest vector data directly from a Figma plugin to bypass the slow cloud APIs

Would love to hear if anyone has optimized Chrome UIA hit-testing, or if you've found ways to bypass the initial cold-hit latency when building the a11y tree in your apps.