Login

r/SelfHostedAI

▲ 1 r/SelfHostedAI+1 crossposts

Should I spend $1,000 on an RTX 4060 Ti 16GB PC for local AI, or wait for something better?

​

Hi everyone,

I'm a hobbyist who enjoys experimenting with local AI models, image generation, and other AI-related projects.

My current hardware:

Gaming laptop: Intel i5-9300H, 16GB RAM, GTX 1650 4GB

Home lab mini PC: Intel i5-1300T, 32GB RAM

Office PC: NVIDIA RTX A4500 with 20GB VRAM

Due to my limited hardware at home, I can only run most of my AI experiments on my office PC.

Recently, I came across a used PC listed for $1,000 with the following specs:

GPU: RTX 4060 Ti 16GB

CPU: Ryzen 7 5700G

RAM: 32GB DDR4

Storage: 1TB SSD

With hardware prices continuing to rise, I'm tempted to buy it. However, I'm wondering whether it would be better to save my money and wait for something more capable.

I initially considered a unified-memory system, such as a Ryzen AI Max device, NVIDIA DGX Spark, or Apple Silicon Mac, since having access to more memory for larger models would be great. Unfortunately, those options are significantly more expensive.

This is purely a hobby, so there's no urgent need to upgrade. I can live without a dedicated AI PC at home, but having one would give me more freedom to experiment.

What would you do in my situation?

Would you buy the RTX 4060 Ti system for $1,000, look for a better used GPU or PC, or save up for a unified-memory system?

I'd appreciate any suggestions, especially from people running local LLMs and image-generation models on a budget.

reddit.com
u/neon-r — 5 days ago
▲ 12 r/SelfHostedAI+2 crossposts

Llama.WASM + WebGPU = Local LLMs

Over the weekend I built a technical proof of concept showing an LLM running client-side in a browser using Llama.cpp compiled to WASM with WebGPU.

github.com
u/Numerous-Fan8138 — 5 days ago
▲ 6 r/SelfHostedAI

I built a panel that lets your AI agent manage your own VPS over MCP

I run a handful of VPSes across Hetzner and DigitalOcean, and I got tired of SSHing into each one just to check whether something was up. So I built ServerOS, a panel for servers you already own, and gave it an MCP server so Claude (or any MCP client) can look after them too.

How it works:

- You run one command on your server. It installs a small daemon (written in Rust, open source: github.com/serveroshq/daemon) that finds what's already running, like Docker containers, systemd services and databases, and leaves it all alone. Nothing gets migrated or reinstalled.

- Each workspace gets its own MCP endpoint. You connect it in Claude or your MCP client of choice, sign in with OAuth, and pick what the agent is allowed to do.

- The agent gets 109 tools: list machines and services, read and search logs, restart things, deploy from a git repo, roll back, set up uptime monitors, manage status pages, run backups, open firewall ports, install package updates and so on.

Things I can now just ask:

- "Why is the shop API slow today?" It pulls the logs and stats and tells me what it found.

- "Deploy the latest main of my repo to web-1 and tell me when it's healthy"

- "Set up a monitor for every site on db-main and put them on a status page"

- "Which of my servers haven't been backed up this week?"

The part I worried about most was letting an AI loose on real servers, so:

- Permissions are split into read, write and commands. You can give an agent read-only access and nothing else.

- Anything destructive, like rebooting, removing a service or restoring a backup, needs the machine or service name typed back as confirmation. The agent can't just fire it off.

- Every call is logged in the workspace activity as "done through MCP", against the person who connected it.

- Raw shell access is its own permission, off unless you turn it on.

It's a paid product (from £5/month for one machine), with a 14-day free trial of Pro. I'm one of the two founders, so I'd really like honest feedback, especially from anyone already letting agents touch their infra. What would you want the agent to be able to do, and what would you never let it do?

serveros.com

u/KnownObligation9934 — 5 days ago
▲ 1 r/SelfHostedAI+1 crossposts

Suggestion for AI agents

Could someone please suggest free AI agents or frameworks that can help me build projects independently?

AI agents are becoming expensive to use because they consume a lot of tokens, and some tools are discontinuing models or changing their features, such as Antigravity.

I’m looking for reliable, free or open-source AI agents, coding frameworks, or self-hosted solutions that can help me build projects on a limited budget.

If you know any useful tools, GitHub repositories, frameworks, or resources, please share their names or links. I would really appreciate your suggestions and personal experiences. Thank you!

reddit.com
u/mhashir28 — 5 days ago
▲ 1 r/SelfHostedAI+1 crossposts

How to with a 128gb evo-x2, MacBook Pro 48gb & M5 Pro, and 4080 super (16gb) desktop.

What’s the best way to utilize this setup?

I want to develop local llm products for SMB, learn AI, and build a great memory system. I really need a “life AI” also, that handles family, health, hobbies, career, etc. in addition to dev & startup work.

Ultimately, I’d like a local LLM to be my daily driver for everything and escalate to frontier models to only learn new things.

I’ve mad ADHD and rapid fire shit to AI and get it more sidetracked than a human in the same situation. I’ve got a decent memory layer now that’s handling this pretty well. but am I reinventing the wheel?

Currently I’m using code as a front end for all of these things. With some internal orchestrators, and leads and an oracle that knows all - but never acts, only directs other orchestrators and leads.

The 128gb evo-x2 is running qwen 35b MoE. About 500-900tok/s PP and ~50tok/s decode. All of its work is focused on business products not as a daily driver currently.

Curious what others are doing, and what everyone’s thoughts are…

reddit.com
u/roadtripsarefun — 5 days ago
▲ 151 r/SelfHostedAI+7 crossposts

LlamaStash 0.0.2 — a zero-overhead terminal launcher for llama.cpp (TUI + CLI + OpenAI-compatible proxy, Linux/macOS/Windows)

I built LlamaStash to scratch a personal itch: I run local models through llama.cpp on AMD Strix Halo and got tired of writing the same llama-server wrapper script for the tenth time.

Ollama and LM Studio both wrap llama.cpp but hide too much (and cost real performance). Raw llama-server is fast but tedious. LlamaStash is the middle ground.

What it does:

  • llamastash init — first-run wizard. Detects your hardware (CUDA / ROCm-HIP / Metal / Vulkan / CPU), installs llama-server, scans your existing HuggingFace / Ollama / LM Studio model caches, recommends a GGUF that fits your VRAM, downloads it, writes a tuned config, smoke-launches it.
  • TUI + CLI + daemon + OpenAI-compatible proxy in one Rust binary. The proxy at 127.0.0.1:11435/v1 lets OpenCode, Cline, the OpenAI SDKs, and llm-cli work as-is. There's also an opt-in --ollama-compat mode that takes port 11434 and answers the byte-exact "Ollama is running" handshake.
  • Multi-model concurrency with per-model port allocation, /health-probed state machine, intelligent context auto-fit (sidesteps llama.cpp's --fit collapse on Linux iGPUs).
  • Agent-friendly CLI: every TUI capability has a CLI subcommand, --json is a stable agent contract, documented exit codes per failure class.
  • In-TUI HuggingFace browser with search, sort, paginate, per-file hardware fit, download with cancel.

On performance — this is the part that matters for this sub.

LlamaStash spawns the unmodified upstream llama-server. So the wrapper should add zero overhead. I measured it. Across AMD APU (Ryzen AI Max+ 395), Apple Silicon, and NVIDIA, on four model sizes (small E2B Q4, mid 31B Q4, large 27B Q8, large MoE 35B-A3B Q8), every cell matches raw llama-server within ≤1%.

Cross-tool numbers on AMD APU (decode tok/s / TTFT ms on chat_turn):

Tool small mid large_dense large_moe
LlamaStash 86.9 / 51 9.8 / 467 7.4 / 417 42.6 / 181
raw llama-server 86.0 / 51 9.9 / 468 7.4 / 414 42.7 / 186
LM Studio 2.16.0 91.1 / 187 11.6 / 1477 7.9 / 1274 37.0 / 683
Ollama 0.24.0 50.4 / 223 4.8 / 1092 2.6 / 1745 12.1 / 476

LM Studio wins decode on small/mid/large_dense (their Vulkan path is well-tuned on gfx1151) but loses on the MoE and pays a 1-1.5s TTFT tax from its OpenAI shim. Ollama is consistently slower, and its RAG prefill is catastrophic (cold prefill every rep — 4 min on a 31B). Mac and NVIDIA tables are in the benchmarks page.

Methodology, variance gates, fairness rules, and per-cell JSONs are all checked in. The harness is reproducible: make bench-end-to-end. Tear it apart.

What it's not:

  • Not an Ollama fork or replacement (though --ollama-compat exists for tools that auto-detect Ollama).
  • Not a model hub.
  • Not a llama.cpp fork. Same upstream binary.
  • Not a hosted service. Loopback-only in 0.0.2. LAN + auth + TLS are on the roadmap.

Install:

curl -fsSL https://llamastash.dev/install.sh | sh   # macOS + Linux one-shot
irm https://llamastash.dev/install.ps1 | iex        # Windows 11 (PowerShell, no admin)
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash
brew install llamastash/llamastash/llamastash       # Homebrew (macOS + Linuxbrew)
yay -S llamastash                                   # Arch Linux (AUR — source build)
yay -S llamastash-bin                               # Arch Linux (AUR — prebuilt binary)
yay -S llamastash-git                               # Arch Linux (AUR — main checkout)
cargo install llamastash                            # any Rust toolchain

Then llamastash init and you're up.

Platform: Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.

Honest tradeoffs: Single-author project. Bug reports especially welcome on hardware I don't own. The OpenAI-compat surface covers chat/completions, embeddings, rerank; Anthropic /v1/messages shim is coming.

Repo: https://github.com/llamastash/llamastash

Blog post with the full story: https://deepu.tech/introducing-llamastash

Benchmark methodology: https://deepu.tech/benchmarking-llamastash

Happy to answer questions in the thread.

u/deepu105 — 11 days ago
▲ 10 r/SelfHostedAI

Building a local-first personal AI / “Jarvis” system: looking for lessons from people who made one useful, durable, and safe.

I am building a private personal project called **TREVIS**. The long-term goal is not a novelty chatbot; it is a tablet-accessible, always-available personal operating system with durable memory, local AI where practical, and carefully approved automation. A customized "Jarvis" for an entrepreneur focused on lifestyle & business needs that has a staff of "Ai agents" to communicate to his organizations & private groups.

I have a Windows 11 desktop with an RTX 4070 12 GB GPU, 32 GB RAM, and a Samsung Android tablet, and maybe a mini-PC to add to the mix. I want to keep recurring AI costs low by using local models where they are good enough, then selectively using paid models only for higher-value work. If this model is doable I want to help those that can benefit from this model. The simplest approaches lead to bigger & better things.

My current conclusion is that I should **not** try to build one giant “Jarvis” repository from scratch. Instead, I want TREVIS to own the policy and memory layer while reusing proven components:

- Obsidian for my human-editable knowledge map and notes
- Ollama for local models
- Open WebUI as a future tablet/browser interface
- A local coding agent such as OpenCode to help develop the system
- Later, possibly Letta for agent working memory, n8n for automation, FastMCP for narrow tools, and Home Assistant for voice/physical integrations

The part I intend to custom-build is deliberately smaller:

- directives, priorities, and editable settings
- canonical long-term memory (decisions, projects, sources, preferences, audit history)
- permissions and human-approval gates
- model-cost routing: local by default, paid escalation when justified
- evaluations so the system can propose improvements without silently changing itself

My intended operating loop is:

```text
Capture → organize → propose → review → approve → act → audit → improve rules
```

I am intentionally avoiding public exposure, bulk MCP installs, unattended writes, autonomous spending/messages, and giving browser agents access to personal credentials. I want every important capability to be useful manually before making it agentic, unless there's recent development to speed things up without a large overhead of paying for models to do it.

I would appreciate practical feedback from people who have built or operated similar local-first systems:

  1. What did you build too early and later regret?
  2. Which components were durable and worth adopting rather than rebuilding?
  3. How did you separate your notes/knowledge base from an agent’s working memory and chat history?
  4. What is the smallest reliable always-on setup for a tablet-accessible personal system?
  5. Which local models/tools were actually useful with an RTX 4070 12 GB VRAM GPU?
  6. How did you prevent agent “self-improvement” from becoming uncontrolled configuration or code changes?
  7. If you use MCP, browser agents, or automation, what permission and approval model held up in practice?

I am especially interested in postmortems, architecture diagrams, self-hosting lessons, backup/security practices, and examples where people reduced paid-token usage without making the system unusable.

I am not asking for access to anyone’s private system, credentials, or confidential workflows. Links to maintained open-source projects, public build logs, and lessons learned are ideal.

Thanks—my goal is to build this patiently enough that it becomes a useful second brain and development environment, rather than an expensive pile of agent tools.

reddit.com
u/Local-Painter-5808 — 14 days ago