Building a local-first personal AI / “Jarvis” system: looking for lessons from people who made one useful, durable, and safe.
I am building a private personal project called **TREVIS**. The long-term goal is not a novelty chatbot; it is a tablet-accessible, always-available personal operating system with durable memory, local AI where practical, and carefully approved automation. A customized "Jarvis" for an entrepreneur focused on lifestyle & business needs that has a staff of "Ai agents" to communicate to his organizations & private groups.
I have a Windows 11 desktop with an RTX 4070 12 GB GPU, 32 GB RAM, and a Samsung Android tablet, and maybe a mini-PC to add to the mix. I want to keep recurring AI costs low by using local models where they are good enough, then selectively using paid models only for higher-value work. If this model is doable I want to help those that can benefit from this model. The simplest approaches lead to bigger & better things.
My current conclusion is that I should **not** try to build one giant “Jarvis” repository from scratch. Instead, I want TREVIS to own the policy and memory layer while reusing proven components:
- Obsidian for my human-editable knowledge map and notes
- Ollama for local models
- Open WebUI as a future tablet/browser interface
- A local coding agent such as OpenCode to help develop the system
- Later, possibly Letta for agent working memory, n8n for automation, FastMCP for narrow tools, and Home Assistant for voice/physical integrations
The part I intend to custom-build is deliberately smaller:
- directives, priorities, and editable settings
- canonical long-term memory (decisions, projects, sources, preferences, audit history)
- permissions and human-approval gates
- model-cost routing: local by default, paid escalation when justified
- evaluations so the system can propose improvements without silently changing itself
My intended operating loop is:
```text
Capture → organize → propose → review → approve → act → audit → improve rules
```
I am intentionally avoiding public exposure, bulk MCP installs, unattended writes, autonomous spending/messages, and giving browser agents access to personal credentials. I want every important capability to be useful manually before making it agentic, unless there's recent development to speed things up without a large overhead of paying for models to do it.
I would appreciate practical feedback from people who have built or operated similar local-first systems:
- What did you build too early and later regret?
- Which components were durable and worth adopting rather than rebuilding?
- How did you separate your notes/knowledge base from an agent’s working memory and chat history?
- What is the smallest reliable always-on setup for a tablet-accessible personal system?
- Which local models/tools were actually useful with an RTX 4070 12 GB VRAM GPU?
- How did you prevent agent “self-improvement” from becoming uncontrolled configuration or code changes?
- If you use MCP, browser agents, or automation, what permission and approval model held up in practice?
I am especially interested in postmortems, architecture diagrams, self-hosting lessons, backup/security practices, and examples where people reduced paid-token usage without making the system unusable.
I am not asking for access to anyone’s private system, credentials, or confidential workflows. Links to maintained open-source projects, public build logs, and lessons learned are ideal.
Thanks—my goal is to build this patiently enough that it becomes a useful second brain and development environment, rather than an expensive pile of agent tools.