u/rahulsrajput77

Hi everyone,

I want to build a personal voice-activated AI assistant/agent that can execute tasks across both my PC (Windows) and mobile device based on voice commands.

Here is what I am aiming for:

Cross-Platform Control: Capable of automating tasks on PC (managing files, running scripts, app controls, browser automation) and basic mobile actions.

Voice-to-Action: Fast Speech-to-Text (STT) → LLM reasoning / Tool Calling → Execution → Text-to-Speech (TTS).

Architecture/Backend: Flexible to run via API (OpenAI/Gemini/Claude) or locally (Ollama/vLLM) with proper function calling / agentic frameworks.

Questions for the community:

1. What frameworks or agent architectures do you recommend for bridging OS-level actions (e.g., LangChain, AutoGen, CrewAI, or direct Python OS/pyautogui scripting)?
2. How should I handle the mobile ↔ PC sync and remote execution securely (e.g., local Flask/FastAPI server, WebSockets, or Home Assistant integration)?
3. Are there existing open-source projects or GitHub repos doing this well that I can study or contribute to?

Thanks in advance for the guidance!