JARVIS is a local-first AI command center designed to bring together voice interaction, memory, planning, automation, integrations, and personal knowledge management into one coherent assistant.
The project began as a voice-controlled desktop HUD inspired by the idea of a true personal AI assistant, but has grown into something larger — a life and work operating system that manages tasks, reminders, calendar context, family logistics, meal planning, business projects, communications, files, media, and long-term memory.
AI becomes dramatically more useful when it has context, memory, tools, and governance.
JARVIS is built to run locally, preserve ownership of personal data, and connect selectively to cloud services only when useful.
It currently supports:
Voice and text commands
Local task, note, reminder, and goal management
Memory capture and Obsidian-style vault output
Google Calendar, Gmail, Drive, Docs, and Sheets integration
Slack and Telegram connectivity
Trello card and project workflows
Plex media search and playback
Weather and morning planning
Pantry, recipes, family meal planning, and shopping lists
Governed Second Brain memory with Data Studio
Action Briefs and open-loop tracking
Wake-word detection with approval-gated external actions
01The problem
Most AI assistants are either:
Conversational but disconnected from real work
Powerful but stateless
Integrated but cloud-first
Automated without enough visibility and control
Working across multiple projects, clients, tools, documents, conversations, and planning contexts — with information spread across Gmail, Slack, Trello, local notes, and voice thoughts — the challenge was not just "talk to an AI."
The real questions were:
How do you give AI enough context to be useful?
How do you preserve what matters over time?
How do you avoid losing important thoughts?
How do you let the assistant take action without becoming risky?
How do you keep the interface fast, local, and understandable?
02The approach
JARVIS is a local desktop AI hub with a Python core, browser-based HUD, structured SQLite database, encrypted local secrets (Windows DPAPI), voice input, ElevenLabs speech output, and OpenAI reasoning.
The HUD provides:
- A central visual display and activity log
- Voice controls with concise spoken summaries
- Center-screen readouts for long-form responses
- Configuration panel and mission/domain rail
- Media playback and action approval interface
- Data Studio for complex context assembly
Cloud services are treated as providers, not as the source of truth.
A key interaction design decision: long AI answers display in the center screen while the spoken reply stays concise. Voice gives the summary. The HUD carries the detail.
External actions are approval-gated, following a draft before action principle — the assistant can prepare, summarize, search, and organize freely, but sending messages, modifying calendars, or controlling external apps all require explicit approval.
03The build
Second Brain Memory Architecture
A governed memory system with four layers: captures (raw inbound material), memory_facts (durable extracted knowledge and decisions), open_loops (commitments, follow-ups, blockers), and memory_versions (correction history). Provenance and review built in — not just a pile of chat logs.
Voice-First Interaction with Smart Display
Voice commands with ElevenLabs speech output, tuned so short answers are spoken and long answers become readable HUD center-screen responses. Wake-word handling uses local validation — audio duration, voice ratio, voiced-frame streaks, and crest factor — to prevent accidental activation from typing or non-speech sounds.
Data Studio & Context Desk
A deliberate workspace for complex workflows too large for voice. Collect documents, notes, uploaded files, project updates, and context before asking JARVIS to synthesize something — enabling multi-project status updates, planning sessions, and second brain review.
Approval-Gated Automation
The assistant prepares, summarizes, searches, and organizes freely. Higher-risk external actions — sending messages, modifying calendars, creating Gmail drafts, controlling apps — require explicit user approval. Automation that stays useful without becoming reckless.
Multi-Service Integration
Google Calendar, Gmail, Drive, Docs, and Sheets. Slack and Telegram. Trello card and project workflows. Plex media search and playback. Weather and morning planning. Pantry, recipes, family meal planning, and shopping lists — all accessible through a single local interface.
04Results
Single command center across 10+ services and tools — calendar, email, messaging, media, notes, tasks, and family logistics in one place
Structured second brain with provenance tracking, open-loop surfacing, correction history, and review workflows — not just saved chat logs
All data stored locally in SQLite with DPAPI-encrypted credentials. Cloud services are optional providers, not the source of truth
Draft-before-action governance with approval gates on all external writes — automation that can be trusted without constant supervision
05Project views
JARVIS is becoming a local AI operating system for life and work — not just a chatbot, but a memory system, planning system, interface, automation layer, and personal context engine.
The biggest lesson from the project:
An AI assistant is not defined by the model alone. The real product comes from memory, interface design, tool boundaries, latency, source tracking, review workflows, user trust, and data ownership.
Three principles that shaped every decision:
- Voice needs restraint. Spoken AI should be concise. Screens should carry complexity.
- Context needs structure. A second brain cannot be a pile of chat logs — it needs captures, facts, links, review states, source trails, and correction history.
- Automation needs governance. Draft before action. Approval gates on external writes. Trust built through transparency.
The foundation is now clear: a useful personal AI assistant needs to remember, reason, act carefully, and meet the user where the work actually happens.