Project · BluIn progress

Pocket Voice Capture

Tap, speak a thought, and Centra files it — idea, reminder, rant, log — and remembers.

ESP32-S3 (LilyGO T-Watch S3)Touch / e-ink displayMEMS micDRV2605 hapticLiPo + USB-C chargingEdgeAgent (store-and-forward)whisper (STT)qwen3 (classify)MQTTFastAPI

41%complete

Current phaseIn progressB1

Local capture + STT

Capture speech and turn it into text on the device (or via Centra when online), buffering everything so nothing is lost offline.

25%

Next upAudio capture loop (push-to-talk)

Done whenSpeak with Wi-Fi off → Blu transcribes and buffers locally

View all phases →

Blu is a small handheld pocket device — a touch / e-ink screen with buttons to scroll, like a smartwatch. Tap to record a thought; it shows the transcription and lets you send or re-record, with a buzz (and chime) on success. Everything lands in Centra, which classifies each note by type — idea, project, daily, rant, log, alert, reminder, and more — schedules the time-based ones, and alerts you. Blu caches the latest notes and works offline, syncing when a link appears. Rechargeable over USB-C. It's the first module on the brain and the proving ground for the resilient-edge contract.

Architecture

Blu is a handheld ESP32-S3 pocket device (screen + buttons/touch + mic + haptic + battery) running the resilient EdgeAgent; a capture module on the brain transcribes (whisper), classifies each note by type with the local LLM, schedules the time-based ones, and notifies. Blu caches recent notes and works offline, syncing when a link appears.

Device (pocket)tap, speak, confirm — in your hand

ESP32-S3 (LilyGO T-Watch S3)Touch / e-ink display + scroll buttonsMEMS microphoneDRV2605 haptic (buzz)speaker (success chime)LiPo + USB-C charging

Edge runtimeoffline-first; never waits for the brain

EdgeAgent store-and-forward outboxLocal cache of recent notesHeartbeats + MQTT Last-WillBattery-aware conserve mode

Capture module (brain)understand + remember

whisper STTLLM classify → category (idea/project/daily/rant/log/alert/reminder)due-time extraction + schedulernotifier (device buzz/TTS, opt-in ntfy)

Note modelhow it's stored & shown

note: text, summary, category, due_at, status/notes + /categories APIbrowse & scroll by category on-device

Data flow

Tap → record; STT shows the transcription on the screen.
Confirm or re-record; on send, a buzz + chime, and the note syncs to Centra (buffered offline).
Centra classifies the note by type and, if it's time-based, schedules it.
Reminders/alerts fire back to the device (buzz + spoken); everything else is filed by category.
Blu caches the latest notes to browse offline, and re-syncs when a link returns.

Phase rail

B0Define & scaffold B1Local capture + STT B2Understand: classify + schedule B3Notify + sync B4aFirmware in a web simulator (Wokwi)B4Pocket device — T-Watch S3 B5On-device UX + polish

Progress log

2026-08-18
Blu is born as the first project. Folder, spec, edge agent, and reminder service scaffolded; phases B0–B5 defined with real hardware engineering.
2026-08-19
Blu is live on the brain: blu-01 (battery) registers and reminder.capture events flow device → MQTT → gateway → NATS → reminder-service → Postgres (stored as captured). B0 done; pivoted to B1 — real audio capture + on-device whisper STT next.
2026-08-19
B2 intent parsing live: reminder-service parses each captured note with the local LLM (qwen3) into a task + due time and marks it scheduled — e.g. 'water the plants at 7pm' → 'water the plants' @ 19:00; a note with no time stays captured. Remaining B2: the due-time scheduler. B1 real audio capture is blocked pending hardware (ESP32 mic, B4) or a host-side audio helper — a container can't grab the laptop mic.
2026-08-19
B2 done + B3 core, with phase-end tests. Scheduler verified: a past-due reminder went scheduled→notified within one tick and the device spoke 'Reminder: feed the cat'. B3: notify fan-out (device TTS + opt-in ntfy) and ack/snooze endpoints work; offline replay remains. Security pass: parameterized SQL, malicious status filter rejected (400), missing id 404, device_id sanitized before the MQTT command topic, whisper bound to 127.0.0.1 only. Built whisper STT + a host mic helper (scripts/blu-mic.py) so B1 voice works with no hardware (needs your mic to test). Added B4a: validate the ESP32-S3 firmware in Wokwi (free web sim) before buying hardware.
2026-08-19
B4a firmware written: projects/blu/firmware/wokwi (ESP32-S3 sketch + diagram) — Wi-Fi + MQTT publish of reminder.capture, retained registration, Last-Will, heartbeat, and a store-and-forward outbox; BOOT button or 20s auto-capture. Pending the user's Wokwi run against a public broker to confirm events arrive.
2026-08-19
Dropped B4a/Wokwi — the web simulator is slow and paywalls compilation, not worth it. The firmware sketch stays in the repo, reusable on a real ESP32-S3. Real-device path now: the laptop mic via scripts/blu-mic.py (B1) today; a physical ESP32-S3 (B4) when purchased. Full stack verified healthy (11 services on 48600-48612): local LLM on GPU, event pipeline, and reminder parse/schedule/notify all working.
2026-08-19
Blu re-scoped to the real product: a pocket handheld (touch/e-ink screen + buttons + mic + haptic + battery, USB-C) that captures a thought, which Centra auto-classifies by type. Shipped classification (LLM → idea/project/daily/rant/log/alert/reminder — verified: 'wifi keeps dropping'→rant, tax form→reminder, wired the board→daily) with a /notes + /categories API. Target hardware: LilyGO T-Watch S3 (touch + mic + DRV2605 haptic + battery). Phases updated: B2 classify+schedule (done), B4 pocket device (T-Watch S3), B5 on-device UX + polish.