Rivertalk/river_connect/.agent/rules/audio-streaming.md

Audio Recording & Streaming Architecture

This project implements an asynchronous and real-time audio messaging system using Phoenix Channels for binary streaming and Oban for background processing.

1. Components & Data Flow

• Capture: AudioRecorder JS Hook (assets/js/hooks/audio_recorder.js) utilizes the MediaRecorder API with audio/webm;codecs=opus.
• Transport: Binary chunks are transmitted via Phoenix Channels (audio:lobby).
• Persistence: The server appends incoming chunks to a .webm file located in priv/static/uploads/.
• Finalization: An audio_end event triggers an Oban job (RiverConnect.Workers.ProcessRecording) to verify the file and update the database status.
• Serving: Static audio files are exposed via a dedicated Plug.Static entry in RiverConnectWeb.Endpoint.

2. Channel Protocol (AudioChannel)

• audio_start: %{"id" => id}. Initializes the server-side recording state and creates the target file.
• audio_chunk: {:binary, payload}. Appends binary data to the file and broadcasts the chunk for real-time consumption.
• audio_end: %{"duration_ms" => ms, "id" => id}. Persists the AudioMessage record and enqueues the processing worker.

3. Real-time Streaming

• During an active recording session, the channel broadcasts audio_chunk events to all subscribers.
• Subscribed clients use AudioContext.decodeAudioData to process and play the binary stream with minimal latency.

4. Database Schema (AudioMessage)

• user_id: Identifies the sender (Guest or Authenticated).
• file_path: Relative URL to the static resource.
• duration_ms: Integer representing the length of the audio.
• status: State machine values ("processing", "ready").
• Note: The schema must @derive {Jason.Encoder, ...} for binary serialization over the socket.

5. Environment & Infrastructure

• Windows Support: mix commands in background processes may require explicit PATH configuration for Erlang/Elixir binaries.
• Static Configuration: Dynamic directories in priv/static require explicit inclusion in the Endpoint configuration to be served correctly.

6. Implementation Notes

• Server Restarts: Modifications to endpoint.ex or UserSocket require a server restart to take effect.
• Hardware Access: Web-based recording requires secure context (HTTPS/Localhost) and explicit browser microphone permissions.
• Latency Management: Real-time playback relies on prompt chunk processing; production environments may require a jitter buffer for consistency.