Dovodi_Backend/docs/web_egress_architecture.md

Web Egress Recording Architecture

This document describes how online-class recording works after replacing the old PlugNMeet recorder with LiveKit Web Egress.

Goal

We no longer rely on the built-in PlugNMeet recorder for class recordings.

Instead:

• PlugNMeet is still the meeting server and source of classroom events
• LiveKit Egress is responsible for recording the rendered meeting page
• Django remains the source of truth for sessions, recordings, lessons, and attendance-related data
• The chat/FastAPI service acts as the bridge between Django and LiveKit Egress

Main Services

1. conference_client

Path:

• Online Class/conference_client/

Role:

• Renders the online classroom UI
• Provides the Rec button for professor recording control
• Hosts the same meeting page used by normal users and the recorder bot
• Applies recorder-specific UI rules when the access token belongs to the recorder

Important points:

• The recorder joins this same frontend using a special access token
• Some UI branches are customized for isRecorder / web_egress=1
• The recorder should bypass landing/prejoin steps and go directly into the meeting

2. conference_server / PlugNMeet

Path:

• conference_server/conference_server/

Role:

• Creates and runs meeting rooms
• Issues meeting join tokens
• Sends meeting lifecycle webhooks
• Continues to provide room state, participant lifecycle, and room end events

Important points:

• PlugNMeet still owns the real classroom
• Attendance and automatic session closing logic still depend on PlugNMeet room/webhook behavior
• Only the recording backend changed, not the meeting backend itself

3. Django backend

Path:

• backend/apps/course/

Role:

• Manages CourseLiveSession
• Generates join URLs and meeting access tokens
• Tracks LiveSessionUser
• Stores LiveSessionRecording
• Creates course lessons automatically from saved recordings
• Controls recording start/stop through backend APIs

Important files:

• live_session.py
• course.py
• webhook.py
• signals.py
• web_egress.py

4. Chat / FastAPI bridge service

Path:

• chat/chat/

Role:

• Receives recording start/stop requests from Django
• Talks to LiveKit Egress
• Tracks the current egress job state
• Receives LiveKit webhook callbacks
• Uploads the final recorded file back to Django

Important files:

• web_egress_apis.py
• main.py
• settings.py

5. LiveKit + Egress stack

External/runtime project on server:

• LiveKit server
• Redis
• Caddy/L4 proxy
• LiveKit Egress

Role:

• Accepts recording jobs
• Opens the target classroom URL in a browser-like environment
• Captures the rendered meeting page
• Saves the output file to shared storage
• Emits webhook events back to the chat/FastAPI bridge

High-Level Flow

flowchart LR
  A["Professor clicks Rec in conference_client"] --> B["Django recording API"]
  B --> C["WebEgressClient in Django"]
  C --> D["Chat/FastAPI /api/web-egress/start/"]
  D --> E["LiveKit Egress job created"]
  E --> F["Recorder bot opens conference_client join URL"]
  F --> G["Rendered meeting page is recorded"]
  G --> H["Egress stores output file"]
  H --> I["LiveKit webhook -> chat/FastAPI"]
  I --> J["chat/FastAPI uploads file to Django callback"]
  J --> K["Django creates LiveSessionRecording"]
  K --> L["Django signal creates CourseLesson"]

Recording Start Flow

Frontend

Professor presses the recording button in:

• useCloudRecording.tsx

That calls:

• POST /api/courses/online/room/recording/

Payload:

{
  "action": "start",
  "room_id": "room-16-1781432247"
}

Django

Handled in:

• CourseLiveSessionRecordingAPIView.post()

What Django does:

1. Resolves the current live session from the meeting token
2. Checks that the caller is a moderator
3. Marks session status as starting
4. Sends a start request to the Web Egress bridge service
5. Stores returned egress_id
6. Marks the session as recording

Important session fields used:

• web_egress_id
• web_egress_status
• web_egress_started_at
• web_egress_stopped_at
• web_egress_error
• recording_title

Chat / FastAPI bridge

Handled in:

• POST /api/web-egress/start/

What it does:

1. Registers a job for the given session_id and room_id
2. Builds the join URL for the recorder bot
3. Calls LiveKit Egress to start a browser-based recording
4. Stores current job state

Recorder Bot

The recorder joins the classroom using a special access token and URL generated by Django:

• ?access_token=...&web_egress=1&skip_landing=1&room_id=...

This is important because:

• the recorder uses the real classroom UI
• it must bypass landing/prejoin checks
• it must render the same content users see

Recording Stop Flow

Manual stop

Professor presses Rec again:

• frontend calls POST /api/courses/online/room/recording/ with "action": "stop"

Django:

1. verifies the session
2. sends stop request to Web Egress bridge
3. marks session as stopping

Automatic stop on session end

This is critical.

If the professor ends the meeting while recording is still active, Django must still stop Web Egress before the session is fully closed.

This is now handled centrally by:

• stop_session_web_egress_if_active(...)

Used from:

• course.py
• webhook.py
• admin.py
• live_session.py

This ensures the final active recording is not lost when the class is closed.

Finalization Flow

After stop:

1. LiveKit Egress finishes processing the output file
2. LiveKit sends webhook events to the chat/FastAPI bridge
3. The bridge locates the output file in shared storage
4. The bridge uploads the file to Django callback endpoint
5. Django creates LiveSessionRecording
6. Django updates CourseLiveSession.web_egress_status

Important Django callback endpoint:

• POST /api/courses/online/room/recording/web-egress/callback/

Handled in:

• CourseLiveSessionWebEgressCallbackAPIView

Shared Storage

The recording file is not sent directly from LiveKit to Django first.

Current pattern:

1. Egress writes file to shared disk
2. Chat/FastAPI reads the file
3. Chat/FastAPI uploads file to Django

Example shared path on server:

• /najm/HabibMeetSocket/shared/recordings/imamjavad/web-egress/<room-id>/

This means:

• LiveKit Egress and chat/FastAPI must both be able to access the same recording storage
• file permissions and mount paths are important

Session and Recording Data Model

CourseLiveSession

Represents the live class session.

Important responsibilities:

• room identity
• live/ended status
• moderator auto-close timing
• current Web Egress recording state

LiveSessionUser

Represents user presence inside a live session.

Used for:

• attendance-like tracking
• determining who is online
• moderator-exit auto-close logic

Important note:

Even with webhooks, Django also performs fail-safe registration when direct join-token flows are used, so presence data is not lost when some event paths are bypassed.

LiveSessionRecording

Represents a final recorded file stored in Django.

Used for:

• listing recordings
• generating lessons
• attaching final media files to courses

Lesson Creation Flow

When a LiveSessionRecording is created, Django signal logic automatically creates a lesson.

Handled in:

• signals.py

What it does:

1. Creates or reuses a chapter for the recording date
2. Creates Lesson
3. Creates CourseLesson
4. Uses session title or subject as lesson title
5. Adds part 1, part 2, etc. when multiple recordings belong to the same session

Late title update behavior

If professor enters a custom recording title near session end, old recordings and already-created lessons must also be renamed.

This is now handled by syncing titles after recording_title is updated, so early-created lessons do not keep stale names.

Why We Kept PlugNMeet Events

The recorder changed, but PlugNMeet room lifecycle still matters.

Things that should still come from PlugNMeet or session logic around it:

• room end state
• participant join/leave tracking
• presence rows in LiveSessionUser
• moderator exit detection
• automatic session closure

So the architecture is intentionally split:

• PlugNMeet remains the meeting authority
• Web Egress becomes only the recording engine
• Django links both worlds together

Important Endpoints

Django

• POST /api/courses/online/room/recording/
  • start or stop recording
• GET /api/courses/online/room/recording/?room_id=...
  • read recording status
• POST /api/courses/online/room/recording/web-egress/callback/
  • receive final uploaded recording from bridge service
• POST /api/courses/online/room/set-recording-title/
  • set custom recording title for active session
• POST /api/courses/plugnmeet/webhook/
  • receive room and participant lifecycle events from PlugNMeet

Chat / FastAPI bridge

• POST /api/web-egress/start/
• POST /api/web-egress/stop/
• GET /api/web-egress/status/
• POST /api/web-egress/livekit-webhook/
• GET /api/web-egress/health/

Environment and Runtime Dependencies

Django

Relevant envs:

• ONLINE_CLASS_WEB_EGRESS_ENABLED
• ONLINE_CLASS_WEB_EGRESS_SERVICE_URL
• ONLINE_CLASS_WEB_EGRESS_SERVICE_TOKEN
• ONLINE_CLASS_WEB_EGRESS_CALLBACK_TOKEN
• ONLINE_CLASS_WEB_EGRESS_TIMEOUT

Chat / FastAPI bridge

Must know:

• Django upload/callback target
• callback token
• shared recordings root
• LiveKit/Egress credentials

LiveKit/Egress

Must be configured with:

• correct output path
• webhook target to chat/FastAPI
• access to the shared recordings directory

Current Operational Notes

• The recorder is a hidden meeting participant named Session Recorder
• The recorder uses the real meeting frontend, not a separate rendering app
• The frontend has recorder-specific behavior to skip entry/landing flow
• Presence and auto-close logic must continue to work independently of recording
• Recording stop should be triggered both manually and during session close

Summary

The current architecture is a three-layer recording pipeline:

1. PlugNMeet hosts the class
2. Django decides when recording starts/stops and stores final course data
3. Chat/FastAPI bridges Django to LiveKit Egress

LiveKit Egress records the rendered classroom page, but PlugNMeet remains the source of meeting truth.

That split is the key design decision:

• do not move classroom lifecycle into Egress
• only move the recording responsibility into Egress