SignSync/project-specifications.md

SignSync Web App - Project Specifications

1. Product Summary

SignSync is a browser-based practice and comparison tool for sign language interpreters. Users watch a YouTube source video, record their own interpretation via webcam, save that recording, and share a link so a second interpreter can record against the same source. The app then plays the source plus both interpretation videos in sync for side-by-side review.

2. Core Objectives

1. Let interpreters practice against any YouTube video.
2. Capture webcam + microphone recordings directly in-browser.
3. Persist recordings with lightweight user metadata.
4. Generate a shareable link for collaboration.
5. Compare two interpreters in synchronized playback.

3. User Roles

1. First interpreter: Creates an initial recording and share link.
2. Second interpreter: Opens shared link and adds a second recording.
3. Viewer/reviewer: Uses playback controls to compare source + interpretations.

4. Primary User Flows

Flow A: Create first recording

1. User opens app.
2. User pastes YouTube URL (or raw video ID) and clicks Load Video.
3. User enters name/email.
4. User records interpretation from webcam.
5. User stops recording and previews it.
6. User clicks Save Recording.
7. Backend stores metadata + video file and returns unique link ID.
8. Frontend shows full share URL (?share=<id>) and copy button.
9. Saved recording appears in "First Recording" panel.

Flow B: Add second recording from shared link

1. User opens shared link with ?share=<unique_link>.
2. App loads existing record and cues stored YouTube video.
3. If second recording does not exist:
   • First recording video is shown blurred with overlay text: "Recording done, waiting for other person..."
   • Playback controls (play/pause, timeline seek) are disabled.
   • Form + recorder are shown for the second user.
4. User records and submits second video.
5. Backend updates same row with name_2, email_2, recorded_video_path_2.
6. Frontend refreshes shared data and shows both videos immediately (no reload needed).
7. First video blur is removed and playback controls are enabled.
8. Once both videos exist, form + recorder are hidden.

Flow C: Synchronized comparison

1. User presses global play/pause button.
2. App controls YouTube + local videos together.
3. Timeline slider seeks all videos simultaneously.
4. First interpretation is always offset +2 seconds (headstart).
5. User can swap left/right interpreter panels.

5. Functional Requirements

5.1 YouTube Loading

1. Must support:

• Full URLs (youtube.com/watch?v=...)
• Short URLs (youtu.be/...)
• Embed URLs
• Raw 11-char video IDs

2. Invalid input must show alert: Please enter a valid YouTube URL.
3. Default loaded video ID: wLM5bzt1xks.
4. Use YouTube IFrame API (https://www.youtube.com/iframe_api).
5. Show title + duration once available.

5.2 Webcam Recording

1. Request getUserMedia with:

• Video: 640x480
• Audio: true

2. Use MediaRecorder with preferred mime:

• video/webm;codecs=vp9 if supported
• fallback video/webm

3. Record in 1-second chunks.
4. Show recording timer (mm:ss) and active indicator.
5. After stop, create preview player using URL.createObjectURL.
6. Provide Discard and Re-record actions.
7. On camera failure, show: Unable to access webcam. Please ensure camera permissions are granted.

5.3 Form Validation

1. Name required (non-empty).
2. Email required and regex-validated in UI.
3. Submission blocked if no recorded blob.
4. Show inline validation errors in form.

5.4 Recording Submission Logic

1. New recording (no share context):

• POST /api/recordings
• Include name, email, video, youtubeVideoUrl

2. Second recording (share context, second not yet present):

• POST /api/share/:uniqueLink/second-video
• Include name, email, video

3. On success for first recording:

• Show success message
• Generate and display share URL
• Load saved recording into first video panel immediately (no reload needed)

4. On success for second recording:

• Show success message
• Refresh shared recording data and show second video immediately (no reload needed)

5. On server/network failure:

• Show friendly error message.

5.5 Shared Recording Behavior

1. If URL has share query param, app must load recording via GET /api/share/:uniqueLink.
2. If record includes YouTube URL, cue corresponding source video.
3. If first/second local video exists, load into respective HTML5 video elements.
4. If second recording exists, disable additional recording UI.
5. If second recording does not yet exist (isSecondUserPending):
   • First video is blurred (filter: blur(15px)) with pointer events disabled.
   • Overlay text "Recording done, waiting for other person..." is shown centered on the blurred video.
   • Play/pause button and timeline slider are disabled.

5.6 Playback + Sync

1. Global play button label toggles:

• ▶ Play
• ⏸ Pause

2. Pressing play should reset all media to start state:

• YouTube at 0
• First local video at 2 seconds
• Second local video at 0

3. Timeline reflects YouTube current time (poll every 250ms while playing).
4. Seek operation:

• YouTube seek to t
• First local video seek to min(t + 2, duration1)
• Second local video seek to min(t, duration2)

5. If user presses play/pause on local videos, YouTube should sync play/pause.

5.7 Video Panels and Swap

1. Show two local video panels in one row on desktop.
2. Left panel is "First Recording" and marked with This video has 2 seconds headstart.
3. Right panel is "Second Recording".
4. If content missing, show placeholders:

• No recording yet
• Waiting for second recording...
• Record your interpretation above (in shared state with missing second video)

5. Swap button (⇄) exchanges displayed first/second recording content.
6. Swap disabled until second recording exists.

6. UI/UX Requirements

6.1 Page Layout (top to bottom)

1. YouTube URL input + Load button
2. YouTube title + duration
3. Embedded YouTube player
4. User form + webcam recorder (when allowed)
5. Timeline slider with current/duration text
6. Two local recording panels + swap button
7. Global play/pause control

6.2 Responsive Behavior

1. Mobile breakpoint around 600px.
2. URL input stack vertically on mobile.
3. Local video panels stack vertically on mobile.
4. Swap button rotates 90 degrees on mobile.

6.3 Visual Style Baseline

1. Dark UI theme.
2. Blue primary actions (#2563eb).
3. Red recording actions (#e63946).
4. Rounded cards, slider, and buttons.

7. Technical Architecture

7.1 Frontend

1. Stack: React + Vite.
2. Main components:

• YouTubePlayer (orchestrator + sync + state)
• WebcamRecorder (capture and preview)
• UserForm (metadata input + validation)

3. Frontend API base URL: http://localhost:3001.
4. Share state is URL-driven (window.location.search).

7.2 Backend

1. Stack: Node.js + Express + SQLite.
2. Middleware:

• cors()
• express.json()
• multer for multipart uploads

3. Upload constraints:

• max file size: 100MB
• accept only video/* mimetypes

4. File naming pattern:

• recording_<timestamp>.webm

5. Storage path:

• public/media/ (local filesystem)

7.3 Data Persistence Model

Single table: recordings

• id INTEGER PK AUTOINCREMENT
• unique_link TEXT UNIQUE NOT NULL
• name TEXT NOT NULL
• email TEXT NOT NULL
• recorded_video_path TEXT NOT NULL
• youtube_video_url TEXT NOT NULL
• name_2 TEXT NULL
• email_2 TEXT NULL
• recorded_video_path_2 TEXT NULL
• created_at DATETIME DEFAULT CURRENT_TIMESTAMP

Unique link generation: 8 random bytes hex string.

8. Backend API Contract

POST /api/recordings

Creates first recording.

Multipart fields:

• name (required)
• email (required)
• youtubeVideoUrl (optional, stored as empty string if missing)
• video file (required)

Success 200:

• success
• id
• uniqueLink
• recordedVideoPath
• message

Validation failure 400:

• error: Name, email, and video file are required

GET /api/recordings

Returns all recordings ordered by newest first.

GET /api/recordings/:id

Returns one recording by numeric ID.

404 if not found.

GET /api/share/:uniqueLink

Returns one recording by share token.

404 if not found.

POST /api/share/:uniqueLink/second-video

Adds second interpreter video to existing row.

Multipart fields:

• name (required)
• email (required)
• video file (required)

Errors:

• 404 recording not found
• 400 second video already recorded
• 400 missing required fields

Success 200:

• success
• recordedVideoPath2
• message

9. Error Handling and Edge Cases

1. Invalid YouTube URL input blocks load.
2. Camera permission denied shows inline error.
3. Missing recording on submit blocked client-side.
4. Backend returns generic 500 on server exceptions.
5. Shared link with invalid token should show not-found behavior (current app logs and remains mostly empty).
6. If local video shorter than seek target, clamp to video duration.
7. Newly uploaded video files may not be immediately available from Vite's dev server. A loadWithRetry mechanism retries .load() up to 5 times at 500ms intervals using native addEventListener('error') on the video element.
8. WebM files from MediaRecorder lack seek indices. Setting currentTime before metadata loads crashes the demuxer. Only set currentTime in onLoadedMetadata handlers.
9. React's synthetic onError prop on <video> elements is unreliable for media errors. Use native event listeners instead.

10. Browser/Platform Requirements

1. Modern Chromium/Safari/Firefox with support for:

• navigator.mediaDevices.getUserMedia
• MediaRecorder
• URL.createObjectURL
• Clipboard API (navigator.clipboard.writeText)

2. Desktop-first, mobile supported via responsive CSS.

11. Security and Privacy Baseline

1. No authentication.
2. No authorization; share link is access mechanism.
3. No encryption-at-rest beyond host defaults.
4. User email stored in plaintext SQLite.
5. CORS enabled broadly (default permissive).
6. Uploaded files are stored locally and directly web-accessible via /media/....

12. Local Development and Run Commands

1. Install dependencies: npm install
2. Run frontend + backend: npm run dev:all
3. Frontend only: npm run dev (Vite default port 5173)
4. Backend only: npm run server (port 3001)

13. Implementation Notes for Another AI Builder

1. Keep functionality equivalent to this baseline rather than redesigning feature behavior.
2. Preserve the 2-second headstart offset for first recording in all sync/seek/play reset paths.
3. Keep shared-link mode logic URL-driven.
4. Ensure form+recorder are hidden once both recordings exist.
5. Use multipart uploads and SQLite schema exactly unless explicitly changing architecture.
6. If deploying production, ensure /media static files are served from the same origin expected by frontend paths.
7. Video loading after upload uses a belt-and-suspenders approach: callback refs trigger loadWithRetry when video elements mount, and backup useEffect hooks watching sharedRecording.recorded_video_path / recorded_video_path_2 catch edge cases where the callback ref alone is insufficient.
8. Always clean up previous native error listeners before adding new ones on the same video element to prevent duplicate retry loops.
9. Never set currentTime on a video element before its metadata has loaded — use onLoadedMetadata handlers instead.

14. Acceptance Criteria

1. User can load a YouTube video and see title/duration.
2. User can record webcam video with audio and preview it.
3. User can submit first recording and receive usable share link.
4. Shared link opens same source video and first recording.
5. Second user can add second recording exactly once.
6. App can play/pause/seek source + local videos in sync.
7. First recording consistently plays with +2s offset.
8. User can swap left/right recording panels.
9. UI works on desktop and mobile widths.
10. After saving first recording, video appears immediately in the first video panel without page reload.
11. After saving second recording, video appears immediately in the second video panel without page reload.
12. Second user sees first video blurred with overlay text before recording, with playback controls disabled.