Kosu E-learning Platform

Kosu - End-to-end E-learning Platform

Large video uploads · Queue-based payment webhooks · Device-aware session control

Role: Solo developer across frontend and backend. I designed the architecture, implemented the product workflow, and handled most of the deployment path myself.

Next.js 15 · NestJS · PostgreSQL · Redis/BullMQ · AWS S3/CloudFront · Docker · GitLab CI/CD

Kosu is a learning platform for learners, instructors, and operators. The main engineering challenge was handling expensive work such as video encoding and payment webhooks without slowing down the user-facing API for checkout, lesson playback, or progress tracking.

System Architecture

flowchart LR
  Client[Client Web App<br/>Next.js] -->|HTTPS| API[NestJS API]
  API -->|SQL| DB[(PostgreSQL)]
  API -->|Cache/Session/Progress| Redis[(Redis)]
  API -->|Enqueue| Queue[BullMQ Queues]
  Queue --> Worker[NestJS Worker]
  Worker --> DB
  Worker --> Redis
  Worker --> Storage[(AWS S3)]
  Storage --> CDN[CloudFront CDN]
  Client -->|HLS/MP4| CDN
  API <--> Payment[Stripe / VNPay / SePay]
  API --> Obs[Observability<br/>Pino + Sentry + Health checks]
  Worker --> Obs

The system is split into three runtimes: API for synchronous user-facing work, Worker for heavy jobs, and Scheduler for cron-driven tasks. PostgreSQL stores relational data, Redis handles hot-path state such as sessions and progress, and S3 + CloudFront deliver media.

That split matters because the product regularly mixes latency-sensitive requests with expensive asynchronous work. Background video processing should never block enrollment, checkout, or lesson playback.

Video Upload & Processing Pipeline

Problem: Large video uploads can choke the main API, fail halfway through unreliable networks, and leave instructors with no clue which processing step is still running.

Solution: Use multipart upload directly to S3, then hand the processing flow to BullMQ workers with explicit step-by-step progress tracking.

Implementation highlights:

The client uploads directly to S3 using multipart presigned URLs
The API only coordinates metadata and enqueues processing jobs
Workers run a seven-step flow: download, inspect, thumbnail, HLS encode, MP4 encode, upload artifacts, cleanup
The frontend polls progress every few seconds and shows the current step
Paid video delivery uses signed CloudFront cookies

Result: The upload flow became production-safe and understandable to end users. Instructors can see whether the system is still uploading, encoding, or publishing rather than staring at an ambiguous spinner.

sequenceDiagram
  participant Client
  participant API
  participant DB
  participant Redis
  participant Queue
  participant Worker
  participant S3
  participant CDN as CloudFront

  Client->>API: POST /files/video/initiate
  API->>DB: create file (UPLOADING)
  API->>S3: create multipart + presigned URLs
  API-->>Client: file_id + upload URLs
  Client->>S3: PUT chunk parts (parallel)
  Client->>API: POST /files/video/complete(parts)
  API->>Redis: init progress hash
  API->>Queue: enqueue video flow jobs
  API->>DB: file -> PENDING
  loop every 3s
    Client->>API: GET /files/video/:id/progress
    API->>Redis: read step/percentage
    API-->>Client: progress response
  end
  Queue->>Worker: deliver job
  Worker->>S3: download/upload artifacts
  Worker->>DB: update file statuses + metadata
  Worker->>Redis: update progress steps
  Client->>API: GET /files/:id/cookies
  API-->>Client: signed CloudFront cookies
  Client->>CDN: GET .m3u8/.ts with cookies
  CDN->>S3: fetch cached/origin segments
  CDN-->>Client: stream video

Queue-first Payment Webhook

Problem: Payment webhooks are easy to get wrong. If the system performs all business logic synchronously, slow database writes or duplicate gateway callbacks can create inconsistent enrollment records.

Solution: Treat the webhook as a fast acknowledgment endpoint. Verify the signature, enqueue the job, and let workers process the real business state inside a transaction.

Implementation highlights:

The webhook endpoint validates the payload and returns 200 ACK immediately
Workers update payment and order state transactionally
Idempotency is enforced through transaction IDs and final-state guards
The design supports multiple gateways without changing the core pattern

Result: Payment handling remains durable under retries and duplicate events while the public webhook stays fast and predictable.

sequenceDiagram
  participant Client
  participant API
  participant DB
  participant Redis
  participant Queue
  participant Worker
  participant Payment as Payment Gateway

  Client->>API: Create order + payment intent
  API->>DB: create order/payment_record (PENDING)
  API-->>Client: payment URL / client secret
  Client->>Payment: complete payment
  Payment->>API: webhook callback
  API->>API: verify signature + validate payload
  API->>Queue: enqueue payment job
  API-->>Payment: 200 ACK (fast)
  Queue->>Worker: process payment job
  Worker->>DB: idempotent update payment/order in transaction
  Worker->>Redis: invalidate cart/session related cache
  Worker->>Queue: enqueue enrollment job
  Worker->>DB: create course_student enrollment
  Client->>API: poll order/payment status
  API->>DB: read final status
  API-->>Client: PAID + enrolled

Multi-device Authentication

Problem: A single refresh token stored on the user record only supports one active session. Logging in on a second device invalidates the first device and makes session revocation coarse-grained.

Solution: Move to session-based auth. Each device gets its own session row, tokens carry session_id and device_id, and refresh token rotation uses compare-and-swap semantics.

Implementation highlights: