Digital video is a chain of discrete steps, each of which can be tuned or replaced in software:

1. Photon capture → electrical charge on a CMOS/CCD photosite.
2. Demosaicing the Bayer or X‑Trans pattern into RGB.
3. Linear → logarithmic tone mapping (dynamic‑range compression).
4. Color–space conversion (e.g. RAW → YC​BCr or RGB).
5. Compression (intra‑frame & inter‑frame).
6. Packetising into a container (.mp4, .mov, .mkv …).
7. Transmission, storage, or real‑time playback.

Note — Each stage exposes programmable hooks: sensor registers, ISP parameters, encoder presets, transport latency budgets, & more.

2.1 CMOS vs CCD

CMOS dominates modern cameras for its rolling‑shutter read‑out, lower power, and per‑pixel ADCs. CCD still appears in high‑end broadcast where global‑shutter artefacts matter.

2.2 Critical Metrics

• Resolution = W × H pixels (▲ impacts bandwidth & DRAM).
• Pixel size µm → well‑depth (dynamic range) vs. low‑light noise.
• SNR₁, Read Noise, DSNU/PRNU.
• Native bit‑depth (10‑, 12‑, 14‑bit) before companders.
• Frame‑rate, exposure time, and blanking.

Note — On‑die PLLs & I²C registers expose most parameters; use v4l2‑ctl (Linux) or vendor SDKs to poke them.

3.1 RAW → RGB → Y′C_BC_R

Programmatic processing usually follows:

// Pseudocode
rawFrame = sensor.read();                     // Bayer 12‑bit
rgbFrame = demosaic(rawFrame, 'VNG');         // edge‑aware
rgbLinear = applyBlackLevel(rgbFrame);
rgbGamma  = applyTransfer(rgbLinear, sRGB);
yuv420    = convertColor(rgbGamma, ITU‑R BT.709);

3.2 Chroma Sub‑sampling

Scheme	Band‑width	Use‑case
4:4:4	100 %	Keying / VFX
4:2:2	66 %	Broadcast / ProRes HQ
4:2:0	50 %	Streaming / H.264

• MIPI CSI‑2 — mobile SoCs & SBCs (Raspberry Pi / Jetson).
• USB UVC — class drivers for webcams.
• GigE Vision / CoaXPress — industrial machine vision.
• SDI / HDMI — uncompressed HD/4K transport over coax.
• SRT / WebRTC — low‑latency IP contribution.

Note — Many “USB cameras” simply bridge MIPI + ISP + UVC ⇢ USB3.

5.1 Universal Controls

• ExposureTime (µs) — beware rolling‑shutter overlap.
• Gain/ISO — digital amplification after ADC.
• WhiteBalanceGain{R,G,B}.
• FrameDuration / FPS.
• RegionOfInterest (binning & windowing).

5.2 API Examples

/* Swift – AVFoundation */
let device = AVCaptureDevice.default(.builtInWideAngleCamera,
                                     for: .video,
                                     position: .back)!
try device.lockForConfiguration()
device.setExposureModeCustom(duration: CMTimeMake(value: 1, timescale: 120),
                              iso: 200, completionHandler: nil)
device.whiteBalanceMode = .locked
device.unlockForConfiguration()

// C++ – OpenCV (V4L2 backend)
cv::VideoCapture cap(0, cv::CAP_V4L2);
cap.set(cv::CAP_PROP_FRAME_WIDTH, 1920);
cap.set(cv::CAP_PROP_FPS,         60);
cap.set(cv::CAP_PROP_EXPOSURE,    -4);  // auto‑exposure: ‑1, manual: value

6.1 Real‑Time Preview

/* JS – getUserMedia */
const stream = await navigator.mediaDevices.getUserMedia({ video: { width: 1280 }});
videoElem.srcObject = stream;

6.2 Batch Recording with FFmpeg

$ ffmpeg -f avfoundation -framerate 30 -i "0" \
         -c:v libx264 -preset veryfast -crf 22 out.mp4

6.3 Low‑Latency NDI / WebRTC

Use libndi_send_create() or mediasoup‑client to inject encoded frames into production pipelines with ~ 100 ms glass‑to‑glass.

7.1 GPU Shaders / Compute

Leverage Metal Performance Shaders, CUDA, or OpenCL for de‑bayer, noise‑reduction, and LUTs at >200 FPS.

7.2 Hardware Encoders

• Apple VTEncoder (VTCompressionSessionEncodeFrame).
• NVIDIA NVENC (cuvidEncodePicture).
• Intel QuickSync (MFXVideoENCODE_EncodeFrameAsync).

7.3 Transcoding Matrix

Profile	Bit‑rate @1080p 30 fps	Latency (≈)
H.264 Baseline	2‑4 Mb/s	< 100 ms
H.265 Main	1‑2 Mb/s	200‑300 ms
AV1 Main	0.6‑1.5 Mb/s	800 ms+

• Genlock / Tri‑Level‑Sync for multi‑camera SMPTE.
• PTS/DTS stamping → playback order vs. decode order.
• Double‑/triple‑buffering avoids tearing; CVPixelBufferPool on iOS.
• Clock drift management via ntpd / PTP (IEEE 1588).

Popular wrappers embed audio, timecode, and user data:

• .mov / QuickTime — rich metadata keys.
• .mxf — OP‑Atom vs. OP‑1A for broadcast interchange.
• .mkv — open container with EBML structure.

Note — Use sidecar JSON/XMP if your metadata schema exceeds container support.

• Profile memory copy paths (DMA, zero‑copy, mmap()).
• Enable perf stat, powermetrics for CPU/GPU throttling.
• Use ffprobe ‑show_packets to detect CTS/DTS gaps.
• Apply adaptive bitrate (ABR) ladders for live OTT.

• OBS Studio – rapid pipeline validation.
• MediaInfo – container/codec inspection.
• qv4l2, CameraControl – register tweaking.
• NDI Studio Monitor – network feed QC.

• SMPTE ST‑2110 (IP‑based video production).
• ITU‑R BT.2020 / BT.2100 (HDR, Wide Color Gamut).
• MIPI Alliance CSI‑2 Spec. (paid).
• ISO/IEC 23008‑2 (HEVC / H.265).
• FFmpeg Docs & AVFoundation Programming Guide.