57 lines
No EOL
4.4 KiB
Markdown
57 lines
No EOL
4.4 KiB
Markdown
# PyFaceBlur Developer & AI Guide
|
|
|
|
This document provides architectural context, data flow maps, and strict guidelines for human developers and AI coding assistants working on the `py-faceblur` codebase.
|
|
|
|
> **Note to AI Assistants:** Read this entire document before proposing architectural changes or adding dependencies to this project.
|
|
|
|
## Architecture & Data Flow
|
|
|
|
The project is structured as a linear, sequential CLI pipeline. It extracts frames, detects faces, clusters them by identity, asks the user for input, and re-encodes the video.
|
|
|
|
### 1. Module Responsibilities
|
|
|
|
- **`src/faceblur/app.py`**: The main entry point. Orchestrates the sequential CLI flow using `rich` for UI/progress and `questionary` for user prompts.
|
|
- **`src/faceblur/video.py`**: Uses `ffmpeg` (via subprocess) to extract frames from the input video at a specified interval.
|
|
- **`src/faceblur/detect.py`**: Wraps the **UniFace** library. Uses RetinaFace for bounding boxes and landmarks, and ArcFace for generating 512-dimensional facial embeddings.
|
|
- **`src/faceblur/cluster.py`**: Takes the 512-dim embeddings and runs **DBSCAN** (from `scikit-learn`) using the **cosine distance** metric to group identical faces into clusters.
|
|
- **`src/faceblur/blur.py`**: Contains the image manipulation logic. Responsible for drawing blurs (Gaussian, pixelate, etc.) and handling the linear interpolation of bounding boxes between keyframes.
|
|
- **`src/faceblur/encode.py`**: Probes the video, auto-detects hardware encoders, and pipes raw BGR bytes from OpenCV to an `ffmpeg` subprocess to re-encode the video with blurs applied.
|
|
- **`src/faceblur/cli.py`**: The legacy/debug CLI entry point. It runs the pipeline but dumps raw frames and face crops to disk instead of re-encoding a video.
|
|
|
|
### 2. The Data Pipeline
|
|
|
|
1. **Extraction**: Video -> Every Nth Frame saved to `/tmp/.../frames/`
|
|
2. **Detection**: Frame -> `FaceData` dataclass (contains `id`, `bbox (x1, y1, x2, y2)`, `embedding`, `landmarks`).
|
|
3. **Clustering**: List of `FaceData` -> List of `Cluster` dataclasses.
|
|
4. **Interpolation (Encoding)**:
|
|
- A lookup table of keyframes is built.
|
|
- For frames between keyframes, bounding boxes are **linearly interpolated**.
|
|
- If a face enters or leaves, the bounding box is held **static** to the nearest known location to prevent split-second privacy exposures.
|
|
5. **Muxing**: OpenCV reads frames -> Blur applied -> Piped as `rawvideo` to FFmpeg -> FFmpeg copies original audio and encodes video.
|
|
|
|
## Core Dependencies
|
|
|
|
- **UniFace (`uniface`)**: The core AI engine for detection and recognition. Do not swap this out for MediaPipe, dlib, or plain OpenCV Haar cascades. It uses ONNX runtime under the hood.
|
|
- **FFmpeg**: Required on the host system. Used for both initial frame extraction and final video re-encoding.
|
|
- **OpenCV (`opencv-python`)**: Used exclusively for image reading/writing and drawing the actual blur arrays. *Not* used for video demuxing/muxing (FFmpeg handles that).
|
|
- **scikit-learn**: Used purely for the `DBSCAN` clustering algorithm.
|
|
- **Rich & Questionary**: Used for the terminal user interface.
|
|
|
|
## Guidelines & Strict Rules for AI Agents
|
|
|
|
When modifying this codebase, adhere to the following rules:
|
|
|
|
1. **Hardware Encoding Fallback is Sacred**:
|
|
In `encode.py`, the `find_best_encoder()` function meticulously probes the system for `av1_vaapi`, `hevc_vaapi`, `h264_vaapi`, `h264_nvenc`, etc., using a `1280x720` test video. Do not simplify or break this logic. If you modify it, ensure hardware encoders are prioritized over CPU (`libopenh264`).
|
|
|
|
2. **Bounding Box Format**:
|
|
Bounding boxes are strictly formatted as `(x1, y1, x2, y2)`. Do not attempt to use `(x, y, w, h)` anywhere in the pipeline.
|
|
|
|
3. **Dependency Discipline**:
|
|
Do not add heavy dependencies (like PyTorch or TensorFlow) unless explicitly requested by the user. UniFace running on ONNX is specifically chosen to keep the footprint light (~50MB).
|
|
|
|
4. **UI Paradigm**:
|
|
The CLI uses a synchronous, sequential prompt flow (`questionary`). Do not introduce asynchronous UI frameworks (like Textual) or background worker threads. The application blocks while waiting for user input, and blocks while processing with a `rich` progress bar.
|
|
|
|
5. **Temp File Cleanup**:
|
|
The pipeline generates gigabytes of raw frames in a temp directory during execution. Ensure any new exit paths or error states properly trigger the `shutil.rmtree(temp_dir)` cleanup block in `app.py`. |