IP-Adapter FaceID Plus v2

Про Tencent (ARC Lab)

Засновано 1998 · Shenzhen, China

IP-Adapter is a family of open-source image-prompt adapters developed by the Applied Research Center (ARC Lab) at Tencent. The original IP-Adapter paper (Ye et al. 2023, arXiv 2308.06721) introduced a decoupled cross-attention scheme that lets a pretrained text-to-image diffusion model accept image prompts as a parallel conditioning channel without retraining the backbone. The FaceID family extends this idea with face-identity embeddings (extracted by InsightFace's ArcFace recogniser) instead of CLIP image embeddings, achieving high identity preservation. IP-Adapter FaceID Plus v2 is a refined version released by author Hu Ye in late 2023 / early 2024 combining ArcFace identity vectors with CLIP image features for better likeness and prompt control. The code and weights are released open-source on GitHub (tencent-ailab/IP-Adapter).

Архітектура

Cross-attention adapter (decoupled image prompt) on top of SDXL/SD1.5

IP-Adapter FaceID Plus v2 is an adapter module attached to a frozen Stable Diffusion XL (or SD 1.5) backbone. The architecture is based on the IP-Adapter design (Ye et al. 2023): a small MLP projector maps an external image embedding into the cross-attention space of the diffusion U-Net, and a parallel set of cross-attention layers is added so that text and image conditioning operate on decoupled keys/values. The 'FaceID' line replaces the CLIP image encoder with InsightFace's ArcFace face recogniser, which produces a 512-dim identity embedding focused on facial geometry rather than appearance. The 'Plus v2' variant combines the ArcFace embedding with a CLIP image embedding of the face crop, giving the model both identity (from ArcFace) and pose/lighting cues (from CLIP). At inference the user supplies one or more reference face images plus a text prompt, and the adapter steers the diffusion process toward a portrait that matches the identity while following the prompt. Training used a large face dataset with paired identity vectors and aligned face crops.

Параметри

~100M adapter parameters on top of a frozen SDXL or SD1.5 backbone

Контекст

75 токенів

Можливості

• High-quality face-identity preservation from one reference image
• Works on top of SDXL and SD 1.5 with frozen weights
• Compatible with LoRAs, ControlNets and other adapters
• Open-source under non-commercial / academic terms
• Strong prompt control for clothing, scene and style while preserving face
• Used as the backbone for many consumer 'AI photo' apps in 2024
• Best for: avatar generation, personalised marketing, headshot apps, character consistency.

Навчання та ліцензія

Trained on large public face datasets (LAION subsets, FFHQ-style data) with InsightFace embeddings extracted per face. Exact composition is not disclosed.

Ліцензія: Open-source weights under the Tencent IP-Adapter license, which allows research and non-commercial use. Commercial use generally requires negotiation with Tencent ARC Lab.

Тестування безпеки: No formal red-team report. The adapter is identity-preserving by design and has obvious deepfake misuse potential; downstream operators are responsible for consent and abuse controls.