TPIPS: The Many Senses of Visual Similarity

We introduce TPIPS: an image similarity metric that is text-conditioned.

Loading teaser…

A single scalar distance cannot say how two images are alike. A reference is compared against fixed candidates A/B/C/…: prior methods (LPIPS, DreamSim, SSIM, L2) each pick one candidate overall, while TPIPS picks a different one per text aspect. Hover any method or aspect (tap on mobile) to see its per-candidate scores on the gauges beside each image; click an image to zoom.

Quick Start

TPIPS supports Python >=3.10. Install matching PyTorch and torchvision builds from the official PyTorch installer, then install TPIPS:

pip install tpips

You can then use TPIPS to compute aspect-conditioned similarity between two images.

import torch
import tpips
from PIL import Image

model = tpips.load_model("embedding", device="cuda")

image_a = Image.open("a.jpg").convert("RGB")
image_b = Image.open("b.jpg").convert("RGB")

with torch.inference_mode():
    similarity = model.similarity(image_a, image_b, factor="lighting")
    distance = model.distance(image_a, image_b, factor="lighting")

print(similarity.item())  # higher means more similar
print(distance.item())    # lower means more similar

The factor argument is free-form text, specifying the visual aspect that similarity is conditioned on (e.g., "lighting", "the pose of the person", "number of ducks"). Pass factor="overall" (or None, "") for prompt-independent similarity.

For model details and the complete API reference, see the TPIPS documentation.

Aspect-Conditioned Retrieval

Given a query photo, we apply TPIPS to retrieve its nearest neighbors conditioned on different visual aspects. Retrieval is done from OpenImages, a dataset of 1.74M real-world images. Conditioning on overall recovers ordinary overall similarity. To encourage retrieval diversity, we adjust the ranking score by subtracting 0.3 of the overall similarity from the aspect similarity. Use the arrows to switch queries; click any image to zoom.

Loading retrieval…

Similarity on a video

The same aspect-conditioned similarity extends to video: pick a clip, pick a factor, and pick a reference frame — the curve shows how similar every other frame is to that reference along the chosen aspect, and updates live as the video plays.

Loading video demo…

How it works

1. Collect aspect-conditioned similarity from humans

To train a text-conditioned similarity model, we curate triplets of synthetic images and ask 5 different annotators an odd-one-out question for each visual aspect. To test generalization, a second smaller held-out set covers outputs from external algorithms—image editing, compositing, novel-view synthesis and image-to-3D—annotated with a 2AFC protocol.

Dataset collection figure — **Datasets.** (left) Odd-one-out judgments on synthetic triplets. (right) 2AFC judgments on outputs of external vision algorithms.

2. Learn aspect-conditioned pairwise similarity

When an image is the odd-one-out of a triplet conditioned on an aspect, the similarity between the two non-chosen images should intuitively be the greatest. We predict this odd-one-out selection by applying a softmax over the pairwise similarities, and train the embeddings using a cross-entropy loss.

Learning objective figure — **Learning similarity from odd-one-out data.** Given this triplet (x₁, x₂, x₃) and "lighting" condition in this example, x₁ is the odd-one-out with its warmer lighting. All three image pairs are compared using a similarity function, *f_θ*, conditioned by the aspect (orange arrows). This yields three pairwise similarities *s₁₂*, *s₁₃*, *s₂₃*. A softmax is then applied to these similarities to predict the odd-one-out distribution (ŷ, pink bar chart). The model is trained by actual human judgements using cross entropy.

More Results

Results on our dataset

Across recent open and proprietary vision-language models, embedding models, and activation-difference baselines, TPIPS gives the closest agreement with human aspect-conditioned similarity judgments — both in-domain (odd-one-out) and out-of-domain (2AFC on real vision-algorithm outputs).

Human agreement bar plot — **Human agreement.** Ours (red) vs. baselines across early/mid/late fusion families. Left is in-domain, odd-one-out results. Right is out-of-distribution 2AFC results. Error bars are standard error.

**Qualitative comparison.** (Left) TPIPS predicts odd-one-out choices that aligns with human's judgement more often that other baselines. (Right) Likewise, TPIPS agrees with human more often for out-of-distribution 2AFC set, too.

Compositional retrieval

BibTeX

@article{wang2026tpips,
      title={The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric},
      author={Wang, Sheng-Yu and Nitzan, Yotam and Hertzmann, Aaron and Zhu, Jun-Yan and Shechtman, Eli and Efros, Alexei A. and Zhang, Richard},
      journal={arXiv preprint arXiv:2607.18237},
      year={2026}
}