The Many Senses of Visual Similarity:
A Text-Prompted Image Perceptual Metric

We introduce TPIPS: an image similarity metric that is text-conditioned.

A single scalar distance cannot say how two images are alike. A reference is compared against fixed candidates A/B/C/…: prior methods (LPIPS, DreamSim, SSIM, L2) each pick one candidate overall, while TPIPS picks a different one per text aspect. Hover any method or aspect (tap on mobile) to see its per-candidate scores on the gauges beside each image; click an image to zoom.

"pip install tpips" (Coming Soon!)

pip install package and code release will be out soon. Stay tuned!

Aspect-Conditioned Retrieval

Given a query photo, we apply TPIPS to retrieve its nearest neighbors conditioned on different visual aspects. Retrieval is done from OpenImages, a dataset of 1.74M real-world images. Conditioning on overall recovers ordinary overall similarity. To encourage retrieval diversity, we adjust the ranking score by subtracting 0.3 of the overall similarity from the aspect similarity. Use the arrows to switch queries; click any image to zoom.

Similarity on a video

The same aspect-conditioned similarity extends to video: pick a clip, pick a factor, and pick a reference frame — the curve shows how similar every other frame is to that reference along the chosen aspect, and updates live as the video plays.

How it works

1. Collect aspect-conditioned similarity from humans

To train a text-conditioned similarity model, we curate triplets of synthetic images and ask 5 different annotators an odd-one-out question for each visual aspect. To test generalization, a second smaller held-out set covers outputs from external algorithms—image editing, compositing, novel-view synthesis and image-to-3D—annotated with a 2AFC protocol.

2. Learn aspect-conditioned pairwise similarity

When an image is the odd-one-out of a triplet conditioned on an aspect, the similarity between the two non-chosen images should intuitively be the greatest. We predict this odd-one-out selection by applying a softmax over the pairwise similarities, and train the embeddings using a cross-entropy loss.

More Results

Results on our dataset

Across recent open and proprietary vision-language models, embedding models, and activation-difference baselines, TPIPS gives the closest agreement with human aspect-conditioned similarity judgments — both in-domain (odd-one-out) and out-of-domain (2AFC on real vision-algorithm outputs).

Compositional retrieval