One image, many right answers
Image–text matching is one-to-many. The photo below comes with five human captions, and each one picks out something different. A model that compresses the image into a single vector has to average all of those readings into one point.
Five ground-truth captions for the same image. Hover them: each emphasises a different part of the scene.
Embedding sets
Represent every image and every caption with vectors instead of one, so different vectors can cover different readings of the same sample.
Degenerate sets
How you compare two sets decides what the sets learn. Existing set similarities train one element and ignore the rest, or pull all of them onto the same point.
Match one-to-one
Score a pair of sets by the optimal one-to-one assignment between their elements, and add two losses that keep the elements of a set apart.
Two ways a set goes wrong
Sparse supervision MIL · PVSE
Multiple-instance similarity keeps only the single best pair between two sets. The passes gradient to one element per set, so the others are never trained and never used at retrieval time.
Set collapse Smooth-Chamfer · Set Div
Smooth-Chamfer averages a soft maximum over all pairs, so every element gets gradient. But every element is pulled toward the same soft average of the other set, and the set shrinks to one point. The paper proves this with Jensen's inequality.
Maximal Pair Assignment Similarity
Take the matrix of cosine similarities between an image's elements and a caption's elements. Find the permutation that pairs every image element with a different caption element and makes the total similarity as large as possible. Only those pairs count, each scaled exponentially so strong matches weigh more.
Cosine similarity matrix · drag a cell up or down to change it
Where the gradient goes · line width =
Gold outlines mark the cells that receive gradient; the small number in each cell is its share. Rows are image elements , columns are caption elements . Everything here is computed live from the formulas in the paper.
Watch what each objective does to a set
A toy version of training on one matching image–caption pair. Four image elements (triangles) and four caption elements (circles) live on a circle, and each panel follows the gradient of one similarity function. This isolates the pull between a positive pair, which is where the three objectives differ.
Both sets start clustered, as they do in the real model where every element is an offset from one global feature. Elements closer than 5° count as merged; an image element is aligned when a caption element is within 8°. Smooth-Chamfer uses the α set above. Illustration only: two dimensions, one positive pair, no negatives. The measurements on the real models are in Evidence.
Six terms, two of them new
Matching removes the force that causes collapse. Two new losses then actively keep each set spread out. Select a term to see what it does.
Triplet loss with hardest negatives
For every image, the matching caption must score higher than the hardest non-matching caption in the batch by a margin , and the same in the other direction. The score is our , so this is where the one-to-one matching enters training.
= 0.2 (ResNet-152 + Bi-GRU), 0.3 (Faster R-CNN + Bi-GRU), 0.15 (Faster R-CNN + BERT).
Global Discriminative loss
Each set element is the global feature plus a learned offset. If the offsets are small, all elements sit on top of the global feature. This loss pushes every element away from its own global reference, so each element has to carry information the global feature does not.
= 0.1 (0.05 on Flickr30k with Faster R-CNN + Bi-GRU), = 0.6 or 0.8, = 0.5.
Intra-Set Divergence loss
Penalises high similarity between every pair of different elements inside the same set, for images and for captions. Elements that drift toward each other are pushed apart, so each one learns a different aspect of the sample.
= 0.1 (0.05 on Flickr30k with Faster R-CNN + Bi-GRU), = 0.6 or 0.8, = 0.5. is the number of elements per set.
Maximum Mean Discrepancy
Minimises the MMD between the distribution of image embedding sets and the distribution of caption embedding sets, which stops the two modalities from drifting apart early in training.
= 0.01 in every configuration.
Diversity regulariser
Penalises similar element slots inside the set prediction module. It is kept from prior work; on its own it does not prevent the collapse shown below, which is why GD and ISD are added on the final embeddings.
= 0.01 in every configuration.
Contrastive loss
Training with only the hardest negative can be unstable in the first epochs. A lightly weighted contrastive term aligns all matching image–caption pairs in the batch and smooths early training.
= 0.001 with Faster R-CNN features; not used with ResNet-152.
Feel the margin
GD and ISD share one shape: . Drag the coloured element around the circle and change and .
Two encoders, two sets, one matching
MaxMatch keeps the dual-encoder design that makes retrieval fast: images and captions are embedded independently, so image sets can be pre-computed and indexed. Step through the pipeline.
Training Optimal one-to-one matching
Scores inside the triplet and contrastive losses use with the optimal assignment, so every element receives gradient from a different partner.
Inference Top-k, no matching
At test time the one-to-one constraint is dropped: the most similar element pairs are selected directly. That removes the assignment step and keeps retrieval fast, while still using the diverse sets.
Original architecture figure from the paper
Figure 2 of the paper. Left: model overview. Right: the Global Discriminative loss pushes elements away from the global embedding, and the Intra-Set Divergence loss keeps elements within a set distinct.
The sets really are different
Three models trained with the same architecture and three different similarity functions. Average the element-to-element similarity over every matching image–caption pair in the test set and the failure modes show up directly.
Values from Figure 4 of the paper. Each panel uses its own colour scale, as in the paper. Hover a cell to read it.
A structure you can see
In a t-SNE of the learned space, MIL leaves most elements scattered and unaligned, and Smooth-Chamfer stacks all four elements of a sample on the same spot. With MaxMatch, the same element index from images and captions lands in the same region, and the four regions stay separate.
Figure 3 of the paper. Colours are set elements; dots are image embeddings, crosses are text embeddings.
Explore our embedding space
A t-SNE of the elements of 250 test images and one caption each, from our Faster R-CNN + BERT model. Points group by matched element pair: an image element and the caption element it is assigned to land in the same cluster. Their index numbers can differ, so a cluster may mix two colours. Hover a point to light up all eight elements of that image–caption pair.
Every element earns its place
Evaluate MS-COCO 5K using only one element of the image set at a time. If a set has collapsed, one element is as good as four. If supervision was sparse, most elements are useless alone.
Circular variance of the embedding sets on Flickr30k (Table 4 of the paper). A lower means the elements of a set point in nearly the same direction.
Better retrieval, same backbones
Recall@K and RSUM on the standard MS-COCO and Flickr30k test splits, against methods that use the same visual and text encoders. RSUM is the sum of R@1, R@5 and R@10 in both directions.
Full table for this setting
What each piece contributes
RSUM on MS-COCO 5K (Faster R-CNN + BERT, single model) as the similarity function and the loss terms change. Two things stand out.
The matching alone helps. With the same Div + MMD regularisers, swapping Smooth-Chamfer for Maximal Pair Assignment moves RSUM from 439.57 to 441.23.
GD and ISD help any similarity, and help ours most. They lift Smooth-Chamfer to 444.10, and Maximal Pair Assignment to 446.53, the best configuration.
RSUM on Flickr30k when our similarity and losses are plugged into PVSE (432.6 → 450.6). The method is not tied to one architecture.
RSUM on Flickr30k with a larger ResNeXt-101 + BERT backbone, vs 551.9 for Set Div and 550.9 for VSE∞.
Four elements, four points of view
Queries from the MS-COCO and Flickr30k test sets, run through our Faster R-CNN + BERT model and picked by hand. For each query we retrieve with every element of its set on its own, as in the paper's qualitative analysis: an element's score for a candidate is its best cosine against the candidate's elements.
These examples were chosen because the four elements disagree in interesting ways. On many test queries all four elements return the same top result.
Against Set Div, from the paper
The paper runs the same per-element retrieval for Set Div (trained with Smooth-Chamfer). In its examples, Set Div returns the same or nearly the same result for all four elements, while MaxMatch returns different results that still fit the query.
Where it stands, and how to cite it
@inproceedings{alomari-etal-2025-maximal,
title = "Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval",
author = "Alomari, Hani and
Sivakumar, Anushka and
Zhang, Andrew and
Thomas, Chris",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.1533/",
doi = "10.18653/v1/2025.acl-long.1533",
pages = "31769--31785"
}
We acknowledge Advanced Research Computing at Virginia Tech for computational resources and technical support. MaxMatch builds on the architecture of Set Div (Kim et al., CVPR 2023) and PVSE (Song & Soleymani, CVPR 2019).