TowerBridgeBridging the Dual-Encoder Modality Gap via Weight-Space Programming
Virginia Tech
In one minute
Attenuate the directions that favor hard negatives. Do not remove them.
CLIP-style dual encoders embed images and text with two separate towers. The two modalities occupy distinct regions of the shared space, separated by a modality gap, which is the usual suspect when a wrong match outranks the correct one. We show what should change in the weights where the two towers meet.
Insight
A paired statistic identifies directions where the two towers agree more on hard negatives than on matched pairs. Removing the leading ones, which align with the common mean and the gap, degrades retrieval on every checkpoint. Shrinking them by 10% improves every one, and the gap's size is largely preserved.
Discovery
We arrive at this correction through weight-space programming. Guided by the statistic, an LLM agent that may edit any component of either tower improves COCO test retrieval on all 13 CLIP checkpoints, and 12 of its 13 final patches edit an output projection.
Method
TowerBridge distills and refines the agent's recurring correction into a closed-form recipe. Computed in seconds from 25,000 labelled pairs, it improves all 21 evaluated CLIP checkpoints and 4 of 5 dual encoders from other families, with no agent and no model-specific tuning.
The problem
A wrong image outranks the correct one
Image–text retrieval ranks candidates by the cosine similarity of two embeddings, one from each tower. A caption can retrieve the wrong image even when its correct match is available.
These high-scoring incorrect matches are hard negatives. Many of them are hubs: images that are retrieved for many captions they do not match. For OpenAI's ViT-B/16, 41% of text-to-image misses on COCO land on hub images.
The two modalities also sit apart in the shared space, separated by a modality gap. Closing it is the natural remedy, yet in our experiments removing the gap degrades retrieval on every checkpoint. Fine-tuning needs gradient-based optimization and hyperparameter search, and test-time re-ranking needs a bank of queries or reference samples and leaves the model unchanged.
The examples on the right are real COCO validation queries. Switch between the zero-shot model and the edited model to see the correct photo replace a hub.
The paired statistic
Where do the two towers agree on the wrong match?
To guide an edit, we need to know which parts of the shared representation favor wrong matches. We compare matched pairs with their hard negatives, the incorrect candidates with the highest similarity scores.
For each caption we collect its five highest-scoring wrong images, and for each image its 25 highest-scoring wrong captions. With unit-normalized image embeddings êv and caption embeddings êt, the paired statistic is the d × d matrix
The first term measures agreement on matched pairs. The other two measure agreement on hard negatives, weighted equally across the two retrieval directions.
- The trace is the margin. The trace of D is the mean similarity margin between matched pairs and hard negatives.
- The eigenvalues split it by direction. The eigenvalues of sym(D) = (D + D⊤) / 2 decompose this margin into contributions along orthogonal directions.
- Negative eigenvalues are targets. A negative eigenvalue marks a hard-negative direction: one where hard negatives agree more than matched pairs.
The insight
A mild shrink helps. Removal hurts.
The two leading hard-negative directions align with the common mean of the embeddings and with the modality gap. They carry useful content, which a mild shrink retains. Drag the strength from a mild shrink to full removal and watch retrieval change on 24 CLIP checkpoints.
Which directions matter
Removing the two leading directions costs 48.6 COCO validation RSUM on average, and centering each modality costs 20.2. Shrinking the same two directions by 10% gains 7.8, about 60% of the gain of one full round.
The choice of directions matters as much as the strength. A 10% shrink of the hard-negative directions gains +12.5 on average, against at most +0.3 for as many top principal directions chosen without pair labels.
Bridging
The correction widens the margin to the hardest negative and reduces hubness, while the modality gap remains largely unchanged: its median size changes by +0.1% over 18 checkpoints. We call this effect bridging.
The discovery
Weight-space programming with an LLM agent
We arrive at this correction through weight-space programming: an LLM agent, Astra, inspects both towers and writes short programs that directly edit their weights without gradients. We design and supply the paired statistic, and the agent chooses where and how to intervene.
What the agent can do
In each session the agent works on one checkpoint. It has read-only access to both towers' weights and to 35,000 COCO training images with five captions each. It can compute over the weights without gradients, inspect activations and per-head contributions, and examine the paired statistic for pairs of tower components.
Every edit is a closed-form program that can be inspected and re-run without the agent. Programs may scale attention heads, remove directions from heads or MLPs, add low-rank updates, apply a shared matrix to both projections, or rewrite up to four whole tensors. Backpropagation, iterative fitting and regression to data targets are disallowed.
A candidate replaces the current patch only if its COCO validation score is higher by more than two paired standard errors and ImageNet accuracy drops by at most 0.5 points. The test sets are sealed: no tool can read them, and each final patch is evaluated once after its session ends.
Paired evidence guides the search. In controls, removing the statistic reduces the agent's gains more than removing access to the raw weights.
The method
TowerBridge: a closed-form recipe in three steps
Nine of the 13 sessions produce projection edits that attenuate directions with negative eigenvalues. We distill the ViT-B/32 session's uniform shrink and refine it with a second round. The recipe takes seconds for each model and requires no agent.
The whole method in a few lines
# ev, et: unit embeddings of 5,000 images, 25,000 captions def towerbridge(Wv, Wt, ev, et, pairs): S = eye(d) for s in (0.10, 0.05): # two rounds iv = normalize(ev @ S.T) it = normalize(et @ S.T) # hard negatives are re-mined each round negs = mine_hard_negatives(iv, it, pairs) D = paired_statistic(iv, it, pairs, negs) lam, V = eigh((D + D.T) / 2) U = V[:, : d // 4] # d/4 smallest S = (eye(d) - s * U @ U.T) @ S return S @ Wv, S @ Wt # same architecture
The subspace size, the two strengths and the numbers of hard negatives are fixed across checkpoints. The matrix S is recomputed from each checkpoint's own embeddings.
Why a uniform shrink
With fixed embeddings and mined negatives, the bilinear hard-negative margin equals ⟨M, sym(D)⟩, where M = S⊤S is the metric induced by the edit. We bound the contraction in each direction and in total:
Under these bounds a maximizer is M = I − c UkU⊤k: a uniform shrink of the k directions with the smallest eigenvalues. Setting k = d/4 and c = 2s − s2 = 0.19 recovers the first round.
This analysis concerns the bilinear surrogate and motivates the form of the step. Under the cosine score used for retrieval, the uniform shrink also outperforms eigenvalue-weighted shrinkage at matched first-order gain, by 8.0 COCO validation RSUM on average across 13 checkpoints.
Overview figure from the paper

Results
One fixed recipe improves every CLIP checkpoint we evaluate
TowerBridge applies the same rule to each checkpoint and computes its correction from that model's own embeddings. All values are changes in RSUM, the sum of recall at 1, 5 and 10 in both retrieval directions, relative to the zero-shot checkpoint.
Beyond CLIP
The same correction significantly improves Flickr30k retrieval, within the ImageNet tolerance, on 4 of 5 evaluated dual encoders from other families. For SigLIP, which lacks a vision projection, the map is applied as a linear layer after the vision encoder.
Fewer hub errors
On held-out COCO pairs, the margin to the hardest negative widens in both directions on all 21 checkpoints, and hubness decreases on all 21.
Ablations and comparisons
What makes the correction work
Each design choice of the recipe is tested on its own: which directions are shrunk, how they are weighted, whether hard negatives are re-mined, which towers are edited, and how many images are needed.
How it compares, and how it combines
TowerBridge is written into the checkpoint. It involves no gradient-based optimization or hyperparameter search, needs no test-time reference bank, and adds no inference cost, so it can be combined with test-time re-ranking.
Scope
The recipe requires labelled image–text pairs and currently targets dual encoders. Trained maps and re-rankers with an in-domain bank can achieve larger retrieval gains. Overall SugarCrepe accuracy is nearly unchanged (+0.06 points across 21 checkpoints): replace and swap improve by 1.1 and 1.6 points, while add decreases by 1.9.
Release
The paper is under submission. We will release code, prompts and logs. The interactive figures on this page are drawn from the same result files as the paper's figures and tables.
Citation
BibTeX
@misc{alomari2026towerbridge,
title = {TowerBridge: Bridging the Dual-Encoder Modality Gap
via Weight-Space Programming},
author = {Alomari, Hani and Atabuzzaman, Md and Thomas, Chris},
year = {2026},
note = {Under submission}
}