Under submission

TowerBridgeBridging the Dual-Encoder Modality Gap via Weight-Space Programming

Hani Alomari, Md Atabuzzaman, Chris Thomas

Virginia Tech

+16.8COCO test RSUM, mean over 21 CLIP checkpoints
21/21CLIP checkpoints improve, 8 never seen by the agent
+6.6Flickr30k test RSUM, with ImageNet accuracy within 0.5 points
8 sto build the edit for ViT-B/16, with no added inference cost

In one minute

Attenuate the directions that favor hard negatives. Do not remove them.

CLIP-style dual encoders embed images and text with two separate towers. The two modalities occupy distinct regions of the shared space, separated by a modality gap, which is the usual suspect when a wrong match outranks the correct one. We show what should change in the weights where the two towers meet.

Insight

A paired statistic identifies directions where the two towers agree more on hard negatives than on matched pairs. Removing the leading ones, which align with the common mean and the gap, degrades retrieval on every checkpoint. Shrinking them by 10% improves every one, and the gap's size is largely preserved.

Discovery

We arrive at this correction through weight-space programming. Guided by the statistic, an LLM agent that may edit any component of either tower improves COCO test retrieval on all 13 CLIP checkpoints, and 12 of its 13 final patches edit an output projection.

Method

TowerBridge distills and refines the agent's recurring correction into a closed-form recipe. Computed in seconds from 25,000 labelled pairs, it improves all 21 evaluated CLIP checkpoints and 4 of 5 dual encoders from other families, with no agent and no model-specific tuning.

The problem

A wrong image outranks the correct one

Image–text retrieval ranks candidates by the cosine similarity of two embeddings, one from each tower. A caption can retrieve the wrong image even when its correct match is available.

These high-scoring incorrect matches are hard negatives. Many of them are hubs: images that are retrieved for many captions they do not match. For OpenAI's ViT-B/16, 41% of text-to-image misses on COCO land on hub images.

The two modalities also sit apart in the shared space, separated by a modality gap. Closing it is the natural remedy, yet in our experiments removing the gap degrades retrieval on every checkpoint. Fine-tuning needs gradient-based optimization and hyperparameter search, and test-time re-ranking needs a bank of queries or reference samples and leaves the model unchanged.

The examples on the right are real COCO validation queries. Switch between the zero-shot model and the edited model to see the correct photo replace a hub.

The paired statistic

Where do the two towers agree on the wrong match?

To guide an edit, we need to know which parts of the shared representation favor wrong matches. We compare matched pairs with their hard negatives, the incorrect candidates with the highest similarity scores.

For each caption we collect its five highest-scoring wrong images, and for each image its 25 highest-scoring wrong captions. With unit-normalized image embeddings êv and caption embeddings êt, the paired statistic is the d × d matrix

D=𝔼matched pairs[êvê⊤t]−12𝔼hard-negative images[ê′vê⊤t]−12𝔼hard-negative captions[êvê′⊤t]

The first term measures agreement on matched pairs. The other two measure agreement on hard negatives, weighted equally across the two retrieval directions.

  • The trace is the margin. The trace of D is the mean similarity margin between matched pairs and hard negatives.
  • The eigenvalues split it by direction. The eigenvalues of sym(D) = (D + D⊤) / 2 decompose this margin into contributions along orthogonal directions.
  • Negative eigenvalues are targets. A negative eigenvalue marks a hard-negative direction: one where hard negatives agree more than matched pairs.

The insight

A mild shrink helps. Removal hurts.

The two leading hard-negative directions align with the common mean of the embeddings and with the modality gap. They carry useful content, which a mild shrink retains. Drag the strength from a mild shrink to full removal and watch retrieval change on 24 CLIP checkpoints.

Which directions matter

Removing the two leading directions costs 48.6 COCO validation RSUM on average, and centering each modality costs 20.2. Shrinking the same two directions by 10% gains 7.8, about 60% of the gain of one full round.

The choice of directions matters as much as the strength. A 10% shrink of the hard-negative directions gains +12.5 on average, against at most +0.3 for as many top principal directions chosen without pair labels.

Bridging

The correction widens the margin to the hardest negative and reduces hubness, while the modality gap remains largely unchanged: its median size changes by +0.1% over 18 checkpoints. We call this effect bridging.

The discovery

Weight-space programming with an LLM agent

We arrive at this correction through weight-space programming: an LLM agent, Astra, inspects both towers and writes short programs that directly edit their weights without gradients. We design and supply the paired statistic, and the agent chooses where and how to intervene.

What the agent can do

In each session the agent works on one checkpoint. It has read-only access to both towers' weights and to 35,000 COCO training images with five captions each. It can compute over the weights without gradients, inspect activations and per-head contributions, and examine the paired statistic for pairs of tower components.

Every edit is a closed-form program that can be inspected and re-run without the agent. Programs may scale attention heads, remove directions from heads or MLPs, add low-rank updates, apply a shared matrix to both projections, or rewrite up to four whole tensors. Backpropagation, iterative fitting and regression to data targets are disallowed.

A candidate replaces the current patch only if its COCO validation score is higher by more than two paired standard errors and ImageNet accuracy drops by at most 0.5 points. The test sets are sealed: no tool can read them, and each final patch is evaluated once after its session ends.

Paired evidence guides the search. In controls, removing the statistic reduces the agent's gains more than removing access to the raw weights.

The method

TowerBridge: a closed-form recipe in three steps

Nine of the 13 sessions produce projection edits that attenuate directions with negative eigenvalues. We distill the ViT-B/32 session's uniform shrink and refine it with a second round. The recipe takes seconds for each model and requires no agent.

The whole method in a few lines

# ev, et: unit embeddings of 5,000 images, 25,000 captions
def towerbridge(Wv, Wt, ev, et, pairs):
    S = eye(d)
    for s in (0.10, 0.05):          # two rounds
        iv = normalize(ev @ S.T)
        it = normalize(et @ S.T)
        # hard negatives are re-mined each round
        negs = mine_hard_negatives(iv, it, pairs)
        D = paired_statistic(iv, it, pairs, negs)
        lam, V = eigh((D + D.T) / 2)
        U = V[:, : d // 4]           # d/4 smallest
        S = (eye(d) - s * U @ U.T) @ S
    return S @ Wv, S @ Wt            # same architecture

The subspace size, the two strengths and the numbers of hard negatives are fixed across checkpoints. The matrix S is recomputed from each checkpoint's own embeddings.

Why a uniform shrink

With fixed embeddings and mined negatives, the bilinear hard-negative margin equals ⟨M, sym(D)⟩, where M = S⊤S is the metric induced by the edit. We bound the contraction in each direction and in total:

(1 − c) I ⪯ M ⪯ I,tr(I − M) ≤ ck

Under these bounds a maximizer is M = I − c UkU⊤k: a uniform shrink of the k directions with the smallest eigenvalues. Setting k = d/4 and c = 2s − s2 = 0.19 recovers the first round.

This analysis concerns the bilinear surrogate and motivates the form of the step. Under the cosine score used for retrieval, the uniform shrink also outperforms eigenvalue-weighted shrinkage at matched first-order gain, by 8.0 COCO validation RSUM on average across 13 checkpoints.

Overview figure from the paper
Method overview figure from the paper: the agent session at the top and the TowerBridge recipe at the bottom.

Results

One fixed recipe improves every CLIP checkpoint we evaluate

TowerBridge applies the same rule to each checkpoint and computes its correction from that model's own embeddings. All values are changes in RSUM, the sum of recall at 1, 5 and 10 in both retrieval directions, relative to the zero-shot checkpoint.

Beyond CLIP

The same correction significantly improves Flickr30k retrieval, within the ImageNet tolerance, on 4 of 5 evaluated dual encoders from other families. For SigLIP, which lacks a vision projection, the map is applied as a linear layer after the vision encoder.

Fewer hub errors

On held-out COCO pairs, the margin to the hardest negative widens in both directions on all 21 checkpoints, and hubness decreases on all 21.

Ablations and comparisons

What makes the correction work

Each design choice of the recipe is tested on its own: which directions are shrunk, how they are weighted, whether hard negatives are re-mined, which towers are edited, and how many images are needed.

How it compares, and how it combines

TowerBridge is written into the checkpoint. It involves no gradient-based optimization or hyperparameter search, needs no test-time reference bank, and adds no inference cost, so it can be combined with test-time re-ranking.

Scope

The recipe requires labelled image–text pairs and currently targets dual encoders. Trained maps and re-rankers with an in-domain bank can achieve larger retrieval gains. Overall SugarCrepe accuracy is nearly unchanged (+0.06 points across 21 checkpoints): replace and swap improve by 1.1 and 1.6 points, while add decreases by 1.9.

Release

The paper is under submission. We will release code, prompts and logs. The interactive figures on this page are drawn from the same result files as the paper's figures and tables.

Citation

BibTeX

@misc{alomari2026towerbridge,
  title  = {TowerBridge: Bridging the Dual-Encoder Modality Gap
            via Weight-Space Programming},
  author = {Alomari, Hani and Atabuzzaman, Md and Thomas, Chris},
  year   = {2026},
  note   = {Under submission}
}