Abstract

Interactive prompting is an appealing approach to vision–language fusion using frozen uni-modal transformers, yet recent progress often relies on increasingly complex prompting architectures. A natural question arises: instead of refining prompt designs, can fusion be improved more effectively by directly adapting internal representations within attention layers? Our analysis, from a representation-centric perspective, suggests that within each frozen attention layer, prompt tokens have limited direct control over the value representations of original modality tokens and their query–key interactions, motivating a lightweight alternative that targets these internal attention representations rather than increasing prompting complexity. Specifically, we use this analysis to guide where lightweight adaptation is applied: we investigate the cross-attention mechanism and propose combining value-only low-rank adaptation with a key–query replacement strategy, yielding a simple and parameter-efficient fusion design. Across common multimodal fusion benchmarks, the proposed method consistently outperforms prior prompting-based fusion baselines while requiring fewer trainable parameters. These results, along with further ablations, support representation-centric adaptation as an effective principle for prompting-based vision–language fusion in the frozen-encoder setting.