Abstract

In this work, we introduce SOARA (Subspace Orthogonal Adaptation via Rotational Alignment), a novel family of parameter-efficient fine-tuning (PEFT) algorithms that navigate the manifold of pretrained weights through geometric alignment. While existing SVD-based methods prioritize magnitude-driven adaptation, for example by modifying singular values (SALT, SVFT) or reparameterizing basis vectors via additive low-rank updates (PiSSA), they often overlook the intrinsic rotational symmetries of the feature space. In contrast, SOARA treats the pretrained principal singular vectors as a fixed basis and adapts the model by learning lightweight rotational transformations within these subspaces. This parameterization preserves orthogonality of the rotated basis while allowing alignment with downstream task geometries. We provide two distinct paths for enforcing orthogonal parametrization of rotational matrices: (1) parametrization via regularization, and (2) parameterizations via sequential Givens rotations or butterfly decompositions. These rotations offer a constrained alternative to additive or purely scale-based adaptation schemes. Our empirical results on large-scale architectures, including ViT-B16 and DeBERTa-v3-base, demonstrate that SOARA matches or exceeds the performance of state-of-the-art PEFT methods across vision and language benchmarks. SOARA attains these results with a small parameter footprint, indicating that rotational alignment is a useful parameterization for preserving and repurposing pretrained feature manifolds.