Abstract

Training deep neural networks on gigapixel Whole Slide Images (WSIs) poses significant GPU memory challenges. Multiple Instance Learning (MIL) circumvents this by processing a bag of patches per WSI; however, memory limits typically force the feature encoder—whether a Convolutional Neural Network (CNN) or a foundation model—to remain frozen. In this work, we investigate whether end-to-end (E2E) training improves MIL performance and out-of-domain generalisation. To overcome the memory bottleneck, we introduce a memory-efficient strategy that divides computations into manageable chunks. For CNNs, this involves dynamically offloading intermediate activations to CPU RAM. For ViT-based MIL encoders, we process patches in micro-batches and recompute each micro-batch during backpropagation, recovering the full-bag encoder gradient without retaining all encoder activations simultaneously. Across PANDA/TCGA-PRAD and CAMELYON17/16, we find that E2E training is useful but not uniformly beneficial: its external effect depends on the adaptation recipe and evaluation setting. In particular, combining full E2E ResNet-18 (RN18) training with WSI-level augmentation gives the strongest RN18 transfer to TCGA-PRAD, and E2E combined with WSI-level augmentation and Learning without Forgetting allows H0-mini to match or exceed much larger frozen foundation models there. On CAMELYON16, however, frozen foundation models remain stronger. These results suggest that E2E training is a conditional but important tool for WSI model adaptation rather than a universal replacement for frozen encoders.