In the quest to interpret complex cellular responses, self-supervised learning (SSL) methods have been developed, promising powerful generic representations. However, their performance on biologically critical tasks such as mechanism of action (MoA) classification remains limited. We argue that no single model—no matter how sophisticated or generalisable—can produce representations that are optimal for all downstream tasks, as different objectives impose conflicting requirements. To address this, we propose a novel framework called Task-guided Representation exaptation (TRex). In TRex, a generic (possibly self-supervised) model first extracts broad and rich morphological embeddings, which are then refined by a lightweight adaptation network optimised for biological relevance linked to the specific downstream tasks. This modular design enables rapid and resource-efficient transformation of generic features into biologically meaningful, task-focused representations — without the need to retrain large-scale models. We evaluate TRex on a 20-plate Cell Painting dataset spanning two cell lines and show that MoA-based adaptation not only significantly improves MoA classification performance (doubling the mAP), but also enhances compound recognition. Our results highlight the limitations of static, generic representations and demonstrate the utility of task-aware adaptation for maximising the biological relevance of morphological profiling.