What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models

Kanatas, Angelos-Nikolaos; Kong, Yuexuan; Alonso-Jiménez, Pablo; Serra, Xavier; Bogdanov, Dmitry

What Makes a Good Layer?

Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models

Angelos-Nikolaos Kanatas¹, Yuexuan Kong^2,3, Pablo Alonso-Jiménez¹, Xavier Serra¹, Dmitry Bogdanov¹

¹ Music Technology Group, Universitat Pompeu Fabra · ² Deezer Research · ³ Nantes Université, École Centrale Nantes, CNRS, LS2N
ISMIR 2026

arXiv · soon Code · soon

Raw per-layer records: data/

Music foundation models are used as frozen feature extractors, but which layer you read from is usually a guess. We map 12 models layer by layer: label-free geometry picks the right layer for most MIR tasks, tonal tasks need a new pitch-transposition equivariance measure, and together they replace an exhaustive layer search with a shortlist of a few proxy-ranked layers that beats trainable multi-layer fusion.

Findings

No universal proxy. Across 12 models, 3 pre-training paradigms and 15 MIR tasks, no single label-free property predicts layer quality everywhere.

Geometry works — off tonal. Intrinsic dimension is the strongest general proxy (mean ρ = 0.76), and straighter temporal trajectories transfer better, most consistently for beat tracking.

Tonal tasks break every standard metric. Key, chord and pitch show weak, sign-inconsistent correlations: static geometry and augmentation-induced invariance are the opposite of what these tasks need.

PTE closes the gap. A self-supervised circle-of-fifths diagnostic — the only metric with a consistent tonal signal (mean ρ = 0.80), and the only one that holds both within and across models.

A few proxy-ranked layers are enough. The top-3 land 0.4 pp from the per-task oracle and match it on 58% of model–task pairs, using 4–16× fewer probe runs; the default final layer is 4.3 pp worse.

Metric-guided fusion beats trainable fusion. Averaging those three layers is the strongest fusion method (p < 0.001): 0.3 pp from the oracle, exceeding it in 41% of model–task pairs, while every trainable baseline trails — and the margin grows when labels are scarce.

But values don't rank models. Effective rank tracks hidden size (ρ = 0.82), not quality (ρ = 0.02) — these metrics choose layers within a model, not winners across architectures.

Extended Results

The paper reports aggregates. The Layer Atlas hosts the breakdowns behind them — best-layer scores for every model × task, sortable, with per-task columns, CSV/LaTeX export, and the raw per-layer records as JSON.

Layer-wise downstream results — every model × task, best layer per cell Live
Layer cheat sheet — the three-layer band to probe first, per model and task family, with depth rules of thumb Live
Layer explorer — interactive per-layer curves for any model and task Live
Layer selection & fusion — proxy-guided vs. trainable fusion vs. the best single layer, per model and per task Live
Metric–task correlations — per task and per model, beyond the paper's paradigm-level averages Live
Intrinsic-metric profiles across depth — overlaid on downstream performance in the Layer Explorer Live

Abstract

Music foundation models are commonly used as frozen audio feature extractors, yet selecting which layer to extract from remains largely heuristic. Current practice defaults to fixed depths or multi-layer fusion, with limited understanding of why certain layers transfer better across downstream tasks or how representation quality varies with depth and pre-training paradigm. We conduct a systematic layer-wise analysis of 12 music foundation models spanning three pre-training paradigms (masked modeling, autoregressive modeling, and contrastive learning), characterizing their hidden representations through intrinsic geometric and transformation-based properties. Correlating label-free representation-quality metrics with layer-wise performance across 15 downstream tasks, we find that several metrics track layer quality for genre classification, emotion recognition, automatic tagging, and beat tracking, albeit with varying strength across tasks and pre-training paradigms. However, all metrics fail on tonal tasks such as key estimation and chord recognition, indicating that no single property serves as a general proxy for representation quality across music information retrieval tasks. To address this gap, we introduce a pitch-transposition equivariance measure that captures properties missed by these standard metrics, providing a consistent indicator of tonal quality across model families. Finally, we show that intrinsic metrics can serve as effective proxies for layer selection, matching or outperforming trainable multi-layer fusion methods, particularly in limited-data settings.

BibTeX

@inproceedings{kanatas2026goodlayer,
  title     = {What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models},
  author    = {Kanatas, Angelos-Nikolaos and Kong, Yuexuan and Alonso-Jim{\'e}nez, Pablo and Serra, Xavier and Bogdanov, Dmitry},
  booktitle = {Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)},
  year      = {2026}
}