Layer selection

For each model and task, the best single layer (in bold) is compared with the middle and last layers, with the layers ranked highest by a label-free metric (the top layer, the best of the top three, or the top three or five averaged or concatenated), and with fusion of all layers, without or with training. Small numbers give the difference from the best single layer.

MethodMERT-v1-95MMERT-v1-330MMusicFM (MSD)MuQ (iter)OMAR-RQ (multifeature)MusicGen-SMusicGen-MMusicGen-LYuE-s1-0.5BMT2MERT-v2-30sMERT-v2-FullSong
Single layers
Best single layer63.465.264.664.567.068.665.567.266.670.167.067.0
Middle layer61.164.364.361.961.666.355.254.154.369.353.953.5
Last layer61.058.462.258.064.663.263.464.966.659.665.065.4
Layers ranked highest by a label-free metric
Top 163.4+0.064.9-0.362.0-2.664.5+0.067.0+0.063.2-5.464.8-0.766.4-0.863.8-2.8———
Best of top 363.4+0.065.2+0.063.9-0.764.5+0.067.0+0.068.1-0.565.5+0.066.9-0.363.8-2.8———
Top 3, averaged61.8-1.663.6-1.659.5-5.164.4-0.166.2-0.867.8-0.865.8+0.3—67.0+0.4———
Top 3, concatenated62.8-0.663.7-1.558.5-6.163.3-1.265.3-1.766.9-1.763.5-2.0—65.0-1.6———
Top 5, averaged62.1-1.365.2+0.060.8-3.863.7-0.866.6-0.467.7-0.966.6+1.168.2+1.067.4+0.8———
Top 5, concatenated60.4-3.061.8-3.561.3-3.361.2-3.360.2-6.866.4-2.263.9-1.666.5-0.762.5-4.1———
All layers, no training
Average64.8+1.465.4+0.263.5-1.164.5+0.063.3-3.768.0-0.663.7-1.865.8-1.568.0+1.470.8+0.764.0-3.064.1-2.9
Concatenation59.7-3.726.7-38.561.0-3.663.3-1.261.7-5.366.0-2.662.3-3.262.4-4.861.8-4.868.9-1.264.3-2.764.2-2.8
All layers, trained
Weighted sum62.0-1.462.9-2.362.2-2.363.1-1.464.8-2.266.2-2.465.6+0.165.6-1.664.4-2.270.0-0.166.0-1.065.7-1.3
Hierarchical convolution60.6-2.860.1-5.161.8-2.863.0-1.563.8-3.262.9-5.762.4-3.160.6-6.661.4-5.269.1-1.062.5-4.563.7-3.3
Attention over layers57.2-6.261.5-3.759.8-4.861.7-2.860.6-6.462.0-6.658.1-7.460.2-7.061.7-4.966.0-4.161.5-5.560.9-6.1

We continue to update these results with more models and downstream tasks.