Get in touch,

Maarten Poirot17 min read

Comparing playlist recommenders

I like building playlists, and the best part is turning up something in my own library I had forgotten I owned. That got me curious about how music recommenders pick the next song, and whether the elaborate ones earn their complexity. So before building anything clever, I tested which signals actually work: with the method names hidden, I scored 1,100 proposed tracks on whether they belonged. One of the most obvious strategies did worse than random. I then scored another 1,100 on the same lists while the recommender learned from each score. That did not clearly beat a ranking that ignored those scores.

Key takeaways

  • Global popularity is a bad playlist expander on this library. Conditioning it on tags helps; content and graph similarity help more.
  • The useful complexity jump is onto an item similarity graph, not past it. SVD on that same graph failed; wrapping the good channels in a hybrid barely moved the mean.
  • A global “add if score is at least 6” rule is the wrong product metric. Thresholds have to be playlist-specific.
  • Letting the hybrid learn from my scores during a session added +0.20 points on average, and the interval includes zero. The gain that does show up is right after a track I rejected.

I will go over the motivation, the hypotheses, the methods, how the 30-second samples were chosen, the 25k similarity matrix, the results, the live session, the limits, and what I would actually conclude.

I like building playlists

Putting a playlist together is a hobby of its own, and the best part is turning up a track I bought years ago and then never played. That gets harder as a library grows. With tens of thousands of files, every new playlist ends up built from the few hundred songs I happen to remember.

What is missing is playlist expansion: given a seed list, propose other songs from the library that fit it. Streaming services do this constantly, on their own catalogue. My own music library, which I listen to through Plex, mostly plays back the files and playlists that are already in it.

Before building that, I wanted to check my assumptions. The first half is one listener, one 25k pilot subset, eleven curated seed playlists, and frozen ranked lists. The second half uses the same lists and the same listener, but the ranking updates while I score.

What I thought going in

I expected three kinds of signal to matter, in different amounts depending on the list. These were prior beliefs, not findings:

  • Sonic characteristics. House, hard rock, or a ballad playlist might be mostly about production, tempo, and timbre.
  • Metadata. Genre, artist neighbourhood, decade, language. Dutch-language party music seemed like a place where embeddings of the sound would miss the thing that actually defines the list.
  • Popularity. A party or “current hits” list might want tracks that lots of people play, even if they are not the closest sonic neighbours.

The ensemble hypothesis was that a hybrid which reweights those channels per playlist would beat any single family. The experiment can support or embarrass that story. It cannot turn a hypothesis into a result by restating it at the end.

How the comparison was run

The design is eleven seed playlists, ten frozen recommenders, and one blinded 100-track trial per playlist. Each method ranked tracks that were not already on the seed playlist, from a 25,000-track catalogue. I built each trial by round-robin assignment, so each method owned about 9 to 11 exclusive slots. A track could still appear on several methods’ lists; overlapping credit inflates graph, hybrid, and tag, so the ranking below uses exclusive ownership only.

I scored every proposal from 0 to 9 on a numeric keypad: 0 means it does not belong, 9 means it belongs easily. The graded scale keeps more information than a yes/no click, and it is fast to operate one-handed while a preview plays. Method identity was hidden. All eleven trials were submitted (1,100 / 1,100).

The screen below is everything I had to go on: which playlist I was rating against, the track, its cover, and the keypad. Nothing on it says which recommender proposed the song.

the blind rating screen on a phone, with the playlist name at the top, album art, the track title, a 30-second preview, and a 0 to 9 keypad under the question how similar this track is to the playlist style
the blind rating screen on a phone, with the playlist name at the top, album art, the track title, a 30-second preview, and a 0 to 9 keypad under the question how similar this track is to the playlist style

The seed lists covered eleven different kinds of listening. During scoring I used only the intended fit as my criterion:

Playlist type What “fit” meant
Current hits Mainstream songs that feel current
Crowd-pleasers Familiar songs that work for a broad audience
Dutch party classics Dutch-language party music
80s-90s nostalgia Synth-led nostalgia
Dance floor High-energy dance tracks
Bossa nova / jazz Relaxed bossa nova and jazz
Soul / easy listening Slower soul and easy-listening tracks
Power ballads Large, dramatic ballads
Indie discovery Less obvious indie tracks worth discovering
Hard rock Hard-rock tracks that fit together
Classic soft rock Road-trip-friendly classic soft rock

Ten frozen recommenders, simple to complex. Several names in the original plan were substituted, not faked: no Spotify popularity API, no LightGCN, no SASRec, no user-track ALS.

Method What it actually used
random Uniform sample of eligible catalogue tracks
popularity Last.fm playcount plus Plex views
tag_popularity Popularity times overlap with seed tags, genre, or decade
tag Cosine to a seed tag-vector centroid (tag SVD, no artist-name leakage)
sonic k-nearest neighbours on acoustic descriptors plus Discogs-EffNet embeddings (a music-specific neural embedding)
item_item Last.fm similar-track scores mapped onto the local catalogue
implicit_mf SVD item factors of that similarity graph, not a user-track model
graph Personalized PageRank on similar-track plus shared-tag edges
hybrid Learned non-negative fusion of the channels above
multistage Union of strong-channel candidates, then a diversity rerank (maximal marginal relevance) for artist spread and flow
Seed playlist
25k catalogue
Ten frozen rankers
100-item blinded mix
Numpad score 0 to 9

Controls (typically two held-out seed members and two hard negatives) sat in the trial so I would see some known goods and known bads. They are in the pooled histograms. They are not in the method ladder.

Thirty seconds from an informative section

Every track was judged from one deterministic 30-second window, chosen to be the most representative part of the song. That rules out the two easy options: the opening 30 seconds of the file, and picking the window by compressed file size (that trick prefers silence or noise). I rated from that clip, the track name, and the album art.

Candidate 30-second windows were scored every 2 seconds from conventional acoustic features:

score=0.40 representativeness+0.35 recurrence+0.15 activity+0.10 stability−edge penalties\mathrm{score} = 0.40\,\mathrm{representativeness} + 0.35\,\mathrm{recurrence} + 0.15\,\mathrm{activity} + 0.10\,\mathrm{stability} - \mathrm{edge\ penalties}

Representativeness is how close the window summary is to the track-wide feature centroid. Recurrence is similarity to non-adjacent sections elsewhere in the track (choruses and riffs beat one-off bridges). Activity asks for enough loudness and rhythmic or spectral content. Stability penalizes long silence and abrupt internal jumps. Intros, outros, and fades get a mild extra penalty. The design also prefers starts near beats or low-onset boundaries, and after the coarse pick it can nudge the start by about 3 seconds for a cleaner cut. I then rendered AAC at 96 kbit/s with very short fades so the clip does not click.

That one window is what I heard. The sonic recommender is a different use of 30-second audio: Discogs-EffNet sees three excerpts (the thumbnail, a non-overlapping middle, and another distinct region), pooled into an embedding. Rating from a chorus-like slice is not the same as embedding three slices.

Why I added 16 GB of RAM in the process

The hard part of training on 25,000 tracks was memory, not modelling. Several methods share a dense item-item graph, and a 25,000 by 25,000 float32 adjacency matrix is about 2.5 GB before anyone copies it. PageRank and SVD make extra working copies, so that stage peaked around 8-9 GB, which the eight gigabytes in the machine could not hold. I added 16 GB to bring it to 24 GB, sized for train-recommenders. The extra memory was useful for the rest of the server anyway.

The same matrix is not one named recommender. It feeds:

  • item_item (direct neighbour scores)
  • implicit_mf (SVD on that graph)
  • graph (Personalized PageRank, with shared-tag edges as well)
  • then hybrid and multistage, which consume those channels

The SVD rung used that graph as if it were a collaborative matrix. It is not a user-track model. That distinction matters when the rung collapses.

What the ratings show

Some playlists are far more selective than others, and that difference matters more than any pooled average. Across all 1,100 scores the mean is 3.46 and 31% are zeros, but the current-hits playlist averages 4.14 with 45% of its scores at 6 or higher, while the 80s-90s nostalgia playlist averages 2.50 with 18% at 6 or higher. A product rule that adds a track only at 6 would accept almost half the current-hits trial and almost none of the bossa or nostalgia trials.

The pooled histogram below is the 0-9 scale as I actually used it: a large spike at zero, then a broad middle, not a clean accept/reject split.

All 1,100 ratings on one axis. Zeros dominate and the rest spread out without a gap, so no score stands out as the obvious place to start accepting tracks.

Playlist means make the same point without pooling first. Current hits and soul / easy listening sit at the high end; classic soft rock and 80s-90s nostalgia sit at the low end.

Mean 0-9 fit by seed playlist. Hover a bar for the share of scores at 6 or higher.

Popularity lost; the graph did the work

On exclusive credit, popularity is worse than random (0.20 vs 0.55). The fitted hybrid put zero weight on raw popularity. Tag-conditioned popularity recovers to 2.88. Tag cosine, sonic kNN, and item-item then sit in the 4.5-4.9 range.

The largest drop on the ladder is item-item to implicit_mf (-3.34 mean points). The largest gain is implicit_mf to graph (+3.55). Hybrid edges the graph by 0.03 points; multistage sits with them. Playlist-clustered bootstrap intervals (resample the eleven playlist means) overlap for hybrid, graph, and multistage. Those intervals ask how the average moves if we reweight these playlist types. They are not a p-value for another listener, and they do not pick a winner.

Mean fit by method, simple to complex. Hover for the 95% interval over the 11 playlist means.

Held to a threshold that lets through at most 1 in 5 random tracks, the methods surface 656 tracks at about 2% random contamination. That number needs its bar explained, because a classical ROC curve is not available here: it would need a binary label for every candidate and a true-positive rate against the whole catalogue, and I have neither. Scores are “does this belong on this playlist?”, and the negative class I can actually calibrate is the ~10 random items in each trial. So for each playlist I took the lowest integer threshold at which at most 1 in 5 of those randoms pass, then counted how many non-random, non-control tracks also pass. Because random is almost all zeros, that threshold is 1 on most lists (not a zero), with current hits the strict exception at 7. It is a weak bar, and it is the honest operating-point analogue of an ROC here.

Tracks found at a per-playlist false-positive rate of 1 in 5, calibrated on random negatives. Popularity barely appears; tag, graph, and fusion dominate.
The same count at a 1 in 10 false-positive rate. The ranking does not flip; the bar is just stricter on a few playlists.

Learned hybrid weights, for the record: sonic 0.27, tag 0.25, item-item 0.20, graph 0.15, implicit_mf 0.07, tag_popularity 0.06, popularity 0.

The right channel still depends on the playlist

Which family wins changes from one playlist to the next, and raw popularity never wins on any of them. Grouping the methods into families (baseline, popularity, metadata, sonic, collaborative, fusion) and taking mean exclusive scores per playlist, sonic leads on hard rock, power ballads, dance-floor, 80s nostalgia, and classic soft rock. Fusion or collaborative lead on Dutch-language party, current hits, soul/easy listening, bossa/jazz, crowd-pleasers, and indie.

That is the part of the original hypothesis that survived: language-and-culture lists are not the same problem as a production niche. Dutch party classics are the clean example. Sonic mean 2.30; collaborative 5.96; fusion 6.17. Hard rock flips it: sonic 6.10, fusion 5.11. Power ballads are even more sonic-led (6.80 vs fusion 4.28).

Cells in the grid below are noisy (about 10 exclusive items per method per playlist), so family averages are the stable read, not any single square.

Playlist type by signal family. Each tile shows its mean fit, and the more blue a tile is, the better that family did. Popularity stays washed out almost everywhere, while sonic and fusion trade the lead row by row. Hover a cell for the number of tracks behind it.

Treated as qualities rather than named products, tags and sound are the channels that help, and global popularity actively hurts. A regression of the human score on z-scored channel features plus playlist intercepts (n = 1,058, R squared = 0.50) puts tag-conditioned popularity, tag cosine, and sonic similarity on the positive side, and global popularity at -1.26 points per standard deviation once playlist identity is in the model. There are no coefficient standard errors in the export, so I treat this as a ranking of channels, not a significance table.

Channel coefficients after playlist intercepts. Popularity points the wrong way; tag and sonic point the right way.

Letting it learn did not clearly help

The frozen lists do not say whether the recommender should keep updating while I listen. I scored the same eleven playlists again, another 1,100 tracks, method still hidden. Averaged over the three copies that learned, updating did not clearly beat ignoring the scores. The gap is +0.20 points (95% interval -0.07 to +0.48). A test that flips the sign of each playlist’s gap gives p = 0.14, and the learning copies were ahead on 6 of the 11 lists. With eleven playlists, a gap under about 0.3 points will not show up, and this one does not.

Four copies, one session

Each session played 100 tracks and rotated through four copies of a hybrid, 25 tracks each. A track already on the seed list, or already played in the session, was not proposed again. Only the copy that proposed a track learned from my score of it.

One copy is frozen. It is trained once and then ignores every score. It is not the hybrid rung in the ladder above: that rung used k-nearest-neighbour sonic scoring, decade tags, and a training split.

The other three start from the same profile and then update, each from its own tracks only. Positive adds a track to its profile when I score it 6 or higher. Signed does that too, and also moves the profile away from tracks I score below 4.5, the midpoint of the scale. Bandit shifts weight toward the channels whose scores agreed with mine.

Seed playlist
Four hybrid copies
Next track, method hidden
Score 0 to 9
Owning copy updates

The four copies draw on the same top of the ranking, so the frozen copy has to reach deeper as the session goes on. Its median pick was rank 35, and its ratings fall from 6.55 inside its top 25 to 5.49 at ranks 51-100. A gap against the learning copies is partly the frozen copy having to dig that far.

A small average, and a few larger slices

The means were 6.05 for the frozen copy, 6.19 for positive, 6.19 for signed, and 6.37 for the bandit. Positive and signed landing on the same mean is the totals matching, not the same tracks. The bandit is the largest single gap, at +0.32 (p = 0.19). The intervals over the eleven playlist means overlap.

Mean 0-9 fit for the four copies. Hover a bar for the interval over the 11 playlist means, and for the share of scores at 6 or higher.

These means sit near 6, while the hybrid rung above sits near 5, because these sessions only played tracks a hybrid had already ranked near the top. The same tracks also scored 1.43 points higher in the live sessions than in the August and September trials (487 pairs, interval 0.77 to 2.21). After subtracting that shift, the live copies sit about level with the best frozen methods, and the shift is larger than the gaps between those methods. On that shifted comparison the bandit lands 0.17 below the old hybrid rung and 0.09 below multistage. I treat that as a coarse ranking across two designs, not a winner.

Which copy leads still depends on the playlist. The bandit is ahead on Dutch party classics and on power ballads. Positive is ahead on indie discovery. The frozen copy is narrowly ahead on hard rock. Soul / easy listening already has a high frozen score, and the learning copies do not improve it on average. Rows in the grid below run from the largest average gain over the frozen copy down to the largest loss.

Mean fit by playlist type and copy. More blue is a higher mean. Hover a cell for that mean and for how far the three learning copies sit from the frozen one.

After I scored a copy’s own previous pick at 3 or lower, its next pick beat the frozen copy’s pick at the same point in the session by about 0.6 to 1.2 points, with the bandit at the high end. After a pick I had scored 6 or higher, the gap was about 0 to 0.3. The frozen copy keeps serving the same region of its list. The learning copies can leave that region.

Seed lists of 30 tracks or fewer gained +0.46, across 6 lists. The 5 longer lists lost 0.11. That split is descriptive. Indie discovery has 6 seed tracks and power ballads have 9, and both gained. Soul / easy listening has 67 and lost the most. Splitting the same gaps on the August and September static-trial means, which do not share this session’s noise, gives +0.15 below the median and +0.26 above it. I would not read this as learning helping on hard playlists.

Leaving the frozen list pays for the bandit

Early in a session the learning copies mostly agree with the frozen ranking: about 8% of their picks in the first quarter sit outside its current top 25. By the last quarter that share is 42% for positive, 61% for signed, and 73% for the bandit. The grid below is the whole session, not the quarters: where each copy’s picks sat in the frozen ranking, among tracks not yet played.

Share of each copy's picks by their place in the frozen ranking. The frozen row is entirely in ranks 1-4 because that copy always takes the current top unshown track. Hover a cell for the mean rating of those picks.

Compared at the same point in the session, the bandit’s picks from outside that top 25 still beat the frozen copy by about half a point. Signed’s departures are close to a tie. Positive’s are uneven, including a loss of about 0.6 points on ranks 5-25.

Rebuilding the frozen ranking offline reproduced the tracks that copy actually played: 96% of its picks before a change on 2 November that made the next track start faster, and every pick after that change. Each copy’s first pick, before any learning, was within half a point of the frozen copy. That is one track per playlist, about what noise looks like. I later edited 61 of the 1,100 scores. Leaving those edited scores out does not move the result.

What this does not show

One rater: me. Twelve opportunistic same-playlist retests moved by about 4 points on average. That is too few pairs, and too messy a design, to quote a reliability coefficient. Seventy-six tracks rated on different playlists are a different question: the same recording can fit one list and not another, which is the point of playlist expansion, not rater noise.

Cells are small. Random-calibrated false-positive rates use about ten negatives per playlist, so thresholds jump in whole score points. The catalogue is 25k, not 50k. implicit_mf is graph SVD, so its failure does not indict a proper user-track model I did not run.

The twelve retests above are a within-trial question. The 487 pairs in the live section are a different comparison, across months, and that 1.43-point shift is larger than the method gaps it is meant to correct.

What I would ship, and what I would not claim

Supported. On this library, suggesting tracks that sound like the seed, share its tags, or sit nearby on a Last.fm similarity graph beats chance and beats global popularity. The jump that matters is onto that graph. A late-fusion hybrid and an MMR rerank sit in the same cluster as PageRank; I cannot name a winner. Playlist type still shifts which family leads. Calibrate “suggest / skip” per list. From the live sessions, the supported slice is narrower: after I rejected a track, the next suggestion from a learning copy beat the frozen one; the shorter seed lists gained while the longer ones did not; and the bandit’s picks from outside the frozen top 25 still held up.

Not supported, including some of the original hypotheses. I did not show that raw popularity is the right lever for party playlists. Tag-conditioned popularity helps; global playcount does not. I did not show that an ensemble which reweights channels per playlist beats the graph: the hybrid was trained across these eleven lists and landed 0.03 points above PageRank with overlapping intervals. I also did not show that learning during a session beats a frozen copy of the hybrid. The three learning copies average +0.20, and the interval includes zero. The bandit, which reweights channels from my scores on that playlist, was the largest of them at +0.32 and still not distinguishable from zero. I did not show that the result holds for other listeners or the full 50k library.

Default ranking, if I productize this, looks like graph or hybrid, with sonic kept for genre-homogeneous lists and tags kept instead of Last.fm playcount. I would not ship online learning as an upgrade on this evidence. I would not ship the implicit-MF rung in the form I evaluated. I would not use a universal cutoff of 6.

If you want the same experiment on your own library, the scoring interface is the easy part: 0-9 on the numpad, 30 seconds of an informative section, methods hidden. The expensive part, on 25k tracks, is admitting that a dense item-item matrix is a memory problem long before it is a modelling problem.