Note: On August 3, 2019, Spotify Insights will be no more. But all the data stories you’ve come to enjoy will be available in Spotify’s newsroom, For The Record. Head over to the site not just for data insights, but also cultural trends, how-tos, artist interviews, and more. Want to stay on top of all our latest news and stories? Follow us on For The Record’s Twitter feed, @spotifynews.
What makes a hit song? People have been chasing that formula since the earliest days of the recorded music industry, and nobody has found it. One company that tries, Next Big Sound, estimates its success rate at picking songs that will soon make the Billboard 200 (based on data from Spotify, Instagram, and other sources) at only 20 percent.
Here’s another prediction: Nobody will ever predict, with total accuracy, which songs will reach the pinnacle of the charts. That is not to say it’s impossible to make a song with a good chance of doing well, or to figure out what kinds of songs are more likely to become hits given listening data, the cultural preferences of the time, and/or the instincts of pro hitmakers.
It’s a tricky thing, as demonstrated by new research into the audio attributes of over 25,000 songs on the Billboard 100 from 1958 to 2013. The trick: To be a hit, a song should sound different from anything on the charts, but not so different that it falls off of the cultural radar of the time.
To decide what makes a song conventional or an outlier, Noah Askin (Assistant Professor of Organizational Behavior at INSEAD, in Paris) and Michael Mauskapf (PhD student in Management & Organizations at Northwestern University’s Kellogg School of Management, in Chicago) used audio analysis from The Echo Nest at Spotify to create a new metric called Song Conventionality (methodology below).
It’s ‘Only’ At The Top
Their graph shows that songs in the top 20 show the least amount of conventionality out of any section of the Billboard Hot 100 over time. The farthest outliers, from a musical perspective (based on audio attributes and genre as described below), are the winners:
If a song is too weird, it’s unlikely to make the charts at all, of course; songs at the top of the charts are more similar to each other than stuff from obscure genres of limited (if passionate) appeal.
But within the charts, songs at the top are more likely to sound unconventional than songs in the middle. At the bottom of the Hot 100, we see a bit more deviation from the popular musical conventions of the time, but still nowhere near as much as within the top 20.
Are these findings statistically significant? Yes.
“These graphs are just a descriptive representation of the data; when we run our explanatory models, and control for a host of other effects,” responded Mauskapf. “We find that the relationship between conventionality and chart position is statistically significant (e.g., for songs that appear on the charts, higher levels of conventionality tend to hurt their chart position, except for those songs that are exceptionally novel).“
So ironically, in order for large swaths of the population to connect with a song, it has to sound different from the other stuff that’s popular at the same time. We appear to crave convention, but crave something different most of all.
Unconventionality Reigns Among the Hits
Let’s take a closer look at the very top of the chart, where the same effect can be seen, with a larger effect the closer you get to the coveted Number 1 spot:
The top song is the least conventional of the top 10. The top 10 are less conventional than the top 20.
If these results are any indication, if an artist and their people wants to put something out that has a good chance of making it to the very top of the charts, they should make something that stands out from the pack by moving in a different musical direction than everyone else’s releases.
So, the moral of the story: Do something different. What, exactly? That’s the hard part.
(As if on cue, as we prepared to post the article you’re reading now, we spotted an article from Slate about how varied the hits were this year, jibing with this research.)
From Askin and Mauskapf:
- “When evaluating cultural products, attributes matter, above and beyond social influence dynamics and symbolic classifications like genre.
- “Attributes shape performance outcomes directly and indirectly, through a relational ecosystem of cultural products we call ‘cultural networks.’
- “Songs that are slightly less conventional than average tend to outperform their peers on the charts.
- “Nevertheless, predicting hit songs is nearly impossible to do, because performance is largely contingent on a song’s relationship to other songs that are produced and released contemporaneously.”
Behind The Scenes
“We used The Echo Nest’s attributes to build a ‘song conventionality’ measure and construct networks of songs for each week of the Billboard Hot 100,” explained Askin and Mauskapf in a summary shared with Spotify Insights. “[The below figure] shows one such network, in which the ‘nodes’ are songs and the ‘ties’ between them represent shared genre affiliations and greater-than-average attribute overlap.”
“Our findings suggest that the crowding of attributes within a cultural network can hinder songs’ movement up the charts.”
Here’s a depiction of one song network they made showing their audio and genre similarities (explanation below):
“The spatial relationship [in the chart above] is a function of both a commonly-used network layout algorithm (Fruchterman-Reingold) and of attribute similarity, such that the greater the distance between two songs–>the more dissimilar those songs are across the Echo Nest attribute space (measured using cosine similarities). Colors represent genres; not surprisingly, songs of the same genre tend to cluster together, and certain clusters(e.g., rock and pop) tend to be more sonically similar than others (e.g., rock and funk.soul). Notice however that some songs do not fit the genre clustering pattern, and act instead as brokers between two or more genres (e.g., Little Latin Lupe Lu).”
For any other music scientists who happen to be reading this, here’s some further background on how this research was done.
“1) First, we used a cosine similarity measure to assess the overall degree of Echo Nest audio attribute overlap for each song pair on a particular chart. Put another way, for each song on every chart, we calculated 99 cosine similarity measures to represent the degree of attribute overlap with every other song on that chart. Cosine similarities vary from 0 to 1, and are a common way to measure “distance” across a multi-dimensional attribute space.
“2) The above measure represents songs’ raw attribute similarity, but two songs that have similar sonic attributes may be perceived differently if they are embedded in different genres. Because listeners’ perceptions of a song’s attributes are likely to be influenced by genre affiliation(s), we wanted to weight each song pair’s cosine similarity by the average attribute overlap of those songs’ “home” genres. To do this, we calculated yearly attribute averages for each genre, and then used the same cosine similarity equation to measure the average attribute overlap of each genre pair. The resulting weights were then applied to the raw similarity measures for each song pair. For example: if one rock song and one folk song had a raw cosine similarity of 0.75, and the average cosine similarity between rock and folk is 0.8, then that genre-weighted cosine similarity for those two songs would be 0.75 * 0.8 = 0.6.
“3) After we had calculated genre-weighted cosine similarity measures for each song pair on each chart, we calculated the mean. The resulting value represents each song’s “conventionality” score for a given week. The higher a song’s conventionality score, the more alike that song is to other songs on the chart.
“The average genre-weighted song conventionality score across Hot 100 songs was a little under 0.8, which suggests that, for the most part, songs that achieve some level of popular success are very much alike. In our analysis, we try to tease apart small variations in this measure to explain why, controlling for the effects of genre, artist popularity, and a host of other factors, some songs tend to do better than others.”
(Top photo courtesy of Flickr/Eva Rinaldi)