---
title: "Week 8 · Chroma Features"
author: "John Ashley Burgoyne"
date: "20 February 2019"
output: 
    html_notebook:
        theme: flatly
---

Welcome to your first R Markdown file of Computational Musicology! RStudio has made a helpful [cheat sheet][1] you can use to learn more about all of the features of R Markdown.

## Set-up

We also begin by loading the libraries we need for analysis. Notice that I've added a secret `spotify.R` file to keep my login credentials secret.

```{r}
library(tidyverse)
library(spotifyr)
source('spotify.R')
```

## Functions

We need some helper functions this week. You may copy these chunks directly into your own files or add them -- without the surrounding chunk markers! -- to your `spotify.R` file.

### Getting the audio analysis

```{r}
#' Get a tidy audio analysis from Spotify.
#'
#' spotifyr returns Spotify's audio analysis as a large list. This function
#' uses list columns to create a structure that works more richly within the
#' tidyverse.
get_tidy_audio_analysis <- function(track_uri, ...) 
{
    get_track_audio_analysis(track_uri, ...) %>% 
        list %>% transpose %>% as_tibble %>% 
        mutate_at(vars(meta, track), . %>% map(as_tibble)) %>% 
        unnest(meta, track) %>% 
        select(
            analyzer_version,
            duration,
            contains('fade'),
            ends_with('confidence'),
            bars:segments) %>% 
        mutate_at(
            vars(bars, beats, tatums, sections), 
            . %>% map(bind_rows)) %>% 
        mutate(
            segments =
                map(
                    segments,
                    . %>% 
                        transpose %>% as_tibble %>% 
                        unnest(.preserve = c(pitches, timbre)) %>% 
                        mutate(
                            pitches = 
                                map(
                                    pitches, 
                                    . %>% 
                                        flatten_dbl %>% 
                                        set_names(
                                            c( 
                                                'C', 'C#|Db', 'D', 'D#|Eb', 
                                                'E', 'F', 'F#|Gb', 'G',
                                                'G#|Ab', 'A', 'A#|Bb', 'B'))),
                            timbre = 
                                map(
                                    timbre,
                                    . %>% 
                                        flatten_dbl %>% 
                                        set_names(
                                            c(
                                                'c1', 'c2', 'c3', 'c4', 
                                                'c5', 'c6', 'c7', 'c8',
                                                'c9', 'c10', 'c11', 'c12'))))))
}
```

### Norms and distance functions

```{r}
#' Normalise vectors for Computational Musicology.
#'
#' We use a number of normalisation strategies in Computational Musicology.
#' This function brings them together into one place, along with common
#' alternative names.
compmus_normalise <- compmus_normalize <- function(v, method = "euclidean")
{
    ## Supported functions
    
    harmonic  <- function(v) v * sum(1 / abs(v))
    manhattan <- function(v) v / sum(abs(v))
    euclidean <- function(v) v / sqrt(sum(v^2))
    chebyshev <- function(v) v / max(abs(v))
    clr       <- function(v) {lv <- log(v); lv - mean(lv)}
    
    ## Method aliases
    
    METHODS <-
        list(
            harmonic  = harmonic,
            manhattan = manhattan,
            L1        = manhattan,
            euclidean = euclidean,
            L2        = euclidean,
            chebyshev = chebyshev,
            maximum   = chebyshev,
            aitchison = clr,
            clr       = clr)
    
    ## Function selection
    

    if (!is.na(i <- pmatch(method, names(METHODS))))
        METHODS[[i]](v)
    else 
        stop('The method name is ambiguous or the method is unsupported.')
}

#' Compute pairwise distances for Computational Musicology in long format.
#'
#' We use a number of distance measures in Computational Musicology.
#' This function brings them together into one place, along with common
#' alternative names. It is designed for convenience, not speed.
compmus_long_distance <- function(xdat, ydat, feature, method = "euclidean")
{
    
    feature <- enquo(feature)
    
    ## Supported functions
    
    manhattan <- function(x, y) sum(abs(x - y))
    euclidean <- function(x, y) sqrt(sum((x - y) ^ 2))
    chebyshev <- function(x, y) max(abs(x - y))
    pearson   <- function(x, y) 1 - cor(x, y)
    cosine    <- function(x, y)
    {
        1 - sum(compmus_normalise(x, "euc") * compmus_normalise(y, "euc"))
    }
    angular   <- function(x, y) 2 * acos(1 - cosine(x, y)) / pi
    aitchison <- function(x, y)
    {
        euclidean(compmus_normalise(x, "clr"), compmus_normalise(y, "clr"))
    }
    
    ## Method aliases
    
    METHODS <-
        list(
            manhattan   = manhattan,
            cityblock   = manhattan,
            taxicab     = manhattan,
            L1          = manhattan,
            totvar      = manhattan,
            euclidean   = euclidean,
            L2          = euclidean,
            chebyshev   = chebyshev,
            maximum     = chebyshev,
            pearson     = pearson,
            correlation = pearson,
            cosine      = cosine,
            angular     = angular,
            aitchison   = aitchison)
    
    ## Function selection
    
    if (!is.na(i <- pmatch(method, names(METHODS))))
        bind_cols(
            crossing(
                xdat %>% select(xstart = start, xduration = duration),
                ydat %>% select(ystart = start, yduration = duration)),
            xdat %>% select(x = !!feature) %>% 
                crossing(ydat %>% select(y = !!feature)) %>% 
                transmute(d = map2_dbl(x, y, METHODS[[i]])))
    else 
        stop('The method name is ambiguous or the method is unsupported.')
}
```

### Unwrapping chroma vectors

```{r}
#' Gathers chroma vectors into long format.
#'
#' Gathers chroma vectors into long format for Computational Musicology.
compmus_gather_chroma <- function(data)
{
    data %>% 
    mutate(pitches = map(pitches, bind_rows)) %>% unnest(pitches) %>% 
    gather("pitch_class", "value", C:B) %>% 
    mutate(pitch_class = fct_shift(factor(pitch_class), 3))
}
```


## Chromagrams

One of the new helper functions, `get_tidy_audio_analysis`, does just what it says, for one track at a time. Spotify's audio analysis is thorough, and basic [documentation][2] is available for everything in it.

Let's pull down the analysis of a recording of Steve Reich's 'Music for Pieces of Wood' (well worth a listen if you don't know the piece). For reasons that will become clearer next week, one needs to `select` the desired field out of the analysis -- in this case, `segments` -- and then `unnest` it.

Spotify segments have a lot of information inside them, but this week, we'll focus just on `start`, `duration`, and `pitches`: the three tools we need to make a chromagram.

```{r}
wood <- 
    get_tidy_audio_analysis('6IQILcYkN2S2eSu5IHoPEH') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)
```

The key to making a chromogram is `geom_tile`. It is powerful but has it's wrinkles. Much of the code in the next block is code that you will simply need to copy every time you use it; next week we'll develop a better understanding of what is going on.

First, we want to choose a normalisation for the chroma vectors: Manhattan, Euclidean, or Chebyshev, using the new helper function `compmus_normalise` (with a little help from the `map` function, which we'll see more of next week). *The name of the normalisation is the only thing in that line of code that you need to change.* 

Then we need to convert the data to so-called long format: a new row for each pitch class. There is a helper function `compmus_gather_chroma` to do that.

Finally we can plot, but beware that ggplot *centres* each tile on the *x* or *y* coordinate instead of using the left corner. Use the duration to make a correction for it.

```{r}
wood %>% 
    mutate(pitches = map(pitches, compmus_normalise, 'chebyshev')) %>% 
    compmus_gather_chroma %>% 
    ggplot(
        aes(
            x = start + duration / 2, 
            width = duration, 
            y = pitch_class, 
            fill = value)) + 
    geom_tile() +
    labs(x = 'Time (s)', y = NULL, fill = 'Magnitude') +
    theme_minimal()
```

### Your turn

What is going on here? Try a different normalisation -- and then try a different piece!

## Dynamic Time Warping

In order to take the step from chromagrams to [dynamic time warping][3], we need to choose an appropriate distance. Distance metrics usually form conceptual pairs with norms, although there are no standard distance metrics to use after Chebyshev normalisation.

Both Aitchison and angular distances have solid theoretical underpinning for chroma vectors. The Manhattan (a.k.a. *total variation distance* in this case) and the cosine pseudo-distance are faster to compute and are often good enough. The cosine distance, in particular, is extremely popular in practice.

| Domain                      | Normalisation | Distance  |
| ----------------------------|---------------|-----------|
| Non-negative (e.g., chroma) | Manhattan     | Manhattan |
|                             |               | Aitchison |
|                             | Euclidean     | cosine    |
|                             |               | angular   |
|                             | Chebyshev     | [none]    |

Let's look at seven recordings of Josquin des Prez's 'Ave Maria'. The comment lines break them down into three pitch levels.

```{r}
tallis <- 
    get_tidy_audio_analysis('2J3Mmybwue0jyQ0UVMYurH') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)
chapelle <- 
    get_tidy_audio_analysis('4ccw2IcnFt1Jv9LqQCOYDi') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)
cambridge <- 
    get_tidy_audio_analysis('54cAT1TCFaZbLOB2i1y61h') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)

###

oxford <- 
    get_tidy_audio_analysis('5QyUsMY40MQ1VebZXSaonU') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)
chanticleer <- 
    get_tidy_audio_analysis('1bocG1N8LM7MSgj9T1n3XH') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)

###

hilliard <- 
    get_tidy_audio_analysis('2rXEyq50luqaFNC9DkcU6k') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)
gabrieli <- 
    get_tidy_audio_analysis('4NnJ4Jes8a8mQUfXhwuITx') %>% 
    select(segments) %>% unnest(segments) %>% 
    select(start, duration, pitches)
```

The `compmus_long_distance` helper function gets everything ready for `geom_tile`. It takes two data frames (don't forget to normalise the chroma vectors), the feature we want to compute distance over (`pitches` in our case), and any of the distance measures in the table above. It returns a long table ready for plotting, with `xstart`, `xduration`, `ystart`, and `yduration`.

```{r}
compmus_long_distance(
    tallis %>% mutate(pitches = map(pitches, compmus_normalise, 'chebyshev')),
    chapelle %>% mutate(pitches = map(pitches, compmus_normalise, 'chebyshev')),
    feature = pitches,
    method = 'euclidean') %>% 
    ggplot(
        aes(
            x = xstart + xduration / 2, 
            width = xduration,
            y = ystart + yduration / 2,
            height = yduration,
            fill = d)) + 
    geom_tile() +
    scale_fill_continuous(type = 'viridis', guide = "none") +
    labs(x = 'The Tallis Scholars', y = 'La Chapelle Royale') +
    theme_minimal()
```
### Your turn

What's going on here? Try different distance metrics to see what works best.

How do the patterns change if you compare recordings at two different pitch levels?

Are there any recordings in your corpus for which this technique could work? Do you think it would work with cover songs?

### Advanced work

If you want to compute actual alignments, look into the R [dtw package][4].

### WARNING

The output of `geom_tile` can be very large if sent through `ggplotly`. You made need to make these visualisations static.


[1]: https://github.com/rstudio/cheatsheets/blob/master/rmarkdown-2.0.pdf
[2]: https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-analysis/
[3]: https://www.youtube.com/watch?time_continue=62&v=gsYhDp2VXMo
[4]: https://cran.r-project.org/web/packages/dtw/index.html
