Direct Labelling of Form of Classical-Period Piano Sonata Movements From Audio Recordings

Musical form is defined as the overall structure of a music piece. The labelling of musical form types (for the purpose of, e.g., querying online music databases) by utilizing raw audio alone is a relatively unexplored area in the field of music information retrieval research. This study investigates the use of self-similarity matrices based on features derived from the raw audio as input into a convolutional neural network to label eight form types found in the movements of piano sonatas from the Classical period, composed by Mozart, Beethoven, Haydn, Clementi and Czerny. Specifically, the focus on pieces for solo piano allows for the use of piano roll features which are generated from the raw audio by state-of-the-art piano transcription software. This work entails the first time that passing the entire self-similarity matrix to a convolutional neural network for the purposes of overall musical form recognition is proposed and explored. The method circumvents the potential difficulties related to inferring form labels in a bottom-up manner based on audio segment boundary detection and segment matching, by directly generating form labels from the audio. Self-similarity matrices based on velocity piano rolls (that contain values that relate to the velocity of the notes being played) were found to outperform other self-similarity matrix types and achieved a macro average ROC-AUC score of 0.823 and a coverage score of 2.045 on a custom data set which was compiled from verified musicological sources. The study is posed as a multi-label classification problem rather than a multi-class classification problem as different form labels were found for several piano sonata movements.