[ISMIR / NeurIPS 2022] Symbolic Music Motif Generation

Introduction

One of the tasks that requires the highest level of creativity is the creation of music. And if you consider the process of composing music, this is easy to understand. One must first come up with the overall musical theme and then consider a main melody as well as chords and an instrumental composition that go with that melody. After ideation, it additionally requires many engineering processes to reach real music such as vocals, lyrics, arrangements, and mixing.

Therefore, we considered methods in which AI might aid artists in the process of writing and performing songs. As a result of this, we began researching generative models that convey the motifs behind creation. The first step in musical composition is to generate a motif: a clue that becomes the central theme of the story. So, rather than generating the entire song at once, we decided it would be more appropriate to generate the essential 4 or 8 measures to help the artist turn them into good music.

In this post, I will be introducing two papers published at ISMIR 2022 and the NeurIPS 2022 Workshop on Machine Learning for Creativity and Design as a result of the above.

Background

Music can be expressed in many forms. Among these forms, the MIDI (Musical Instrument Digital Interface) form contains information about the instruments used and meta information related to music (signature, tempo, beat). The music is played through the note information (note on/off time, pitch, velocity) of each instrument. In particular, standardized conventions are established for digital musical instruments, making them simple to handle. Although the sound quality varies greatly depending on the virtual sound pack used and has the drawback of being unable to express the fine vibrations of string instruments, it was determined that it was sufficient to convey the motif due to its ability to handle data in a discrete space. Out of the many methods of expressing MIDI as a matrix, we used a binary representation(y∈{0,1}^{Time X Pitch X Inst})quantized in units of sixteenth notes.

Symbolic Music Loop Generation with Neural Discrete Representations^[1]

The biggest characteristic of the musical domain is the repetition of everything, from brief rhythmic patterns to phrases and verses. As a result, it may not be possible to accurately reflect many repetitive patterns when relying just on the transformer's self-attention. In order to solve this issue, we 1) concentrated on generating loops, and 2) employed discrete latent codes to express repetitive patterns in finite space.

The core frame of music is a loop, which is a musical unit consisting of 4 or 8 measures. Simply playing loops over and over or blending a few loops together properly can complete a piece of music. Therefore, loops can be a good motif for artists. Also, because the length to be generated is short, the error accumulation of autoregressive models is able to be reduced.

Thus, we designed a one-class loop detector that extracts loops from the Lakh MIDI Dataset (LMD)^[2] as part of the data-set preparation process. The reason is that there is no loop set for MIDI, and the existing methods of extracting loops have exclusively relied on detecting autocorrelated peaks from structural patterns in a song^{[3], [4]}.

Figure 1. The loop extraction process and performance evaluation from a one-class loop detector^[1]

If you listen to several music samples, you will notice visible beginnings, endings, and rules that are only present in loops. For example, assuming a loop of 8 measures, there are patterns such as [A - B - A - B - A - B - C - D], [A - B - C - D - A - B - C - D], and [A - B - A - B - A - B - A - B]. In the above examples, the first and fifth measures are similar. We attempted to extract loops from LMD by using a data-driven approach to learn these patterns. When only normal data is given, our loop detector, based on the anomaly detection model Deep SVDD^[5] operates under the principle of properly estimating the probability distribution and then regarding probabilistically low or outlier samples as abnormal. In other words, as shown in (b) of Figure 1, the model is trained to determine whether or not there is a loop using the probability value of the unseen sample, and it is assumed that the loop set is comprised of normal data. As learning data, a loop audio set gathered from the music community Looperman was used. As shown in (a) of Figure 1, audio and MIDI data have different formats. Despite this, we thought that converting to a bar-to-bar correlation matrix would be useful for discriminating loops, as it would represent the progression pattern of a song. The distance from the center in the embedding space was defined as a loop score score, which is described as probability above. The lower the value, the more the sample's structure resembles a loop. As a result, as shown on the right side of Figure 1, it was verified that the random set indicated a higher loop score than the loop set, which was significantly lower.

The loop set was then compressed using VQ-VAE^[6] for loop generation, and LSTM was used to model the compressed discrete code. This is because it is assumed that there is a finite number of codes that can express rhythm patterns that humans find familiar. When Transformer was applied as an autoregressive model, however, perhaps due to a lack of data, the performance in full sampling mode did not come out as desired.

Table 1. Performance evaluation^[1]

As shown in Table 1, the metric below was used to evaluate the performance.

- Loop Score (LS): Evaluation of how close to the loop structure samples are generated using the learned loop detector
- Unique Pitch (UP): Evaluation of the average number of pitches used in one measure, which means how generated samples are similar with training sets in terms of harmony
- Note Density (ND): Evaluation of the average number of notes used in one measure which means how generated samples are similar with training sets in terms of rhythms
- Precision & Recall (P & R)^[7]: Measurement of the overlapped ratio between the training set and the generated set, which means an expression of the fidelity and diversity of the sample
- Density &Coverage (D & C)^[8]: An index more robust with regard to outliers than P & R

However, in order to calculate P & R and D & C, latent features must be extracted from the pre-trained classifier. Since there was no classifier suitable for this study in the MIDI domain, referring to Naeem et al.^[8], randomly initialized networks were used as feature extractors. (The paper referenced reported that random networks can also be sufficiently utilized as feature extractors.) Our model outperformed Music Transformer^[9] and MuseGAN^[10] in terms of sample fidelity and diversity, and it also received a high score on the human listening test. Additionally, utilizing the learned loop detector, we were able to control the sample fidelity and diversity of the generative model, known as rejection sampling^[11]. [paper] [code] [demo]

Instrument Separation of Symbolic Music by Explicitly Guided Diffusion Model^[12]

There are many different ways to arrange music, and it is also possible to achieve the effect of arranging music by playing the same piece of music on different instruments. This is the same principle that applies when playing the same piece of music on an acoustic guitar, for example, to add a classic feel to a piece of music initially played on the piano.

In this study, when the label of the instrument is removed and a mixture (x∈{0,1}^{Time X Pitch})with only notes is given, we designed an arrangement generative model (Mixture2Music) that restores music (y∈{0,1}^{Time X Pitch X Inst}). Basically, it is a matter of deciding which instrument to assign each note to. As this problem can generate multiple solutions, this is an ill-posed problem and is quite similar to the problem of image segmentation or colorizing in computer vision. The goals that our model must achieve are 1) diversity: it must be able to generate a variety of music, and 2) consistency: it must be able to generate music while keeping as many of the original notes as possible.

Figure 2. Diffusion model for Mixture2Music^[12]

To this end, we utilized a diffusion model (DDPM^[13], DDIM^[14]), which is a likelihood-based model, in an effort to express the diversity of songs. In general, the likelihood-based generative model expresses different modes better than the GAN approach, and the diffusion model in particular has fewer limits on the model structure than VAE or normalizing flow. Additionally, we forced the predicted value of the reverse process not to generate unused notes by multiplying by the mixture in order to maintain the notes of the original song. As a result, our model was successful in producing outcomes that were both diverse and consistent. [paper] [code] [demo]

Conclusion

We introduced 1) loop extraction using structural patterns of music and loop generation research using discrete representation and 2) arrangement generation forcing strong consistency to the diffusion model, both of which we had begun to inspire artists. Through the research mentioned in the former, it was possible to generate high-quality loops with short lengths but high versatility. Through the latter, we proposed a new method of forcing constraints on the diffusion model so that music could be arranged using various methods. We intend to generate long music in the future through ongoing research, reflect repeated patterns in various layers, and develop it into a form that is easy for humans to control.

▶Instrument Separation of Symbolic Music by Explicitly Guided Diffusion Model (Link)