Teaching Composition to a Computer      ... for people who don't know music

Table of Contents


Algorithmic and generative music composition tends to fall under the vast umbrella of Computational Creativity, a relatively new field of research.

Computational Creativity
  • Computational creativity is a multidisciplinary endeavour that is located at the intersection of the fields of artificial intelligence, cognitive psychology, philosophy, and the arts. Read more…

The goals of computational creativity are

These are rather lofty goals, but represent incredibly important directions for A.I. research, and for our own understanding of creativity. As the field has grown, there’s been a shift from ‘mere generation’ to evaluation methods – e.g. what are the measures of creativity, what distinguishes creative work from uncreative work?

There’s a whole slew of interesting work that’s been done already, some examples are:

Although here, the focus will be on music composition.

Common Techniques

Most techniques focus on ‘mere generation’ of music – that is, there isn’t much in the way of evaluation techniques – and are generally applications of machine learning research. The most common approach is to ‘learn’ a style of composition from a source corpora, and generate music that is similar to the source.

Markov Chains

Markov Chains are incredibly simple and generally have interesting output, hence their popularity. Markov based compositions tend to lack any high-level structure, although there have been attempts to remedy this problem. One of the most interesting involves a scheme similar to generating and filling in madlibs [5].

The Madlib Approach

  1. Structure Generation: Sketch out the piece by identifying something like a rhyme scheme for phrases and/or motifs; e.g. ABAA BBCB.

  2. Chord Generation: Place chords by assigning each a root note to each ‘blank’.

  3. Melody Generation: Generate notes corresponding to the chord for each measure.

You can here it here.

Genetic Algorithms

The algorithmic version of natural selection. Given an initial population, there are generally three steps to a genetic algorithm.

  1. Selection – Assign every member of the population a fitness score, which measures how ‘good’ that member is.
  2. Mating – Combine the most members with the highest fitness scores to create a new generation of ‘children’.
  3. Mutation – Randomly change the values of the ‘children’.

For example, consider a situation where you have 10 Markov Chains from 10 different places, and your fitness function is just the probability of outputting a C#. The steps would then be:

  1. Selection – Choose the five Markov Chains with the highest probability of outputting a C#.
  2. Mating – For every possible pair from the previous step, create a new Markov Chain that uses half of the probabilities from one parent, and half from the other.
  3. Mutation – Randomly change the probabilities for the children generated from the previous step.

Neural Networks

Neural Networks are the algorithmic version of neurons in the brain. The way it works is that each node can be ‘on’ or ‘off’, and if a neuron is ‘on’, it has a probability of activating connected neurons (e.g. turning them ‘on’ as well). Do this a bunch of times until you get to the output neurons, and you have the result.

This method has been very successful in practice [3], although doesn’t vary much in style.

Previous Work

Functional Scaffolding for Musical Composition

Functional scaffolding for musical composition (FSMC) by Hoover et. al. [1] details a method of music generation which enables a user to create complete multipart arrangements from a single monophonic track without the need for musical expertise. This is an example of computer-assisted creativity, where a novice can create something beyond their expertise with the help of a computer.

In essence, the source monophonic melody serves as the scaffold or musical starting point of the system. FSMC then allows the user to explore potential accompaniments from this scaffold. Each instrument in the scaffold is generated using neural networks. Users can navigate the musical space by using an interface (MaestroGenesis, in the paper), and thums-up or thumps-down accompaniments to evolve the neural networks to help the system generate music more in line with what they’re looking for.

Examples of the kind of output this system generates can be found here.

An Unnatural Selection

A live generative music project by Arne Eignfeldt [2] which generates music during a performance, meant to be sight-read by musicians.

From the program notes:

Musical ideas are born, are passed on to new generations and evolved, and eventually die out, replaced by new ideas

In short, a genetic algorithm is used to unfold the music over time. Here, each instrument is a different population, each phrase an individual with the rhythmic and melodic material of the phrase acting as its chromosomes.

So like natural selection, individuals in a population breed together to produce new ‘offspring’ which may be subject to random mutations. Interestingly, there is no fitness function in this work; all individuals pass to the next generation.

Here’s an overview:

The following corpora was used:

An Unnatural Selection: I - Imagination is a Way

To quote the author:

Whether the system is computationally creative is a more difficult matter to determine. While I echo Cope’s desire that “what matters most is the music” (Cope 2005)1, I am fully aware of Wiggins reservations that “with hand-coded rules of whatever kind, we can never get away from the claim that the creativity is coming from the programmer and not the program” (Wiggins 2008)2

My Work

I’ve only just begun working on generative music and so my approaches are rather naive and lack any solid theoretical background (though hopefully improving).

Melody Generator for Lyrics

This system receives a lyrical phrase as input and generates a melody for it. I’m working on this project for my Master’s thesis in Computer Science. It’s still just getting started, but thus far the way it works is:

  1. Break the lyrical phrase into individual syllables
  2. Initially, assign a quarter note to each syllable
  3. Increase the duration of syllables that are vowels by a random amount.
  4. Assign pitches to the syllable durations

Currently, pitches are assigned using a Markov Chain trained on folk music. The analysis of syllables is done using CMUdict (the Carnegie Mellon Pronouncing Dictionary), which yields the IPA (International Phonetic Alphabet) pronunciation for each word. This allows us to check for syllables that are vowels.


B.A.E.S.I.L. (an acronym for BAESIL Automates Exploratory Songwriting Interactively, Live) is a composition tool which uses generative music methods to help a user write a song, initially targeted at game developers. BAESIL is largely inspired by the concept of casual creators [4].

Essentially, BAESIL is used to help a user discover the music they want, rather than compose it from scratch.

To generate music, BAESIL uses a pair of Markov Chains – one for pitch and one for duration. Users upload music to serve as an initial corpora. Users can then edit, regenerate, delete, and cut/copy/paste notes as they desire.

The eventual goal of BAESIL is to use genetic algorithms so that the user can train the system to generate the kind of music they want, by using a “thumbs-up”/”thumbs-down” type of system. The user can also bookmark sections that they like and paste them wherever they’d like – essentially like a word processor.

The way I plan to implement the evolutionary process is as follows:

Currently, I’m not satisfied with this approach as I don’t think this will have great results, but it serves as an initial direction. Essentially, I’m working to build upon the work of [1] in this project.

Future Work

Generative music is still a relatively new concept, in the sense that there is a great deal of work to be done. More specifically, the elephant in the room is the analysis of musicial structure. Most of these methods involve generating music given an initial corpora without much scrunity to sound compositional practices.

Given the breadth and difficulty of this task, lots of interesting methods exist but none that can satisfactorily write a piece of music that sounds like Bach’s interpretation of Coltrane, for example.

See Also

  1. Evaluating Musical Metacreation
    – Arne Eigenfeldt, Philippe Pasquier and Adam Burnett [pdf]
  2. Critical issues in evaluating freely improvising interactive music systems
    – Adam Linson, Chris Dobbyn and Robin Laney [pdf]
  3. Player Responses to a Live Algorithm: Conceptualising computational creativity without recourse to human comparisons?

    – Oliver Bown [pdf]

  4. Collaborative Composition with Creative Systems: Reflections on the First Musebot Ensemble
    – Arne Eigenfeldt, Oliver Bown and Benjamin Casey [pdf]
  5. Metacreation Lab


  1. Generating a Complete Multipart Musical Composition from a Single Monophonic Melody with Functional Scaffolding
    – Amy K. Hoover, Paul A. Szerlip, Marie E. Norton, Trevor A. Brindle, Zachary Merritt and Kenneth O. Stanley [pdf]
  2. Generative Music for Live Musicians: An Unnatural Selection
    – Arne Eigenfeldt [pdf]
  3. Composing Music With Recurrent Neural Networks
    – Daniel Johnson [article]
  4. Casual Creators
    – Kate Compton and Michael Mateas [pdf, blog]
  5. Algorithmic Compositions Based on Discovered Musical Patterns
    – Shan Man-Kawn and Chiu Shih-Chuan

  1. Cope, D. 2005. Computer models of musical creativity, Cambridge: MIT Press. 

  2. Wiggins, G. 2008. Computer Models of Musical Creativity: A Review of Computer Models of Musical Creativity by David Cope, Literary and Linguistic Computing, 10(1), 109–116.