Generating Music

Recently I’ve undertaken a personal project (computoser) that is a bit strange. I tried to write software that generates music at random. Good music, that is, because “random” usually generates noise. The idea, of course, is not at all new – there has been research on the topic and there is software that attempts to generate music. I generally group this software in three categories:

  • software for helping composers – it’s the composers who handle most of the things, but the software can suggest some motifs, variations, etc. to help the composer
  • programs that take functions or other mathematical concept and try to transform it to frequencies, based on some arbitrary mapping
  • programs that generate music adhering to composition rules

Out of these three, the first one is very specialized and not suitable for a large audience, and the second is not producing results that a sane person would listen to (the mathematical/physics rules that these algorithms try to employ are already used at the lower level of music – e.g. note frequencies double each octave; additionally, the mapping between the functions/sequences to notes are often arbitrary, so even though the the Fibonacci numbers are “nice”, there is no definitive way of mapping them to music that is guaranteed to sound good). The third group can generate “listenable” music, but it needs to properly define the rules that composers adhere to when writing music. Some of these programs use fractals, other use pictures, text, or whatever to generate music. In fact, these are all different ways to seed the random.

So, I obviously I went for the third option, implemented the algorithm, which is obviously the hard part. There is more music to it than programming, so it would probably suffice to say that there is a lot of random.nextInt() all over the place to pick different routes in the generation process, so that each piece may sound a bit different. And the choices include the note lengths, the scale, the interval between consecutive pitches (notes), maintaining a proper contour, and more. Later I realized I have used something similar to Markov chains, because there are probabilities for these decisions. But in addition to the algorithm, there was a technical challenge: how do I play the generated midi file in the browser?

  • browsers don’t play midi. Flash doesn’t play midi
  • generating an mp3 from midi is a CPU-heavy process and if 10 people open the page at the same time and trigger the music generation, the server dies (a micro EC2-instance, yes, but that wouldn’t scale even on a medium instance)

I considered a lot of options. The Web Audio browser API is a workable option, and with the help of Michael Deal’s js library one can indeed play midi in the browser. However, there are complications – you’d have to load the soundbank on the client, and the cross-browser support is not exactly guaranteed. Initially the library did not support multiple channels, but now it does, so it looks like a very viable option. It is always an option to just send the MIDI for download and let the user player with whatever player is installed on their machine, but that’s not good user experience.

So I went for a different approach – a scheduled job runs every couple of minutes and generates a new track, renders it to mp3 (via wav) and stores it. Then whenever a user comes, he gets served a newly generated track. If a lot of users come at the same time, this is recorded, and the scheduled job starts generating more tracks. If it can’t cope, then old tracks are served to the user, but such that are not listened by more than one or two other people.

The end result, Computoser, is now live, and I think it is capable of generating pretty nice music.

The UI is using Bootstrap (I’m not much of a designer) and the player is jPlayer, which plays the mp3. The whole thing runs on Amazon EC2 (free tier, for now) and stores the generated music in an S3 bucket. The programming stack is Java and Spring MVC, with MySQL and EhCache. Nothing fancy, but the focus was more on the music generation part and less on the architecture, and as I didn’t want to spend time learning new technologies, I picked the ones I’m familiar with. One of the reasons to pick Java was that there are nice high-level APIs for working with MIDI – jMusic (I’m using that one) and jFugue, which let you work with the proper terms (scales, chord, instruments, notes, rests, etc.) rather than with low-level midi instructions. Oh, and the track titles are generated by a tiny linguistic algorithm that tries to construct fairly proper English phrases based on a small dictionary and a couple of predefined structures (and it needs a lot more work).

P.S. Added to Github, paper with details linked in the Readme

52 thoughts on “Generating Music”

  1. “implemented the algorithm, which is obviously the hard part, but I won’t discuss it in details”

    No GitHub, no algorithmic details… so basically, anyone arriving here from a certain “Hacker News” site can quietly admire your black-box magic and then show themselves out the door… okayface.jpg ;D

  2. Well, the reason I didn’t go in details is that it has too much music terms. The general overview is that I have coded as many composition rules as I can. What are these rules? Well, choosing a better interval, choosing good note lengths, retaining a proper contour. Most of that is randomly choosing between one option or the other, which an actual composer would also do. I added these examples to the post, so that readers can get a bit better picture.

  3. This is cool. I did something similar for my college thesis with Markov models, cellular automata, genetic algorithms and the like-also used jMusic. 😀

  4. @mhy – could you share your results? I’ve spent a lot of time listening to other attempts in the area.

  5. I also made something similar that just generates the midi, using Markov models and genetic algorithms.

    Will you share this on github?

  6. by “results”, I meant midi/mp3’s – mine are out there for getting (open your firebug, check the URL and grab the mp3). (One of the reasons this is not on github is that I don’t like git (using mercurial here)).

    I will probably eventually share the code, but it needs to become more complete and less patchy.

  7. I’ve been thinking about this a lot too – I think your result it pretty awesome as a starting point. I was thinking of a music “stream” that mutated according to user votes, but your result is interesting and probably more easily achievable.

  8. I had this idea about two years ago when I was first learning Fourier Transforms. The idea I had was that if every signal existing had a unique Fourier Transform, how about taking FTs of music you like[sample data], finding something unique between all these transforms to produce a transform which you can convert to audible music.

    But, obviously I never went so far to actually try this thing out. Any Signals expert here care to comment on whether it would work?

  9. The music is good but the instruments are a too primitive and harsh to listen to for long. I’d like to hear longer pieces with gentler instruments.

    One thing I like is the lack of an overt emotional quality to this music. It tends to avoid cliché and keeps me interested. Well done with your algorithms. I’ll stay tuned for developments. Apart from the intellectual challenge, I guess the exciting thing about composing in algorithms is you’re moulding not a single piece but actually producing an open ended music stream, with the possibility of generating unanticipated delights.

  10. Exactly 🙂

    The soundbank is really something to improve, as many comments suggested. I will look for something better.

  11. Browsers don’t play MIDI? Well, maybe not…but computers usually do and the browser just lets the computer handle it. You won’t be able to uue anything but GM, though. 🙁
    I never bothered trying to do it on the web, since the desktop is much more versatile. I’ve been fiddling with a music creation program since 1999 or thereabouts, the result being here:

    And the results of the result, as it were are in the Mu[sic] pages of the same site.


  12. >choosing a better interval, choosing good note lengths, retaining a proper contour. Most of that is randomly choosing between one option or the other, which an actual composer would also do.

    It might be a good idea to make the algorithm heuristic – let the software learn what the best options are based on what the listeners like.

  13. Sounds pretty good.
    Do I have to sign in with facebook just to generate somethin myself?

    I’ve got a desktop app that does SOME algorithmic composition.
    Mostly it just comes up with a chord progression and arpeggiates it different ways. Not a very complete song, but, eh, it’s a start. My main focus has been piano practice. If ya install the thing at and then load up the “RandomSong.mid”, it’s recalc’d each time you load it.

    I would be interested in hearing about the algorithms you use 🙂 (Not the midi=>mp3 part, just the note generation)

  14. @StephenHazel no, signing up is optional – everything runs fine without it, you just don’t get a collection of what you have previously liked.

    As for the note generation – it really isn’t something that strange – just picking an interval at random, according to some rules (e.g.prefer perfect intervals). Assign a range for the piece, so if a note should go above that range, it doesn’t, and so on.

  15. I read an book in 1968 and it was readable only 3 first pages, all others were pure math infested with math, and heard a decent result back in 60s made on one of the 1st computers occupying several big rooms. Yes, it was based on Markov’s sequence. Since then nothing was even close to the result I heard.
    Since then I was fascinated with the idea. I found that if 10 people with the same cultural background listen to 10 different melodies, the will rate them almost the same. It means that we have an algorithm of comparing music, of rating it. We need to recognize this algorithm.
    That will resolve the problem.

  16. multiple parts – bass, mainline, chord, drums. All optional except for the mainline. The others follow the mainline in some ways (the bass is currently not following, just using the same scale)

  17. @Bozho

    It’s pretty terrible code, so I’m looking long-term to rewrite it and put a web front-end on it. Maybe I’ll release it some day 😀

  18. Seems to be broken at the moment? (‘Sorry, a problem occured. Please click ‘next’ or refresh the page’, and the getMidi link is a 404)

  19. I would like to know the effect of the variations of the slop cursor . To level music setting and to. level rhythm map. In other words what difference exists among slop 0 and slop -6 ?.
    Besides, why in midimage 3 the shrink button has disappeared ? Are the same results gotten acting with the values of scan (step) and max phrase? Thanks in advance for answer. Giampiero

  20. I’m amazed, I must say. Seldom do I encounter a
    blog that’s both educative and amusing, and without a doubt,
    you’ve hit the nail on the head. The problem is something that too few folks are speaking
    intelligently about. I am very happy I stumbled
    across this in my search for something relating to this.

  21. magnificent put up, very informative. I ponder why the other specialists of
    this sector don’t understand this. You must proceed your writing.
    I am sure, you have a great readers’ base already!

  22. I have no intention of knowing my weight, nor do I really care.
    This will help eliminate the unconscious snacking that is so much a
    part of being overweight. I made an effort to crank out 100 kettle
    bell swings split up, but generally settled with 80.

  23. Hmm no technical details about anything. what a shame. I suppose we take your word for it that this algorithm exists and that it is in fact random?

  24. Excellent blog! Do you have any hints for aspiring
    writers? I’m hoping to start my own site soon but I’m a little lost on everything.
    Would you recommend starting with a free platform like WordPress or go for a paid option?
    There are so many options out there that
    I’m completely overwhelmed .. Any suggestions? Thanks!

  25. Good to get the background. I am looking for an algorithm that is not based on Western music, since even when I input supposedly random notes I still get patterns and rhythms that are distinctly Western. Do you know of any? Thanks.

  26. Well, you can try different scales – e.g. Turkish, Indian. They give a distinct non-Western feel

Leave a Reply

Your email address will not be published.