Generating Music
Recently I’ve undertaken a personal project (computoser) that is a bit strange. I tried to write software that generates music at random. Good music, that is, because “random” usually generates noise. The idea, of course, is not at all new – there has been research on the topic and there is software that attempts to generate music. I generally group this software in three categories:
- software for helping composers – it’s the composers who handle most of the things, but the software can suggest some motifs, variations, etc. to help the composer
- programs that take functions or other mathematical concept and try to transform it to frequencies, based on some arbitrary mapping
- programs that generate music adhering to composition rules
Out of these three, the first one is very specialized and not suitable for a large audience, and the second is not producing results that a sane person would listen to (the mathematical/physics rules that these algorithms try to employ are already used at the lower level of music – e.g. note frequencies double each octave; additionally, the mapping between the functions/sequences to notes are often arbitrary, so even though the the Fibonacci numbers are “nice”, there is no definitive way of mapping them to music that is guaranteed to sound good). The third group can generate “listenable” music, but it needs to properly define the rules that composers adhere to when writing music. Some of these programs use fractals, other use pictures, text, or whatever to generate music. In fact, these are all different ways to seed the random.
So, I obviously I went for the third option, implemented the algorithm, which is obviously the hard part. There is more music to it than programming, so it would probably suffice to say that there is a lot of random.nextInt() all over the place to pick different routes in the generation process, so that each piece may sound a bit different. And the choices include the note lengths, the scale, the interval between consecutive pitches (notes), maintaining a proper contour, and more. Later I realized I have used something similar to Markov chains, because there are probabilities for these decisions. But in addition to the algorithm, there was a technical challenge: how do I play the generated midi file in the browser?
- browsers don’t play midi. Flash doesn’t play midi
- generating an mp3 from midi is a CPU-heavy process and if 10 people open the page at the same time and trigger the music generation, the server dies (a micro EC2-instance, yes, but that wouldn’t scale even on a medium instance)
I considered a lot of options. The Web Audio browser API is a workable option, and with the help of Michael Deal’s js library one can indeed play midi in the browser. However, there are complications – you’d have to load the soundbank on the client, and the cross-browser support is not exactly guaranteed. Initially the library did not support multiple channels, but now it does, so it looks like a very viable option. It is always an option to just send the MIDI for download and let the user player with whatever player is installed on their machine, but that’s not good user experience.
So I went for a different approach – a scheduled job runs every couple of minutes and generates a new track, renders it to mp3 (via wav) and stores it. Then whenever a user comes, he gets served a newly generated track. If a lot of users come at the same time, this is recorded, and the scheduled job starts generating more tracks. If it can’t cope, then old tracks are served to the user, but such that are not listened by more than one or two other people.
The end result, Computoser, is now live, and I think it is capable of generating pretty nice music.
The UI is using Bootstrap (I’m not much of a designer) and the player is jPlayer, which plays the mp3. The whole thing runs on Amazon EC2 (free tier, for now) and stores the generated music in an S3 bucket. The programming stack is Java and Spring MVC, with MySQL and EhCache. Nothing fancy, but the focus was more on the music generation part and less on the architecture, and as I didn’t want to spend time learning new technologies, I picked the ones I’m familiar with. One of the reasons to pick Java was that there are nice high-level APIs for working with MIDI – jMusic (I’m using that one) and jFugue, which let you work with the proper terms (scales, chord, instruments, notes, rests, etc.) rather than with low-level midi instructions. Oh, and the track titles are generated by a tiny linguistic algorithm that tries to construct fairly proper English phrases based on a small dictionary and a couple of predefined structures (and it needs a lot more work).
P.S. Added to Github, paper with details linked in the Readme
Recently I’ve undertaken a personal project (computoser) that is a bit strange. I tried to write software that generates music at random. Good music, that is, because “random” usually generates noise. The idea, of course, is not at all new – there has been research on the topic and there is software that attempts to generate music. I generally group this software in three categories:
- software for helping composers – it’s the composers who handle most of the things, but the software can suggest some motifs, variations, etc. to help the composer
- programs that take functions or other mathematical concept and try to transform it to frequencies, based on some arbitrary mapping
- programs that generate music adhering to composition rules
Out of these three, the first one is very specialized and not suitable for a large audience, and the second is not producing results that a sane person would listen to (the mathematical/physics rules that these algorithms try to employ are already used at the lower level of music – e.g. note frequencies double each octave; additionally, the mapping between the functions/sequences to notes are often arbitrary, so even though the the Fibonacci numbers are “nice”, there is no definitive way of mapping them to music that is guaranteed to sound good). The third group can generate “listenable” music, but it needs to properly define the rules that composers adhere to when writing music. Some of these programs use fractals, other use pictures, text, or whatever to generate music. In fact, these are all different ways to seed the random.
So, I obviously I went for the third option, implemented the algorithm, which is obviously the hard part. There is more music to it than programming, so it would probably suffice to say that there is a lot of random.nextInt() all over the place to pick different routes in the generation process, so that each piece may sound a bit different. And the choices include the note lengths, the scale, the interval between consecutive pitches (notes), maintaining a proper contour, and more. Later I realized I have used something similar to Markov chains, because there are probabilities for these decisions. But in addition to the algorithm, there was a technical challenge: how do I play the generated midi file in the browser?
- browsers don’t play midi. Flash doesn’t play midi
- generating an mp3 from midi is a CPU-heavy process and if 10 people open the page at the same time and trigger the music generation, the server dies (a micro EC2-instance, yes, but that wouldn’t scale even on a medium instance)
I considered a lot of options. The Web Audio browser API is a workable option, and with the help of Michael Deal’s js library one can indeed play midi in the browser. However, there are complications – you’d have to load the soundbank on the client, and the cross-browser support is not exactly guaranteed. Initially the library did not support multiple channels, but now it does, so it looks like a very viable option. It is always an option to just send the MIDI for download and let the user player with whatever player is installed on their machine, but that’s not good user experience.
So I went for a different approach – a scheduled job runs every couple of minutes and generates a new track, renders it to mp3 (via wav) and stores it. Then whenever a user comes, he gets served a newly generated track. If a lot of users come at the same time, this is recorded, and the scheduled job starts generating more tracks. If it can’t cope, then old tracks are served to the user, but such that are not listened by more than one or two other people.
The end result, Computoser, is now live, and I think it is capable of generating pretty nice music.
The UI is using Bootstrap (I’m not much of a designer) and the player is jPlayer, which plays the mp3. The whole thing runs on Amazon EC2 (free tier, for now) and stores the generated music in an S3 bucket. The programming stack is Java and Spring MVC, with MySQL and EhCache. Nothing fancy, but the focus was more on the music generation part and less on the architecture, and as I didn’t want to spend time learning new technologies, I picked the ones I’m familiar with. One of the reasons to pick Java was that there are nice high-level APIs for working with MIDI – jMusic (I’m using that one) and jFugue, which let you work with the proper terms (scales, chord, instruments, notes, rests, etc.) rather than with low-level midi instructions. Oh, and the track titles are generated by a tiny linguistic algorithm that tries to construct fairly proper English phrases based on a small dictionary and a couple of predefined structures (and it needs a lot more work).
P.S. Added to Github, paper with details linked in the Readme
is this on github?
Nope.
“implemented the algorithm, which is obviously the hard part, but I won’t discuss it in details”
No GitHub, no algorithmic details… so basically, anyone arriving here from a certain “Hacker News” site can quietly admire your black-box magic and then show themselves out the door… okayface.jpg ;D
Well, the reason I didn’t go in details is that it has too much music terms. The general overview is that I have coded as many composition rules as I can. What are these rules? Well, choosing a better interval, choosing good note lengths, retaining a proper contour. Most of that is randomly choosing between one option or the other, which an actual composer would also do. I added these examples to the post, so that readers can get a bit better picture.
This is cool. I did something similar for my college thesis with Markov models, cellular automata, genetic algorithms and the like-also used jMusic. 😀
@mhy – could you share your results? I’ve spent a lot of time listening to other attempts in the area.
Bozho: Why don’t you share your results before asking others to do so?
I also made something similar that just generates the midi, using Markov models and genetic algorithms.
Will you share this on github?
by “results”, I meant midi/mp3’s – mine are out there for getting (open your firebug, check the URL and grab the mp3). (One of the reasons this is not on github is that I don’t like git (using mercurial here)).
I will probably eventually share the code, but it needs to become more complete and less patchy.
In the university of my city, a research group built something similar: http://www.youtube.com/watch?v=bD7l4Kg1Rt8
For http://muziki.org I have solved the challenge of playing MIDI in the browser by using Flash and Tonfall (http://code.google.com/p/tonfall/). The user edited scores are parsed by a Perl MIDI library server side, and passed back as JSON.
Tonfall allows to use soundbanks (“sound sheets”) to create a better quality audio.
I’ve been thinking about this a lot too – I think your result it pretty awesome as a starting point. I was thinking of a music “stream” that mutated according to user votes, but your result is interesting and probably more easily achievable.
I had this idea about two years ago when I was first learning Fourier Transforms. The idea I had was that if every signal existing had a unique Fourier Transform, how about taking FTs of music you like[sample data], finding something unique between all these transforms to produce a transform which you can convert to audible music.
But, obviously I never went so far to actually try this thing out. Any Signals expert here care to comment on whether it would work?
Listening to the algorithm. It sounds beautiful.
>I don’t like git (using mercurial here)
Me too; see: https://en.wikipedia.org/wiki/Mercurial#Source_code_hosting
The music is good but the instruments are a too primitive and harsh to listen to for long. I’d like to hear longer pieces with gentler instruments.
One thing I like is the lack of an overt emotional quality to this music. It tends to avoid cliché and keeps me interested. Well done with your algorithms. I’ll stay tuned for developments. Apart from the intellectual challenge, I guess the exciting thing about composing in algorithms is you’re moulding not a single piece but actually producing an open ended music stream, with the possibility of generating unanticipated delights.
Exactly 🙂
The soundbank is really something to improve, as many comments suggested. I will look for something better.
Browsers don’t play MIDI? Well, maybe not…but computers usually do and the browser just lets the computer handle it. You won’t be able to uue anything but GM, though. 🙁
I never bothered trying to do it on the web, since the desktop is much more versatile. I’ve been fiddling with a music creation program since 1999 or thereabouts, the result being here: http://www.persongo.net/BFP/MIDImage.aspx
And the results of the result, as it were are in the Mu[sic] pages of the same site.
Dan
I really am impressed! Great work. This would be great with a chip-tunes sound.
>choosing a better interval, choosing good note lengths, retaining a proper contour. Most of that is randomly choosing between one option or the other, which an actual composer would also do.
It might be a good idea to make the algorithm heuristic – let the software learn what the best options are based on what the listeners like.
Nice back ground music
no code? see ya
Sounds pretty good.
Do I have to sign in with facebook just to generate somethin myself?
I’ve got a desktop app that does SOME algorithmic composition.
Mostly it just comes up with a chord progression and arpeggiates it different ways. Not a very complete song, but, eh, it’s a start. My main focus has been piano practice. If ya install the thing at http://PianoCheetah.com and then load up the “RandomSong.mid”, it’s recalc’d each time you load it.
I would be interested in hearing about the algorithms you use 🙂 (Not the midi=>mp3 part, just the note generation)
@StephenHazel no, signing up is optional – everything runs fine without it, you just don’t get a collection of what you have previously liked.
As for the note generation – it really isn’t something that strange – just picking an interval at random, according to some rules (e.g.prefer perfect intervals). Assign a range for the piece, so if a note should go above that range, it doesn’t, and so on.
I read an book in 1968 and it was readable only 3 first pages, all others were pure math infested with math, and heard a decent result back in 60s made on one of the 1st computers occupying several big rooms. Yes, it was based on Markov’s sequence. Since then nothing was even close to the result I heard.
Since then I was fascinated with the idea. I found that if 10 people with the same cultural background listen to 10 different melodies, the will rate them almost the same. It means that we have an algorithm of comparing music, of rating it. We need to recognize this algorithm.
That will resolve the problem.
@Bozho to come up with chord progression using this “mapping”:
http://mugglinworks.com/chordmaps/mapC.htm
Then I arpeggiate with patterns that are explained in…
http://www.reddit.com/r/piano/comments/ldnnj/how_to_play_pop_piano_ballads/
So do you use just one bass line and one melody?
And somehow put chords between them?
multiple parts – bass, mainline, chord, drums. All optional except for the mainline. The others follow the mainline in some ways (the bass is currently not following, just using the same scale)
@Bozho
It’s pretty terrible code, so I’m looking long-term to rewrite it and put a web front-end on it. Maybe I’ll release it some day 😀
Bozho, do you find that Computoser music reminds you of music you have composed?
I have to admit I have never composed anything more than a simple motif.
this should def be on github
Seems to be broken at the moment? (‘Sorry, a problem occured. Please click ‘next’ or refresh the page’, and the getMidi link is a 404)
What browser are you using? It works for me.
You may also want to have a look at SoundHelix, a nice algorithmic random music composer written in Java.
Check out some examples here: http://www.soundhelix.com/audio-examples
I would like to know the effect of the variations of the slop cursor . To level music setting and to. level rhythm map. In other words what difference exists among slop 0 and slop -6 ?.
Besides, why in midimage 3 the shrink button has disappeared ? Are the same results gotten acting with the values of scan (step) and max phrase? Thanks in advance for answer. Giampiero
To be honest, I didn’t get the question at all…
i am more interest in the second categories software, could you recommend some?
Well done! I like your idea and current implementation actually works pretty well I would say :).
Asking questions are really fastidious thing if you
are not understanding anything entirely, except this paragraph
provides fastidious understanding yet.
Visit my website … garage door repair
This is my first time pay a quick visit at here and i am in fact
happy to read all at alone place.
Yes! Finally someone writes about georgia power company atlanta ga bill
pay.
I’m amazed, I must say. Seldom do I encounter a
blog that’s both educative and amusing, and without a doubt,
you’ve hit the nail on the head. The problem is something that too few folks are speaking
intelligently about. I am very happy I stumbled
across this in my search for something relating to this.
magnificent put up, very informative. I ponder why the other specialists of
this sector don’t understand this. You must proceed your writing.
I am sure, you have a great readers’ base already!
I have no intention of knowing my weight, nor do I really care.
This will help eliminate the unconscious snacking that is so much a
part of being overweight. I made an effort to crank out 100 kettle
bell swings split up, but generally settled with 80.
Hmm no technical details about anything. what a shame. I suppose we take your word for it that this algorithm exists and that it is in fact random?
I’ve written a paper about it and it will be published anytime now
Excellent blog! Do you have any hints for aspiring
writers? I’m hoping to start my own site soon but I’m a little lost on everything.
Would you recommend starting with a free platform like WordPress or go for a paid option?
There are so many options out there that
I’m completely overwhelmed .. Any suggestions? Thanks!
Good to get the background. I am looking for an algorithm that is not based on Western music, since even when I input supposedly random notes I still get patterns and rhythms that are distinctly Western. Do you know of any? Thanks.
Well, you can try different scales – e.g. Turkish, Indian. They give a distinct non-Western feel