Here’s an article on midimutant we did with Aphex Twin for MagPi Magazine, written by Sean McManus. Most of the work on this project recently has revolved around exploring custom hardware using old FM synth chips from games consoles, but with any luck there should be some more evolved DX7 sounds around here soon.
So I’ve been getting a into FM synthesis lately, and after giving the TX7 a spin I’ve been really wanting to try evolving sounds for the later synth chips Yamaha made for games consoles and soundcards. At that point in history FM was one of the only realistic ways you could synthesise complex sound as there wasn’t enough time (processing) for general purpose DSP, and there wasn’t enough space (memory) for sample playback to be an option.
One of the most famous of these range of synth chips was the YM2612 which was used in the Sega Megadrive/Genesis consoles. I got hold of a few to try, but it’s been quite a challenge to get them working so I thought probably worth documenting the process. The YM2612 is programmed using 5 control pins and 8 data pins. You send one byte to the chip at a time – first an address to write to, then the data to write, which allows you to set a couple of hundred parameters that define 6 separate sounds. The memory on the chip is organised into a map of the global settings (top left) and the voice settings in two banks:
There isn’t much documentation for these things, the main reference is for a similar chip which is written in Japanese. More usefully there are software versions in emulators, and quite a few bits and pieces online from other people’s projects and some translations of the docs. Usefully the Sega documentation included a piano test sound which is great as you have something that *should* actually make a noise as long as everything else is working.
The first problem was figuring out the timing – despite having some examples it took me a while to realise that there is a specific order that you have to set the status pins, and that there is one change at one time that actually triggers (latches) the data transfer to the synth:
This is a timing diagram, I’m used to seeing them in datasheets, but it’s the first time I’ve needed to understand one properly (with a bit of help from David Viens). It tells us that for writing, the data (D0-D7 pins) are actually read shortly after the WR pin is set low – to add to the fun, some of the pins (denoted by the line over them) are 0v for on, 5v for off. You can also read data from the synth, but whatever address you supply you only get a single status byte that tells you if it’s currently busy, and if either of it’s two internal clocks have overflowed. The idea I assume is that you would use them for timing music accurately without the need to do it in your main CPU.
The audio coming out needs to be amplified too – if you connect it directly to a speaker there is a danger of overloading the DAC and causing it to burn out, so you need a bit of analogue circuitry to do this. Here is an example of triggering and glitching (by writing to random addresses) the example piano sound:
I started off using an Arduino Nano, but didn’t have much success so switched to atmega328 which was a bit easier to set up. One of the problems is that ideally you need to be able to control the 8 data GPIO pins in one go (to set the 8bit value), which isn’t possible from the nano as none of the ports have 8 bits. You also have to supply the YM2612 with a clock pulse at around 8Mhz – which took me a while to get right, I tried both dividing the atmega’s 16Mhz clock by 2 and outputting it on a pin as well as a separate external oscillator and both eventually worked – except the oscillator leaked quite badly into the audio output (there is probably a fairly simple fix to isolate this).
One potential problem is dodgy vintage chips, I tried two suppliers on eBay, one Chinese shop and another in the UK – all the ones I’ve tested so far worked fine, I’m happy to say. The best way of testing them is to use a second hand Sega Megadrive console and replacing the YM2612 it has with a socket you can easily plug in candidates for testing – I managed without but it would have come in handy sanity wise to know for sure that the chips worked.
One thing that I wasted a lot of time doing was ignoring the output entirely and just trying to get the status bit to change by setting the internal timer – the idea was to make sure the chip logic was working before worrying about the audio amplifier circuitry. However, this requires the read sequence timing to work as well as the write timing – and as soon as you can listen to the output you get small clues from the DAC noise, which actually indicates the chip is running or not even if there is no sound triggered.
Here are some links I found helpful: the wikipedia entry for the pinout, full software documentation (for the registers) and a great blog post about putting it all together, with schematics – in French. Here is a hardware synth module with similar YMF262 chips.
As the egglab camouflage experiment continues, here are some recent examples after 40 or so generations. If you want to take part in a newer experiment, we are currently seeing if a similar approach can evolving motion dazzle camouflage in Dazzle Bug.
Each population of eggs is being evolved against a lot of background images, so it’s interesting to see the different strategies in use – it seems like colour is one of the first things to match, often with some dazzle to break up the outline. Later as you can see in some of these examples, there is some quite accurate background matching happening.
It’s important to say that all of this is done entirely by the perception from tens of thousands of people playing the game – there is no analysis of the images at any point.
AI in games is a hot topic at the moment, but most examples of this are attempts to create human-like behaviour in characters – a kind of advanced puppetry. These characters are also generally designed beforehand rather than reacting and learning from player behaviour, let alone being allowed to adapt in an open ended manner.
Geo was an free software game I wrote around 10 years ago which I’ve recently rediscovered. I made it a couple of years after I was working for William Latham’s Computer Artworks – and was obviously influenced by that experience. At the time it was a little demanding for graphics hardware, but it turns out the intervening years processing power has caught up with it.
This is a game set in a large section of space inhabited by lifeforms comprised of triangles, squares and pentagons. Each lifeform exerts a gravitational pull and has the ability to reproduce. It’s structure is defined by a simple genetic code which is copied to it’s descendants with small errors, giving rise to evolution. Your role is to collect keys which orbit around gravitational wells in order to progress to the next level, which is repopulated by copies of the most successful individuals from the previous level.
Each game starts with a random population, so the first couple of levels are generally quite simple, mostly populated by dormant or self destructive species – but after 4 or 5 generations the lifeforms start to reproduce, and by level 10 a phenotype (or species) will generally have emerged to become an highly invasive conqueror of space. It becomes an against the clock matter to find all the keys before the gravitational effects are too much for your ship’s engines to escape, or your weapons to ‘prune’ the structure’s growth.
I’ve used similar evolutionary strategies in much more recent games, but they’ve required much more effort to get the evolution working (49,000 players have now contributed to egglab’s camouflage evolution for example).
What I like about this form of more humble AI (or artificial life) is that instead of a program trying to imitate another lifeform, it really just represents itself – challenging you to exist in it’s consistent but utterly alien world. I’ve always wondered why the dominant post-human theme of sentient AI was a supercomputer deliberately designed usually by a millionaire or huge company. It seems to me far more likely that some form of life will arise – perhaps even already exists – in the wild variety of online spambots and malware mainly talking to themselves, and will be unnoticed – probably forever, by us. We had a little indication of this when the facebook bots in the naked on pluto game started having autonomous conversations with other online spambots on their blog.
Less speculatively, what I’ve enjoyed most about playing this simple game is exploring and attempting to shape the possibilities of the artificial life while observing and categorising the common solutions that emerge during separate games – cases of parallel evolution. I’ve tweaked the between-levels fitness function a little, but most of the evolution tends to occur ‘darwinistically’ while you are playing, simply the lifeforms that reproduce most effectively survive.
You can get the updated source here, it only requires GLUT and ALUT (a cross platform audio API). At one time it compiled on windows, and should build on OSX quite easily – I may distribute binaries at some point if I get time.
As part of this we are expanding the patterns possible with the HTML5 canvas based pattern synthesiser to include geometric designs. Anna and Laura are interested in how camouflage has evolved to disrupt perception of movement so we need a similar citizen science game system as the eggs, but with different shapes that move at different speeds.
Here are some test mutations of un-evolved random starting genomes:
This is an example pattern program:
This was a chance to do some detective work on the massive amount of genetic programming data we’ve amassed over the last few months, figure out ways to visualise it and create large prints of the egg pattern generation process. I selected family trees of eggs where mutations caused new features that made them difficult for people to spot, and thus resulted in large numbers of descendants. Then I printed examples of the eggs at different stages to see how they progressed through the generations.
We also ran the egglab game in the gallery on a touch screen which accidentally coincided with some great coverage in the Guardian and Popular Science, but the game kept running (most of the time) despite this.
The Poly (or Royal Cornwall Polytechnic Society) was really the perfect place for this exhibition, with its 175 year history of promoting scientists, engineers and artists and encouraging innovation by getting them together in different ways. Today this seems very modern (and would be given one of our grand titles like ‘cross-displinary’) but it’s quite something to see that in a lot of ways the separation between these areas is currently bigger than it ever has been, and all the more urgent because of this. The Poly has some good claims to fame, being the first place Alfred Nobel demonstrated nitroâ€glycerine in 1865! Here are some pages from the 1914 report, a feel for what was going on a century ago amongst other radical world changes:
9,000 players, 20,000 games played and 400,000 tested egg patterns later we have over 30 generations complete on most of our artificial egg populations. The overall average egg difficulty has risen from about 0.4 seconds at the start to 2.5 seconds.
Thank you to everyone who contributed their time to playing the game! We spawned 4 brand new populations last week, and we’ll continue running the game for a while yet.
In the meantime, I’ve started working on ways to visualise the 500Mb of pattern generating code that we’ve evolved so far – here are all the eggs for one of the 20 populations, each row is a generation of 127 eggs starting at the top and ordered in fitness score from left to right:
This tree is perhaps more useful. The ancestor egg at the top is the first generation and you can see how mutations happen and successful variants get selected.
We’ve released our latest citizen science camouflage game Egglab! I’ve been reporting on this for a while here so it’s great to have it released in time for Easter – we’ve had coverage in the Economist, which is helping us recruit egg hunters and 165,000 eggs have been tested so far over the last 3 days. At time of writing we’ve turned over 13 generations starting with random pattern programs and evolving them with small mutations, testing them 5 times with different players and picking the best 50% each time.
Here is an image of some of the first generation of eggs:
And this shows how they’ve developed 13 generations later with the help of many thousands of players:
We can also click on an individual egg and see how it’s evolved over time:
And we see how on average the time taken to find eggs is changing:
Technically this project involves distributed pattern generation on people’s browsers using HTML5 Canvas, making it scalable. Load balancing what is done on the server over three machines and a Facebook enabled subgame – which I’ll use another blog post to explain.
I’m putting the final pieces together for the release of the all new Project Nightjar game (due in the run up to Easter, of course!) and the automatic pattern generation has been a focus right up to this stage. The challenge I like most about citizen science is that along with all the ‘normal’ game design creative restrictions (is it fun? will it work on the browser?) you also have to satisfy the fairly whopping constraints of the science itself, determining which decisions impact on the observations you are making – and being sure that they will be robust to peer review in the context of publication – I never had to worry about that with PlayStation games 🙂
With this game, similar to the last two, we want to analyse people’s ability to recognise types of pattern in a background image. Crucially, this is a completely different perception process from recognition of a learned pattern (a ‘search image’), so we don’t want to be generating the same exact egg each time from the same description – we don’t want people to ‘learn’ them. This also makes sense in the natural context of course, in that an individual bird’s eggs will not be identical, due to there being many many additional non-deterministic processes happening that create the pattern.