How to design a tangible programming language – Pattern Matrix at Algomech (part 2)

Once we acknowledge that weaving and programming are part of the same technological timeline, we can begin to look at the history of weaving as a eight thousand year long tale of human relationship with digital technologies – and use this long view to research new approaches to software engineering, a field with a much less developed history and many interesting problems to solve.

Using augmented reality to display dynamic information on a tangible programming language.
Using augmented reality to display dynamic information on a tangible programming language.

(Follows Part 1 here.)

One of our threads of investigation is using the pattern matrix as a general purpose tangible programming system – one that we can use for controlling swarms of robots, programming different types of weaving systems and describing complex processes, such as live musical systems.

The magnetic system on the new pattern matrix consists of four hall effect sensors on every location you can place a block. There are four unique ways you can arrange the magnets – which means four types of programming block are possible. As we want to reuse these physical blocks for various uses and programming languages, we decided to use abstract shapes to denote the block types to begin with. Each of the four blocks can be rotated and flipped to give 32 total possible orientations, or symbol ‘tokens':

All combinations of token orientation with four magnets
All combinations of token orientation with four magnets

However, only 16 of these orientations are actually unique. We can only determine flip orientation on the circular block, and only rotation on the rectangular and triangular ones – where flipping them makes no difference to the magnets. The square block is a kind of special one, as we can tell both rotation and flip orientation, so it can represent eight tokens in total all by itself:

All unique tokens and orientations possible. With mid-grey shapes the flip (which side is up) is irrelevant.
All unique tokens and orientations possible. With mid-grey shapes the flip (which side is up) is irrelevant.

It’s important to note at this point that the parallels with tablet weaving are no coincidence: rotating and flipping arrangements of four binary elements for this magnetic system are the same actions as those performed when weaving using tablets. Weaving in the pattern matrix is more than a subject, it’s built directly into it’s mode of operation.

Next we need to test the applicability of this tangible programming system for wider uses. The other cultural phenomena the Penelope project is involved in is livecoding – so is it possible to use the pattern matrix to introduce a weaving centred programming technology in a very different context, not to describe weaving but to generate music in a performance such as an Algorave? This is something that Ellen first pioneered at our weavecoding performance at The Museum of casts of classical sculptures in Munich, but the new pattern matrix has better capabilities for a general purpose programming language.

Having 16 states of four blocks is indeed limiting for a language, but not too limiting. Some types of programming language, such as a string rewriting system like a Lindenmayer system are particularly well suited to this. They are also surprisingly Turing complete languages, able to represent any other programming language in existence, given enough space and time.

Here is a quick example of how this works in text form – a string rewriting system is simply a list of search-replace actions that are carried out in a consistent order. The original example, used to model the growth pattern of algae – consists of a starting string: “A” and two replacement rules, replace “A” with “AB” and replace “B” with “A”. If we run these two rules over and over on the same bit of text we gradually ‘grow’ a pattern like this:


n = 0 : A
n = 1 : AB
n = 2 : ABA
n = 3 : ABAAB
n = 4 : ABAABABA
n = 5 : ABAABABAABAAB
n = 6 : ABAABABAABAABABAABABA
n = 7 : ABAABABAABAABABAABABAABAABABAABAAB

On the pattern matrix we use four of the rows to represent four different rules that replicate in this manner (each made of 5 possible symbols, as it’s a 5×5 grid), which we run 8 times on the starting string (A) to create a musical sequence. Four of the tokens represent these rules (A,B,C and D), the remaining tokens represent musical actions – changes in pitch, rests and sound triggers. There is massive variety of potential patterns, you can control the amount of recursion by the number of rule reference tokens you use – to control the resulting length of the sequences, and thus the complexity of the music. Interestingly we also need a ‘no operation’ (NOP) instruction that does nothing – as in low level assembler languages. We need this as a way to be able to shift timing in the musical sequence by one instruction.

A musical language in 16 instructions.
A musical language in 16 instructions.

With a tangible programming language like this it’s also very important to consider how you categorise instructions by shape – as you can quickly switch between similar operations by simply rotating or flipping tokens, while switching between different shapes takes longer (as you need to pick up a new block) so should represent bigger changes if possible.

Four rules are plenty for generating hugely complex sequences, so we can use the fifth bottom row to control global parameters like scale, synchronisation options (for our slub collaborative sync protocol) or switch between more banks of sounds for greater variety.

Slub performance including the pattern matrix at the Brighton British Science Festival Algorave
Slub performance including the pattern matrix at the Brighton British Science Festival Algorave

The first time we tried this out was at the British Science Festival Algorave in Brighton. A projection was set up with a camera to show the pattern matrix being used, and while technically everything went fine (other than some syncing difficulties), it highlighted a key problem with tangible programming languages. With no dynamic feedback other than the state of the blocks on the pattern matrix, it’s very difficult to tell what is happening during a performance, it’s hard to understand what musically is resulting from the changes you are making.

In order to find a way around this we designed an augmented reality ‘layer’ to place over the pattern matrix, which gives feedback on the currently triggered notes and the paths between the recursive string production rules. We use fluxus and it’s AR feature, which was written by Gabor Papp – which is based on the ARToolkit library. We use a printed out marker to find the plane and camera scale of the centre of the pattern matrix in the image from a USB camera. Once this is done the marker can be removed (as neither the camera or pattern matrix moves) and we can use millimetres as units and place objects over the block locations in 3D space. When the sensors detect a change we can display this new information, also updating the current position in the sequence playback to give feedback on the current sound playing.

Pattern matrix livecoding as part of slub performance at the Algomech Algorave
Pattern matrix livecoding as part of slub performance at the Algomech Algorave. Pix thanks to Dan Hett

As an initial trial the AR improved things when first tried out at the Algomech Algorave in Sheffield, it makes the pattern matrix easier to understand and perform with – and also provides some feedback for the audience in a projection. In a last minute change we switched from Latin characters to Linear A, an undeciphered ancient Greek script – a reference to Flavia’s work on the Penelope project. This is actually preferable to Latin characters as the musical language represents meaning in a way that that actual glyph used is irrelevant – it’s better if it can’t be ‘read’ or confused with another meaning by anyone (still alive).

So it seems that AR could be one of the items in our toolbox for further tangible programming experiments. Perhaps we can better explain the structural changes caused by livecoding the weaving notation for the warp weighted loom by having a dynamic weave structure ‘floating’ on top of the tokens, alongside the loom simulation. This could also be of use for describing tablet weaving actions with these blocks, which would need to be more abstract than the binary weaving notation.

Another area to explore is the design of the blocks themselves, moving away from the abstract shapes, we can design them for specific purposes. Similar to our work on viruscraft, where we have more closely explored the correspondence between physical form (receptors and structural protein arrangements) and tangible interfaces, it seems that these shapes may be worth considering more closely now the sensor matrix is working well.

Pattern Matrix at Algomech (part 1)

I’m writing this on the train with a slightly sleep deprived brain fizzing and popping from thoughts, ideas and conversations from this year’s Algomech festival in Sheffield. The Penelope project took a significant role in the festival, with the group’s participation in the Unmaking Symposium, the exhibition and also testing our latest weavecoding technology at the Algorave. I’ll be writing more on the algorave in a subsequent post.

During the symposium we discussed the critical, liberating and potentially dangerous aspects of Unmaking in a wide variety of contexts – from reverse engineering knitwear and classical Greek dance to discovering the untapped abilities of classical musical instruments when human limitations become a secondary consideration. The symposium also provided us with a good opportunity to take stock of our own group’s current directions and thinking in regard to the Penelope project.

Pix thanks to
Pix thanks to James Vanderhoven

comp

We also had our own corner of the Algomech exhibition, which included Jacquard woven experiments, the Quipu sonification and visualisation and the first public trials of the new pattern matrix V2.

exhib

As is our usual practice, we used this exhibition to get essential feedback on our new design for the pattern matrix, as well as the interpretation of what we are doing – being there in person talking to people allows you to very quickly determine what works, and adapt the focus based on the responses you get.

We discussed the long view on digital technology, the role of weavers in foundations of western mathematics and simply the practicalities of the technology we are developing – the constant stream of visitors represented a wide range of different ages and backgrounds. These aspects of the pattern matrix seemed significant (in no particular order):

  • The construction technique was immediately interesting to most people, specifically the material, beech wood spalting and open frame construction.
  • Related to this, there seems little association with the pattern matrix as a ‘device’ in the sense of a ‘gizmo’ – which is interesting as the Raspberry Pi and other PCBs are clearly visible. We’ve noticed this at some level with the previous version but with the wood construction this effect is much more pronounced.
  • It is seen as being game like, e.g. a “70’s educational toy”. There is an expectation that it is something to be ‘played’, and similarly its potential as a musical sequencer is a common observation.
  • The understandability of the magnet sensing seems a key ingredient. There is no other particularly hidden magic like computer vision or RFID involved, and polarity and digital arrangements of magnets are easily explained and experienced by holding the tokens together.
  • Having some extra circuit boards and wood cut parts to hand, originally intended as backups – were great for people who wanted to know more about what was going on. In future we should also have the token block parts to show as well.
  • The different shaped blocks were immediately appealing – they seemed to invite experimentation more than alternating the binary tokens by flipping them. To follow this up we need to investigate ways to use different shapes to configure thread colour at the same time as structure in a better way than we are now. The black/white sides could define structure while the shape could correspond to the colour for the specific thread. It also indicates that using shaped tokens as instructions for tablet weaving is worth experimenting with quite soon.
  • Younger children focused on the blocks alone (and of course tried all sorts of things no one else did, like stacking them) but slightly older children worked out they were having an effect on the weaving process and generally could patiently work themselves it out without any explanation required.
  • Having Anni Albers’ ‘On Weaving’ book next to pattern matrix helped with older visitors, perhaps representing a more conventional and authoritative source of information to introduce the concepts of notation, structure and pattern in weaving.
Pix thanks to James Vanderhoven
Pix thanks to James Vanderhoven
The 8-way tangible colour switching instruction
An 8-way tangible colour switching instruction

Physical vs digital – a false dichotomy

More general concepts that came up in conversation included a common theme during Algomech, exploring the inescapably fuzzy boundaries of concepts such as digital, physical and analogue. The myth of the “real world” being analogue and the “virtual world” being digital is a troublesome one to a weaver.

The ‘anti-device’ effect of the pattern matrix has the potential to explore this conundrum, as it represents a seemingly acceptable demonstration of the physical nature of the digital, and that forms of digital technology have inhabited the world of the reassuringly physical for many millennia of human invention.

Check your supply chains

One aspect of the pattern matrix I picked on for the symposium which came up in the exhibition as well was the fact that the beech wood came from a single tree in Cornwall courtesy of Aaron Moore – and used this as an example of our design philosophy of taking on the myth of collapse rather than the myth of infinite abundance.

img_20171107_145340

Feedback from Penelope team members

Having a 5X5 grid initially was thought to be excessive, as most ancient weaves can be expressed with a 4X4 matrix. This larger capability turns out to be more important than we first thought, as it means you can demonstrate the importance of odd and even numbers in the mathematics of weaving.

There is a problem with the single colour change block due to a faulty use of the code from the old version. This has always been a somewhat temporary feature so we should sort this out properly (e.g. using other shapes for colour across the matrix) before we use it next.

We can also try using an augmented reality approach to show the weave structure directly over the grid, so it’s easier to see how the token block changes relate to specific crossings. This could be displayed alongside the current warp weighted loom rather than as a replacement for it.

Adventures with FM sound and the YM2612

So I’ve been getting a into FM synthesis lately, and after giving the TX7 a spin I’ve been really wanting to try evolving sounds for the later synth chips Yamaha made for games consoles and soundcards. At that point in history FM was one of the only realistic ways you could synthesise complex sound as there wasn’t enough time (processing) for general purpose DSP, and there wasn’t enough space (memory) for sample playback to be an option.

img_3528

One of the most famous of these range of synth chips was the YM2612 which was used in the Sega Megadrive/Genesis consoles. I got hold of a few to try, but it’s been quite a challenge to get them working so I thought probably worth documenting the process. The YM2612 is programmed using 5 control pins and 8 data pins. You send one byte to the chip at a time – first an address to write to, then the data to write, which allows you to set a couple of hundred parameters that define 6 separate sounds. The memory on the chip is organised into a map of the global settings (top left) and the voice settings in two banks:

registers

There isn’t much documentation for these things, the main reference is for a similar chip which is written in Japanese. More usefully there are software versions in emulators, and quite a few bits and pieces online from other people’s projects and some translations of the docs. Usefully the Sega documentation included a piano test sound which is great as you have something that *should* actually make a noise as long as everything else is working.

The first problem was figuring out the timing – despite having some examples it took me a while to realise that there is a specific order that you have to set the status pins, and that there is one change at one time that actually triggers (latches) the data transfer to the synth:

timing

This is a timing diagram, I’m used to seeing them in datasheets, but it’s the first time I’ve needed to understand one properly (with a bit of help from David Viens). It tells us that for writing, the data (D0-D7 pins) are actually read shortly after the WR pin is set low – to add to the fun, some of the pins (denoted by the line over them) are 0v for on, 5v for off. You can also read data from the synth, but whatever address you supply you only get a single status byte that tells you if it’s currently busy, and if either of it’s two internal clocks have overflowed. The idea I assume is that you would use them for timing music accurately without the need to do it in your main CPU.

The audio coming out needs to be amplified too – if you connect it directly to a speaker there is a danger of overloading the DAC and causing it to burn out, so you need a bit of analogue circuitry to do this. Here is an example of triggering and glitching (by writing to random addresses) the example piano sound:

I started off using an Arduino Nano, but didn’t have much success so switched to atmega328 which was a bit easier to set up. One of the problems is that ideally you need to be able to control the 8 data GPIO pins in one go (to set the 8bit value), which isn’t possible from the nano as none of the ports have 8 bits. You also have to supply the YM2612 with a clock pulse at around 8Mhz – which took me a while to get right, I tried both dividing the atmega’s 16Mhz clock by 2 and outputting it on a pin as well as a separate external oscillator and both eventually worked – except the oscillator leaked quite badly into the audio output (there is probably a fairly simple fix to isolate this).

One potential problem is dodgy vintage chips, I tried two suppliers on eBay, one Chinese shop and another in the UK – all the ones I’ve tested so far worked fine, I’m happy to say. The best way of testing them is to use a second hand Sega Megadrive console and replacing the YM2612 it has with a socket you can easily plug in candidates for testing – I managed without but it would have come in handy sanity wise to know for sure that the chips worked.

yamaha_ym2612_top_metal_die_shot
(image source)

One thing that I wasted a lot of time doing was ignoring the output entirely and just trying to get the status bit to change by setting the internal timer – the idea was to make sure the chip logic was working before worrying about the audio amplifier circuitry. However, this requires the read sequence timing to work as well as the write timing – and as soon as you can listen to the output you get small clues from the DAC noise, which actually indicates the chip is running or not even if there is no sound triggered.

ops

Here are some links I found helpful: the wikipedia entry for the pinout, full software documentation (for the registers) and a great blog post about putting it all together, with schematics – in French. Here is a hardware synth module with similar YMF262 chips.

My source (and soon more stuff) is here.

Creative Design Informatics for Horticultural Awareness at the End of the World Garden

Thanks to Paul Chaney who runs The End of the World Garden, we had an opportunity to trial a short workshop based on our Farm Crap App and prototype Allotment Lab on his two-acre forest garden site in Cornwall. This was our contribution to the Bank Holiday Weekend Haymaking Extravaganza (along with a bit of hay-making too).

We started by looking in detail at the Farm Crap App, how it arose from an intersection of governmental policy and the needs of farmers, providing a way to quantify the nutrients present in natural manures that are added to the land. We were able to map a section of the End of the World Garden intended for future agricultural experimentation, and based on its crop history and soil type calculate the additional nutrients required for growing different crops.

img_20170826_160520

One of our discoveries were the limitations of the underlying DEFRA data, being based on a particular approach to farming it uses a pretty broad brush – the necessary simplicity of the data in some areas (for example in the range of crops present) is not so well suited to smallholders. For example high yield crops that maintain profitability are less important to smallholders who are able to use slower growing strains. It was also interesting to work with people who had less experience with farming (me included) and trying to work out the difference between ‘dribble bars’ and ‘splash plates’ and other arcane muck spreading technology.

We also tested our prototype Allotment Lab. This has been designed to provide a couple of trial walk through experiments based on compost quality and soil type estimation for a more general audience – allotment owners, gardeners and smallholder farmers included.

img_20170826_175749

We tested the soil in a small area of field, attempting to form different shapes with our hands and a little water, in order to estimate its specific type and consistency. The soil in this part of Cornwall consists of a top layer of ‘sandy clay’ (actually containing larger granite particles, rather than sand) and a lower layer of ‘silty clay’ laid down from melt water during the last ice age. Paul dug down to find the lower layer so we could check the difference between them using the test.

tex

img_20170826_175132

This workshop was useful in that it highlighted the limitations of top-down governmental data and the potential for citizen science to allow people to gather their own knowledge based on their specific circumstances. One of our challenges is coming up with a range of experiments that can augment official data and allow farmers, allotment owners and others to ask questions, collect data and make decisions that will help them in increasingly difficult social, climatic and economic circumstances.

Viruscraft: Genetic model connected to a tree visualisation

The genetic model we were working on previously has now been ported into a browser compatible form and connected to a new tree visualisation that displays the species that emerge as the host population adapts to a virus infection. It’s still a prototype with rough edges, but have a play with it here, some example pics:

tree7

This is one of the earlier attempts, which I like the look of, but later cleaned up versions are a bit clearer to read what is going on.

tree17

tree18

Firstly the genetic part is working in that the population evolves to get gradually better at coping with the virus infection (the fitness score increases) – it takes a bit too long at the moment, but it’s great to be able to see this working in realtime as it happens with new species branching off older ones.

One early observation is that this has the potential to show why diversity is beneficial. If you modify the virus (fitness function) at a point when there are lots of different species present, the chances are that a few of them will be resilient enough to the infection to expand into the new environmental niche, and things eventually continue as before. If you alter the virus when there is only a single really well evolved species that is a bit too good at coping with the existing virus – the chances are that you will cause the population to go extinct as it won’t be able to adapt. This is analogous to the situation with bananas: “To carry on growing the same genetic banana is stupid”.

A big chunk of the work here was actually spent optimising the code. It’s pretty amazing how developed browsers are for development – I’ve been using the profiler in the chromium browser to locate slowness and keep the frame rate as near 60 frames per second as possible. I just noticed taking the screenshot for this post the slowest single part seems to be the debug text rendering, strangely enough.

prof

Technology after Collapse

The philosophy underlying contemporary ‘seamless’ technology production seems to be one of endless energy, bountiful resources and waste being someone else’s problem. Naive working assumptions of some form are a requirement when designing for the future, but do we believe in these enough now to make them useful? Flashy ‘aspirational’ tech videos of ever thinning devices disappearing into the ‘cloud’ seem to be less common than they once were, so perhaps not.

Whenever a philosophy starts to look shaky, there are huge opportunities to try different ideas. For example, what happens if we instead use collapse as a working assumption for design? A sudden global societal collapse may be as unlikely as fusion power coming along to ‘solve’ climate change – but as a working assumption it shines a different light on usability – for a future that will mostly likely be somewhere between these two extremes. It’s also a view that is shared by some areas of research (particularly military) and I suspect by quite a few internal corporate future planning departments.

This research direction was triggered by Amber who found a paper called “Unplanned Obsolescence: Hardware and Software After Collapse”, this quote is from one of the papers it cites:

“In our present world of virtually unlimited resources, at least from the consumer’s perspective, acquiring the newest piece of technology is often considered a desirable lifestyle choice (e.g., for early adopters). A future of collapse might see a different picture…”

Sustainable Interaction Design: Obsolescence in a Future of Collapse and Resource Scarcity, Christian Remy, Elaine M. Huang

For this weeks seminar at the Institute for Music and Media in Düsseldorf, Julian Rohrhuber and I wanted to explore this area a little with the students. This is something that has been present in FoAM Kernow’s work for a long time – partly as we work closely with environmental researchers so it’s hard to avoid, but it’s also inherited from the history of the FoAM network.

excavate
We started by discussing the great Atari video game burial of 1983 and the comparative durability of the technology used at this time in history compared to that of today. This was an important aspect of the What Remains, so we looked at how we recycled old cartridges to store new software on for that project.

The specific scenario of collapse used in this type of research is one where all electronics production has ceased, all logistic and communication channels are destroyed, damaged or restricted. This could be due to a global crash, or an area becoming cut off from the rest of the world. Each component in our devices has a lifetime, most of them less than 10 years. While its possible that we could salvage components and repair our devices, there is also the issue of the knowledge required – much of it currently restricted to specialist silos of expertise.

“Social networks or institutions of people interested in computer repair could be invaluable for sourcing parts and maintaining skills needed to keep computing alive until devices and power are no longer scarce.”

Unplanned Obsolescence: Hardware and Software After Collapse, Esther Jang, Matthew Johnson, Edward Burnell, Kurtis Heimerl

tangle
Quote from Radix’s Adventures in Tech #3: Tanglebots, assemble! by Fiona Campbell-Howes

The starting point for our Tanglebots workshop for example, is an e-waste scrapheap we search through for things to take apart and reuse. Sustainability issues like this are often better presented implicitly, to set the scene – which then leads to a more interesting situation than starting with identical ‘kits’. You get to explore internal parts of cheap toys like dissecting natural organisms, and this process fits well with the haphazard reality of building technology, and working with family groups leads to knowledge sharing of the kind that would be needed to keep skills alive after collapse.

For the practical part of our seminar we wanted to focus on internet infrastructure, although originally developed to be resilient to war situations – the current scale involved, with undersea cables and centralised servers makes it one of the weakest aspects of our societies infrastructure to collapse, along with mobile technology:

“Long distance networking and information sharing becomes difficult with the decay of Internet infrastructure. Therefore, it may be crucial to establish communication channels, file sharing practices, and communities before the breakdown of mobile computing.”

Unplanned Obsolescence: Hardware and Software After Collapse

Some of the problems involved include software updates – as these become more challenging users become more prone to malware and viruses as they can’t be protected as effectively by the software authors. How would we manage software distribution post-collapse?

morse-dat
Unknown signal, a composite morse/data packet combination.

Global data transmissions are happening all the time on short wave radio. Certain frequencies bounce from the ionosphere or micro-meteorite trails to cover the globe, and data ‘modes’ are used rather than voice as they can travel further at lower power. There is even a slow-scan TV data mode which is used by the International Space Station to transmit images to amateur radio operators around the world – and regular contacts are made with people on the ground using shortwave frequencies.

iss

We spent the afternoon scanning the airwaves using the websdr.org network (no radio hardware required). You can choose your global location and frequency band (and by extension the time of day too – as transmissions travel further in the night time). The web interface allows you to record sound as wav files, which we analysed using Audacity.

In order to figure out what a particular transmission is, there are some clues you can use. The first is the frequency used, which you can look up via this chart as there are rules for what frequencies can be used for. We also found these example data recordings which we used to figure out the type of transmission data based on listening.

Here are some example recordings we made:

The first one was a bit of an enigma for a while – it sounds like really slow badly tuned music. We figured out it was probably an RTTY text messaging protocol called JT65 – for low power, long range short text communication:

jt65

The second one is still unknown, but there are a few clues. It’s a periodic burst of high speed data transmitted on a frequency band (6765 to 7000 kHz) that includes ‘FAX’ modes. It’s likely that this is some kind of weather map – but it doesn’t sound quite like any of the examples I’ve heard.

The third recording is a collection of simultaneous morse transmissions on similar frequencies. They all seem to be machine generated, a lot of repetition – probably some kind of beacons. I’ve tried to decipher them below but no hidden messages unfortunately!

decoded

This seminar was a good crash course for us all, and a way to get a handle on the sheer amount of activity on these not too well known global communication channels. A lot of these concepts are making their way into technology we are building, for example the penelope pattern matrix where we are using wood construction and modular circuits that we can adapt for different uses.

Future radio directions could be to set up a packet radio server to host some websites, even on a very tiny range (perhaps simply transmitted/received over audio) so we wouldn’t need a licence.

Additional reading material:

Pattern Matrix PCBs arrived & first tests

After triple checking the schematics and design files and ordering 80 PCBs (50 sensors and 30 i2c boards) there was an anxious wait for them to arrive and do some initial tests to find out if there were any mistakes. We now have enough boards to make two new pattern matrix devices, one 4X4 and one 5X5 – the plan is to evaluate the design and refine it for future builds.

img_20170628_105103

img_20170628_105226

img_20170628_110233

The picture below shows the first test boards populated and plugged it into the Pi – it’s much neater than the lego and breadboard prototype! The good news is that it seems to work so far, the only problem I’ve had is with the hall effect sensors, the pads are a tiny bit too close together for my skills. After a couple of tricky situations fixed with a de-soldering pump, I think I’ve come up with a strategy that works. I can bend the outer pins away from each other and solder the central pin first – then bend them back to finish the outer ones and being very careful not to bridge the pads.

img_20170628_173331

The blue jumpers on the square i2c boards allow you to program the device channel that the two expanders use – these could alternatively be hard soldered, but it’s good to have the option to reuse the parts or reconfigure a pattern matrix so we can add different sensors etc.

img_20170628_185854

For reference, the KiCad 3D viewer models look pretty close.

i2c-render

sensor-render

Viruscraft: building a ‘reasonably accurate’ genetic game world simulation

The concept for the viruscraft game is to have a realtime genetic model or simulation of the host evolution which is adapting to the properties of a virus you are building (either on screen or via a tangible interface as part of an exhibit). This model needs to be realistic, but only up to a point – it can be more of a caricature of biology than a research model would need to be, as our intentions are educational rather than biological research.

Using our previous species prototype as a starting point, we have a network of connected locations that can be inhabited by organisms. These organisms can jump to neighbouring locations and be infected by others in the same place at the same time. Now we need to figure out how different species of these organisms could emerge over time that evolve immunity to a virus – so we can build up a family tree (phylogeny) similar to the ones we created for the egglab game but that is responding to the viruses that you create in realtime as you play. The evolution itself also has to happen fast enough that you can see effects of your actions ‘quickly enough’, but we’ll worry more about that later.

For a job like this we need to move back from fancy visualisations and graphics and try to get some fundamental aspects working, using standard tools like graphviz to understand what is going on to save time.

The first thing to do is to add a fixed length genetic string to each individual organism, this is currently 40 elements long and is made from biologically based A,T,C and G nucleotide symbols. We chose these so we can use biological analysis tools to test the system as we go along just like any other genetic process (more on that below). The organisms can also reproduce by spawning copies of themselves. When they do this they introduce random errors in the genetic code of their offspring which represents mutation.

Previously we were using a ‘SIRS’ model for virus infection (susceptible -> infected -> recovered -> susceptible), based on 4 global parameters that determined the probability of jumping from one state to the next. Using the genetics, the probability of infection is now different for every individual based on:

1. Is a virus infected individual in the current location?

2. If so, use our genetic code to determine the probability of catching it. Currently we use the ratio of A’s to T’s in the genetic string as a totally arbitrary place-holder ‘fitness function’, the lower the number the better. AAAAAAT is bad (fitness: 6) while TTTTTTA is good (fitness: 0.1666) – so we would expect the A’s to disappear over time and the T’s increase in the genetic strings. This number also determines the probability of dying from the disease and (inversely) the probability of gaining immunity to it.

3. A very small ‘background infection’ probability which overrides this, so the virus is always present at a low level and can’t die out.

The next thing we need is a life cycle for the organism – this needs to include the possibility of death and the disease model is now a ‘SIR’ one, as once recovered, individuals cannot go back to being susceptible again.

state

All the other non virus related probabilities in the simulation (spawning offspring, moving location, natural death) are currently globally set – to make sure we are seeing evolution based only on disease related behaviour for now.

This model as it is could form the foundation of a world level visualisation – seeing organisms running around from place to place catching and spreading your virus and evolving resistance to it. However this is only half the story we want to tell in the game, as it doesn’t include our time based ‘phylogenetic’ family tree view. For this, we still need to figure out how to group individuals into species so we can fully visualise the effects of your virus on the evolution of all the populations as a whole.

First we need to decide exactly what a species is – which turns out to be quite an arbitrary concept. The rather course approach that seems to work here is to say that two organisms represent two distinct species if more than a quarter of their genes are different between them.

We can now check each organism as it’s born – and compare its genome against a ‘blueprint’ one that represents the species that it’s parent belongs to. If it’s similar enough we add it to its parents species, if it’s too different we create a new species for it. This new species will have a copy of its genome as the ‘blueprint’ to compare all its descendants with. This should mean we can build up a set of related species over time.

If we run the simulation for 5000 time steps we can generate a phylogenetic family tree at the end, using the branch points between species to connect them. We are hiding species with only 1 member to make it simpler, and the population is started off with 12 unique individuals. Only one of which (species 10) is successful – all the later species are descendants of that one:

test

The numbers here are the ID, fitness and size of population for each species. The colours are an indication of population size. The fitness seems to increase towards the right (as the number drops) – which is what we’d expect if new species are emerging that cope better with the virus. You can imagine changing the virus will cause all this to shift dramatically. The “game mechanic” for viruscraft will all be about tinkering with the virus in different ways that changes the underlying fitness function of the host, and thus the evolution of the populations.

As we used standard biological symbols for our genetic code, we can also convert each species into an entry in a FASTA format text file. These are used by researchers to determine population structure from limited information contained in genetic samples:


> 1 0.75 6
TGCTCTTGCGTACTAGACTGTTGACATCTCCACCGGATAA
> 3 0.46153846153846156 5
TGGTTTTCTGCTGTGGGGATAACCTGCCACTCAGTGGTGA
> 5 0.6153846153846154 171
CACTATCGCTCATTGCACTGTCGTGGTTTTAGTAACGAGC
...

In the FASTA file in the example above, the numbers after the ‘>’ are just used as identifiers and are the same as the tree above. The second line is the blueprint genome for the species (its first individual). We can now visualise these with one of many online tools for biological analysis:

phylo_tree

This analysis is attempting to rebuild the first tree in a way, but it doesn’t have as much information to go on as it’s only looking at similarity of the genetic code. Also 40 bases is not really enough to do this accurately with such a high mutation rate – but I think it’s a good practice to keep information in such a way that it can be analysed like this.

PCB design for pattern matrix 2

This is the pattern matrix 2 tangible sensor schematic, which is fairly simple – just 4 hall effect sensors and a capacitor to smooth out any noise on the power supply.

sensor-schematic

We need to make hundreds of these for the Penelope Project, and we can save some costs by using the built in pull up resistors in the MCP32017 to get a decent signal from the sensors. The difficulty with this PCB is arranging the sensors so they align with the magnets in the tangible programming block in the optimum manner. From tests with the prototype Lego rig, this isn’t actually too critical – but it’s set up so the lead length can be tweaked a bit at soldering time.

sensor-render

This took me about 20 variations to finally get right, but the circuit is just about simple enough that it can be made single sided – this is good because the top side will be partly exposed, while the lower side with all the copper traces can be protected. It’s good practice to have large areas of copper left connected to ground, partly as it’s a common connection needed all over the board, partly for stability but also it reduces the amount of chemicals required to etch the circuit – as only the parts around the traces need to be removed.

sensor-render2

The i2c expander board is a little more complicated. The design is made to be modular so we can stack up any number of these connected to the Raspberry Pi for different arrangements of sensors. Each board can deal with 8 sensor locations (each comprising 4 individual hall effect sensors). Their job is to convert the digital signals from each sensor into serial data (using the i2c protocol) so the Raspberry Pi can read them all just using 2 wires, plus 2 for power.

i2c-schematic

Each board can be configured to a separate i2c device address to tell it apart from the others using jumper connectors. This one had to be 2 sided, but I managed it without any ‘vias’ (holes to pass traces from one side of the board to another). I also added a power indicator LED as a last minute addition.

i2c

I’ve been learning the open source Kicad software to design these, which is now used by CERN for building the LHC, so it’s pretty fully featured! The idea is that you draw the schematic first, link each component with a physical ‘footprint’, then switch to the PCB design stage. Other software I’ve used in the past tries to route everything in one go for you (and can come up with some pretty strange and messy results). Kicad works in a semi-automatic manner – you need to draw each trace by hand, but it routes them around components and other traces, and suggests the shortest path for you. This is quite a lot better than a fully automatic approach as you have more control over the end result, and easily end up with a decent placement of all the parts.

i2c-render

This project is of course open hardware, and can be found on github here.

Pattern Matrix 2 haptic experiments

One of the potential future additions to our tangible programming hardware is haptic feedback, using sound/vibration to provide a extra channel of information through your fingers when programming with tangible blocks. We wanted to test this before designing the PCB hardware in case we could add it to the system simply – this was initiated by watching a reverse engineering video on youtube (I think by bigclivedotcom) where a technique was mentioned for sneakily reusing input lines (of which we have hundreds) as outputs when they are not being read, by using diodes to separate the signals.

img_20170602_164024

Piezo transducers are often used as cheap speakers, and they can also provide some touch feedback by using lower frequencies. We tested one to see if it interfered with the magnetic fields, and the diaphragm doesn’t seem to be ferrous metal (I’m not actually sure what it is made from) so it can be placed right over the sensors with no effect. A bit more Lego to attach it to the prototype sensor unit, and to see we can feel it through the tangible block when it’s touching the sensor.

The first attempt consisted of simply plugging a speaker into the i/o line from the MCP23017 port expander, switching between input and output on the Pi and adding a diode to prevent the output voltage getting to the sensor.

circuit1

The problem is all the circuits we’re using run on 3.3V, but the piezo speaker I’m using is rated up to 20V – so it’s just not strong enough to feel it. The Pi also provides 5V pins, so a second attempt to use a simple driver circuit (single transistor common emitter amplifier) to boost the voltage:

circuit2

You can hear this version in the video above – it’s playing a sound using a frequency crudely determined by the 4 bit value the block orientation represents whenever it changes. This is louder (loud enough at least to make a recording) but still not enough to feel easily, even when you put the piezo between the sensor and the block. So it seems the best way to get this to work properly will be via a separate circuit, not something we can slot into the existing input hardware. Another advantage of doing it separately is that we can treat it more like a multichannel audio setup with a dedicated processor that is not interrupted by the sensor reading. One solution I saw was to use the same H bridges as used to drive motors from microcontrollers as these take much higher voltages – this will be something to try later on.