The last months have been booked solid with production on various projects, so I’m very behind with blogging. This means that there are a few loose threads that I need to look back on and write about, one of which is the Viruscraft world.
This is a screenshot of the current ‘alpha version’ of viruscraft we tested with the custom tangible interface (more on that soon) during the second game testing workshop. You can read Amber’s report on this workshop here.
It took a while to develop this planet, we started with a more conventional system:
This was a planet modelled in blender, the idea being to create islands and regions for different host species to live. We automatically made zones for different species to live (sea and land mainly but also higher terrain is possible) and then populated this with host organisms. There are two problems with this approach, one is that it’s a static model – we could have a set of variations on planets but e can’t change the terrain. One of the initial obvious references we talked about to the Wellcome Trust was Minecraft – which allows players to modify a procedurally generated terrain. Another problem with this is that if we look at our initial mood board for the project, where we collect artistic references:
There are lots of interesting ideas here, including perhaps using geomtric virus shapes in an interesting way. There is a problem I’ve seen on many games projects where if there isn’t a strong artistic style at the start, a more photorealistic logic takes over and pushes things into less and less interesting territory. So we decided to try something a little stranger and more interesting – could we create worlds built from triangular sections of terrain in the same was as viruses are created from triangular proteins?
There is also a distinct advantage to a procedural technique we can do with modular world building because the shape of the land affects how viruses spread – the number and physical location of population clusters has a big affect on how difficult it is, as a virus – to spread and become successful. The more dynamic and controllable we can make the world, the more interesting the game will be.
We chose a shape called a ‘Pentakis icosidodecahedron‘ – basically an 80 sided sphere built from triangles. One mistake I made with this was to not read the mathematical fine print – it took quite a lot of swearing before I figured out that it’s not regular, some of the triangles are equilaterals and others are isosceles. This means we have to squash and stretch the components to fit precisely, otherwise there are gaps between them.
Once we can arrange the triangles in the right places we need to make sure the terrain edges match up. It turns out there are 8 possible arrangements – from all sea, all land to the 6 varieties of crossings that can occur between the edges of the triangles:
We can also control the probability of a triangle of containing land, and use a slider as a sea level setting. Roughly speaking, the more unified the planet – the less, bigger populations possible and the easier it is for your virus to spread. The more fragmented and diverse the populations, the harder it is.
We’ll be adding more controls here in the future, so you’ll be able to customise your planet quite a bit more. We then made some animated low-poly host creatures to populate the planet. These have receptors for the virus that match the different shaped ligands – they breed and evolve to gain resistance to your virus over time.
In order to place them correctly on the right habitat we created some test ‘flat worlds’ to make sure everything was lined up correctly. It works by calculating barycentric coordinates in each triangle based on the particular terrain section, so we can generate a random location and check if its land or sea.
Another possibility is to have different themed planets, an ice world, urban planet, etc – perhaps rather than creating them you could explore a galaxy planet by planet. One important idea that came from the feedback would be to use the ‘difficulty setting’ that we’ve figured out from the terrain composition to provide tutorial levels that you could work through in increasing difficulty.
Like many we have tended to outsource this work to China, where the costs allow us to do short-run prototyping with our tiny budgets. Are these lower costs simply due to scale and the Communist support of the Maker community there? Or more worryingly, is this due to reduced regulations in the areas of employment rights and environmental protection?
Either way, outsourcing this kind of fabrication sometimes doesn’t make sense. For Viruscraft we need a finer understanding of the materials involved to design our tangible virus interface. We need to detect the ‘shapes’ of receptors attached to a physical virus capsid, and the first approach we want to try is using voltage detection of the kind used in things like the Makey Makey. If we give each virus receptor three bits of information we can represent seven different receptor shapes, which is plenty for our needs. We need some kind of connection point that can read this information – a printed circuit board with exposed conductive ‘touch pads’ embedded in the virus capsid seems a good way to proceed.
This is also a chance to experiment with PCB manufacture, which is pretty messy and involves very nasty chemicals. In ‘doing it ourselves’, the theory is we can learn ways to minimise environmental degradation which could impact our designs in future, even if we end up having to outsource their manufacture.
The general concept is that copper clad boards are etched with acid in patterns that control the flow of electricity when components are soldered to them. We will be using some (quite old) sodium persulfate crystals here, added to water to form an acid bath for the board, which will be immersed until the exposed areas of copper are dissolved. We wanted to try two approaches to creating a ‘resist’ that would prevent the etching, one using insulation tape (actually horse leg bandage tape – well, we are based in a rural setting!) – the other simply using permanent marker to draw the circuit.
First we design a outline and print it out at the correct scale, you can see this in the background in the photo below. As we are using triangular plugs/sockets, we need to be able to ‘read’ the receptor in any orientation, so 18 pads are needed in total.
Then we completely cover a copper clad board in insulation tape. One of the big lessons from this process is that etching large areas uses up more acid and takes longer – so you only want to expose the absolute minimum.
We stuck the template to the board and used it to guide cuts that go all the way through the tape to expose the copper.
We are trying two different methods and designs here – one using just the tape cut-out where we create 18 separated islands which we can drill though and solder to create raised touchpads, and another drawing a more complex circuit with a sharpie felt tip pen. The second approach provides a flat connection area, as no soldering is needed on the touch pads themselves – but we can still solder connections on the edges of the board (this becomes a bit clearer later).
Here we have dissolved the sodium persulfate crystals to create the acid bath – note we are doing this outside, as the process creates fumes you don’t really want to be breathing. We started the temperature at 50 degrees – you are supposed to maintain this but we didn’t have a way to do that. The etching took about an hour and a half, it would probably be more like 10 minutes at the proper temperature, but it doesn’t seem to matter if you are not in a hurry.
Afterwards we store the acid as it can be used again, the blueness indicates the amount of dissolved copper – when this gets too dark it stops working, then you can neutralise the acid by adding bicarbonate of soda and take it to the tip for safe disposal. Obviously the labelling is important – not just the fact it’s toxic but the date it was made, for future use – it should last a few months.
Here is the etched board after washing well in water and drying – no copper visible!
The next step is to remove all the resist – both the tape (and the glue that comes with it), and the marker pen. That requires a bit of alcohol and elbow grease.
Here we can see the copper and mostly nice clean tracks – there was one place which required some work with a scalpel to break a remaining connection, but we didn’t expect it to be this good for our first attempt!
Now the PCBs can be separated…
And finally – tested on a Raspberry Pi, using the GPIO pins for now to read the receptor type directly. On this PCB we have two of the three sets of connectors working – so six values in total. This redundancy means that the more that match, the more confident we can be that we have the correct reading.
Things to try next are some tweaks to the Pi so it can read the receptor more sensitively as currently it needs nearly the full 3.3V to register on the digital pins. The Makey Makey seem to do this by using large pull up resistors to detect the presence of tiny voltages and filtering the noisy signal this creates in software.
It’s not clear at this point if we will be making the final connectors in this way, but it would be nice if we can (and get a bit better at it) – at least we are now able to quickly iterate their design ourselves.
Other options to be looked into include using a small CNC mill to drill out PCB tracks, and more exotically – the use of micro-organisms to eat copper.
Here’s an article on midimutant we did with Aphex Twin for MagPi Magazine, written by Sean McManus. Most of the work on this project recently has revolved around exploring custom hardware using old FM synth chips from games consoles, but with any luck there should be some more evolved DX7 sounds around here soon.
The trick to start understanding this (I still have quite some way to go) came from only thinking about a single square tablet. If we follow the paths of each of the four threads while turning the square 90 degrees at a time we can see how tablet weaving is a combination of a weave (up and down movement) and a braid (left and right), as it twists the threads in relation to each other.
From this sketchy starting point it was possible to create two 3D objects to represent each twist, one for clockwise and another for anticlockwise. If you colour the separate threads appropriately and combine them together you get something like this:
While this looks fancy, it’s wrong. The threads may be in the correct form conceptually, but woven structure comes about as a relationship between the positioning of the threads and the tension applied to them. Many of the threads above should be pulled straight and push others out of the way to give a pattern that was actually straight stripes of colour, rather than chevrons. So how can we add tension?
One way to approach this problem would be to use a physical simulation of the kind usually applied to cloth simulation, and ‘relax’ the the threads to achieve a realistic result, using a stochastic approach to iteratively tighten them within collision constraints, until it ‘looked right’. The problem with this is that it wouldn’t lead to a deeper understanding of what is going on here. This in a way is related to a bigger issue with AI and machine learning, where techniques like artificial neural networks can be trained to solve problems well enough to be useful, for example in speech recognition – but do not provide any new knowledge about language or understanding of deeper scientific issues.
So if we want to understand some of the ‘thread logic’ of table weaving, we are can approach this in a more symbolic manner. Can we add additional straightened threads to our two twisted ones?
As with the twists, there need to be two forms of straightening – left or right twist to straightened threads, and then we need to get back from a straightened thread to a left or right twist.
Notice that some of these shapes connect, while others are incompatible. We can start with the original twisted weave above, and process it to pull the threads straight. In order to do this we need to know the past and future actions of the weaver, or the current twist in the context of those before and after it. This makes sense, as when weaving structure emerges fully a few wefts behind the current one you are weaving – only as the tension is applied to the fabric does it take form.
The rules to describe this turn out to be well represented as a diagram. The nodes are the 3D shapes required and the edges are the actions of the weaver (the special ‘floating’ state change interestingly depends on the action before the last one – memory does seem important in tablet weaving).
For example, we can ‘left twist’ repeatedly (the top right state) as the arrow points to itself. If we start going in the other direction we then need to pass through two straightening states to get to a full ‘right twist’. If we start going backwards and forwards in smaller numbers of turns then more complex things happen.
When we process the first weave with these rules, you can see some of the straightening effects. The tension on the threads means that some cover up others, e.g none of the yellow threads are now visible on the top of the fabric at all.
The structure is more visible here than on a real weaving as the threads are thinner that they would be for the resulting weave which would be more densely packed together (this is less realistic but helps to understand what is going on).
How do we know if any of this is correct? The only way to test this for sure is against real weave. We can try out different sequences of actions and see if the model matches. As indicated above, tablet weaving is a technique that comprises several categories of weaves – these define some specific types of structure we can test.
Type 1: Repeated twists and turn back
Most normal tablet weave consists of twisting repeatedly 90 degrees in the same direction and weaving a weft each time. In practice there is only so far you can go in the same direction before the unwoven warp threads behind the tablets get tangled up, so you need to change direction and go the other way until they are untangled, providing some symmetry to the pattern. The first example has all the tablet threads aligned in the same sequence – and we weave 8 turns one way and 8 turns back again. You can see in the middle when we change direction we create a short straightened ‘float’ section which causes the tension to pull the threads straight here.
One of the further mysteries that our first tablet weaving simulations couldn’t previously recreate were situations where the pattern on the back and the front of the weave were not opposite of each other. This is highly unusual in weaving, but this model seems to represent this correctly. Here the actions are the same as the first example – 8 one way and then the other, but the thread colours in the tablets are offset from one another so they are staggered and you get the diagonal patterns.
Type 2: Single faced double weave
Part of the complexity of tablet weaving is because it is a kind of double weave – there are two intertwined weave structures happening at the same time. If we repeat two wefts of 90 degrees one way followed by two more the other direction, the two weaves remain on the same side of the textile – which can be seen clearly if we colour them appropriately. This example keeps the white weave on the top side with the brown one on the lower side.
Type 3: Degenerate floats
The third type of weave is not really a weave but a breakdown of the process caused by only weaving single 90 degree turns backwards and forwards repeatedly. This means half of the threads are not incorporated into the weave and ‘float’ along the surface on both sides.
While the language to fully describe the tablet weaving has yet to be developed properly, you can have a go yourself with this model which is currently online here (takes a few moments to render at first).
This gets us closer to a working model of tablet weaving, and provides something we can start to use for a more advanced aims of the Penelope project. For example, can we use the pattern matrix to tangibly livecode tablet weaving? Does this make it possible to explore and explain this type of weaving?
If this kind of textile wasn’t complicated enough, people in ancient times combined multiple weaving techniques, for example tablet weaving and warp weighted weaving in the same piece of fabric. Creating a kind of ‘grand unified’ weaving model is an additional future challenge, so we can start to understand better the thought processes involved in these advanced techniques.
The future is in short supply at the moment, particularly in a country with so many things changing – it is a constant surprise that there are so few ideas about where we could be headed. At the smaller scales too, at FoAM Kernow, in common with many organisations there is a feeling we tend to be so fixated on the current problems (do we need to invoice for that project, have we replied to this important email, have we reached an agreement on the next steps for this other project), that we miss the long view. Why are we doing this? What are we doing it for? Do we know if this is the right approach? These seem difficult to answer when stuck in short term thinking.
One of the central parts of FoAM’s research for some time has been future preparedness. Through a series of projects Maja and Nik have been building up a collection of methods that allow an individual or organisation to picture itself in a future under a range of conditions called scenarios. Scenarios are designed to be extremes, or caricatures of the possible â€“ if you can describe them well enough, you can envisage how you or your organisation would adapt to them. The goal is to reach a point where change is something that can be reasoned about in advance, rather than reacted to as it happens.
Our initial question was (as FoAM Kernow): â€œWhat do we want to be doing? â€ broken down a bit more specifically into these points:
What would be choose to do?
What makes a FoAM Kernow project?
What are our selection criteria & success criteria?
How to say no?
How to back out of projects where contributions are less relevant?
What is our role as collaborators?
What is the nature of our collaborations?
After this we went through a series of exercises to build up a comprehensive list of things both internal and external that could cause changes to our organisation. These were quite diverse, but we seemed to focus on external things a bit more than internal ones e.g.: inaccessibility of lab equipment, the rise in nostalgia as a form of solutionism, and that more people are active in politics than there used to be.
The aim is to figure out two significant ‘critical uncertainties’ (interestingly, these are not necessarily the most essential ones, but the most uncertain) which we can use to build our scenarios from. We ended up with these:
The way are living is DOOMED! âŸ· The status quo is FINE!Continuous learning & exploring the new âŸ· doing what you know. same old.
The first uncertainty defines the amount to which our society has the capacity to deal with the many big problems currently facing it. The second one we took to have meaning both internally and externally â€“ society being more or less progressive and open to new ideas, as well as the requirement that we would need to learn new things, or whether our existing skill set is sufficient.
The next thing we did was to draw them as two axes and split them into four scenarios which represent the extremes of these uncertainties. We were then able to describe the world and our place in it by focusing on each one in turn, importantly giving them names which really helped give them character. This part was really fun, as they really are caricatures, and resulted in quite a lot of laughter â€“ even (or especially) when describing what might be quite uncomfortable futures.
Poundland [the way we live is fine, we only do what we know]
This is a world where everything works out fine without any new knowledge needed. It’s hopeful in that we dodge some bullets, but a lot of top down control needs to be exerted for this world to operate, so it’s restrictive in a conservative, traditional way.
â€œStasis reigns in Poundland. While class inequality is as bad as it always was, there isn’t much questioning authority. Most people defer decisions to the powers that be. The ruling class is conservative, praising traditional values. Critical education has made way for edutainment and propaganda. Hard Brexit happens seamlessly, the borders are locked down, society surveilled. There are a lot less people in Poundland than there were in the UK. Climate change is not as bad as people thought in the 2010s. A new ice age has arrived and we’re coping. With appropriate urgency, solutions can be found to maintain reasonably comfortable lifestyles.
On the surface, FoAM is doing everything right, yet has not much visible power. We receive government funding to start a new university for applied sciences and technologies, focusing on mitigating effects of the Ice Age. Beneath this benign face, we are guerrilla teachers. Using workshops and other forms of adult education, we prepare our students to think for themselves. With the university as a front, we are hiding in plain sight.â€
To a certain extent this seems to be a world that is close to the vision of groups like those behind Vote Leave â€“ a future where the UK is a â€œSingapore on the shores of Europeâ€, deregulated to the point where we can kickstart our economy and closeted workforce into higher productivity. It’s a bit like the future described in this book (reading of which was a bit of a follow up for this scenario):
The problems in society identified in this book seem pretty good – but the unshakable belief in the free market pushes issues like human rights, health, quality of life and the environment into irrelevancies with only passing mentions â€“ the assumption (as in this scenario) that these things will ‘somehow just be fixed’, so prosperity can continue.
Silicon Wharf [the way we live is fine, with a thirst for knowledge]
Silicon Wharf is a progressive, shiny future where everything is solvable by big investment in new knowledge. It’s a world of expensive institutional buildings with odd angled walls, carpet tiles and people wearing lanyards, it’s a very EU future vision – and seems mildly nostalgic and quaint, perhaps because it is a world reliant on abundance of many things.
â€œThe Silicon Wharf is ruled by technocrats (both in government and corporate sector), with rationality and logic as the dominant virtues. It is a node in a globalised network of excellence, where the “best minds” work on technological solutions to climate change. There is a lot of funding for (blue sky) research, as long as the results are presented as innovative products and services that can be scaled up to produce social and/or environmental impact (e.g. carbon sequestration into diamonds for the luxury market). There are mega infrastructure projects under way, championed by China. Brexit has been cancelled and Cornwall has become a key player in the Europe of Regions. The world is increasingly open and there is much cross- border exchange (of knowledge, people, stuff).
The FoAM network has expanded to hundreds of studios and has formed its own NoE (Network of Excellence). The network works on long-term technology research and development and disruptive education. We focus on technologies that are relevant, accessible, tangible and open source. The knowledge we gain through R&D we pass on to anyone who is interested, in workshops and other p2p educational formats. We teach people to make things that can solve their immediate issues, and teach them to teach each other. We get to engage in interesting translation and dissemination activities, which take us out to different contexts, regions and cultures. We’re therefore continuously learning and expanding our horizons.â€
Life in this world seems quite easy but somehow dangerous too – just imagine the amount of administration we’d be doing, there are lots of spreadsheets here! The general openness to new learning would make this an exciting place though, with lots of travel possible, due to new innovation excellence in transportation technology.
IRK (Independent Republic of Kernow) [the way we live is doomed, with a thirst for knowledge]
This is in some ways a collapse scenario, nothing has worked out but there is still a thirst for new ways â€“ and because acceptance that the old ways are broken are unavoidable, there are opportunities to attempt much bigger alterations and experiments.
â€œThe Independent Republic of Kernow is a good place to live in highly uncertain times. Urban centres have become danger zones of poverty, disease and violence, so the predominantly rural IRK receives many urban emigrants and refugees. While there is a low life-expectancy in IRK, life is filled with buzzing experimentation and excitement. People dare to take risks, knowing that their survival depends on learning from failure. There are different political and economic experiments (including UBI, a new tithe system, etc.). As the NHS collapsed, new forms of Medical Citizen Science are on the rise. A plethora of infrastructure projects appear and disappear. Hybrid land-sea infrastructure is the most experimented with, as IRK has become a seasonal island. New Dawn traders deploy their sail ships to bring in goods from further afield, but most of the produce is locally sourced (short-chain economies, including initiatives like New Lina, End of The World Garden…). There isn’t much leisure or time to rest, so burnout has become a plague. Some burnt-out people created Circles of Stability, where they tell each other fables of bygone stable lifestyles. While these circles often promote hopeless nostalgia, most of the world has moved to new myths: biomimetic stories of humans becoming social-insects.
FoAM has moved into the Eden Project. We are issuing our own currency (FoAMcoin) and are part of several trade networks (incl. the well established Feral Business Network). The main dome is a productive garden where we grow food and medicine for our collaborators and wider network (in collaboration with other horticulturalists and farmers in the area). What we can’t use ourselves, we barter for other goods. One of the domes is converted into the warehouse for New Dawn traders, another is devoted to agritech experiments and another is focusing on our communication infrastructure projects, such as waterproof mesh networks. We’re trading produce, but primarily we exchange skills. We aim to cultivate a skill-ecosystem that includes a wide range of people with diverse knowledge, talents and skills. We travel more, using slow modes of transport. Some of us stay in Eden, while others opt for journeyer lifestyles with no fixed studio and no desk (work). FoAM thrives in this trans-local republic, and even though we know we might die soon, we’ll die happy and fulfilled.â€
So, appropriately IRK is the opposite of Poundland, life has the potential to be good but generally short â€“ whereas in Poundland stability was rigidly enforced and life seemed hard. When we were discussing this scenario there was a lot of laughter and general surprise that this would be the best one in a lot of ways, even though everything is doomed.
Swamp.ac.uk [the way we live is doomed, we only do what we know]
The last scenario is, by all rights the worst one possible â€“ there is no desire or drive for learning new things, so everything is irretrievably broken. However, one of the significant things that happens when futurecrafting is finding that what you thought were the worst case scenarios have features of them that are still interesting and desirable. This is the scenario of escapism, VR and games â€“ it also has the best parties. As scenarios are extremes, they are not going to happen – but they do give indications of situations that we might recognise in future so we can connect with these desirable outcomes, even if they are a result of an overall situation we might not wish for.
â€œThe way humans live is doomed, and we have accepted it. It’s too late to change anything anyway. We’re living in a Trump-inspired world, in what is left of the UK. There is no funding for research, no electric car subsidies, but regional funding and (corrupt) local philanthropy is growing. It’s all about who you know. Universities are small, focusing on keeping the local population occupied (some dedicated to spreading ‘the word’ of the regional philanthropic magnates). Brexit is still in (perpetual) negotiation, leading to additional economic, social and political paralysis and fragmentation. A sense of powerlessness and inequality permeates everything, leading to a simmering, low level civil war. In their despair people turn to escapism and nostalgia. VR games and end of the world parties are plentiful, with people crowding together ‘to go out with a bang’.
FoAM has realised that the only way to survive in this world is to build a stable institution ourselves, to become one of the power players (in loose alignment with local rulers). We abandoned our individual passions; instead, all our energy goes into collective endeavours. Our institute is known for developing high quality and transparent technologies. Our biggest department focuses on techniques and technologies for maintenance and repair of crumbling infrastructure for everyday life. Our other line of work is less known in the mainstream, but hugely popular with the escapist underground; we organise exorbitant crypto-parties and citizen science game-fests (VirusCraft became hugely popular, funding most of our non-profit programmes). Accesslab is our public outreach programme, helping people make sense of the nonsense they’re faced with every day. Seemingly dealing with short term issues, we’re actually teaching (subversive) long-term thinking.â€
It seems that the only hope in this world would be to somehow nurture a shift to a more IRK like situation, attempting to show in small ways some way out of the powerlessness. The significance of AccessLab in this world is interesting, as it would be increasingly important here to connect a science starved of funding with the reality of the situation to keep it relevant, as well as making it useful for providing potential new avenues.
Of course we live in neither the worst or best of all possible worlds, but thinking of the extremes helps us to be prepared for moves in the middle ground; combinations of these futures are more likely. We can now use these scenarios to plan projects, for each potential new thread of investigation we can consider how many of these scenarios would it work under, and the more the better.
We’ve had quite a lot of success over the past couple of years with really popular projects we’ve nurtured into existence ourselves, which seem to strike a chord and get a lot of attention – Sonic Kayaks, AccessLab and midimutant for example. These scenarios are a new tool for us to continuously gauge the relevance of what we are doing, and provide hints for how to make an idea useful regardless of how the future pans out.
Once we acknowledge that weaving and programming are part of the same technological timeline, we can begin to look at the history of weaving as a eight thousand year long tale of human relationship with digital technologies – and use this long view to research new approaches to software engineering, a field with a much less developed history and many interesting problems to solve.
One of our threads of investigation is using the pattern matrix as a general purpose tangible programming system – one that we can use for controlling swarms of robots, programming different types of weaving systems and describing complex processes, such as live musical systems.
The magnetic system on the new pattern matrix consists of four hall effect sensors on every location you can place a block. There are four unique ways you can arrange the magnets – which means four types of programming block are possible. As we want to reuse these physical blocks for various uses and programming languages, we decided to use abstract shapes to denote the block types to begin with. Each of the four blocks can be rotated and flipped to give 32 total possible orientations, or symbol ‘tokens’:
However, only 16 of these orientations are actually unique. We can only determine flip orientation on the circular block, and only rotation on the rectangular and triangular ones – where flipping them makes no difference to the magnets. The square block is a kind of special one, as we can tell both rotation and flip orientation, so it can represent eight tokens in total all by itself:
It’s important to note at this point that the parallels with tablet weaving are no coincidence: rotating and flipping arrangements of four binary elements for this magnetic system are the same actions as those performed when weaving using tablets. Weaving in the pattern matrix is more than a subject, it’s built directly into it’s mode of operation.
Next we need to test the applicability of this tangible programming system for wider uses. The other cultural phenomena the Penelope project is involved in is livecoding – so is it possible to use the pattern matrix to introduce a weaving centred programming technology in a very different context, not to describe weaving but to generate music in a performance such as an Algorave? This is something that Ellen first pioneered at our weavecoding performance at The Museum of casts of classical sculptures in Munich, but the new pattern matrix has better capabilities for a general purpose programming language.
Having 16 states of four blocks is indeed limiting for a language, but not too limiting. Some types of programming language, such as a string rewriting system like a Lindenmayer system are particularly well suited to this. They are also surprisingly Turing complete languages, able to represent any other programming language in existence, given enough space and time.
Here is a quick example of how this works in text form – a string rewriting system is simply a list of search-replace actions that are carried out in a consistent order. The original example, used to model the growth pattern of algae – consists of a starting string: “A” and two replacement rules, replace “A” with “AB” and replace “B” with “A”. If we run these two rules over and over on the same bit of text we gradually ‘grow’ a pattern like this:
n = 0 : A
n = 1 : AB
n = 2 : ABA
n = 3 : ABAAB
n = 4 : ABAABABA
n = 5 : ABAABABAABAAB
n = 6 : ABAABABAABAABABAABABA
n = 7 : ABAABABAABAABABAABABAABAABABAABAAB
On the pattern matrix we use four of the rows to represent four different rules that replicate in this manner (each made of 5 possible symbols, as it’s a 5×5 grid), which we run 8 times on the starting string (A) to create a musical sequence. Four of the tokens represent these rules (A,B,C and D), the remaining tokens represent musical actions – changes in pitch, rests and sound triggers. There is massive variety of potential patterns, you can control the amount of recursion by the number of rule reference tokens you use – to control the resulting length of the sequences, and thus the complexity of the music. Interestingly we also need a ‘no operation’ (NOP) instruction that does nothing – as in low level assembler languages. We need this as a way to be able to shift timing in the musical sequence by one instruction.
With a tangible programming language like this it’s also very important to consider how you categorise instructions by shape – as you can quickly switch between similar operations by simply rotating or flipping tokens, while switching between different shapes takes longer (as you need to pick up a new block) so should represent bigger changes if possible.
Four rules are plenty for generating hugely complex sequences, so we can use the fifth bottom row to control global parameters like scale, synchronisation options (for our slub collaborative sync protocol) or switch between more banks of sounds for greater variety.
The first time we tried this out was at the British Science Festival Algorave in Brighton. A projection was set up with a camera to show the pattern matrix being used, and while technically everything went fine (other than some syncing difficulties), it highlighted a key problem with tangible programming languages. With no dynamic feedback other than the state of the blocks on the pattern matrix, it’s very difficult to tell what is happening during a performance, it’s hard to understand what musically is resulting from the changes you are making.
In order to find a way around this we designed an augmented reality ‘layer’ to place over the pattern matrix, which gives feedback on the currently triggered notes and the paths between the recursive string production rules. We use fluxus and it’s AR feature, which was written by Gabor Papp – which is based on the ARToolkit library. We use a printed out marker to find the plane and camera scale of the centre of the pattern matrix in the image from a USB camera. Once this is done the marker can be removed (as neither the camera or pattern matrix moves) and we can use millimetres as units and place objects over the block locations in 3D space. When the sensors detect a change we can display this new information, also updating the current position in the sequence playback to give feedback on the current sound playing.
As an initial trial the AR improved things when first tried out at the Algomech Algorave in Sheffield, it makes the pattern matrix easier to understand and perform with – and also provides some feedback for the audience in a projection. In a last minute change we switched from Latin characters to Linear A, an undeciphered ancient Greek script – a reference to Flavia’s work on the Penelope project. This is actually preferable to Latin characters as the musical language represents meaning in a way that that actual glyph used is irrelevant – it’s better if it can’t be ‘read’ or confused with another meaning by anyone (still alive).
So it seems that AR could be one of the items in our toolbox for further tangible programming experiments. Perhaps we can better explain the structural changes caused by livecoding the weaving notation for the warp weighted loom by having a dynamic weave structure ‘floating’ on top of the tokens, alongside the loom simulation. This could also be of use for describing tablet weaving actions with these blocks, which would need to be more abstract than the binary weaving notation.
Another area to explore is the design of the blocks themselves, moving away from the abstract shapes, we can design them for specific purposes. Similar to our work on viruscraft, where we have more closely explored the correspondence between physical form (receptors and structural protein arrangements) and tangible interfaces, it seems that these shapes may be worth considering more closely now the sensor matrix is working well.
During the symposium we discussed the critical, liberating and potentially dangerous aspects of Unmaking in a wide variety of contexts – from reverse engineering knitwear and classical Greek dance to discovering the untapped abilities of classical musical instruments when human limitations become a secondary consideration. The symposium also provided us with a good opportunity to take stock of our own group’s current directions and thinking in regard to the Penelope project.
As is our usual practice, we used this exhibition to get essential feedback on our new design for the pattern matrix, as well as the interpretation of what we are doing – being there in person talking to people allows you to very quickly determine what works, and adapt the focus based on the responses you get.
We discussed the long view on digital technology, the role of weavers in foundations of western mathematics and simply the practicalities of the technology we are developing – the constant stream of visitors represented a wide range of different ages and backgrounds. These aspects of the pattern matrix seemed significant (in no particular order):
The construction technique was immediately interesting to most people, specifically the material, beech wood spalting and open frame construction.
Related to this, there seems little association with the pattern matrix as a ‘device’ in the sense of a ‘gizmo’ – which is interesting as the Raspberry Pi and other PCBs are clearly visible. We’ve noticed this at some level with the previous version but with the wood construction this effect is much more pronounced.
It is seen as being game like, e.g. a “70’s educational toy”. There is an expectation that it is something to be ‘played’, and similarly its potential as a musical sequencer is a common observation.
The understandability of the magnet sensing seems a key ingredient. There is no other particularly hidden magic like computer vision or RFID involved, and polarity and digital arrangements of magnets are easily explained and experienced by holding the tokens together.
Having some extra circuit boards and wood cut parts to hand, originally intended as backups – were great for people who wanted to know more about what was going on. In future we should also have the token block parts to show as well.
The different shaped blocks were immediately appealing – they seemed to invite experimentation more than alternating the binary tokens by flipping them. To follow this up we need to investigate ways to use different shapes to configure thread colour at the same time as structure in a better way than we are now. The black/white sides could define structure while the shape could correspond to the colour for the specific thread. It also indicates that using shaped tokens as instructions for tablet weaving is worth experimenting with quite soon.
Younger children focused on the blocks alone (and of course tried all sorts of things no one else did, like stacking them) but slightly older children worked out they were having an effect on the weaving process and generally could patiently work themselves it out without any explanation required.
Having Anni Albers’ ‘On Weaving’ book next to pattern matrix helped with older visitors, perhaps representing a more conventional and authoritative source of information to introduce the concepts of notation, structure and pattern in weaving.
Physical vs digital – a false dichotomy
More general concepts that came up in conversation included a common theme during Algomech, exploring the inescapably fuzzy boundaries of concepts such as digital, physical and analogue. The myth of the “real world” being analogue and the “virtual world” being digital is a troublesome one to a weaver.
The ‘anti-device’ effect of the pattern matrix has the potential to explore this conundrum, as it represents a seemingly acceptable demonstration of the physical nature of the digital, and that forms of digital technology have inhabited the world of the reassuringly physical for many millennia of human invention.
Having a 5X5 grid initially was thought to be excessive, as most ancient weaves can be expressed with a 4X4 matrix. This larger capability turns out to be more important than we first thought, as it means you can demonstrate the importance of odd and even numbers in the mathematics of weaving.
There is a problem with the single colour change block due to a faulty use of the code from the old version. This has always been a somewhat temporary feature so we should sort this out properly (e.g. using other shapes for colour across the matrix) before we use it next.
We can also try using an augmented reality approach to show the weave structure directly over the grid, so it’s easier to see how the token block changes relate to specific crossings. This could be displayed alongside the current warp weighted loom rather than as a replacement for it.
So I’ve been getting a into FM synthesis lately, and after giving the TX7 a spin I’ve been really wanting to try evolving sounds for the later synth chips Yamaha made for games consoles and soundcards. At that point in history FM was one of the only realistic ways you could synthesise complex sound as there wasn’t enough time (processing) for general purpose DSP, and there wasn’t enough space (memory) for sample playback to be an option.
One of the most famous of these range of synth chips was the YM2612 which was used in the Sega Megadrive/Genesis consoles. I got hold of a few to try, but it’s been quite a challenge to get them working so I thought probably worth documenting the process. The YM2612 is programmed using 5 control pins and 8 data pins. You send one byte to the chip at a time – first an address to write to, then the data to write, which allows you to set a couple of hundred parameters that define 6 separate sounds. The memory on the chip is organised into a map of the global settings (top left) and the voice settings in two banks:
There isn’t much documentation for these things, the main reference is for a similar chip which is written in Japanese. More usefully there are software versions in emulators, and quite a few bits and pieces online from other people’s projects and some translations of the docs. Usefully the Sega documentation included a piano test sound which is great as you have something that *should* actually make a noise as long as everything else is working.
The first problem was figuring out the timing – despite having some examples it took me a while to realise that there is a specific order that you have to set the status pins, and that there is one change at one time that actually triggers (latches) the data transfer to the synth:
This is a timing diagram, I’m used to seeing them in datasheets, but it’s the first time I’ve needed to understand one properly (with a bit of help from David Viens). It tells us that for writing, the data (D0-D7 pins) are actually read shortly after the WR pin is set low – to add to the fun, some of the pins (denoted by the line over them) are 0v for on, 5v for off. You can also read data from the synth, but whatever address you supply you only get a single status byte that tells you if it’s currently busy, and if either of it’s two internal clocks have overflowed. The idea I assume is that you would use them for timing music accurately without the need to do it in your main CPU.
The audio coming out needs to be amplified too – if you connect it directly to a speaker there is a danger of overloading the DAC and causing it to burn out, so you need a bit of analogue circuitry to do this. Here is an example of triggering and glitching (by writing to random addresses) the example piano sound:
I started off using an Arduino Nano, but didn’t have much success so switched to atmega328 which was a bit easier to set up. One of the problems is that ideally you need to be able to control the 8 data GPIO pins in one go (to set the 8bit value), which isn’t possible from the nano as none of the ports have 8 bits. You also have to supply the YM2612 with a clock pulse at around 8Mhz – which took me a while to get right, I tried both dividing the atmega’s 16Mhz clock by 2 and outputting it on a pin as well as a separate external oscillator and both eventually worked – except the oscillator leaked quite badly into the audio output (there is probably a fairly simple fix to isolate this).
One potential problem is dodgy vintage chips, I tried two suppliers on eBay, one Chinese shop and another in the UK – all the ones I’ve tested so far worked fine, I’m happy to say. The best way of testing them is to use a second hand Sega Megadrive console and replacing the YM2612 it has with a socket you can easily plug in candidates for testing – I managed without but it would have come in handy sanity wise to know for sure that the chips worked.
One thing that I wasted a lot of time doing was ignoring the output entirely and just trying to get the status bit to change by setting the internal timer – the idea was to make sure the chip logic was working before worrying about the audio amplifier circuitry. However, this requires the read sequence timing to work as well as the write timing – and as soon as you can listen to the output you get small clues from the DAC noise, which actually indicates the chip is running or not even if there is no sound triggered.
Thanks to Paul Chaney who runs The End of the World Garden, we had an opportunity to trial a short workshop based on our Farm Crap App and prototype Allotment Lab on his two-acre forest garden site in Cornwall. This was our contribution to the Bank Holiday Weekend Haymaking Extravaganza (along with a bit of hay-making too).
We started by looking in detail at the Farm Crap App, how it arose from an intersection of governmental policy and the needs of farmers, providing a way to quantify the nutrients present in natural manures that are added to the land. We were able to map a section of the End of the World Garden intended for future agricultural experimentation, and based on its crop history and soil type calculate the additional nutrients required for growing different crops.
One of our discoveries were the limitations of the underlying DEFRA data, being based on a particular approach to farming it uses a pretty broad brush – the necessary simplicity of the data in some areas (for example in the range of crops present) is not so well suited to smallholders. For example high yield crops that maintain profitability are less important to smallholders who are able to use slower growing strains. It was also interesting to work with people who had less experience with farming (me included) and trying to work out the difference between ‘dribble bars’ and ‘splash plates’ and other arcane muck spreading technology.
We also tested our prototype Allotment Lab. This has been designed to provide a couple of trial walk through experiments based on compost quality and soil type estimation for a more general audience – allotment owners, gardeners and smallholder farmers included.
We tested the soil in a small area of field, attempting to form different shapes with our hands and a little water, in order to estimate its specific type and consistency. The soil in this part of Cornwall consists of a top layer of ‘sandy clay’ (actually containing larger granite particles, rather than sand) and a lower layer of ‘silty clay’ laid down from melt water during the last ice age. Paul dug down to find the lower layer so we could check the difference between them using the test.
This workshop was useful in that it highlighted the limitations of top-down governmental data and the potential for citizen science to allow people to gather their own knowledge based on their specific circumstances. One of our challenges is coming up with a range of experiments that can augment official data and allow farmers, allotment owners and others to ask questions, collect data and make decisions that will help them in increasingly difficult social, climatic and economic circumstances.
The genetic model we were working on previously has now been ported into a browser compatible form and connected to a new tree visualisation that displays the species that emerge as the host population adapts to a virus infection. It’s still a prototype with rough edges, but have a play with it here, some example pics:
This is one of the earlier attempts, which I like the look of, but later cleaned up versions are a bit clearer to read what is going on.
Firstly the genetic part is working in that the population evolves to get gradually better at coping with the virus infection (the fitness score increases) – it takes a bit too long at the moment, but it’s great to be able to see this working in realtime as it happens with new species branching off older ones.
One early observation is that this has the potential to show why diversity is beneficial. If you modify the virus (fitness function) at a point when there are lots of different species present, the chances are that a few of them will be resilient enough to the infection to expand into the new environmental niche, and things eventually continue as before. If you alter the virus when there is only a single really well evolved species that is a bit too good at coping with the existing virus – the chances are that you will cause the population to go extinct as it won’t be able to adapt. This is analogous to the situation with bananas: “To carry on growing the same genetic banana is stupid”.
A big chunk of the work here was actually spent optimising the code. It’s pretty amazing how developed browsers are for development – I’ve been using the profiler in the chromium browser to locate slowness and keep the frame rate as near 60 frames per second as possible. I just noticed taking the screenshot for this post the slowest single part seems to be the debug text rendering, strangely enough.