The Penelope project is concerned with many things, pattern structures in ancient poetry, comparisons of Andean and Greek mathematics, and the role of liveness in thought processes – to name just a few. We can also add robotics to this list.
A weaver in action is often mistaken for a robot – perhaps it’s the repetitive actions that mask the accompanying thought processes from our eyes. Similarly, a livecoding performer typing code into a laptop can face a problematic interpretation that they are grandstanding their expertise, rather than simply attempting to contribute to a party atmosphere at an Algorave. These activities often seem to be misunderstood in a similar manner but in different directions, perhaps partly due to the fact they both employ technology that is either greatly over or under-played by the ubiquity of their end results.
The Penelope project is concerned with using these technological misunderstandings in order to search for deeper assumptions we may have. One case we’ve discussed for a while has been to combine the situations above, if a weaver livecodes a robot which is operating an ancient loom – what is the response to this?
One way to design these situations well is to use an ‘artistic conceit’, a implicit fictitious assumption that provides a framework to fit things together so they make sense. For example, working with the Quipu database, we used the central idea that the Inca were attempting to communicate with us to naively employ cryptanalysis, visualisation and sonification in an attempt to understand their untranslatable knotted thread codes.
An important reference for the Penelope project for me is Carl Sagan. In his influential series ‘Cosmos’, weaving is mentioned many times as a central ordering principle of time and space for the Ionian thinkers of ancient Greece, who initially wrote down many of the fundamental scientific ideas he is introducing. He embarks on a thought experiment considering what life would be like today without the destruction of the library of Alexandria – which contained much of their discoveries and ancient knowledge from all over the world, now lost forever. His conjecture is that this single event may have set us back as much 500 years in scientific thought, so it’s just possible that without this event occurring, on a different timeline to ours – around about now, the first space ships would be returning from the surveying planets orbiting other stars.
It it this kind of imagination and suspension of disbelief that can give us a new perspective on our current predicaments – and perhaps a more humble position with which to question that our ways of thinking are inevitable, obvious and about as good as they can be.
For example, if the invention of general purpose computation and robotics had occurred in a society much earlier – one founded on the woven cosmos, what could it have looked like, and how would it have worked? What might it tell us about different ways of doing things?
Once we acknowledge that weaving and programming are part of the same technological timeline, we can begin to look at the history of weaving as a eight thousand year long tale of human relationship with digital technologies – and use this long view to research new approaches to software engineering, a field with a much less developed history and many interesting problems to solve.
Using augmented reality to display dynamic information on a tangible programming language.
One of our threads of investigation is using the pattern matrix as a general purpose tangible programming system – one that we can use for controlling swarms of robots, programming different types of weaving systems and describing complex processes, such as live musical systems.
The magnetic system on the new pattern matrix consists of four hall effect sensors on every location you can place a block. There are four unique ways you can arrange the magnets – which means four types of programming block are possible. As we want to reuse these physical blocks for various uses and programming languages, we decided to use abstract shapes to denote the block types to begin with. Each of the four blocks can be rotated and flipped to give 32 total possible orientations, or symbol ‘tokens’:
All combinations of token orientation with four magnets
However, only 16 of these orientations are actually unique. We can only determine flip orientation on the circular block, and only rotation on the rectangular and triangular ones – where flipping them makes no difference to the magnets. The square block is a kind of special one, as we can tell both rotation and flip orientation, so it can represent eight tokens in total all by itself:
All unique tokens and orientations possible. With mid-grey shapes the flip (which side is up) is irrelevant.
It’s important to note at this point that the parallels with tablet weaving are no coincidence: rotating and flipping arrangements of four binary elements for this magnetic system are the same actions as those performed when weaving using tablets. Weaving in the pattern matrix is more than a subject, it’s built directly into it’s mode of operation.
Next we need to test the applicability of this tangible programming system for wider uses. The other cultural phenomena the Penelope project is involved in is livecoding – so is it possible to use the pattern matrix to introduce a weaving centred programming technology in a very different context, not to describe weaving but to generate music in a performance such as an Algorave? This is something that Ellen first pioneered at our weavecoding performance at The Museum of casts of classical sculptures in Munich, but the new pattern matrix has better capabilities for a general purpose programming language.
Having 16 states of four blocks is indeed limiting for a language, but not too limiting. Some types of programming language, such as a string rewriting system like a Lindenmayer system are particularly well suited to this. They are also surprisingly Turing complete languages, able to represent any other programming language in existence, given enough space and time.
Here is a quick example of how this works in text form – a string rewriting system is simply a list of search-replace actions that are carried out in a consistent order. The original example, used to model the growth pattern of algae – consists of a starting string: “A” and two replacement rules, replace “A” with “AB” and replace “B” with “A”. If we run these two rules over and over on the same bit of text we gradually ‘grow’ a pattern like this:
n = 0 : A
n = 1 : AB
n = 2 : ABA
n = 3 : ABAAB
n = 4 : ABAABABA
n = 5 : ABAABABAABAAB
n = 6 : ABAABABAABAABABAABABA
n = 7 : ABAABABAABAABABAABABAABAABABAABAAB
On the pattern matrix we use four of the rows to represent four different rules that replicate in this manner (each made of 5 possible symbols, as it’s a 5×5 grid), which we run 8 times on the starting string (A) to create a musical sequence. Four of the tokens represent these rules (A,B,C and D), the remaining tokens represent musical actions – changes in pitch, rests and sound triggers. There is massive variety of potential patterns, you can control the amount of recursion by the number of rule reference tokens you use – to control the resulting length of the sequences, and thus the complexity of the music. Interestingly we also need a ‘no operation’ (NOP) instruction that does nothing – as in low level assembler languages. We need this as a way to be able to shift timing in the musical sequence by one instruction.
A musical language in 16 instructions.
With a tangible programming language like this it’s also very important to consider how you categorise instructions by shape – as you can quickly switch between similar operations by simply rotating or flipping tokens, while switching between different shapes takes longer (as you need to pick up a new block) so should represent bigger changes if possible.
Four rules are plenty for generating hugely complex sequences, so we can use the fifth bottom row to control global parameters like scale, synchronisation options (for our slub collaborative sync protocol) or switch between more banks of sounds for greater variety.
Slub performance including the pattern matrix at the Brighton British Science Festival Algorave
The first time we tried this out was at the British Science Festival Algorave in Brighton. A projection was set up with a camera to show the pattern matrix being used, and while technically everything went fine (other than some syncing difficulties), it highlighted a key problem with tangible programming languages. With no dynamic feedback other than the state of the blocks on the pattern matrix, it’s very difficult to tell what is happening during a performance, it’s hard to understand what musically is resulting from the changes you are making.
In order to find a way around this we designed an augmented reality ‘layer’ to place over the pattern matrix, which gives feedback on the currently triggered notes and the paths between the recursive string production rules. We use fluxus and it’s AR feature, which was written by Gabor Papp – which is based on the ARToolkit library. We use a printed out marker to find the plane and camera scale of the centre of the pattern matrix in the image from a USB camera. Once this is done the marker can be removed (as neither the camera or pattern matrix moves) and we can use millimetres as units and place objects over the block locations in 3D space. When the sensors detect a change we can display this new information, also updating the current position in the sequence playback to give feedback on the current sound playing.
Pattern matrix livecoding as part of slub performance at the Algomech Algorave. Pix thanks to Dan Hett
As an initial trial the AR improved things when first tried out at the Algomech Algorave in Sheffield, it makes the pattern matrix easier to understand and perform with – and also provides some feedback for the audience in a projection. In a last minute change we switched from Latin characters to Linear A, an undeciphered ancient Greek script – a reference to Flavia’s work on the Penelope project. This is actually preferable to Latin characters as the musical language represents meaning in a way that that actual glyph used is irrelevant – it’s better if it can’t be ‘read’ or confused with another meaning by anyone (still alive).
So it seems that AR could be one of the items in our toolbox for further tangible programming experiments. Perhaps we can better explain the structural changes caused by livecoding the weaving notation for the warp weighted loom by having a dynamic weave structure ‘floating’ on top of the tokens, alongside the loom simulation. This could also be of use for describing tablet weaving actions with these blocks, which would need to be more abstract than the binary weaving notation.
Another area to explore is the design of the blocks themselves, moving away from the abstract shapes, we can design them for specific purposes. Similar to our work on viruscraft, where we have more closely explored the correspondence between physical form (receptors and structural protein arrangements) and tangible interfaces, it seems that these shapes may be worth considering more closely now the sensor matrix is working well.
During the symposium we discussed the critical, liberating and potentially dangerous aspects of Unmaking in a wide variety of contexts – from reverse engineering knitwear and classical Greek dance to discovering the untapped abilities of classical musical instruments when human limitations become a secondary consideration. The symposium also provided us with a good opportunity to take stock of our own group’s current directions and thinking in regard to the Penelope project.
As is our usual practice, we used this exhibition to get essential feedback on our new design for the pattern matrix, as well as the interpretation of what we are doing – being there in person talking to people allows you to very quickly determine what works, and adapt the focus based on the responses you get.
We discussed the long view on digital technology, the role of weavers in foundations of western mathematics and simply the practicalities of the technology we are developing – the constant stream of visitors represented a wide range of different ages and backgrounds. These aspects of the pattern matrix seemed significant (in no particular order):
The construction technique was immediately interesting to most people, specifically the material, beech wood spalting and open frame construction.
Related to this, there seems little association with the pattern matrix as a ‘device’ in the sense of a ‘gizmo’ – which is interesting as the Raspberry Pi and other PCBs are clearly visible. We’ve noticed this at some level with the previous version but with the wood construction this effect is much more pronounced.
It is seen as being game like, e.g. a “70’s educational toy”. There is an expectation that it is something to be ‘played’, and similarly its potential as a musical sequencer is a common observation.
The understandability of the magnet sensing seems a key ingredient. There is no other particularly hidden magic like computer vision or RFID involved, and polarity and digital arrangements of magnets are easily explained and experienced by holding the tokens together.
Having some extra circuit boards and wood cut parts to hand, originally intended as backups – were great for people who wanted to know more about what was going on. In future we should also have the token block parts to show as well.
The different shaped blocks were immediately appealing – they seemed to invite experimentation more than alternating the binary tokens by flipping them. To follow this up we need to investigate ways to use different shapes to configure thread colour at the same time as structure in a better way than we are now. The black/white sides could define structure while the shape could correspond to the colour for the specific thread. It also indicates that using shaped tokens as instructions for tablet weaving is worth experimenting with quite soon.
Younger children focused on the blocks alone (and of course tried all sorts of things no one else did, like stacking them) but slightly older children worked out they were having an effect on the weaving process and generally could patiently work themselves it out without any explanation required.
Having Anni Albers’ ‘On Weaving’ book next to pattern matrix helped with older visitors, perhaps representing a more conventional and authoritative source of information to introduce the concepts of notation, structure and pattern in weaving.
Pix thanks to James VanderhovenAn 8-way tangible colour switching instruction
Physical vs digital – a false dichotomy
More general concepts that came up in conversation included a common theme during Algomech, exploring the inescapably fuzzy boundaries of concepts such as digital, physical and analogue. The myth of the “real world” being analogue and the “virtual world” being digital is a troublesome one to a weaver.
The ‘anti-device’ effect of the pattern matrix has the potential to explore this conundrum, as it represents a seemingly acceptable demonstration of the physical nature of the digital, and that forms of digital technology have inhabited the world of the reassuringly physical for many millennia of human invention.
Check your supply chains
One aspect of the pattern matrix I picked on for the symposium which came up in the exhibition as well was the fact that the beech wood came from a single tree in Cornwall courtesy of Aaron Moore – and used this as an example of our design philosophy of taking on the myth of collapse rather than the myth of infinite abundance.
Feedback from Penelope team members
Having a 5X5 grid initially was thought to be excessive, as most ancient weaves can be expressed with a 4X4 matrix. This larger capability turns out to be more important than we first thought, as it means you can demonstrate the importance of odd and even numbers in the mathematics of weaving.
There is a problem with the single colour change block due to a faulty use of the code from the old version. This has always been a somewhat temporary feature so we should sort this out properly (e.g. using other shapes for colour across the matrix) before we use it next.
We can also try using an augmented reality approach to show the weave structure directly over the grid, so it’s easier to see how the token block changes relate to specific crossings. This could be displayed alongside the current warp weighted loom rather than as a replacement for it.
A new project begins, on the subject of ecology and evolution of infectious disease. This one is a little different from a lot of Foam Kernow’s citizen science projects in that the subject is theoretical research – and involves mathematical simulations of populations of co-evolving organisms, rather than the direct study of real ones in field sites etc.
The simulation, or model, we are working with is concerned with the co-evolution of parasites and their hosts. Just as in more commonly known simulations of predators and prey, there are complex relationships between hosts and parasites – for example if parasites become too successful and aggressive the hosts start to die out, in turn reducing the parasite populations. Hosts can evolve to resist infection, but this has an overhead that starts to become a disadvantage when most of a population is free of parasites again.
Example evolution processes with different host/parasite trade-offs.
Over time these relationships shift and change, and this happens in different patterns depending on the starting conditions. Little is known about the categorisation of these patterns, or even the range of relationships possible. The models used to simulate them are still a research topic in their own right, so in this project we are hoping to explore different ways people can both control a simulation (perhaps with an element of visual live programming), and also experience the results in a number of ways – via a sonifications, or game world. The eventual, ambitious aim – is to provide a way for people to feedback their discoveries into the research.
The UAV toolkit’s second project phase is now complete, the first development sprint at the start of the year was a bit of research into what we could use an average phone’s sensors for, resulting in a proof of concept remote sensing android app that allowed you to visually program different scripts which we then tested on some drones, a radio controlled plane and a kite.
This time we had a specific focus on environmental agencies, working with Katie Threadgill at the Westcountry Rivers Trust has meant we’ve had to think about how this could be used by real people in an actual setting (farm advisors working with local farmers). Making something cheap, open source and easy to use, yet open ended has been the focus – and we are now looking at providing WRT with a complete toolkit which would comprise a drone (for good weather) a kite (for bad weather/no flight licences required) and an android phone so they don’t need to worry about destroying their own if something goes wrong. Katie has produced this excellent guide on how the app works.
The idea of appropriate technology has become an important philosophy for projects we are developing at Foam Kernow, in conjunction with unlikely connections in livecoding and our wider arts practice. For example the Sonic Bike project – where from the start we restricted the technology so that no ‘cloud’ network connections are required and all the data and hardware required has to fit on the bike – with no data “leaking” out.
With the UAV toolkit the open endedness of providing a visual programming system that works on a touchscreen results in an application that is flexible enough to be used in ways and places we can’t predict. For example in crisis situations, where power, networking or hardware is not available to set up remote sensing devices when you need them most. With the UAV toolkit we are working towards a self contained system, and what I’ve found interesting is how many interface and programming ‘guidelines’ I have to bend to make this possible – open endedness is very much against the grain of contemporary software design philosophy.
The “app ecosystem” is ultimately concerned with elevator pitches – to do one thing, and boil it down to the least actions possible to achieve it. This is not a problem in itself, but the assumption that this is the only philosophy worth consideration is wrong. One experience that comes to mind recently is having to make and upload banner images of an exact size to the Play Store before it would allow me to release an important fix needed for Mongoose 2000, which is only intended to ever have 5 or 6 users.
For the UAV toolkit, our future plans include stitching together photos captured on the phone and producing a single large map without the need to use any other software on a laptop. There are also interesting possibilities regarding distributed networking with bluetooth and similar radio systems – for example sending code to different phones is needed, as currently there is no way to distribute scripts amongst users. This could also be a way of creating distributed processing – controlling one phone in a remote location with another via code sent by adhoc wifi or SMS for example.
Last week the weavecoding group met at Foam Kernow for our Cornish research gathering. As we approach the final stages of the project our discussions turn to publications, and which ideas from the start need revisiting. While they were here, I wanted to give local artists and researchers working with code and textiles a chance to meet Ellen, Emma and Alex. As we are a non-academic research organisation I wanted to avoid the normal powerpoint talks/coffee events and try something more informal and inclusive.
One of the original ideas we had was to combine weaving and coding in a performance setting, to both provide a way to make livecoding more inclusive with weaving, and at the same time to highlight the digital thought processes involved in weaving. Amber made vegetarian sushi for our audience and we set up the Jubilee Warehouse with a collection of experiments from the project:
The newly warped table loom with a live camera/projection from underneath the fabric as it was woven with codes for different weaves on post-it notes for people to try.
The tablet/inkle loom to represent ancient weaving techniques.
The pattern matrix tangible weavecoding machine and Raspberry Pi.
A brand new experiment by Francesca with a dancemat connected to the pattern matrix software for dance code weaving!
The slub livecoding setup.
This provided an opportunity for people to try things out and ask questions/provide discussion starting points. Our audience consisted of craft researchers, anthropological biologists, architects, game designers and technologists – so it all went on quite a lot longer than we anticipated! Alex and I provided some slub livecoded music to weave by, and my favourite part was the live weaving projection – with more projectors we could develop this combination of code and weaving performance more. Thanks to Emma for all the videos and photos!
This autumn we are continuing work on the UAV toolkit with Karen Anderson and her research group at the Environment and Sustainability Institute. This time we have a mission to help the Westcountry Rivers Trust by coming up with fast and cheap ways they can build maps of farms to determine water run-off problems, which gives farmers proof they need to get funding to fix pollution issues.
Flight planning operations and photo checking
In order to make the software usable in this case, we decided on two directions. On the one hand there needs to be a simple way to start and stop programs (or “flight modes”) that read sensor data, as well as defining certain global settings, ie. flight altitude, desired image coverage etc. At the same time, the code to define what this does needs to still be programmable in the app – and more complex behaviours need to be possible to support both kites and UAVs. Our philosophy is that it has to be open ended, as we don’t know where the toolkit it might be useful (ie. crisis mapping situations) or what new sensors will be available on a device in the future.
The new main screen
One specific set of new behaviours we need is for kite mapping. We already have the ability to choose when to take pictures based on GPS and altitude, but with a kite there can be lots of turbulence and the camera is in a much less controlled state, flipping around taking shots of the sky etc. So we need to calculate things like jerk from change in acceleration and use orientation sensors to only take photos when the lens is pointing directly down, within some degree of acceptable margin.
Below is a section of the code that calculates if we are pointing down using the magnetometer and accelerometer – the drag drop visual code can now be used to build normal Scheme functions using a touchscreen (a bit like scheme bricks). In fact I managed to do all of this work on the phone. There are now two types of code, the main programs or “flight modes” that you can run from the front screen, and a library of editable functions which they use. This means there are now three levels that the software can be used – using it without needing to see any of the code at all, editing the basic behaviour like which sensor’s data are captured, and finally modifying the more detailed code to make it do completely new things.
Last week I had the chance to help out the Raspberry Pi foundation at their Picademy in Exeter. It was good to meet up with Sam Aaron again to talk livecoding on Pis, and also see how they run these events. They are designed for local teachers to get more confident with computers, programming and electronics to the point where they can start designing their own teaching materials on the second day of the two day course. This is a model I’m intending to use for the second inset teacher training day I’m doing next week at Truro school – it’s pretty exciting to see the ideas that they have for activities for their pupils, and a good challenge to help find ways to bring them into existence in a day.
We also had the ending of Future Thinking for Social Living at the Miners Court summer party last week. We exhibited the map made during the workshops, made lots of tea, and had some fun with the pattern matrix in musical mode out in the garden – I adapted Alex’s music system we used with Ellen in Munich to run on Raspberry Pi so it didn’t require a laptop, or a screen at all – simply a speaker. It was interesting how quickly people got the idea, in many ways music is easier to explain than weaving as listening while coding is multi-sensory.
Ellen’s exhibition in Munich was always going to be a pivotal event in the weavecoding project – one of the first opportunities to expose our work to a large audience. The Museum of casts of classical sculptures was the perfect context for the mythical aspects of weaving, overlooked by Penelope and friends with her subversive woven/unwoven work, we could explore the connections between livecoding and weaving.
Practically we focused on developing the tangible weavecoding exhibit for events later in the week, as well as discussing the many languages we have developed so far for different looms and weaving techniques. One of our discoveries is that none of the models or languages we have created seem sufficient in themselves – weaving could be far too big to be able to be described or solved from a single perspective. We’ve tried approaches describing weave structures from the actions of the weaver, setup of the loom and structure of the fabric – perhaps the most promising is to explor the story of weaving from the perspective of the thread itself.
One of the distinctive things about weaving in antiquity is how multiple technologies were combined to form a single piece of fabric, weaving in different directions, weft becoming warp, use of tablets vs warp weighted weaving. To explain this via the path of a single conceptual thread crossing through itself may make this possible to describe in a more flexible, declarative and abstracted manner than having to explain each method separately as if in it’s own world.
The pattern matrix has now been made into good shape for explaining the relationship between colour and structure in pattern formation. For the first time we also used all 4 sensors per block on the bottom row which meant we could use a special “colour” block that the system recognises from the normal warp/weft ones and use it’s rotation to choose between 8 preset colour settings. This was quite a breakthrough as it had all been theoretical before.
Adding this more complex use of the magnetic patterns meant that Alex could set up the matrix as a tangible interface for his tidal livecoding software meaning Ellen could join us for a collaborative slub weavecoding performance on the Saturday evening. The prospect of performing together was something we have talked about since the very beginning of the project, so it was great to finally reach this point. The reverb in the museum was vast, meaning that we had to play the space a lot, and provide ‘music for looking at sculptures by’:
Midway through the weavecoding project and our researches have thrown up a whole load of topics that either don’t quite fit into our framework, or we simply won’t have time to pursue properly. Here are some of the tangents I’ve collected so far.
Coding with knots: Khipu
One of the cultures I’m increasingly interested in are the Incas. Their empire flourished to up to 37 million people, without the need of money or a written language. We know that some numeric information was stored using Khipu, a knot based recording system which was used in combination with black and white stones to read and calculate. Two thirds of the quipus we have are un-translated, and do not fit into the known numeric coding system – what information do they hold?
Harvard University provides a Khipu Database Project with many surviving examples documented – I’m hoping to run a workshop soon to look through some of this data in a variety of ways.
Tablet weaving NAND gates
Diagram thanks to Phiala’s String Page – the only place I’ve seen tablet weaving explained properly.
There are logic gates in tablet weaving logic. I haven’t fully figured this out yet, but I noticed modelling tablet weaving that you end up basically mapping the combinations of the weaving actions (such as turn direction) and colour as truth tables.
Top face colour based on top left/top right hole yarn in a single card and turn direction (clockwise/counter clockwise)
TL Yarn : TR Yarn : Turn : Top face colour
--------------------------------------------
Black : Black : CCW : Black
Black : Black : CW : Black
Black : White : CCW : Black
Black : White : CW : White
White : Black : CCW : White
White : Black : CW : Black
White : White : CCW : White
White : White : CW : White
Things get stranger when you include twist and combinations of actions with multiple cards. Would it be possible to compile high level programming languages into weaving instructions for carrying out computation? Perhaps this is what the untranslatable quipus are about?
Asemic writing is a post-literate written form with no semantic content. Miles Visman programs procedural asemic languages and hand weaves them. I think this may be an important connection to livecoding at some point.
