How to design a tangible programming language – Pattern Matrix at Algomech (part 2)

Once we acknowledge that weaving and programming are part of the same technological timeline, we can begin to look at the history of weaving as a eight thousand year long tale of human relationship with digital technologies – and use this long view to research new approaches to software engineering, a field with a much less developed history and many interesting problems to solve.

Using augmented reality to display dynamic information on a tangible programming language.
Using augmented reality to display dynamic information on a tangible programming language.

(Follows Part 1 here.)

One of our threads of investigation is using the pattern matrix as a general purpose tangible programming system – one that we can use for controlling swarms of robots, programming different types of weaving systems and describing complex processes, such as live musical systems.

The magnetic system on the new pattern matrix consists of four hall effect sensors on every location you can place a block. There are four unique ways you can arrange the magnets – which means four types of programming block are possible. As we want to reuse these physical blocks for various uses and programming languages, we decided to use abstract shapes to denote the block types to begin with. Each of the four blocks can be rotated and flipped to give 32 total possible orientations, or symbol ‘tokens':

All combinations of token orientation with four magnets
All combinations of token orientation with four magnets

However, only 16 of these orientations are actually unique. We can only determine flip orientation on the circular block, and only rotation on the rectangular and triangular ones – where flipping them makes no difference to the magnets. The square block is a kind of special one, as we can tell both rotation and flip orientation, so it can represent eight tokens in total all by itself:

All unique tokens and orientations possible. With mid-grey shapes the flip (which side is up) is irrelevant.
All unique tokens and orientations possible. With mid-grey shapes the flip (which side is up) is irrelevant.

It’s important to note at this point that the parallels with tablet weaving are no coincidence: rotating and flipping arrangements of four binary elements for this magnetic system are the same actions as those performed when weaving using tablets. Weaving in the pattern matrix is more than a subject, it’s built directly into it’s mode of operation.

Next we need to test the applicability of this tangible programming system for wider uses. The other cultural phenomena the Penelope project is involved in is livecoding – so is it possible to use the pattern matrix to introduce a weaving centred programming technology in a very different context, not to describe weaving but to generate music in a performance such as an Algorave? This is something that Ellen first pioneered at our weavecoding performance at The Museum of casts of classical sculptures in Munich, but the new pattern matrix has better capabilities for a general purpose programming language.

Having 16 states of four blocks is indeed limiting for a language, but not too limiting. Some types of programming language, such as a string rewriting system like a Lindenmayer system are particularly well suited to this. They are also surprisingly Turing complete languages, able to represent any other programming language in existence, given enough space and time.

Here is a quick example of how this works in text form – a string rewriting system is simply a list of search-replace actions that are carried out in a consistent order. The original example, used to model the growth pattern of algae – consists of a starting string: “A” and two replacement rules, replace “A” with “AB” and replace “B” with “A”. If we run these two rules over and over on the same bit of text we gradually ‘grow’ a pattern like this:


n = 0 : A
n = 1 : AB
n = 2 : ABA
n = 3 : ABAAB
n = 4 : ABAABABA
n = 5 : ABAABABAABAAB
n = 6 : ABAABABAABAABABAABABA
n = 7 : ABAABABAABAABABAABABAABAABABAABAAB

On the pattern matrix we use four of the rows to represent four different rules that replicate in this manner (each made of 5 possible symbols, as it’s a 5×5 grid), which we run 8 times on the starting string (A) to create a musical sequence. Four of the tokens represent these rules (A,B,C and D), the remaining tokens represent musical actions – changes in pitch, rests and sound triggers. There is massive variety of potential patterns, you can control the amount of recursion by the number of rule reference tokens you use – to control the resulting length of the sequences, and thus the complexity of the music. Interestingly we also need a ‘no operation’ (NOP) instruction that does nothing – as in low level assembler languages. We need this as a way to be able to shift timing in the musical sequence by one instruction.

A musical language in 16 instructions.
A musical language in 16 instructions.

With a tangible programming language like this it’s also very important to consider how you categorise instructions by shape – as you can quickly switch between similar operations by simply rotating or flipping tokens, while switching between different shapes takes longer (as you need to pick up a new block) so should represent bigger changes if possible.

Four rules are plenty for generating hugely complex sequences, so we can use the fifth bottom row to control global parameters like scale, synchronisation options (for our slub collaborative sync protocol) or switch between more banks of sounds for greater variety.

Slub performance including the pattern matrix at the Brighton British Science Festival Algorave
Slub performance including the pattern matrix at the Brighton British Science Festival Algorave

The first time we tried this out was at the British Science Festival Algorave in Brighton. A projection was set up with a camera to show the pattern matrix being used, and while technically everything went fine (other than some syncing difficulties), it highlighted a key problem with tangible programming languages. With no dynamic feedback other than the state of the blocks on the pattern matrix, it’s very difficult to tell what is happening during a performance, it’s hard to understand what musically is resulting from the changes you are making.

In order to find a way around this we designed an augmented reality ‘layer’ to place over the pattern matrix, which gives feedback on the currently triggered notes and the paths between the recursive string production rules. We use fluxus and it’s AR feature, which was written by Gabor Papp – which is based on the ARToolkit library. We use a printed out marker to find the plane and camera scale of the centre of the pattern matrix in the image from a USB camera. Once this is done the marker can be removed (as neither the camera or pattern matrix moves) and we can use millimetres as units and place objects over the block locations in 3D space. When the sensors detect a change we can display this new information, also updating the current position in the sequence playback to give feedback on the current sound playing.

Pattern matrix livecoding as part of slub performance at the Algomech Algorave
Pattern matrix livecoding as part of slub performance at the Algomech Algorave. Pix thanks to Dan Hett

As an initial trial the AR improved things when first tried out at the Algomech Algorave in Sheffield, it makes the pattern matrix easier to understand and perform with – and also provides some feedback for the audience in a projection. In a last minute change we switched from Latin characters to Linear A, an undeciphered ancient Greek script – a reference to Flavia’s work on the Penelope project. This is actually preferable to Latin characters as the musical language represents meaning in a way that that actual glyph used is irrelevant – it’s better if it can’t be ‘read’ or confused with another meaning by anyone (still alive).

So it seems that AR could be one of the items in our toolbox for further tangible programming experiments. Perhaps we can better explain the structural changes caused by livecoding the weaving notation for the warp weighted loom by having a dynamic weave structure ‘floating’ on top of the tokens, alongside the loom simulation. This could also be of use for describing tablet weaving actions with these blocks, which would need to be more abstract than the binary weaving notation.

Another area to explore is the design of the blocks themselves, moving away from the abstract shapes, we can design them for specific purposes. Similar to our work on viruscraft, where we have more closely explored the correspondence between physical form (receptors and structural protein arrangements) and tangible interfaces, it seems that these shapes may be worth considering more closely now the sensor matrix is working well.

Weavecoding performance experiments in Cornwall

Last week the weavecoding group met at Foam Kernow for our Cornish research gathering. As we approach the final stages of the project our discussions turn to publications, and which ideas from the start need revisiting. While they were here, I wanted to give local artists and researchers working with code and textiles a chance to meet Ellen, Emma and Alex. As we are a non-academic research organisation I wanted to avoid the normal powerpoint talks/coffee events and try something more informal and inclusive.

IMG_1650

One of the original ideas we had was to combine weaving and coding in a performance setting, to both provide a way to make livecoding more inclusive with weaving, and at the same time to highlight the digital thought processes involved in weaving. Amber made vegetarian sushi for our audience and we set up the Jubilee Warehouse with a collection of experiments from the project:

  • The newly warped table loom with a live camera/projection from underneath the fabric as it was woven with codes for different weaves on post-it notes for people to try.
  • The tablet/inkle loom to represent ancient weaving techniques.
  • The pattern matrix tangible weavecoding machine and Raspberry Pi.
  • A brand new experiment by Francesca with a dancemat connected to the pattern matrix software for dance code weaving!
  • The slub livecoding setup.

IMG_1634

This provided an opportunity for people to try things out and ask questions/provide discussion starting points. Our audience consisted of craft researchers, anthropological biologists, architects, game designers and technologists – so it all went on quite a lot longer than we anticipated! Alex and I provided some slub livecoded music to weave by, and my favourite part was the live weaving projection – with more projectors we could develop this combination of code and weaving performance more. Thanks to Emma for all the videos and photos!

IMG_1692

IMG_1564

Weavecoding Munich

Ellen’s exhibition in Munich was always going to be a pivotal event in the weavecoding project – one of the first opportunities to expose our work to a large audience. The Museum of casts of classical sculptures was the perfect context for the mythical aspects of weaving, overlooked by Penelope and friends with her subversive woven/unwoven work, we could explore the connections between livecoding and weaving.

IMG_8477 2

Practically we focused on developing the tangible weavecoding exhibit for events later in the week, as well as discussing the many languages we have developed so far for different looms and weaving techniques. One of our discoveries is that none of the models or languages we have created seem sufficient in themselves – weaving could be far too big to be able to be described or solved from a single perspective. We’ve tried approaches describing weave structures from the actions of the weaver, setup of the loom and structure of the fabric – perhaps the most promising is to explor the story of weaving from the perspective of the thread itself.

IMG_20150510_062737

IMG_20150508_153211

One of the distinctive things about weaving in antiquity is how multiple technologies were combined to form a single piece of fabric, weaving in different directions, weft becoming warp, use of tablets vs warp weighted weaving. To explain this via the path of a single conceptual thread crossing through itself may make this possible to describe in a more flexible, declarative and abstracted manner than having to explain each method separately as if in it’s own world.

IMG_20150508_152045

IMG_20150509_100421

The pattern matrix has now been made into good shape for explaining the relationship between colour and structure in pattern formation. For the first time we also used all 4 sensors per block on the bottom row which meant we could use a special “colour” block that the system recognises from the normal warp/weft ones and use it’s rotation to choose between 8 preset colour settings. This was quite a breakthrough as it had all been theoretical before.

IMG_20150508_194220

Adding this more complex use of the magnetic patterns meant that Alex could set up the matrix as a tangible interface for his tidal livecoding software meaning Ellen could join us for a collaborative slub weavecoding performance on the Saturday evening. The prospect of performing together was something we have talked about since the very beginning of the project, so it was great to finally reach this point. The reverb in the museum was vast, meaning that we had to play the space a lot, and provide ‘music for looking at sculptures by':

slub at Kunsthal Aarhus

Last week Alex and I took to the road on another slub mini-tour starting in Denmark at the Kunsthal Aarhus where we ran a livecoding workshop and performed at the opening of the Aarhus Filmfestival.

IMG_6390

IMG_20131112_132248

The Kunsthal gallery was exhbiting “Systemics #2: As we may think (or, the next world library)” with work by Florian Hecker, Linda Hilfling, Jakob Jakobsen, Suzanne Treister, Ubermorgen, YoHa + Matthew Fuller.

Linda Hilfling and UBERMORGEN’s work comprised an Amazon print on demand hack which was perhaps an even more elaborate version of their previous Google Will Eat Itself. The gallery floor was printed with a schematic describing the processing from the raw material input to the finished printed books.

IMG_20131111_120659

Suzanne Treister’s work called HEXEN 2.0 included alternative/hidden histories of technology presented as densely descriptive tarot cards and prints showing many connections between individuals, events and inventions.

IMG_20131112_174729

Dagstuhl – Collaboration and learning through live coding

Dagstuhl seminars are week long free form meetings between different disciplines centred around computer science. The location is a specially designed complex in the German countryside, and activities include long walks in the surrounding hills, a well equipped and beautiful music room and a well stocked wine cellar.

Our seminar was called ‘Collaboration and learning through live coding’, organised by Alan Blackwell, Alex McLean, James Noble and Julian Rohrhuber and included people from the fields of Software Engineering, Computer Science Education as well as plenty of practising livecoders and multidisciplinary researchers.

IMG_20130919_104725

Discussion was wide ranging and intense at times, and the first job was to sufficiently explain what livecoding actually was – which turned out to require performances in different settings:

1. Explanatory demo style livecoding: talking through it as you do it.
2. Meeting room coffee break gigs: with a closely attentive audience.
3. The music room: relaxed evening events with beer and wine.

So Dagstuhl’s music room was immediately useful in providing a more ‘normal’ livecoding situation. It was of course more stressful than usual, knowing that you were being critically appraised in this way by world experts in related fields! However it paid off hugely as we had some wonderful interpretations from these different viewpoints.

One of the most important for me was the framing of livecoding in terms of the roots of software engineering. Robert Biddle, Professor of Human-Computer Interaction at Carleton University put it into context for us. In 1968 NATO held a ‘Software Components Conference’ in order to tackle a perceived gap in programming expertise with the Soviet Union.

Software_components_lecture_large

This conference (attended my many of the ‘big names’ of programming in later years) led to many patterns of thought that pervade the design of computers and software – a tendency for deeply hierarchical command structures in order to keep control of the arising complexity, and a distrust of more adhoc solutions or any hint of making things up as we go along. In more recent times we can see a fight against this in the rise of Agile programming methodologies, and it was interesting to look at livecoding as a part of this story too. For example it provides a way to accept and demonstrate the ‘power to think and feel’ that programming give us as humans. The big question is accessibility, in a ubiquitously computational world – how can this reach wider groups of people?

IMG_20130918_025213

Ellen Harlizius-Kl├╝ck works with three different domains simultaneously – investigating the history of mathematics via weaving in ancient Greece. Her work includes livecoding, using weaving as a performance tool – demonstrating the algorithmic potential of looms and combinations of patterns. Her work exposes the hidden shared history of textiles and computation, and this made a lot of sense to me as at the lowest level the operations of computers are not singular 0’s and 1’s as is often talked about, but actually in terms of transformations of whole patterns of bits.

Mark Guzdial was examining livecoding through the lens of education, specifically teaching computer science. The fact that so many of us involved in the field are also teaching in schools – and already looking at ways of bringing livecoding into this area, is noteworthy, as is the educational potential of doing livecoding in nightclub type environments. Although here it works more on the level of showing people that humans make code, it’s not a matter of pure mathematical black boxes – that can be the ground breaking realisation for a lot of people.

IMG_20130917_095349

Something that was interesting to me was to concentrate on livecoding as a specifically musical practice (rather than also a visual one) as there are many things about perceiving the process with a different sense from your description of it that are important. Julian Rohrhuber pointed out that “you can use sound in order to hear what you are doing” – the sound is the temporal execution of the code – and can be a close representation of what the computer is actually doing. This time based approach is also part of livecoding working against the notion that producing an ‘end result’ is important, Juan A. Romero said that “if you’re livecoding, you’re not just coding the final note” – i.e. the process of coding is the artform.

IMG_20130917_092532

In terms of a school teaching situation sound is also powerful, as described by Sam Aaron, livecoder and creator of Sonic Pi. A child getting a music program to work for the first time in a classroom is immediately obvious to everyone else – as it is broadcast as sound, inspiring a bit of competition and ending up with a naturally collaborative learning experience.

It’s impossible to cover all the discussions that we had, these are just the ones I happened to get down in my notebook, but it was a great opportunity to examine what livecoding is about now in relation to other practices, where it came from and where it might go in the future.

IMG_20130918_220833

London Algorave at nnnnn

In order to get ourselves prepared for the Dagstuhl livecoding seminar (more on that later), we kicked off with a London Algorave at nnnnn, Ryan Jordan’s noise research laboratory in deepest Hackney. Slub had one of our better performances, which was recorded – watch this space.

*UPDATE*

IMG_20130914_172033

Larger components make larger sounds.

IMG_20130914_220252

Massive synth washes and brutal beats from the rock star livecoders Meta-Ex.

IMG_20130914_220827

Meta-Ex close up.

IMG_20130914_235502

Yee-King’s brand new visual acid generating machine reconfigured our minds.

Slub at the Deershed festival

Deershed is a music festival designed to accommodate families with lots of activities for children. Part of this year’s festival was a Machines Tent, including Lego robot building, Mechano constructions, 3D printing and computer games.

Slub’s daily routine in the Machines Tent started by setting up the Al Jazari gamepad livecoding installation, a couple of hours with Martyn Eggleton teaching Scratch programming on an amazing quad Raspberry Pi machine (screens/processors and keyboards all built into a welded cube).

IMG_20130720_102902

At some point we would switch to Minecraft, trying some experiments livecoding the LAN game world using Martyn’s system to access the Minecraft API using Waterbear, a visual programming language using a similar blocks approach as Scratch and Scheme Bricks.

During the afternoons Alex and I could try some music livecoding experiments. This was a great environment for playful audience participatory performances, with families continually passing through the tent I could use a dancemat to trigger synths in fluxus while Alex livecoded music designed to encourage people to jump up and down.

IMG_20130720_150827

One of the most interesting things for me was to be able to see how lots of children (who mostly didn’t know each other) collaborate and self organise themselves in a LAN game, there was quite a pattern to it with all the groups:

  1. Mess around with Minecraft as usual (make some blocks, start building a house).
  2. Find something built by someone else, destroy a few bricks.
  3. Snap out of the game to notice that the other kids are complaining.
  4. Realise that there are other people in the world – and they are sat around them!
  5. Attempt to fix the damage.

At this point other people would join in to help fix things, after which there would be some kind of understanding reached between them to respect each other’s creations. This has all really inspired me to work on Al Jazari 2 which combines a lot of these ideas.

IMG_20130720_084658

Deershed Festival, Sonic Bike Lab, Fascinate Festival

Preparations for a busy summer, new Al Jazari installation gamepads on the production line:

IMG_20130715_154549

This weekend Alex and I are off to the Deershed Festival in Yorkshire to bring slub technology to the younger generation. We’ll be livecoding algorave, teaching scratch programming on Raspberry Pis and running an Al Jazari installation in between. Then onwards to London for a Sonic Bike Lab with Kaffe Matthews where we’re going to investigate the future of sonic bike technology and theory – including possibly, bike sensor driven synthesis and on the road post-apocalyptic mesh networking.

At the end of August I’m participating in my local media arts festival – Fascinate in Falmouth, where I’ll be dispensing a dose of algorave and probably even more musical robot techno.