Neural Network livecoding and retrofitting ZX Spectrum hardware

An experimental, and quite angry neural network livecoding synth (with an audio ‘weave’ visualisation) for the ZX Spectrum: source code and TZX file (for emulators). It’s a bit hard to make out in the video, but you can move around the 48 neurons and modify their synapses and trigger levels. There are two clock inputs and the audio output is the purple neuron at the bottom right. It allows recurrent loops as a form of memory, and some quite strange things are possible. The keys are:

  • w,d: move diagonally north west <-> south east
  • s,e: move diagonally south west <-> north east
  • t,y,g,h: toggle incoming synapse connections for the current neuron
  • space: change the ‘threshold’ of the current neuron (bit shifts left)

This audio should load up on a real ZX Spectrum:

One of the nice things about tech like this is that it’s easily hackable – this is a modification to the video output better explained here, but you can get a standard analogue video signal by connecting the internal feed directly to the plug and detaching the TV signal de-modulator with a tiny bit of soldering. Look at all those discrete components!

IMG_20140809_125815

Thinking Digital 2014

Last week I had the honour of both performing with Alex and presenting at Thinking Digital 2014. Suzy O’Hara invited me to represent the intersection of art, science and education of FoAM kernow and present the work I’ve been doing with the Sensory Ecology Group at Exeter University. I did a quick Egglab game demo and related some thoughts on working with scientists and how it connects with my experience teaching programming in the classroom.

It was an interesting and unusual venue for me, organiser Herb Kim is very much developing on the TED theme – so lots of extremely well considered, motivating and inspiring talks. Much of the context was one of venture capital and startup business so it was interesting to see an explosive talk by Aral Balkan on the implications of Facebook and Google’s business models on the future of our society (he included some of the other presenter’s companies too). This reminded me very much of the themes we explored in Naked on Pluto, but coming from a new angle.

Personally his talk was challenging to me as he roundly attacked the free software movement, for essentially providing a great sandbox for enthusiasts and well funded companies – but incapable of doing much more in terms of data security for real people. As a designer, he sees this as essentially a design problem – one that these companies have solved for themselves but is utterly lacking in devices such as the Firefox phone OS. For Aral, this is fundamentally a design problem that needs it’s own movement, and new business models to be developed. These business models need to take into consideration long term usability (for which user privacy is an essential feature) rather than ultra short term profit ‘pump and dump’, selling of people’s information for vast amounts – i.e. silicon valley ‘business as usual’.

Two things are apparent to me following this talk – one is that I have been labouring under the impression that a particular focus on design is somehow implicitly tied with specific business practices – simplification as wallpapering over data harvesting, and other tactics. This is very much a short sighted developer view, and is wrong – they can of course service different types of businesses.

The other point came during his 3 slide explanation of how to start your own social network (1. fork a github repo, 2. set up a server and 3. install it). Clearly even this satirical simplification is beyond all but existing software developers (many of whom are working for companies reliant on user surveillance in some indirect or direct way). The challenge for me is that I can’t ultimately see a way to make ‘interface as user experience’ ever converge on anything other than exploitation. Can ‘user experience’ ever regard people philosophically as anything but consumers – regardless of the underlying business model?

The problem in solving that is that we now have two problems – the terrible state of software engineering preventing accessibility (i.e programming at large still stuck in the 70’s) and the lack of understanding in society of what a computer is and how it works. The second of these problems is being addressed in some part by the activities of CodeClub (Aral is [correction: was] a director of this organisation) and similar education initiatives. Regarding pushing software engineering forward, in some way I think recent livecoding takeup by musicians over programmers is a fascinating development here, in terms of showing us how programming – when it’s taken and twisted into very strange and new forms, can start to make sense and work for ‘real people’.

Thoughts on teaching programming with Minecraft and Python

Saturday saw the first dBsCode taster workshop, for budding programmers between 11 and 16. We set up 20 Raspberry Pi’s, which we networked together and used our new procedural Minecraft 3D shape primitives to build a number of projects in Python involving castles, spiders and an infinite house generator.

dbsmcpi-castle

dbscode

Networking was very important – it enabled them to jump into each other’s games which started as a major distraction, but ended up being useful – as people could see what each other were doing and work together.

As part of this, some level of ‘griefing’ was present (where other players interfere in your world), but asking them about this during the breaks, the consensus was that this is part of the culture of Minecraft – I even instructed victims to pull their network cables out, but they saw that as completely unnecessary. Most of them took turns in both constructive/collaborative and programming activity and destructive/graffiti like activities associated with griefing, and this was acceptable to them. The use of code to easily rebuild structure after damage (or being able to remove people with liberal procedural application of lava) was therefore quite an attractive feature of the code approach.

It was common to adapt the programming to fit their understanding of Minecraft, so it seemed natural to take on a hybrid approach to building – using Python to do the heavy lifting (building, or extracting large areas) or repetitive tasks, after which they did the fiddly or more creative bits manually as normal. For example one student used the sphere primitive to extract huge caves underground, and then filled them with little hand made shrines and sculptures.

What was interesting was that to some extent the ‘experts’ at Minecraft (many of whom already document their creations on Youtube channels) found this more challenging in many ways than those less attached to the Minecraft culture – who were more accepting of new ways of doing things.

The manner of programming we used here – running Geany (a lightweight Python IDE) at the same time as Minecraft, was completely in line with livecoding practice, as programs interacted with the Minecraft world in realtime with network messages. The great thing about the rather slow network bottleneck is that you can walk around a structure while the program is creating it, which allows a much better understanding of how the process is working than if it instantly popped into existence.

More procedural Minecraft architecture

More work on the Python teaching environment we’ll be using next week for the Minecraft workshop at dBsMusic. I’m working on various ideas for procedural architecture, using this as a way to demonstrate what programming is about – thinking procedurally/functionally. The code is online here – I’ll be adding some exercises and course materials over the next few days.

Edit: Documentation on how to use, also install.sh should work on Raspberry Pi. Exercises to follow.

dbsmcpi-prims

dbsmcpi-houses2

Many languages: Düsseldorf Institute for Music and Media Seminar

Last week was my first official seminar at the Düsseldorf Institute for Music and Media with Julian Rohrhuber filling my new role as Associate Professor in Critical Programming. I wanted to start by introducing the cultures and history of programming, with a focus on the people who invented programming languages and what they were doing if for – from the early mathematicians to the military/space industry and in more recent times the rise of JavaScript from a language that would only ever be used for “animating buttons” to the language with widest reach.

pf

With that in mind I wanted to try teaching a fluxus workshop using planet fluxus, the version that compiles Scheme to JavaScript rather than the native code version. This is now working in a new url with quite a lot of fixes and now quite a lot of testing carried out on it. I’m pretty pleased with the support for webgl – and plan to use it for some upcoming games, other than needing switching on with some versions of Safari, it otherwise seems pretty widespread and fast.

My second day of teaching was followed by a presentation by Ellen Harlizius-Klück and Alex McLean on weaving, ancient mathematics, programming, mythology and music – which provided a great introduction for a meeting we had the next day on an upcoming project bringing these concepts together.

IMG_20140206_165357

Cornwall “Let’s Get Digital” presentation

Here’s a presentation I gave at the end of last year at a Creative Skills Cornwall meeting at Falmouth University. I introduced the problems of a growing producer/consumer digital divide – the need for more public discourse in the politics of technology and how free software, codeclub, livecoding, algorithmic weaving and sonic bikes can indicate other relationships we can have with technology.

The talk went down really well, but the slides are a little minimal so it might not be super clear what it was all about based just on them 🙂

slide

Raspberry Pi: Built for graphics livecoding

I’m working on a top secret project for Sam Aaron of Meta-eX fame involving the Raspberry Pi, and at the same time thinking of my upcoming CodeClub lessons this term – we have a bunch of new Raspberry Pi’s to use and the kids are at the point where they want to move on from Scratch.

This is a screenshot of the same procedural landscape demo previously running on Android/OUYA running on the Raspberry Pi, with mangled texture colours and a cube added via a new livecoding repl:

IMG_20140108_232857

Based on my previous experiments, this program uses the GPU for the Raspberry Pi (the VideoCore IV bit of the BCM2835). It’s fast, allows compositing on top of whatever else you are running at the time, and you can run it without X windows for more CPU and memory, sounds like a great graphics livecoding GPU to me!

Here’s a close up of the nice dithering on the texture – not sure yet why the colours are so different from the OUYA version, perhaps a dodgy blend mode or a PNG format reading difference:

IMG_20140108_232914

The code is here (bit of a mess, I’m in the process of cleaning it all up). You can build in the jni folder by calling “scons TARGET=RPI”. This is another attempt – looks like my objects are inside out:

IMG_20140109_004111

slub at Kunsthal Aarhus

Last week Alex and I took to the road on another slub mini-tour starting in Denmark at the Kunsthal Aarhus where we ran a livecoding workshop and performed at the opening of the Aarhus Filmfestival.

IMG_6390

IMG_20131112_132248

The Kunsthal gallery was exhbiting “Systemics #2: As we may think (or, the next world library)” with work by Florian Hecker, Linda Hilfling, Jakob Jakobsen, Suzanne Treister, Ubermorgen, YoHa + Matthew Fuller.

Linda Hilfling and UBERMORGEN’s work comprised an Amazon print on demand hack which was perhaps an even more elaborate version of their previous Google Will Eat Itself. The gallery floor was printed with a schematic describing the processing from the raw material input to the finished printed books.

IMG_20131111_120659

Suzanne Treister’s work called HEXEN 2.0 included alternative/hidden histories of technology presented as densely descriptive tarot cards and prints showing many connections between individuals, events and inventions.

IMG_20131112_174729

Dagstuhl – Collaboration and learning through live coding

Dagstuhl seminars are week long free form meetings between different disciplines centred around computer science. The location is a specially designed complex in the German countryside, and activities include long walks in the surrounding hills, a well equipped and beautiful music room and a well stocked wine cellar.

Our seminar was called ‘Collaboration and learning through live coding’, organised by Alan Blackwell, Alex McLean, James Noble and Julian Rohrhuber and included people from the fields of Software Engineering, Computer Science Education as well as plenty of practising livecoders and multidisciplinary researchers.

IMG_20130919_104725

Discussion was wide ranging and intense at times, and the first job was to sufficiently explain what livecoding actually was – which turned out to require performances in different settings:

1. Explanatory demo style livecoding: talking through it as you do it.
2. Meeting room coffee break gigs: with a closely attentive audience.
3. The music room: relaxed evening events with beer and wine.

So Dagstuhl’s music room was immediately useful in providing a more ‘normal’ livecoding situation. It was of course more stressful than usual, knowing that you were being critically appraised in this way by world experts in related fields! However it paid off hugely as we had some wonderful interpretations from these different viewpoints.

One of the most important for me was the framing of livecoding in terms of the roots of software engineering. Robert Biddle, Professor of Human-Computer Interaction at Carleton University put it into context for us. In 1968 NATO held a ‘Software Components Conference’ in order to tackle a perceived gap in programming expertise with the Soviet Union.

Software_components_lecture_large

This conference (attended my many of the ‘big names’ of programming in later years) led to many patterns of thought that pervade the design of computers and software – a tendency for deeply hierarchical command structures in order to keep control of the arising complexity, and a distrust of more adhoc solutions or any hint of making things up as we go along. In more recent times we can see a fight against this in the rise of Agile programming methodologies, and it was interesting to look at livecoding as a part of this story too. For example it provides a way to accept and demonstrate the ‘power to think and feel’ that programming give us as humans. The big question is accessibility, in a ubiquitously computational world – how can this reach wider groups of people?

IMG_20130918_025213

Ellen Harlizius-Klück works with three different domains simultaneously – investigating the history of mathematics via weaving in ancient Greece. Her work includes livecoding, using weaving as a performance tool – demonstrating the algorithmic potential of looms and combinations of patterns. Her work exposes the hidden shared history of textiles and computation, and this made a lot of sense to me as at the lowest level the operations of computers are not singular 0’s and 1’s as is often talked about, but actually in terms of transformations of whole patterns of bits.

Mark Guzdial was examining livecoding through the lens of education, specifically teaching computer science. The fact that so many of us involved in the field are also teaching in schools – and already looking at ways of bringing livecoding into this area, is noteworthy, as is the educational potential of doing livecoding in nightclub type environments. Although here it works more on the level of showing people that humans make code, it’s not a matter of pure mathematical black boxes – that can be the ground breaking realisation for a lot of people.

IMG_20130917_095349

Something that was interesting to me was to concentrate on livecoding as a specifically musical practice (rather than also a visual one) as there are many things about perceiving the process with a different sense from your description of it that are important. Julian Rohrhuber pointed out that “you can use sound in order to hear what you are doing” – the sound is the temporal execution of the code – and can be a close representation of what the computer is actually doing. This time based approach is also part of livecoding working against the notion that producing an ‘end result’ is important, Juan A. Romero said that “if you’re livecoding, you’re not just coding the final note” – i.e. the process of coding is the artform.

IMG_20130917_092532

In terms of a school teaching situation sound is also powerful, as described by Sam Aaron, livecoder and creator of Sonic Pi. A child getting a music program to work for the first time in a classroom is immediately obvious to everyone else – as it is broadcast as sound, inspiring a bit of competition and ending up with a naturally collaborative learning experience.

It’s impossible to cover all the discussions that we had, these are just the ones I happened to get down in my notebook, but it was a great opportunity to examine what livecoding is about now in relation to other practices, where it came from and where it might go in the future.

IMG_20130918_220833