Users > “drivers” of software

I’ve finally had a chance to sit down and comment on John Naughton’s article in the Guardian and it’s manifesto addressed to the education secretary The Rt Hon Michael Gove MP on computer education in the UK. This joins an avalanche of recognition that programming – or “coding” – is suddenly a Good Thing for People To Know.

It is wonderful to read a major media publication pouring scorn on the old idea that “computers are like cars”, “users are drivers” which has dogged perceptions of computers for years. There is a new philosophy here focusing on hacking, in the original sense – an algorithmic literacy of processes which now mediate every aspect of our lives. This is seen as good as a general life skill, and of course good for the economy for encouraging the kind of independent thinking needed for successful startup companies.

This avalanche has to be credited to some extent to game designer David Braben and his Raspberry Pi project, and an extensive PR campaign tweaking people’s memories (including an increasing number of politicians in their 30’s and 40’s) who remember school computers like the BBC micro, and their later influence on what these same politicians like to call the “creative industries”. This of course all seems instinctively good sense for those of us who have been closely watching the boom in popularity of arduino, processing and free software methodologies in hacklabs and fablabs.

However, an approach organised on this scale is unlikely to support such generalist and creative philosophies we are used to. A few days prior to this article we had an announcement of £575m for kitting out schools with computing infrastructure from familiar public sector contractors including Serco and Capita, and a bit more detail on Staffordshire county council who are spending £28m on “Apple products under the iOS lot including iMacs, Mac Books, iPods, iPads, Mac Minis and Lion Server.”

The problem here is that a rejection of “users as drivers” is a rejection of iOS (and to a lesser extent Android) and the app store philosophy. App stores are extremely successful at promoting the idea of phones as appliances (never computers in their own right) and software as small digestible “apps” encapsulated in locked down environments generally marketed as a kind of protection for users. If these models of computing are to grow as expected – they are completely at odds with an accessible understanding we need for this change in education approach, and the creative literacy of algorithms which would follow.

When I’ve interviewed graduates for creative programming jobs the thing which is really most valuable (much more so than knowing the relevant programming language) is exactly the thing that having an “edit source code” button on every app would encourage (as included on OLPC’s sugar, another older example of an education targeted effort). What is needed is a creative lack of respect for software, a cheerful abandonment of the fear that “breaking something” will be your fault rather than that of the system.

Jellyfish forms

The Jellyfish VM is now running in fluxus on android – a kind of executable rendering primitive which defines it’s form (a 3D triangle list) in the same place as it’s behaviour (a vector processing program), in the primitive’s data array.

This provides a 3D procedural modelling VM loosely inspired by the Playstation 2 which is much faster than interpreting scheme (or Java) on ARM devices like android. There are also possibilities for genetic programming or touch based programming interfaces just like Betablocker DS (think 3D animation rather than music).

Triangles are written to by de-referencing a register that defines the start of the model data, which exists in program memory. It can be programmed with raw vectors for each instruction, and right now the VM is run for 10 cycles for each frame, although this will be configurable “in the future”. Here is a simple program that was used to make the screenshots:

(define jelly (build-jellyfish))

(with-primitive jelly
 (program-jellyfish
  (list
 ; data
   (vector 0 0 0)         ; time (increases by 1 each loop)
   (vector 2 2 -3)        ; shuffle data for converting (x y z) -> (z z x)
 ; code follows to build a vertex by rotation around an angle based on time
   (vector LDA 0 0)       ; load current time from address 0
   (vector LDL 135.3 0)   ; load value 135.3 (angle in degrees)
   (vector MUL 0 0)       ; multiply time by angle
   (vector SIN 0 0)       ; makes (sin(angle) cos(angle) 0)
 ; make a spiral by scaling up with time
   (vector LDA 0 0)       ; load time again
   (vector LDL 0.05 0)    ; load 0.05
   (vector MUL 0 0)       ; multiply to get time*0.05
   (vector MUL 0 0)       ; mul rotation vector by time*0.05
 ; move backward in z so we get some depth
   (vector LDA 0 0)       ; load the time again
   (vector LDL 0.03)      ; load 0.03
   (vector MUL 0 0)       ; multiply the time by 0.01
   (vector LDA 1 0)       ; load the shuffle vec from address 1 
   (vector SHF 0 0)       ; shuffle the x to z position
   (vector ADD 0 0)       ; add (0 0 x) to set z on current position
   (vector STI 0 REG_MDL) ; write position to model memory registers
 ; increment the index by 1
   (vector LDA 0 0)       ; load address
   (vector LDL 1 0)       ; load inc
   (vector ADD 0 0)       ; add them together
   (vector STA 0 0)       ; store at address loc
   (vector JMP 2 0))))    ; goto 2

Jellyfish

Lots of travelling lately and for me, time in airports and long train journeys are time for acts of random research. One of the recent fruits is a new imaginary processor (code name Jellyfish), largely based on Betablocker DS but with ideas stolen from the PS2 vector unit.

Like the PS2 VU it’s a vector processor, but each instruction is 96 bits wide (unlike the PS2 which is 128), enough to store 3 32bit values representing 3D vectors or colours. Like other Betablocker variants every combination of bytes are valid as a runnable program, so it will for example happily execute random junk.

There are currently 25 instructions, with vector operations like dot, cross product and length, and various instructions for reordering vectors. It’s targeted at running on ARM processors so it uses fixed point maths for battery friendliness.

This is an ascii-shot of the super exciting debugger running a similarly exciting program to rotate a vector around the z axis.

-- prog -- pc:11
004   sta 13.000000 0.000000
005   sin 0.000000 0.000000
006   sta 14.000000 0.000000
007   lda 14.000000 0.000000
008   lda 15.000000 0.000000
009   crs 0.000000 0.000000
010   sta 16.000000 0.000000
011 > jmp 0.000000 0.000000
012   0.000000 0.000000 0.000000
013   60.000000 0.000000 0.000000
014   -0.125977 -0.992020 0.000000
015   0.000000 1.000000 0.000000
016   0.000000 0.000000 -0.125977
017   0.000000 0.000000 0.000000
-- stk -- stk:0
  0.000000 0.000000 0.000000 : 199
  0.000000 0.000000 0.000000 : 200
> 0.000000 0.000000 -0.125977 : 201
  0.000000 1.000000 0.000000 : 202

Scheme Bricks on Android


A screenshot of scheme bricks running on android. Its actually the next version of scheme bricks really, as it’s been subject to a major rewrite. The original was pretty cluttered with lots of last minute pre-going-on-stage hacks and additions. As in the original, the touchscreen expression selection works by ray casting into the 3D scene with (geo/line-intersect) which is now part of android fluxus, but no dragging or dropping – the fun bit – or even fonts yet.

Still it’s somewhat satisfying to work on fluxus scheme code that runs on Linux, Mac, Windows, Android and PS2!

Pink polygons & multitouch

A new version of android fluxus, with pdata, multitouch capability, scene inspection and a very pink test script included (see code below) fluxus-0.0.2.apk & source. Press the trackball button to edit the scheme code.

(clear)

(define twirl-shape 
  (with-state
   (hint-unlit)
   (build-polygons 40 triangle-strip)))

(define finger-shapes
  (with-state
   (hint-none)
   (hint-unlit)
   (hint-wire)
   (list (build-polygons 30 triangle-strip)
         (build-polygons 30 triangle-strip))))

(define (spiral)
  (line-width 5)
  (pdata-map! (lambda (c) (vector 1 0.29 0.42)) "c")
  (pdata-index-map!
   (lambda (i p)
     (let ((i (* i 0.8)))
       (vmul (vector (sin i) (cos i) 0) (* 0.02 i i))))
   "p")
  (pdata-copy "p" "pref") ; only really needed for animation
  (pdata-map!
   (lambda (p pref)
     (vadd pref (vmul (crndvec) 0.2)))
   "p" "pref"))

(for-each
 (lambda (finger-shape)
   (with-primitive finger-shape (spiral)))
 finger-shapes)

(with-primitive 
 twirl-shape
 (pdata-map!
  (lambda (c)
    (vector 1 0.29 0.42))
  "c"))

; using with-primitive is really slow, so directly use grab
; returns the distance between the objects
(define (get-pinch)
  (grab (car finger-shapes))
  (let ((a (vtransform (vector 0 0 0) (get-transform))))
    (ungrab)(grab (cadr finger-shapes))
    (let ((b (vtransform (vector 0 0 0) (get-transform))))
      (ungrab)
      (vdist a b))))

; store pinch as it's slow to calculate
(define pinch 1)

(every-frame
 (begin
   ; do the twirling
   (with-primitive
    twirl-shape
    (pdata-index-map!
     (lambda (i p)
       (let ((i (* i 0.5)))
         (vmul (vector (sin i) (cos i) (* 2 (cos (* i 10.43)))) 
               (* 5 (sin (* pinch (+ i (* 0.1 (time))) 0.1))))))
     "p"))
   ; check for touch events
   (for-each
    (lambda (touch-id)
      (let ((finger-shape (list-ref finger-shapes touch-id)))
        (with-primitive 
         finger-shape
         (identity)
         (translate 
          (get-point-from-touch touch-id))
         (rotate (vector 0 0 (* (time) 10)))))
      (set! pinch (get-pinch)))
    (get-touch-ids))))

More android tinkering

In order to figure out the source of the crashes I was getting I quickly wrote a linux wrapper around the android code so it could run in a simpler-to-debug environment. This of course was running without any problems, so not much help, but it’ll come in useful anyway.

Next step, try it in the emulator – this exhibited the crash, but I couldn’t get a good stack trace from gdb. While investigating this I decided I could use a simpler method with the hardware, which is much faster than the emulator. addr2line is a program you can use for making sense out of crash reports, you take the address reported in the crash log (using adb logcat to read them from the android device), feed it to addr2line with the library and it tells you the corresponding line number in the source.

It turned out the crashes were down to something more fundamental as they were happening all over the place in the scheme interpreter – so I had a bit of a wider search and it came down to threads, I hadn’t noticed that Android’s GLSurfaceView uses a separate thread for rendering, so I was calling the scheme interpreter in the midst of other evaluations, not too great. A simple fix using java locks around the evaluations and rendering calls.

The next thing was implementing more fluxus features. One major improvement is using with-state to wrap push and pop so they are implicitly scoped. This means you can’t miss (or add extra) a pops or pushes to your state stack manipulations, so something like:

(push) 
(translate (vector 1 2 3)) 
(build-cube) 
(pop)

becomes:

(with-state 
    (translate (vector 1 2 3)) 
    (build-cube))

To do this I had to learn TinyScheme’s macro language which is different to Racket’s, but this isn’t too complex a task:

(define-macro (with-state . args) 
  `(begin (push) (let ((r (begin ,@args))) (pop) r)))

Here we call push then basically “paste” the code given to the with-state function into a begin block, and record it’s return before calling pop. Then we return the value. There is a very similar macro for with-primitive.

There are now quite a few commands ported – there is a new version packaged here, and this is the test script that I’m using:

(clear)
(colour (vector 0 0.5 1 1))
(define cubes
  (map 
   (lambda (p)
     (with-primitive p
       (apply-transform)
       (translate (vector -2.5 2 5))
       (rotate (vector 45 0 0))
       (translate 
        (vector
         (quotient (- p 1) 6)
         0
         (modulo (- p 1) 6)))
       p))
   (build-list
    (lambda (n) 
      (with-state
       (scale (vector 0.5 0.05 0.5))
       (build-cube)))
    36)))

(every-frame
 (for-each 
  (lambda (p)
    (with-primitive 
     p
     (rotate (vmul (vector (sin p) 0 (cos p)) 5))))
  cubes))

Shaving yaks on android

fluxus installed on the emulator

Mostly my android experience has been good so far, it’s been very quick to get things running and be fairly productive. It doesn’t come without it’s share of yak shaving though. I’ve spent quite a lot of time trying to get remote debugging of native code working with my phone (a HTC desire) with little success. This seems to be a fairly well documented problem, the symptoms are messages like this when running ndk-gdb:

ERROR: Could not setup network redirection to gdbserver?
Maybe using –port=
to use a different TCP port might help?

When run with –verbose it seems something a little different is wrong:

run-as: Package ‘am.fo.nebogeo.fluxus’ has corrupt installation

Which looks like some result of having a rooted phone (which I do) as ndk-debug is a possible attack vector for doing nasty things and is therefore very critical of the permissions of the directories on the system. In order to fix this I installed ROM Manager from the market which contains a script to fix permissions. This didn’t work at first by pressing the button in the app so after some poking around I found the shell script in: /data/data/com.koushikdutta.rommanager/files/fix_permissions

It turned out that this script depended on an interesting utility called busybox, which provides tiny versions of GNU programs for phones. The easiest way to install this was to install TitaniumBackup and click on it’s “Problems” button which would download it, then copy it from /data/data/com.keramidas.TitaniumBackup/files into /system/bin/ and run busybox –install -s /system/bin/

Then the script could run, and additionally I needed to run chmod 774 /data/data/* to get make sure everything was in the correct state, but still no luck with gdb. At this point I decided my time would be better spent on making a linux version of the android app for debugging purposes and get it running in the software emulator. More on this soon.

I’m also documenting bits and pieces of my android development notes on FoAM’s wiki here.

Fluxus on Android

I recently took a long train journey and had an idea to use my android phone for taking timelapse photos for Miska Knapek. I had no laptop with me, and realised there was no way I could script anything on the phone itself to do this, or anything more complex. This was the kernel of an idea that ended up as two intense days of hacking fluxus into android.

It’s running as native code (underneath the JVM), and as it’s got the same type of processor as the NDS it shares some code with Betablocker DS which has similar restrictions. There is only a tiny amount of fluxus code ported at the moment, but the scenegraph and core maths is pretty much the same (with some changes to fixed point). I’ve used TinyScheme for the interpreter (the same one Impromptu is based on) which gives you an impressively complete language with a very small footprint.

This is just a quick proof of concept (how far I got is testament to the excellent documentation and ease of portability using C++ with the NDK more than my abilities). It would be nice to expose all the sensors to the script (motion, compass, GPS, camera, wifi) and make it quite general purpose. Writing code with the touch screen is do-able but not very usable – but I have some ideas in this area also…

Get the source and, of course the package. Touch the screen when it’s started up to view and eval the source.