Organisation hacking (part 1)

Over the last six months I’ve been taking a crash course in company formation, treating it like any other investigation into a strange and esoteric technology. Last year I registered FoAM Kernow as a UK non profit organisation in the mould of FoAM Brussels. Starting off with absolutely no knowledge at all (but with a lot of help from FoAM’s wider friends and relations) I found a lot of confusing fragments of information online so I thought I’d document the process here as much as I can. It’s important to state that I’m not a lawyer or professional in these matters at all, so no substitute for proper legal advice – and any corrections would be most welcome.

What does non-profit mean?

A non-profit company is simply one that is not allowed to have shareholders. A for-profit company can pay a set of people who own shares part of the profits called dividends. In a non-profit the money outflow is more tightly controlled and can only go to employees or to pay costs for the company. Contrary to popular belief, there is no limitation on the size of the company, how much it pays employees or the money it makes in total (there are some very large non-profits out there).

In place of shareholders, a non-profit is “limited by guarantee” – a more legally correct term for this type of organisation. Individuals guarantee a set amount against liability (debts). The default is a whopping £1 each.

As far as I understand it “limited by shares” and “limited by guarantee” are the two main different types of organisation in the UK. There are also a cluster of other subcategories: charities, community interest groups, co-operatives and social enterprises for example. These are a little bit more fuzzy it seems, but they tend to be different flavours of limited by guarantee companies with more legal paperwork (especially in the case of charities) to determine the precise purpose and role of the organisation, and ultimately access to profits – what the money can and can’t be used for.

Why form a company?

Having a legal entity with which to work from rather than being an individual (sole trader) is better for larger projects, and bigger institutions are happier collaborating with legal entities. This is partly a matter of indirection or abstraction, e.g. if I get hit by the proverbial bus, the legal entity continues to exist. More importantly, it means multiple people can work together as part of a legal structure with a publicly stated set of shared values (called the articles of association). There is also a well established democratic process to make organisational decisions (more on that later).

Sole traders on the other hand can employ as many people as they want, but the structure would be fixed as a simple hierarchy – and the organisation has no legally defined purpose. Also things like insurance are different, but I won’t get into that yet!

Why form a non profit company?

A non-profit fits well with the goals of FoAM. Generally we focus on exploring ways of doing independent research, we are strict on non-exclusive rights to our work (participating in free software and creative commons) and finding a place between arts and sciences. None of this requires conventional fund raising by selling shares.

Another big reason is an issue of trust. We work mainly with people in spheres of arts and research – and it turns out a lot of people don’t want to work with companies that are run on a for-profit basis. In fact I would go so far as to say that some of the most interesting people we work with are wary of this. There is a somewhat justifiable worry that their contribution may be exploited via pressures to make a return on investments made by third parties.

The downsides are mainly tax related – for-profit companies can use dividends to pay investors (who can be employees too) which are not taxed – this is a very common practice. Usually small companies employ people at the minimum taxable wage then pay the rest by dividends. There is no way to do this with a company limited by guarantee – all payments to people need to be accounted for as normal employment or subcontracting and subject to income tax. It turns out that there are other upsides to being non-profit that may counteract this in the long run as you get treated differently in some contexts (e.g banking). For the next installment (if I find time!) I’d like to talk more about that and the formation process too…

Connecting Flask to Apache

When exploring the cosmos of python powered web frameworks it’s easy to get lost due to fast moving versions. Recently we had a lot of difficulty getting Flask connected via a WSGI script on Apache. This seems to be due to needing the object factory to be called specifically rather than relying on the module namespace to do the work for us, there doesn’t seem to be much help online about this yet. We’re also running python in a virtualenv so the working wsgi script looks like this:

#!/usr/bin/python                                                               
import sys
import logging
import os

logging.basicConfig(stream=sys.stderr)
PROJECT_DIR = "/var/www/yourflasksite/"
activate_this = os.path.join(PROJECT_DIR, 'bin', 'activate_this.py')
execfile(activate_this, dict(__file__=activate_this))

sys.path.append(PROJECT_DIR)
from app import create_app
application=create_app(os.getenv('FLASK_CONFIG') or 'default')
application.secret_key = 'your site's key'

For completeness, here is the apache config that starts it up:

<VirtualHost *:80>
    ServerName your.url
    WSGIScriptAlias / /var/www/yourflasksite/yoursite.wsgi
    <Directory /var/www/yourflasksite/app>
	Order allow,deny
        Allow from all
    </Directory>
    Alias /static /var/www/yourflasksite/app/static
    <Directory /var/www/yourflasksite/app/static/>
        Order allow,deny
        Allow from all
    </Directory>
    ErrorLog ${APACHE_LOG_DIR}/yourflasksite-error.log
    LogLevel warn
    CustomLog ${APACHE_LOG_DIR}/yourflasksite-access.log 
</VirtualHost>

Adventures with i2c

In order to design the next version of the flotsam hardware I need to make it cheaper and easier to build. The existing hardware was very cheap in terms of components but expensive in terms of time it took to construct! With this lesson learned and with a commission on the horizon I need to find a simpler and more flexible approach to communication between custom hardware and the Raspberry Pi – mainly one that doesn’t require so many wires.

i2c is a serial communication standard used by all kinds of components from sensors to LCD displays. The Raspberry Pi comes with it built in, as the Linux kernel supports it along with various tools for debugging. I also have some Atmel processors from a previous project and there is quite a bit of example code for them. I thought I would post a little account of my troubleshooting on this for others that follow the same path, as I feel there is a lot of undocumented knowledge surrounding these slightly esoteric electronics.

The basic idea of i2c is that you can pass data between a large number of independent components using only two wires to connect them all. One is for data, the other is for a clock signal used for synchronisation.

Debugging LEDs
Debugging LEDs

I started off with the attiny85 processor, mainly as it was the first one I found, along with this very nice and clear library. I immediately ran into a couple of problems, one was that while it has support for serial communication built in (USI) you have to implement i2c on top of this so your code needs to do a lot more. The second was with only 8 pins the attiny85 is not great for debugging. I enabled i2c on the Raspberry Pi and hooked it up, ran i2cdetect – no joy. After a lot of fiddling around with pull up resistors and changing voltages between 3 and 5v either no devices were detected, or all of them were, all reads returning 0 (presumably logic pulled high for everything) or noise – nothing seemed to make any difference.

After a while (and trying other i2c slave libraries to no avail) I switched to an atmega328 processor using this library which includes a Makefile! One thing that I’ve noticed that would make things much easier for learning this stuff is more complete toolchains in example code including the right #defines and fuse settings for the processor. However this code didn’t work either at first, and my attempts at using debugging LEDs on PORTB didn’t work until I figured out it was conflicting with the UART i/o used in the example code – after figuring out that this wasn’t part of the i2c code I removed it and the Raspberry Pi could at last see the device with i2cdetect. With the addition of some LEDS I could check that bytes being sent were correctly being written to the internal buffer at the correct addresses.

It finally works!
It finally works!

Reading was another matter however. Most of the time i2cget on the Pi failed, and when it did work it only returned 0x65 no matter what the parameters. I’d already read extensively about the Raspberry Pi’s i2c clock stretching bug and applied various fixes which didn’t seem to make any difference. What did the trick was to remove the clock divide on the atmega’s fuses – by default it runs at 8Mhz but slows the instruction cycles to 1Mhz – without that it could keep up with the Pi’s implementation and all reads were successful. I still had to solve the ‘0x65 problem’, and went into the i2c code to try and figure out what was going on (using 8 LEDs to display the i2c status register). It seems like reading single bytes one at a time is done by issuing a TW_ST_DATA_NACK as opposed to TW_ST_DATA_ACK, as it sends a not-acknowledged for the last byte read. This state is not supported by the library, after fiddling around with it a bit I switched over on the Pi’s side to using the python smbus library, and tried using read_i2c_block_data – which reads 32 bytes at a time. The first byte is still 0x65 (101 in decimal, in the photo above) – but the rest are correct, I’ll need to read a bit more on the i2c protocol to figure that one out (and get the attiny working eventually), but at least it’s enough for what I need now.

Handy collection of pinouts
Handy collection of pinouts

Migrations in django 1.7

For the wild cricket tales project I’m using django again, the all new 1.7 version which has amongst lots of other shiny things has an all new migrations system. This is so you can change the underlying database ‘model’ for your site and it automatically tracks your changes and applies them to the data (via python scripts it generates and you can tweak if needed).

However, if you trash them (my 0001_initial.py was deleted – not sure what by, or why I didn’t have them in version control but that’s another question) it’s helpful to know how to start again from scratch – the official documentation doesn’t have much to say on this topic yet. This is what I cobbled together, and it requires that your current database matches the code state exactly:

Delete all existing migrations:

rm my-app/migrations/*

Remove references to them from the database – presumably you need to be a little more selective here if you are running more than one app:

sqlite3 db.sqlite3 "delete from django_migrations"

Create new starting state:

python manage.py makemigrations --empty my-app

Fake the initial migration – this will leave the migrations system assuming that the current database state match the code but won’t actually apply the changes (i.e. making the tables again):

python manage.py migrate --fake

I’m still a little unclear how migrations in source control work across development/production servers etc, but time will tell…

Scratch -> Lego Mindstorms

A bit of hardware hacking for Troon Primary CodeClub, who have tons of old style Lego Mindstorms they don’t use any more, and after a year of Scratch programming on their PCs are just getting started with Raspberry Pi. We’re using this Scratch modification together with the hardware I’m making which is based on this circuit. The main thing here is an L293D Motor Controller IC which can drive 2 DC motors in both directions. You can write the hardware code in Scratch like this to control the lego motors:

blink11

The most tricky part in this whole endeavour has been physically connecting to Mindstorms. At the moment I’m having to use crocodile clips which won’t work long in normal classroom conditions – but I’m wary of destroying/modifying the connectors as they’re not made any more…

IMG_20140317_111808

How to back up your encryption keys

Without really noticing it I’ve gradually acquired more and more encryption keys without understanding how to back them up properly. Until fairly recently I lazily assumed that remembering the passphrases would be enough in case of my laptop catching on fire, but this is not the case.

I use GPG keys both for authenticity over email, and encryption when sending people passwords for stuff I’m setting up for them. The Ubuntu launchpad also uses GPG for signing packages for which I use a different key. I also run a bunch of servers, for which I use ssh keys to prove my identity, then there is the Android play store, that requires binaries to be signed, using yet another key, which is also shared for OUYA packages too.

The main algorithm in use for authentication in all these cases is called RSA. RSA and similar algorithms generate a pair of keys, one public and one private. The private key data is encrypted using yet another algorithm (usually AES) which is what your passphrase is used for. When it’s needed, you type your passphrase in, it decrypts the RSA private key and uses that to identify you. So it’s vitally important that this key data is backed up as it can’t be recreated from your passphrase. There doesn’t seem very much information online on the practicalities of all this, so I’m documenting the process with links to where I got info here, partly in order to get feedback if it’s wrong!

With ssh it’s just a matter of copying the contents of your .ssh directory – which contain the public and encrypted private key. The android keys are in a file called .keystore, in your home directory.

When it comes to GPG the best way seems to be to list and export them individually with:

gpg --list-secret-keys
gpg --export-secret-keys your-id-number > secret-key.asc

The id number is the part after the slash for each keypair. At the same time, it’s important to back up a revocation key for each key – this allows you to tell the GPG trust network if your identity becomes compromised by someone finding out your key (or losing/forgetting your passphrase, which is perhaps more likely). You can do this with:

gpg --gen-revoke your-id-number

And paste the result into a text file.

So you can take all these files and store them somewhere safe on a usb stick for example. It all needs to be encrypted so it doesn’t matter if it’s found and accessed. I almost made a fundamental mistake and encrypted it with GPG, which would have been like locking my house keys inside the house. Instead you can encrypt the file using AES independently using this command:

openssl aes-256-cbc -in your-key-file.tar.gz -out your-key-file.tar.gz.enc

I’m assuming once this is done, the best practice is to put it in various places to reduce the chances of it getting lost, as it doesn’t matter if it’s accessible. Make sure to use a long passphrase you won’t forget! The advice given here is to use a long randomly generated string and write it on a piece of paper, which is stored in a safety deposit box – this is the approach to take if you are in charge of really important stuff, I’m not sure that I’m at that point yet 🙂

A fluxus workshop plan

I’ve been getting some emails asking for course notes for fluxus workshops, I don’t really have anything as structured as that but I thought it would be good to document something here. I usually pretty much follow the first part of the fluxus manual pretty closely, trying to flip between visually playful parts and programming concepts. I’ve taught this to teenagers, unemployed people, masters students, professors and artists – it’s very much aimed at first time programmers. I’m also less interested in churning out fluxus users, and more motivated by using it as an introduction to algorithms and programming in general. Generally it’s good to start with an introduction to livecoding, where fluxus comes from, who uses it and what for. I’ve also started discussing the political implications of software and algorithmic literacy too.

So first things first, an introduction to a few key bindings (ctrl-f fullscreen/ctrl-w windowed), then in the console:

  1. Scheme as calculator – parentheses and nesting simple expressions.
  2. Naming values with define.
  3. Naming processes with define to make procedures.

Time to make some graphics, so switch to a workspace with ctrl-1:

  1. A new procedure to draw a cube.
  2. Calling this every frame.
  3. Mouse camera controls, move around the cube.
  4. Different built in shapes, drawing a sphere, cylinder, torus.

Then dive into changing the graphics state, so:

  1. Colours.
  2. Transforms.
  3. Textures.
  4. Multiple objects, graphics state persistent like changing a “pen colour”.
  5. Transform state is applicative (scale multiplies etc).

Then tackle recursion, in order to reduce the size of the code, and make much more complex objects possible.

  1. A row of cubes.
  2. Make it bend with small rotation.
  3. Animation with (time).

At this point they know enough to be able play with what they’ve learnt for a while, making procedural patterns and animated shapes.

After this it’s quite easy to explain how to add another call to create tree recursion, and scope state using (with-state) and it all goes fractal crazy.

This is generally enough for a 2 hour taster workshop. If there is more time, then I go into the scene graph and explain how primitives are built from points, faces and show how texture coords work etc. Also the physics system is great to show as it’s simple to get very different kinds of results.

Compiling Scheme to Javascript

Recently I’ve been continuing my experiments with compilers by writing a Scheme to Javascript compiler. This is fairly simple as the languages are very similar, both have garbage collection and first class functions so it’s largely a case of reusing these features directly. There are some caveats though – here is an example of the canonical factorial example in Scheme:

(define factorial
  (lambda (n)
    (if (= n 0) 1
        (* n (factorial (- n 1))))))

The Javascript code generated by the compiler:

var factorial = function (n)
{
    if ((n == 0)) {
        return 1
    } else {
        return (n * factorial((n - 1)))
    }
}

Scheme has tail call optimisation, meaning that recursive calls are as fast as iterative ones – in Javascript however, a function that calls itself will gradually use up the stack. The factorial example above breaks with larger numbers so we need to do some work to make things easier for Javascript interpreters. When considering list processing, fold can be considered a core form that abstracts all other list processing. If we make that fast, we can reuse it to define higher order list processing such as filter, which takes a function and a list and returns another list. The function is called on each element, if it returns false it’s filtered out of the resulting list.

(define filter
  (lambda (fn l)
    (foldl
     (lambda (i r)
       (if (fn i) (append r (list i)) r))
     ()
     l)))

Compiling this with an iterative version of fold – which uses a for loop in Javascript, we generate this code:

var filter = function (fn,l)
{
    return (function() {
        var _foldl_fn=function (i,r) {
            if (fn(i)) {
                return r.concat([i])
            } else {
                return r
            }
        };

        var _foldl_val=[];
        var _foldl_src=l;
        for (var _foldl_i=0; _foldl_i<_foldl_src.length; _foldl_i++) {
            _foldl_val = _foldl_fn(_foldl_src[_foldl_i], _foldl_val); 
        }
        return _foldl_val; })()
    }
}

We have to be a little bit careful that variables created by the compiler don’t clash with those of the user’s code, so we have to make really ugly variable names. You also need to be careful to wrap code in closures as much as possible to control the scope. The problem with this is that there is probably a limit to the amount of nesting possible, especially comparing different implementations of Javascript. A interesting project that has got much further than this is Spock. More context to this work and some actual source code soon…

Android Camera Problems

The DORIS marine mapping platform is taking shape. For this project, touch screens are not great for people wearing gloves in small fishing boats – so one of the things the android app needs to do is make use of physical keys. In order to do that for taking pictures, I’ve had to write my own camera android activity.

It seems that there are differences in underlying camera behaviour across devices – specifically the Acer E330 model we have to take out on the boats. It seemed that nearly every time takePicture() is called the supplied callback functions fail fire. I’ve tested callbacks for the shutter, raw and jpeg and error events and also tried turning off the preview callback beforehand as suggested elsewhere, no luck on the Acer, while it always works fine on HTC.

The only solution I have so far is to close and reopen the camera just before takePicture() is called which seems to work as intended. As it takes some seconds to complete, it’s also important (as this is bound to a key up event) to prevent the camera starting to take pictures before it’s finished processing the previous one as that causes further callback confusion.

import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.hardware.Camera;
import android.hardware.Camera.CameraInfo;
import android.hardware.Camera.PictureCallback;
import android.util.Log;

class PictureTaker 
{
    private Camera mCam;
    private Boolean mTakingPicture;

    public PictureTaker() {
        mTakingPicture=false;
    }

    public void Startup(SurfaceView view) {
        mTakingPicture=false;
        OpenCamera(view);
    }

    private void OpenCamera(SurfaceView view) {
        try {
            mCam = Camera.open();
            if (mCam == null) {
                Log.i("DORIS","Camera is null!");
                return;
            }
            mCam.setPreviewDisplay(view.getHolder());
            mCam.startPreview();
        }
        catch (Exception e) {
            Log.i("DORIS","Problem opening camera! " + e);
            return;
        }
    }

    private void CloseCamera() {
        mCam.stopPreview();
        mCam.release();
        mCam = null;
    }

    public void TakePicture(SurfaceView view, PictureCallback picture)
    {
        if (!mTakingPicture) {
            mTakingPicture=true;
            CloseCamera();
            OpenCamera(view);
            
            try {
                mCam.takePicture(null, null, picture);
            }
            catch (Exception e) {
                Log.i("DORIS","Problem taking picture: " + e);
            }
        }
        else {
            Log.i("DORIS","Picture already being taken");
        }   
    }
}

Re-interpreting history

A script for sniffing bits of supercollider code being broadcast as livecoding history over a network and re-interpreting them as objects in fluxus, written during an excellent workshop by Alberto de Campo and Julian Rohrhuber at /*VIVO*/ Mexico City.

(osc-source "57120")

(define (stringle str)
    (map
        char->integer
        (string->list str)))


;;(osc-destination "osc.udp:255.255.255.255:57120")
;;(osc-send "/vivo" "s" '("fluxus:hola"))

(define (safe l n)
    (list-ref l (modulo n (length l))))

(define (render arg)
    (let ((l (map (lambda (t) (/ t 255)) (stringle arg))))
        (with-state
            (scale (vector (safe l 0)
                    (safe l 1)
                    (safe l 2)))
            (rotate (vmul (vector (safe l 32)
                    (safe l 12)
                    (safe l 30)) 360))
            (colour (vector (safe l 0)
                    (safe l 3)
                    (safe l 4)))

            (build-torus 0.1 1 4 20))))

(clear)
(scale 2)
;;(render "hello 343 323")

(every-frame 
    (begin
        (when (osc-msg "/hist")
            (printf "~a~n" (osc 1))
            (when (osc 1)
                (render (osc 1))))))