ImperialViolet

Persistence (17 Apr 2003)

Over the past few weeks I've been knocking about with ideas (which generally go under the name of Landscape) for holding data between code. Generally a byte stream (be it a pipe or a file) is the highest we get when sharing data. Too much data is far too difficult to get to and too little data is well linked.

One key part of fixing this could be a persistent object store and I've been messing with Python versions of this. Firstly, ZODB is far too slow and Prevayler keeps everything in memory.

Prototype 1

This kept data in XFS extended file attributes and each object was a file.

Pros:

  • Nice kernel interface for non-Python code to use

Cons:

  • Requires XFS
  • Inflexible
Prototype 2/3

This used SQL to store the objects.

Pros:

  • Again, a nice interface for non-Python code
  • SQL has transaction ability
  • Backlinks are handled just by a different SQL query

Cons:

  • SQL is pretty slow - would have to read and write cache
Prototype 4

This uses proxy object that act as far pointers and pickle/unpickle objects on demand

Pros:

  • Deals with any pickleable Python object (e.g. most of them) well

Cons:

  • Pain for anything non-Python to use
  • At the moment the code is a bit flaky. Playing with the garbage collector is a dangerous (and non-deterministic) game.
Ramblings

These are, of course, only prototypes to play about with these ideas. The true way to do it would be to use EROS. But EROS isn't seeing much take-up (I've not even got it to boot) and it might be better to put the neat bits of EROS into Linux, even if they don't all fit.

Prototypes 1 and 2 only support dictionary type data and Prototype 3 has slightly bodged support for other objects (they can have a .type link, pointing to the python code to be imported). Supporting other types of objects is very important for objects like (for example) the mixer which has to get and get its values from/to somewhere else.

The interface is also very important. Finding the right set of ideas to simply abstract data is a very hard problem. The interface to the same is possibly just as hard.

At the moment the interface is shell like but different types of objects will need more than that. It remains to be seen if some objects will need to handle the interface themselves to the level that they do today (e.g. GTK/QT level APIs) or if a system akin to XHTML/CSS will do.