1. Some problem that just doesn’t really exist (or hasn’t existed for a very long time) with relational databases
2. MongoDB
3. Profit!

An example would be the first speaker, who didn’t like normalized data because it had “bad locality”. Now ignoring for a second the difference between a logical and a physical data model, and the existence of the normal forms, if you ever did find that the bottleneck on your joins specifically was seek time, you could pre-compute the join and refresh it whenever anything changed – in Oracle using a materialized view (1996!), a continuous query, the result cache… And that’s on top of the block buffer cache and the query optimizer already being very smart. Another of the same speaker’s examples overlooked the existence of nested tables, saying what you could do with them is impossible in “SQL databases”. It’s claimed that MongoDB is more flexible because it doesn’t constrain you to tables. Well that’s backwards… We don’t work the way we do because tables are a limitation of the technology, we use the relational model because it has sound mathematical underpinnings, and the technology reflects that†. Where’s the rigour in MongoDB’s model?

Another speaker claimed that it was far better for each application to have its own database, and expose all its data through web services. Sounds good, except you now need another technology, a directory to find all these things, since they aren’t just table names or stored procedure names in the one place, and manage access control and auditing. And if you need to touch data across several of them then you’ll need something to coordinate that… We could call it a transaction processing facility, since that’s what IBM called it in 1960. He handwaved over both of those. There were many similar examples.

Another recurring theme was of an organization refreshing its hardware and modifying its architecture, one component of which was introducing MongoDB, yet all the performance gains attributed to it. For example splitting OLTP and OLAP from one database into two, and introducing a delay of a few minutes between data coming in and being available for reporting. Well that will give you a performance boost in any database! If you can tolerate the delay, of course. But if you could, why build it that way in the first place (or having built it, complain that it’s slower than you’d like), and if you can’t, then you can’t do this. In the roadmap they are promising point-in-time recovery in a future release. Oracle had that in 1988, when I had just left primary school.

So anyway, since it’s free‡, there’s no reason not to evaluate MongoDB, and see if it suits your use cases. But just remember that these kids think they’re solving problems that IBM (et al) solved quite literally before they were born in some cases, and the features are probably already there in your existing database/technology stack (I have used Oracle for my rebuttals just because I am most familiar with it, but I expect the same is true for SQL Server and DB/2 as well). Talk to your friendly local DBA…

† I personally predict that in a few years there will be a lot of work re-normalizing the data in MongoDB and its rivals so it can actually be useful. That’s reason enough to become an expert in it. In about 2001, the company I joined then had just completed a massive engineering effort to get off Versant and (back) into Oracle… All this object-database stuff gives me massive deja vu for the 1990s when they were all the rage.

‡ In the same way that Oracle is also free for evaluation purposes. No-one would deny that Oracle is expensive in production! But there is no such thing as cheap or expensive in business, there is only worth the money or not.

Well if you are Apple, I’d say your greatest challenges were scrolling a document, responding to a mouse click and keeping up with a user typing. Hell, you can’t even manage it for this blog post…

… first as tragedy, then as farce.

There is a strange attitude among many in this industry towards what are contemptuously referred to as legacy systems. No-one would ever articulate this of course, because when you say it out loud it sounds ridiculous, but the implicit belief is, in the 70s, 80s, 90s they had: smartphones and AJAX† and Ruby-on-rails and Chrome (and a long shopping list of “modern” technologies) but because they were stupid they chose to use dial-up modems, and dumb terminals, and program in FORTRAN. And because they were stupid, we have nothing to learn from them.

On a similar note, it is a common refrain to hear that newspapers are obsolete. And as a business model, that may well be true – but since the 1980s, newspaper publishing has defined mission-critical computing. Come hell or high water, the paper has to be on the newsstands in the morning. The desktops, the networks, the servers, the presses, the logistics, and all the software and IT have to work. Nowadays, it is as much as anyone can hope for for most software to mostly work, most of the time. If a browser or an operating system crashes or freezes, you take it in your stride and restart it. If a website is down, you might try again later, or you might just not bother. Even major bits of infrastructure are unreliable. The skills required to do serious computing are simply decaying, while individuals such as myself retain and practice the old ways, I don’t think when the last newspaper switches off its presses, that talent will then make its website five-9s reliable… Even the best civil engineer can’t build a castle on a fetid swamp. We’ll have to nuke it from orbit – only way to be sure – and start again.

† I recently used AJAX to build the interface for one of my projects. Even with bolt-ons like Comet, it’s pretty crude and feeble compared to Tcl/Tk… From the last century. I’m sure by the end of this decade it will be as-good‡, only to be swept away by some shiny new thing, and we’ll be back to square one in terms of getting useful work done. Since the 80s, every decade in computing has been a shallow copy of the previous decade. Eventually it will go full circle, like pocketwatches being replaced by wristwatches, to be replaced by clocks on mobile phones in your pocket…

‡ And yet, actually no more powerful or easy to use than Curses, for either developers or end users.

SQL> create view v1 as
select object_type, count(1) as howmany from user_objects group by object_type;

The histogram automatically scales to the width of the current window.

Also, I have been reading Jordan Mechner’s book The Making Of Prince Of Persia†. It’s both fascinating and inspiring. Just before that, I read The Future Was Here, the story of the Commodore Amiga‡. The book is made even more poignant by my Mac inexplicably showing the beach ball as I scroll through a simple web page, or the mighty RHEL servers at work being unable to keep up with my typing. The future is still back in the 80s.

† The original code is also on Github.

‡ I have an A500+ on my desk right now, the best of them IMHO. I might write a post comparing it with the Atari STE, and the BBC with the C64, in the cold light of day as an experienced adult. I have a fine collection of classic machines now, often acquired broken with the intention of repairing them myself. Another time-sink from OCaml work…

• Some .cmi for toplevel internals that used to be installed in`ocamlc -where` are now to be found in `ocamlc -where`/compiler-libs. Add “-I +compiler-libs” where needed.
• Warning 28 is now enabled by default.

The impact of these was that the toplevel prompt wasn’t working in the shell, and one non-fatal warning when compiling, so nothing that would have broken any code, but it’s good to be up-to-date. The necessary changes have been checked in on Github.

Speaking of which, bearing in mind the LinkedIn débàcle, I have a password generator too…

A lot of that was lost in the intervening years. It was certainly possible to program the “next generation”, the 16-bit home computers like the Atari ST and Commodore Amiga, of course. It just wasn’t what they were for. The creators of the BBC went to great lengths to include a fantastic BASIC dialect, whereas ST Basic was notoriously bug-ridden, and Devpac was a third-party product that you had to pay for. I don’t know so much about the Amiga world, but on the ST, the barriers to entry to programmers were certainly higher. Not insurmountably so, for one such as me who had grown up on the BBC and took programmability for granted, but I wonder how accessible I would have found it if it were my first machine. Certainly there was more to learn in order to, as on the BBC, produce a “professional” looking program, one that would operate with GEM for example. But, whether it was truly an unexpected emergent property of the more advanced machines, or the general zeitgeist, programming fell out of the mainstream. Games were the normal use, consoles such as the Sega Megadrive replaced 8-bit micros in some households.

Since then (fast-forwarding over most of the ’90s and all of the ’00s) programming has simultaneously gotten easier and less accessible. How is this possible? Abstraction. Moving further from the machine, or placing more and more layers between the programmer or user and the machine (once those terms were nearly synonymous). It is very easy for a user to write a macro in VBA that gets a lot of useful work done, and I would never advocate taking this type of programming away for that reason. But it encourages thinking of the machine as a “black box”, it is difficult to reason about what it is actually doing, and that in turn discourages the very powerful mindset that it’s all just code, all the way down that is needed to be an actual programmer. This is not necessarily intended as a value judgement; for many people computers are just tools and that’s fine, for them the macro-style approach is highly productive.

But someone has to make the tools, and the question is, is the Raspberry Pi going to nurture a new generation of tool-makers? Abstraction is useful because it allows one to do work without repetitive detail and focus on the problem domain. But I argue that abstraction should only be introduced once the fundamentals are understood. Learning on a machine like the BBC teaches that, no so much the details which become obsolete, but the concepts, (e.g. I do my real work in Python on Linux on x64 not BBC Basic on a Model B, but I use dis and GDB regularly). The Raspberry Pi has video, ethernet, USB, 256M RAM. It comes with a GUI and a web browser. It has more in common with a PlayStation than it has with a BBC Micro (and I make the same criticism of the OLPC). I know a lot of people of my age are excited about getting one and using it as a cheap embedded controller, like an Arduino. But I don’t think it’s a useful teaching tool, or at least, no more useful than a common PC. For that, you’d want something like a FIGnition. I honestly don’t know why that project has been relegated to the sidelines while the Pi gets all the press.

† Thought experiment: If you had a BBC clocked at 2Ghz what “real work” could you not do on it? What would you need to add? What about a BBC with a 65816 instead of a 6502, giving it 16M RAM, and ADFS access to modern storage – but fundamentally the same OS, languages(s), the same model of computation, switch it on, BASIC > prompt and off you go? I am struggling to think of anything…

• RAMAMP RA20+44 for £500. This (I believe) is a board that adds 32K to a standard Model B, which can be used as sideways or shadow RAM. A shadow RAM only Aries B20 goes for £20, same again for a B12 which can take sideways RAM. To put that into perspective, the BBC B when new cost £399.
• Electron +3 for £200, a disk drive for the Acorn Electron, for 5× what an Electron itself goes for.

The retro stuff I bought I use (well, play with!), but at these prices I can only assume people are keeping them in glass display cabinets… I could understand if someone desperately needed some data off old disks for sentimental reasons, but the price for the RAMAMP, given that it lives inside the case and there are functionally identical substitutes (e.g. Aries B32, or B20+B12) for a fraction of the price, is completely inexplicable.

Update

• Commodore 1540 for £205!

But the other reason is a re-awakened interest in the BBC Micro! I had one of these when I was a child, it was my first machine, but even now as a highly experienced adult, it still fascinates, in fact, even more than the first time round. I am intrigued by how much I can do with how little by today’s standards – 32k of memory and a 2Mhz processor. Right now I am playing at running “background” tasks by interrupt handling, e.g. to run 6502 code during the VBI. And with an ethernet interface, the possibilities are endless…

Tho’ of course I have not forgotten about OCI*ML…

The key to this is, to borrow the VCS terminology with which most people are familiar, the agent script. This acts as a proxy between RAC (or CRS, really) and your own code, allowing the cluster to start and stop it, and to check it’s health†. This is very similar to the scripts found on typical Unix systems in /etc/init.d/ that are invoked when changing runlevels – it takes the action to perform as the first parameter, and the body of the action can be any reasonable Unix commands. These must be sufficient to execute the program with no prerequisites, e.g. setting up LD_LIBRARY_PATH, the ORACLE_HOME, and so forth, and checking that the environment is sane, e.g. that necessary directories exist and are writeable. This script should live on a filesystem shared by all RAC nodes, let’s assume it is mounted as /common and the agent is called myapp.sh.

#!/bin/bash

# a simple CRS agent script
#
# 19-OCT-2011   Gaius       Initial version

# set up the environment - on each node, /home/oracle/this-node.env is a
# symlink to the environment variables (e.g. ORACLE_SID for the instance,
# PATH, LD_LIBRARY_PATH etc)
. /home/oracle/this-node.env

# check that the environment is sane, e.g. we can write to the log dir
# CRS looks at the exit code of this script to see if the operation was
# a success
if [ ! -w /common/log ]; then
    exit 1
fi

# parse the command line to see what CRS wants to do
case $1 in
'start')
    # do any prior cleanup, then start myapp and store its PID
    mv /common/log/myapp.log{,.old}
    myapp >/common/log/myapp.log 2>&1 &
    EXITCODE=$?
    echo $! >/common/pids/myapp.pid
    ;;
'stop')
    kill `cat /common/pids/myapp.pid`
    EXITCODE=$?
    ;;
'check')
    # check that 1 process named myapp is running - CRS will
    # automatically do this check on the correct node every 60s (default)
    # and if it returns non-zero take corrective action
    NUMPROCS=`ps -ef|awk '/[m]yapp/ {X += 1} END {print X}'`
    if [ $NUMPROCS -eq 1]; then
        EXITCODE=0
    else
        EXITCODE=1
    fi
    ;;
*)
    echo "Usage: $0 [start|stop|check]"
    ;;
esac

exit $EXITCODE
# End of file

Next on each RAC node, put in a symlink $CRS_HOME/crs/script/myapp.sh → /common/myapp.sh. This ensures that any node can execute the script, but there is only a single copy of it to maintain. Make sure it is executable with chmod. This can be tested on each node by calling it manually in the shell with the parameters and seeing what it does. Next, we register the agent script with the cluster:

$ crs_profile -create myapp -a $CRS_HOME/crs/script/myapp.sh -t application
$ crs_register myapp

This creates a cluster resource called myapp, with an agent script defined by -a, of a type application.

Now, we can start to manipulate our own program with the standard Oracle commands:

$ crs_start myapp
Attempting to start `myapp` on member `oel1`
Start of `myapp` on member `oel1` succeeded.
$ crs_stat myapp
NAME=myapp
TYPE=application
TARGET=ONLINE
STATE=ONLINE on oel1
$ crs_stop myapp
Attempting to stop `myapp` on member `oel1`
Stop of `myapp` on member `oel1` succeeded.
$ crs_stat myapp
NAME=myapp
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

You can see this is running on my Oracle Enterprise Linux test system rather than my usual Debian.

This is a trivial example not suitable for real Production usage, for example, the check action should do more than just verify the process exists (it could be stuck) and the stop should try a clean shutdown, then a hard kill if that does not complete within a certain threshold, and all steps should write comprehensive logging to enable quick troubleshooting (you can see CRS’s own log in $CRS_HOME/log/`hostname`/crsd/crsd.log). Nevertheless it serves to demonstrate how simple it is to HA your own applications (assuming you have RAC already!), and a basis for further development. I have been using this technique in Production for several years now for a variety of purposes.

The official documentation is here. If your application is a server itself, it will also need a VIP of its own, so clients can connect whichever RAC node it is on. A very useful parameter for crs_profile in this case is -r, to make the application depend on its VIP, so the cluster knows to start them in the correct order, on the same node. There are many options viewable with crs_profile -help, including check interval, number of restart attempts, and so on.

† Alert VCS operators will have noted the absence of the “clean” action. You could do this in stop, or in start before actually starting

Here is an example from a recent interview, for an Oracle DBA position. First, I asked the candidate if he knew what was an SCN was, and he did. Next I asked about incremental backups, and he knew what those were too. Thirdly, I asked him if he knew what a physical standby was, and again, he did. All good so far, but these are just facts that can be recited from memory, or looked up on Google (as I did to make those links!). A better measure of a candidate is a realistic problem to solve, so next I described a scenario in which a standby database is stuck on WAIT_FOR_GAP and All defined FAL servers have been attempted, and the archived redo log is no longer on the primary (e.g. deleted by a naive cron job† that catches everything older than a day). He replied that the only way to solve this problem was to completely rebuild the standby, as if setting it up for the first time. I reminded him of the previous three facts we had discussed, and asked again, and got the same reply.

Now a strong candidate would have thought about this (out loud):

1. I can easily find the SCN of the standby database
2. A physical standby database is a block-by-block copy of its primary
3. I can do an incremental backup of the primary from that SCN and apply it to the standby.
4. Then resume log shipping (having stopped deletion on the primary if the previous step would take longer than the retention period)

Of course, in that particular case, it is a documented technique, tho’ if you hadn’t happened to have read it, it’s still obvious, particularly when you have established all the underlying facts already. In fact this is true about most things in IT: if you grasp the underlying principles then, even if you don’t know the specific command or syntax‡ off the top of your head, you can probably figure out the solution, or at least, be able to re-frame the problem in a way that someone else can jump in and help with just the piece you’re missing. At least you know what to ask or what to look for. There was obviously more to the interview than just that, but with a few other other scenario questions, a clear pattern emerges, giving an accurate picture of the candidate. That’s the point of thinking out loud, and freezing up is the last thing you want in a DBA when there is a production outage! Those are the kinds of people I look for, and this experience sets me up very well when I am on the other side of the table.

I wrote a few months ago about a slight sense of ennui with my present role; the time has come to do something about that, so next month I join the dangerous, glamorous world of algo trading. The people that I’ve hired will get along just fine without me (tho’ some of them should have been paying more attention over the last few years to learning Python!) and so will the organization… if it manages to hang onto them.

† Unfortunately, even in these days of RMAN, those are still common
‡ For this reason, I almost never ask candidates to write code on a whiteboard, tho’ sometimes I will have them draw a diagram