Happy New Year

Regular readers may have noticed that I’ve gone a bit quiet of late. Part of this is down to the new job which has been keeping me busy; it builds on my existing skills of course, but there is plenty to learn too.

But the other reason is a re-awakened interest in the BBC Micro! I had one of these when I was a child, it was my first machine, but even now as a highly experienced adult, it still fascinates, in fact, even more than the first time round. I am intrigued by how much I can do with how little by today’s standards – 32k of memory and a 2Mhz processor. Right now I am playing at running “background” tasks by interrupt handling, e.g. to run 6502 code during the VBI. And with an ethernet interface, the possibilities are endless…

Tho’ of course I have not forgotten about OCI*ML…

How To Run Your Own Apps on RAC

RAC, or Real Application Clusters, is Oracle’s proprietary clustering solution, for highly-available databases. Let’s break down the name. The “real” I assume refers to the fact that it is active-active; all nodes in the cluster are available to do useful work, in contrast with active-passive systems such as Microsoft’s where one node is idle, awaiting failure of the first before taking over its services. The “application” is the interesting bit; altho’ most implementations I have seen only use RAC for the database (and code running inside it) it can easily be used as a general-purpose failover clustering solution for any in-house or third-party code, saving the administrative overhead of having one type of cluster for the database and another for the applications. Just run the whole lot on your RAC!

The key to this is, to borrow the VCS terminology with which most people are familiar, the agent script. This acts as a proxy between RAC (or CRS, really) and your own code, allowing the cluster to start and stop it, and to check it’s health†. This is very similar to the scripts found on typical Unix systems in /etc/init.d/ that are invoked when changing runlevels – it takes the action to perform as the first parameter, and the body of the action can be any reasonable Unix commands. These must be sufficient to execute the program with no prerequisites, e.g. setting up LD_LIBRARY_PATH, the ORACLE_HOME, and so forth, and checking that the environment is sane, e.g. that necessary directories exist and are writeable. This script should live on a filesystem shared by all RAC nodes, let’s assume it is mounted as /common and the agent is called myapp.sh.

#!/bin/bash

# a simple CRS agent script
#
# 19-OCT-2011   Gaius       Initial version

# set up the environment - on each node, /home/oracle/this-node.env is a
# symlink to the environment variables (e.g. ORACLE_SID for the instance,
# PATH, LD_LIBRARY_PATH etc)
. /home/oracle/this-node.env

# check that the environment is sane, e.g. we can write to the log dir
# CRS looks at the exit code of this script to see if the operation was
# a success
if [ ! -w /common/log ]; then
    exit 1
fi

# parse the command line to see what CRS wants to do
case $1 in
'start')
    # do any prior cleanup, then start myapp and store its PID
    mv /common/log/myapp.log{,.old}
    myapp >/common/log/myapp.log 2>&1 &
    EXITCODE=$?
    echo $! >/common/pids/myapp.pid
    ;;
'stop')
    kill `cat /common/pids/myapp.pid`
    EXITCODE=$?
    ;;
'check')
    # check that 1 process named myapp is running - CRS will
    # automatically do this check on the correct node every 60s (default)
    # and if it returns non-zero take corrective action
    NUMPROCS=`ps -ef|awk '/[m]yapp/ {X += 1} END {print X}'`
    if [ $NUMPROCS -eq 1]; then
        EXITCODE=0
    else
        EXITCODE=1
    fi
    ;;
*)
    echo "Usage: $0 [start|stop|check]"
    ;;
esac

exit $EXITCODE
# End of file

Next on each RAC node, put in a symlink $CRS_HOME/crs/script/myapp.sh → /common/myapp.sh. This ensures that any node can execute the script, but there is only a single copy of it to maintain. Make sure it is executable with chmod. This can be tested on each node by calling it manually in the shell with the parameters and seeing what it does. Next, we register the agent script with the cluster:

$ crs_profile -create myapp -a $CRS_HOME/crs/script/myapp.sh -t application
$ crs_register myapp

This creates a cluster resource called myapp, with an agent script defined by -a, of a type application.

Now, we can start to manipulate our own program with the standard Oracle commands:

$ crs_start myapp
Attempting to start `myapp` on member `oel1`
Start of `myapp` on member `oel1` succeeded.
$ crs_stat myapp
NAME=myapp
TYPE=application
TARGET=ONLINE
STATE=ONLINE on oel1
$ crs_stop myapp
Attempting to stop `myapp` on member `oel1`
Stop of `myapp` on member `oel1` succeeded.
$ crs_stat myapp
NAME=myapp
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

You can see this is running on my Oracle Enterprise Linux test system rather than my usual Debian.

This is a trivial example not suitable for real Production usage, for example, the check action should do more than just verify the process exists (it could be stuck) and the stop should try a clean shutdown, then a hard kill if that does not complete within a certain threshold, and all steps should write comprehensive logging to enable quick troubleshooting (you can see CRS’s own log in $CRS_HOME/log/`hostname`/crsd/crsd.log). Nevertheless it serves to demonstrate how simple it is to HA your own applications (assuming you have RAC already!), and a basis for further development. I have been using this technique in Production for several years now for a variety of purposes.

The official documentation is here. If your application is a server itself, it will also need a VIP of its own, so clients can connect whichever RAC node it is on. A very useful parameter for crs_profile in this case is -r, to make the application depend on its VIP, so the cluster knows to start them in the correct order, on the same node. There are many options viewable with crs_profile -help, including check interval, number of restart attempts, and so on.

† Alert VCS operators will have noted the absence of the “clean” action. You could do this in stop, or in start before actually starting

How To Interview

One of the things I do at my present job is interview candidates for my team, and occasionally other teams too. I estimate I’ve seen ~150 people over the last 5 years. Of those very few that I do say “hire!” based on 2×1-hour interviews, talking in a meeting room rather than watching them actually do work, the vast majority turn out pretty well, as in, they do turn out to be highly productive. The strategy I use for this is something I call the leap of logic. Firstly, I establish that a candidate knows some basic facts, then I ask a question requiring them to synthesize some new knowledge.

Here is an example from a recent interview, for an Oracle DBA position. First, I asked the candidate if he knew what was an SCN was, and he did. Next I asked about incremental backups, and he knew what those were too. Thirdly, I asked him if he knew what a physical standby was, and again, he did. All good so far, but these are just facts that can be recited from memory, or looked up on Google (as I did to make those links!). A better measure of a candidate is a realistic problem to solve, so next I described a scenario in which a standby database is stuck on WAIT_FOR_GAP and All defined FAL servers have been attempted, and the archived redo log is no longer on the primary (e.g. deleted by a naive cron job† that catches everything older than a day). He replied that the only way to solve this problem was to completely rebuild the standby, as if setting it up for the first time. I reminded him of the previous three facts we had discussed, and asked again, and got the same reply.

Now a strong candidate would have thought about this (out loud):

1. I can easily find the SCN of the standby database
2. A physical standby database is a block-by-block copy of its primary
3. I can do an incremental backup of the primary from that SCN and apply it to the standby.
4. Then resume log shipping (having stopped deletion on the primary if the previous step would take longer than the retention period)

Of course, in that particular case, it is a documented technique, tho’ if you hadn’t happened to have read it, it’s still obvious, particularly when you have established all the underlying facts already. In fact this is true about most things in IT: if you grasp the underlying principles then, even if you don’t know the specific command or syntax‡ off the top of your head, you can probably figure out the solution, or at least, be able to re-frame the problem in a way that someone else can jump in and help with just the piece you’re missing. At least you know what to ask or what to look for. There was obviously more to the interview than just that, but with a few other other scenario questions, a clear pattern emerges, giving an accurate picture of the candidate. That’s the point of thinking out loud, and freezing up is the last thing you want in a DBA when there is a production outage! Those are the kinds of people I look for, and this experience sets me up very well when I am on the other side of the table.

I wrote a few months ago about a slight sense of ennui with my present role; the time has come to do something about that, so next month I join the dangerous, glamorous world of algo trading. The people that I’ve hired will get along just fine without me (tho’ some of them should have been paying more attention over the last few years to learning Python!) and so will the organization… if it manages to hang onto them.

† Unfortunately, even in these days of RMAN, those are still common
‡ For this reason, I almost never ask candidates to write code on a whiteboard, tho’ sometimes I will have them draw a diagram

Two Analogies and An Observation

In the Army, there are basically two kinds of soldiers: enlisted and commissioned. Enlisted soldiers fire weapons, drive vehicles, guard checkpoints and go on patrol and so forth – they are the “hands-on” bit of the Army. You would start off as a Private, get promoted to Corporal, then Sergeant, Colour Sergeant and finally the highest enlisted rank, Sergeant Major. The other type of soldier, the commissioned officers, coordinate many small teams of enlisted soldiers in order to achieve some grand objective. The obvious parallel between the Army and a commercial organization is no coincidence; everything about the commercial world is based on the military, even suits are descended from uniforms. Call it blue/white collar, programmers/managers, associates/partners, workers/party, it’s basically the same as enlisted/commissioned. The latter has explicit power over the former, but at the same time, is completely dependent on them.

Many organizations have a career path that requires that in order to progress beyond a certain level, measured in responsibility, or pay, or title, or any other metric, you must “get a commission”, i.e. become a manager. Using our military analogy a career at such an organization would go Private → Sergeant → Lieutenant → Major, perhaps with the interim ranks too. There’s nothing inherently wrong with this approach; Roman Centurions started out in the ranks and no-one can argue that they weren’t a wildly successful Army in their day. The “commissioning” may involve getting a qualification like an MBA, or it might be informal, an experienced programmer segueing from mentoring juniors to officially leading the team with just a change in title. An alternative approach adopted by some organizations is separate career tracks for technical people and managers, usually at the behest of the former. You might in such an organization, on the technical track, start as a Junior Engineer and work your way up to Principal Engineer. But here’s the thing: the most experienced, most respected, most decorated Sergeant Major in the Army still has to salute the greenest graduate trainee Lieutenant fresh out of training. There is no such thing as parallel career tracks. It is in fact still linear. The engineer track ends on “Architect” (whatever that means) not CTO, and such an organization will inevitably be controlled by people who have little or no experience of doing what the workers actually do. Maybe that doesn’t matter, if your organization has the institutional memory the Army does, but in high-tech it matters very much. I believe that advocating a technical career track is an abdication of responsibility by the engineering staff, and not in their or the wider organization’s best interests. Even if you personally wish to remain technical (and FWIW I do), you will be better off in an organization with a single career track and no shortcuts, so the management you work for is technically savvy.

Speaking of ceilings, there is another phenomenon I have observed, in job ads, and I would therefore assume in the contracts of many working engineers: a compensation package defined as a base salary plus a certain percentage bonus. Let’s say for sake of argument that you have a base of £50k and a 20% bonus. If the company meets all of its objectives, usually defined as a certain level of revenue and a certain level of profit, then you can expect a bonus of £10k. Seems reasonable on the face of it, but let’s dig a little deeper. Where does the objective come from? Well it is actually only finger in the air figures; maybe the person who comes up with them isn’t very good at estimating, either way, you have no influence on this. Who decides whether it has been met or not? The company’s profit is whatever the CFO wants it to be really; maybe it’s in the best interest of the company for tax reasons say, to make no profit this quarter, so he can shuffle some expenditure around: sometimes a company will beat revenue but miss profit by 1%… No bonuses. Or the opposite, if the company makes the expected profit on less revenue, is that really a bad thing, as it means greater efficiency has been achieved, but still, no bonus. And finally, what happens if the company performs brilliantly and smashes all its targets? Well the senior managers pocket multi-hundred-percent bonuses and the rank-and-file get at most 20%. They might do that anyway, regardless of company performance! It’s not an eligibility at all. It’s a cap, and it’s not applied to everyone, particularly not the person who ultimately sets the compensation policy… When you see something like this in a job ad, think of it as a job with no bonus, and be certain that you are happy with the base salary alone. If you’re not, negotiate or keep looking, but don’t tilt at windmills. Too many times I’ve seen people spend a bonus that never materialized.

My final analogy for today will be the restaurant industry. Let’s say you are a top chef, the kind running a Michelin starred establishment. What sort of people would you want to work in your kitchen? Now I am speculating here, having never done this myself, but I would guess that while you would obviously want people who can execute complex tasks quickly without sacrificing quality, and who are willing to work like sled dogs, you are also looking at their potential to become top chefs themselves one day. Not least so you can leave one of them in charge while you are off promoting your book or TV show, but also because there is satisfaction to be found in nurturing talent and seeing it blossom. A good chef goes to the market in the morning and buys whatever is the finest produce of the day, and that is what is on the menu when the restaurant opens for business, not what’s in the recipe book. He improvises when he has to, actively seeks opportunities to experiment and he makes his money by being creative. And an integral part of that is surrounding himself with a great team (we are back to the enlisted/commissioned relationship).

Now consider the manager of a fast food restaurant, or better yet, the manager responsible for the menu of an entire franchise. What sort of people does he want in the kitchen? Well, again speculating, I don’t think he really cares much, so long as they can produce the same burger every time, from lowest-common-denominator ingredients bought in bulk. Towards that end, every aspect of the process is carefully designed to be foolproof, so it can be done by people who are cheap to hire and easy to replace, and any investment is an investment in the process and perhaps the equipment; not the people, because they have a habit of quitting and never looking back! A manager may be thinking about shareholder value, and that’s fine, but I bet there are no programmers thinking that it would be great to have every product perfectly specced and project micro-managed so they can just code flip burgers all day!

The moral of these three stories is, have a care what you ask for, because you might get it, and it might not be what you think it is!

Extra disclaimer: any reference to any organization is purely coincidental, and my opinions are mine alone.

Scrum

Perhaps fittingly during the Rugby World Cup I am now a Certified ScrumMaster. As much as you can be a master of anything after a 2-day course ;-)

While I fully agree with the overall Agile strategy of aiming for a return on investment as early in the project as possible, and iterating and adapting to add features as the external environment and the internal understanding of the problem domain evolve, I do have a number of issues with Scrum as it is commonly implemented. Chiefly that it is dangerous both to careers of individuals and to the viability of software development as a profession if the only touchpoint between technical people and “the business” is the Product Owner. Programmers are as intimately involved with the business as say, accountants†, whether they (programmers) are aware of that or not, yet outside of technology companies how much influence do the programmers have? Over even their own working conditions, let alone the direction of the company? What’s the ratio of ex-CFOs to ex-CTOs in the CEO’s chair‡?

As an individual, you must understand that a typical Scrum team is to the rest of the organization as your computer is to you; a black box, presumably replaceable by any other black box. Stories go in, features come out, the bit in the middle could be anyone, anywhere, so long as they’re cheap… By programming the box, you are creating value and deserve credit for that but the box itself is just a tool; well that is how the Product Owner (or their boss) sees it too!

I urge every member of a Scrum team to regularly talk to the same people the Product Owner talks to. Let them know what you do, how you are creating real business value, even if not directly making money for the organization, and what you can do for them, if they can get it onto the backlog. Don’t hide in the comfort zone of only ever speaking to other programmers, about programming, isolated from the bigger picture. Be an asset, not a “resource”.

† Even in companies whose main product is not software
‡ At my previous employer, the CEO came from the legal department and his deputy from HR!

Failing Gracefully

Let’s say you have a high-performance system, operating beyond it’s design capacity already, and the workload is increasing. What would you like it to do?

• Keep running, but slow down. This is the default behaviour for most applications, as in, what you get if you don’t actually plan for this scenario†. If the load increase is temporary, and the responsiveness of the system isn’t critical, this may be acceptable. But there is still a limit. If a system goes into a death spiral, it’s not useful to anyone, it may be impossible to get any useful diagnostics, and then contagion may spread to any connected system. Not good!
• Shed load, either by refusing new work, or by killing or suspending existing work. But this in turn raises a host of questions. Is there some work which cannot be refused? Can the tasks even be suspended (or rather, can they be resumed again) or must they be killed (and can work done so far be undone)? In what order to kill or suspend? Are there any dependancy relationships between jobs that need to be considered? What happens to these jobs, are they abandoned, resumed, or restarted? Can the rules governing all of this be modified in-flight?
• Give up, as in, just crash. This might be the safest option, at least the system is now in a known state, rather than responding erratically or non-deterministically, assuming it was written to be fail-safe. This raises similar questions, but for the app as a whole. Is there another program that will restart it (and if so, under what conditions)? Is there any cleaning up that needs to be done? What happens to work that was in progress at the time? And resources locked on external systems, what frees them, or notifies those systems, if they need to be?

Ultimately, this is a question about the consequences of the system failing under load, which is likely to be at the worst possible moment, when it is doing the most work because you are doing the most business! What is the least-worst option? Obviously this varies from app to app. Maybe there is no least-worst option, so you spend the money to cope with any conceivable load. That’s assuming you can, that the hardware even exists, that your code can be written to exploit it, that your algorithms themselves remain stable. Some apps are written to push hardware to the limits under a normal workload, all or nothing.

But we have to be aware of what the options are, how close every system is to experiencing this scenario, and how to make systems that are aware of their own responsiveness, or at least able to report it to another system, which is in turn empowered to act in some way. You may choose to provoke a crash in the event of a memory leak and restart, rather than allowing responsiveness to degrade, and achieve a higher overall throughput, for example‡. Or a hybrid strategy of tolerating some slowdown and shedding load. Not choosing is choosing too!

† So, most applications then!
‡ According to an old Zed Shaw article, some production Ruby apps required 400 restarts/day (!)

Learning the Wrong Lessons

I’m going to start with an assumption here, which I think is pretty reasonable, that when everyone learns to program, or to operate a system, they do so in an environment much simpler than one that is used for commercial-level work. There is of course real value in this experience: many of the concepts, once grasped in a simple environment, provide a firm foundation for more advanced concepts that you need later. The canonical example of this is the Hello, world! program. It doesn’t do anything useful, but to get those words on your screen, you have to have correctly installed and configured and run a basic toolchain or IDE: shell/desktop → editor → compiler → new program. But there are some bits of the toolchain that a professional developer uses every day, that you don’t need at all at this stage; probably you don’t need any sort of build tool nor do you need a debugger nor a profiler nor version control. The wrong lesson to learn at this stage would be that actually, these things aren’t ever needed. I have seen it happen, but in general, I don’t think this does happen much, people do grow into the rest of the toolchain as and when they need to and are ready for it. The first time you single-step through your own program in a debugger is like magic! The sysadmin aspect of this is, for example, you quickly learn how upgrading one package can break another – but not until you are ready to understand it, you don’t need to worry about that to get started.

When learning to program with databases, it’s a bit more complex because there are more prerequisites (e.g. you need to have a database to talk to, and you have to know how to include or at least refer to external libraries in your program and so on) but the same basic principle applies: a minimal system on which a Hello, world! equivalent can be written. One application, with one user, talking to one database, also with one user. You can create tables, INSERT and SELECT data, and at this stage, for the programs you write for yourself, you don’t need all that fancy stuff like stored procedures, triggers, referential integrity constraints and so on. And why would you? You’re writing the code yourself, so why not keep it all in the one file (no build tools or version control at this stage remember)? And if you want to run some code before every INSERT, why not just do it? But unfortunately something strange seems to happen here that doesn’t happen with compilers, and I don’t know why: this experience gets wrongly carried forward into the real world, and people insist on trying to do everything in their own code, and using the database only as a dumb datastore. You can perhaps get by without a debugger if you are willing to pepper your C code with #ifdef and printf() statements every other line, and it’s possible to do a big project without real version control. But let’s see how the database fallacy plays out in the real world.

In an enterprise setting, it is not at all unusual to have 25, 50 or 100 applications connecting to a single database, representing thousands of connected users. Some of those applications are mission-critical: without them, the business comes grinding to a halt and starts hemorrhaging money. Changing these is a big deal and requires the say-so of senior management. Some of those applications are as old as the organization itself; some were written before the developers now working on them were born (really!). They are probably written in a dozen or more languages. Even programs written in the same language a decade or two apart might as well be in different languages when dependencies, standards and coding styles are taken into consideration. Some are 100,000+ lines of code in hundreds of source files that over years of modification, aren’t as well structured as they once were, and pre-date widespread unit testing. Now let’s say we have some piece of business logic that we need to enforce across the organization. Maybe it’s a new regulatory requirement, maybe it’s a strategic shift in the business, maybe it’s something trivial seeming that is important to the CEO. So how do we go about doing this?

1. Change every application. This is a massive, massive undertaking, requiring people skilled in all the languages, familiar with the domain knowledge, developers, QA, budget for all these people, all their existing projects put on hold and the cost associated with that, business sign-off, risk management if there is some unforeseen interaction between the old code and new. That’s even if there is one single person in the organization who actually knows what all the applications are. Seeing who and what is connected right now doesn’t help – what about that batch job that runs once a week, or once a year? And you have to do it all again next time!
2. Implement a new business rules engine and modify every application to check with that before making any changes to the database. We have now got one central place in which to update our business logic! But we still need to change every application to now also connect to our new service (and over what protocol? COM / CORBA / RPC / some proprietary thing? The new service needs to speak them all fluently) and get its approval for any database operation, and new code for if that permission is refused. It needs to be as reliable and performant as the most mission-critical application. And it still doesn’t enforce anything, since all the applications still need to talk to the database anyway. Are you going to tell the CFO he can no longer use his beloved Excel® because you haven’t figured out how to write a plugin for it for the new service? Not to mention all the third-party apps that expect a standard database connection and don’t have an API.
3. Add integrity/check constraints to the underlying database, with more complex code implemented in the database’s stored procedure language. From this we get one central location to maintain and we get absolute enforcement of the rules even for applications that we don’t know about! We get one language to express the business rules in, throughout the entire application portfolio and lifecycle, since the language du jour changes far more often than the database. And it runs on a system that is already as mission-critical as the most important applications. Errors are handled by the existing code that handles database errors in those applications. There is still the risk that an application might break of course, but of the three scenarios, this is the least, and only the broken application needs any modification, not all of them (and most SP/database mechanisms will allow an exception to be made for one application if that is really, really necessary). Think this doesn’t matter in your greenfield site or small company? Well if your business succeeds, in 30 years time you will wish you had paid heed to my wisdom!

I don’t think I have said anything too far-fetched or controversial here – yet still there is huge resistance to working this way, option 3. Option 2 I see or hear about all the time – one household name IT vendor even used to recommend it! Until they added the capabilities you needed for 3, then they changed their minds. Very few people are crazy enough to attempt option 1. Some things I often hear:

• It’s difficult to version control SP code deployed in the DB. I don’t think that’s any truer than saying it’s difficult to version control Java code deployed in an app server, which is to say, it isn’t difficult at all, it’s commonplace. And over in Ruby-land, entire books are written just about how to get your code from a development environment into production, something that no other language community seems to struggle with. Yet version controlling a stored procedure is apparently too difficult.
• Stored procedures are hard to test. This is a strange one. For a start, SPs are strongly typed; the compiler will tell you if there’s a code path in or out that doesn’t make sense, and in Oracle at least, will calculate all the dependencies for you. So that’s one set of common unit tests you might need in Ruby eliminated off the bat. To test OO code requires mocking to coerce the object into the internal state required to represent the test scenario – how is setting up test data any different? There is a TAP producer for PL/SQL and other tools besides. There are debuggers and profilers too.
• A stored procedure language is not a fully-featured language. Well, we aren’t trying to write an entire application just in stored procedures! Most dedicated SP languages have all the modern constructs that you would expect, and in Oracle at least, you can use Java Stored Procedures with all the language features OO developers are familiar with, or external procedures in any language. What matters is where the logic is implemented – in one place, close to the data – the actual language is just a detail. PL/SQL compiles to native code and runs in-process with the database; there is no higher-performance architecture than that.
• I don’t want to have to learn another language. Overlooking for a second this is a huge red flag in any developer (especially one that proposes modifying production apps which might be in other languages anyway!) there is a lot to learn to work in any modern environment: a typical Java shop might have Eclipse, WebLogic, Maven, Hudson, Anthill, Subversion, and a whole plethora of others, that you need to learn before writing a single line of application code. A working knowledge of a very high level SP language is straightforward in comparison, and there will more than likely be a specialist or a DBA around to help you too. Not to mention that developer favourite Hibernate comes with its own query language…

Speaking of Hibernate leads me onto ORMs in general, another attempt to avoid writing SQL at any cost. Some say that the use of ORM is necessary due to the impedance mismatch tho’ in my experience this is only an issue when developing in a particularly dogmatic OO style. This comes back to my earlier point: a real enterprise has hundreds of applications, and is writing new ones all the time. A normalized relational database doesn’t paint you into any corners – any query that you haven’t even thought of today can be executed against the data, because the underlying maths is sound. If you have “persisted” “objects” into the database, then you will have to deal with true impedance mismatch: any application that doesn’t use the same ORM layer in the same language can’t see your data, at least not without jumping through hoops – at best you have taken on a burden of technical debt and at worst you have frozen your organization in time technologically. Real programs mix procedural, OO and functional styles as appropriate; this is simply a non-issue for any competent, experienced programmer.

And that’s before we come onto the notorious performance problems of ORMs: it doesn’t know when it is retrieving your objects that you only actually wanted 1 of 20 columns in the underlying table, and that you were planning to sum them – something the DB could have done for you and sent only the result over the network – so it grabs them all, then allocates a full-fledged object for each of them in-memory. Hibernate is particularly bad at joins; it would rather query two whole tables and stitch them together itself – and the developer thinks the database is slow! If your GPS sends you on a roundabout route and you’re late do you say “my car is too slow”?

So what is really happening here? Well, I believe that it just boils down to an old-fashioned demarcation dispute (and two can play at that game – but I prefer to identify with the overall development effort, not a language tribe). For the ubiquitous scenario I have described there is a very clear “right way” to do it, at both technical and business levels, just as no serious developer now would argue that the GOTO statement is the right way to program, nor that we should be writing applications in assembly language. SQL, referential integrity constraints, triggers and stored procedures are powerful tools; if you are a developer and improvising your own solution, you are taking on huge technical debt, perhaps without even realizing. If you are a manager and your developers won’t use them, you might ask why you are paying them to reinvent a wheel that Oracle has had since 1992 (version 7), and IBM long before, and making it square as they do so…