My First MOOC

Just completed the Data Analysis for Life Sciences XSeries on EdX, my first MOOC. I had been meaning to learn R for a while as I’ve seen some cool stuff being done with it (I am mainly a Python guy), and to try one of these online courses, and it was very interesting to take a peek into a field of computing that I’ve had no exposure to, biostats. Obv a course like this barely scratches the surface of a very deep and broad area, but it was enjoyable to do and a good foundation for future practice. And most interesting computation these days is really matrix manipulation/LinAlg so all the skills are very transferrable. My first degree was in Mech Eng and a lot of it was familiar once I had dredged it up from memory.

I actually started out doing the Microsoft Professional Program in Data Science a while ago after reading an article in El Reg but got sidetracked; I expect I’ll finish that one up in the next few months too in my copious free time.. The real skills are in the maths so again, all very transferrable, across industries and technology platforms. It’s free too, you only pay if you want the cert at the end.

I should really update this blog more often… My technology work and interests are quite different now than when I started it all those years ago… I am mainly planning to use so-called “data science” to explore some government data, perhaps I will write up what I discover (if anything) here, or if I create any tooling that might be useful to anyone else.

Posted in R, Random thoughts | Tagged , , , , , | Leave a comment


I have been having a play with some cloud stuff recently, as hinted at in my last post, and have put together some nice Python objects wrapping APIs/command-line tools so I can do things like:

>>> from cloudlib import Aws # or Azure or VBox
>>> cloud = Aws() 
>>> ip = cloud.spinup_vm()

VBox is just my local machine, which I suppose is what people mean by private cloud :-) I have a bunch of other functions for managing them too, so the main code is transparently the same whatever the VMs are actually running on. This has made it easy to run a few scenarios like running Postgres in a VM somewhere, and Barman in a VM somewhere else. Pretty cool!

I also have come to realize that I have been thinking of AWS and Azure as glorified hypervisors but that isn’t really true; with all the services layered on top of them, it makes more sense to consider them to be fully-fledged operating systems in their own right. And they are clearly mature enough by now to be worth investing some time in, which I don’t think was necessarily the case when I first signed up for AWS in 2011 and found it wasn’t as good a fit for our use cases as the vSphere + 3Par platform we were operating at the time.

Speaking of 3Par, I am astonished that HP has bought SGI now. Never would have seen that coming in the ’90s!

Update July 2017 It’s been pointed out to me that Vagrant does this so I have actually switched to using that. I have also settled on Azure as my cloud of choice…

Posted in Cloud, Linux, Postgres, Python, Random thoughts, Virt | Tagged , , , , | Leave a comment

Turbo Boost

We have a couple of machines at home†:

  • Core i7 – 2 cores, hyperthreading, 3Ghz turbo boosting to 3.5Ghz, 16G RAM
  • Core i5 – 4 cores, no HT, 3.2Ghz, 32G RAM

I compared them using the POV-Ray 3.7 benchmark with a single worker thread and no other workload than normal background tasks, the i7 system completed in 13m25s average, the i5 system in 12m58s. Now going on clock speed alone I would expect the i5 to have been 6% faster, it was only 3%, so clearly clock-for-clock the i7 does have some advantages, but with boost it should have beaten the i5 in this scenario. The whole point of Turbo Boost is that if you are only using one core it can accelerate it but I’m not seeing that it actually does in any useful way.

Other results, with 4 worker threads the i7 runs it in 6m13s, with 2 threads in 7m1s. So there is some advantage in HT of about 12% even for a compute bound task. But the i5 system runs with 4 threads in 3m35s, nothing beats real actual cores, at a higher frequency! Which is obvious really. Everything else is a gimmick.

No regrets buying the i7 box mind, it will do everything I need it to, and these results are the perfect excuse to really get to grips with something I’ve been meaning to do for years, and have only dabbled in so far, which is offloading compute in a serious way to Azure.

† Well more than a couple!

Posted in Cloud, Random thoughts | Tagged , , , | Leave a comment

HP Calculators I Have Owned

Over the years I have owned several HP calculators:

  • HP 28s, my first RPN programmable that I used when an engineering student in the mid-90s, and made the jump from GCSE/A-level Casio to a grown-up calculator.
  • HP 48GX, a very serious machine that I never really used the full power of, but would certainly have been used everyday had I continued into mechanical engineering. Upgrade from the 28s for better programming and graphing. Probably bought in 1996, still used for certain tasks.
  • HP 12C which I acquired while working as a consultant in about 1999 and needing to do TVM calculations, then used later while writing financial software to prototype and validate bond/portfolio pricing calculations. This is the one that presently sits on my desk at work and is still regularly used.
  • HP 17B-II which I bought as an upgrade to the 12C but then discovered I preferred the older one, so it was relegated to the status of a backup. This one is on my desk at home.
  • I also seem to have acquired a modern HP 35s somewhere along the way, probably for nostalgia’s sake. This one gets used for things neither the 12C nor the 17B can do, such as the odd bit of trig or binary/hex.

Quality products, built to last, providing their owners with literally decades of faithful service, just new batteries every now and then. Looking at HP now, it’s sad to see how far they have fallen, and I suspect they haven’t hit the bottom yet.

Posted in Random thoughts | Tagged , | Leave a comment

What is Experience?

Like many people of my generation, I started programming at an early age. At 10, I was writing 6502 assembly language. My beloved BBC Micro lacked built-in flood fill, circle and ellipse drawing, and other graphics commands, and I spent many happy hours working on these. I used interrupts and vectors to place a clock in the corner of the screen, of limited use since the Beeb lacked a battery-backed RTC, but, multitasking baby!! Exciting stuff. I dabbled in making games, I did a little 3D and simple physics modelling, I wrote my own version of LOGO†.

I sometimes come across people with similar backgrounds, who count their programming experience as starting from those days‡. Now it is natural that I would feel an affinity here, but real experience has taught me that that experience doesn’t carry over into programming in a commercial setting. The problem is, a normal, working programmer, does an awful lot of things other than write code. Hobby projects don’t teach you to write documentation, or specs for new features, or clear bug reports that anyone can pick up to reproduce the issue. It doesn’t teach you how to work in a team, how to branch and merge, how to do code reviews, how to mentor others. It doesn’t teach you how to manage priorities or dependencies or balance conflicting demands. It doesn’t teach you how to talk to users or sponsors or vendors. It doesn’t teach you how to make intelligent use of technical debt. It doesn’t teach you about how to dive into unfamiliar code, under a tight time constraint, in a business domain you don’t fully understand, and make your change without breaking anything else (or what to do if you do). Dealing with distributed, heterogeneous systems can make things get very complicated, very quickly, in ways you can’t anticipate unless you’ve done it a few times before. There are a million other things… A guy or girl who does this isn’t even exaggerating their experience, they’re demonstrating that they don’t even understand what experience is. People in their early 20s who think their “10 years” experience puts them on a level with someone in their 30s who has 10 years commercial experience plus their teenage and other personal hobbyist projects. Please stop this!

† Then about 14 I discovered other interests and didn’t do much computing again ’til college at 18. FORTRAN baby!!
‡ Or equivalent for their generation.

Posted in BBC, Business, Random thoughts | Leave a comment

Form vs Function in Applications

When you have been around the industry for a little while, you will see that nearly all applications, especially user-facing ones, are made of only three kinds of things: Forms, Reports and Workflows. A Form is simply an interface, a screen or a page, for the user to input some data. It is made of text fields, checkboxes, drop-down menus, and so on. A good Form offers prompting and validation, such as a text field that is supposed to be a number is actually a number and within a sensible range. It can be dynamic, updating fields depending on each other, all sorts of fancy things, and when the user is done, saves the results in a database.

A Report is a screen or page that accesses a database, often one populated at least in part by Form input, filters, sorts and aggregates the records, and presents them in a useful way, for example a graph or a table. Often Reports have a few Form-like elements too, so the user can refine or drill down into the data. A Workflow simply links Forms and Reports together. Behind both you might have Services (APIs). So 4 things, to make (nearly!) any application you can think of.

Let’s consider a website like Amazon. You go to it, and it runs the default Report. You refine that Report into a selection of things you’re interested in, and select one, fill in a Form to buy it, and the database is updated, one item transferred atomically from their inventory to your basket, by calling a Service. Then in the warehouse someone runs a Report that tells them what to pack and who to ship to, then they fill in the Form to update the status. There’s nothing in this, fundamentally, that couldn’t have been done on a 1971 IBM 3270, it just might not look as pretty to modern eyes, but all the functionality and interactivity would be there. If you’re buying a book, do you really need a photo of the picture on the cover to decide?

Once you start thinking at this level, suddenly it doesn’t matter that the lifetime of the current fashionable web page generation language is only a few years, or the latest Javascript framework will only be around for a few months. Let junior programmers (who I’ll define as those who define themselves primarily in terms of their language rather than their domain) worry about that and worry about running just to keep up with their peers jumping on the latest bandwagon. Concentrate on the meat of the application, and use tried and true languages and platforms to do it. This isn’t an anti-technology rant by the way, just a piece of advice for those ready for the next level.

Posted in Business, C++, Oracle | Leave a comment

Can anyone learn to code?

Anyone can learn to code, we are constantly told these days, but is it true? Yes… in the same sense that it is true that anyone can learn to play a musical instrument. Many people do, and many of them get a great deal of personal satisfaction from it, play in bands, write songs, meet new people, and lots of other good things.

But if a man selling guitars and guitar lessons tells you that playing the guitar is a guaranteed route to a well paid, secure career, then you would be wise to take that with a pinch† of salt. Or worse, if he tries to convince you that your kids should just learn to play the guitar and join a Silicon Roundabout startupband, instead of studying traditional subjects.

Similarly anyone can learn to cook, but the skills required to cater a dinner party for some friends don’t scale up to catering a formal reception for hundreds, or running a restaurant. Not even Jamie Oliver is stupid enough to try to convince anyone his books and TV shows will teach you that. If schools taught people to cook meals from basic ingredients then as a nation we would be healthier, wealthier and happier, but they can’t even manage that… Why does anyone think they can teach “coding”?

† Handful

Posted in Business, Random thoughts | Leave a comment