Andrew E. Scott

In my previous post, I talked about how I’m using a Raspberry Pi to run a Facebook backup service and provided the Python code needed to get (and maintain) a valid Facebook token to do this. This post will be discussing the actual Facebook backup service and the Python code to do that. It will be my second Python program ever (the first was in the previous post), so there will likely be better ways to do what I’ve done, although you’ll see it’s still a pretty simple exercise. (I’m happy to hear about possible improvements.)

The first thing I need to do is pull in all the Python modules that will be useful. The extra packages should’ve been installed from before. Also, because the Facebook messages will be backed-up to Gmail using its IMAP interface, the Google credentials are specified here, too. Given that those credentials are likely to be something you want to keep secret at all costs, all the more reason to run this on a home server rather than on a publicly hosted server.

from facepy import GraphAPI
import urlparse
import dateutil.parser
from crontab import CronTab
import imaplib
import time

# How many status updates to go back in time (first time, and between runs)
MAX_ITEMS = 5000
# How many items to ask for in each request
REQUEST_ITEMS = 25
# Default recipient
DEFAULT_TO = "my_gmail_acct@gmail.com" # Replace with yours
# Suffix to turn Facebook message IDs into email IDs
ID_SUFFIX = "@exportfbfeed.facebook.com"
# Gmail account
GMAIL_USER = "my_gmail_acct@gmail.com" # Replace with yours
# and its secret password
GMAIL_PASS = "S3CR3TC0D3" # Replace with yours
# Gmail folder to use (will be created if necessary)
GMAIL_FOLDER = "Facebook"

Before we get into the guts of the backup service, I first need to create a few basic functions to simplify the code that comes later. Initially, there’s a function that is used to make it easy to pull a value from the results of a call to the Facebook Graph API:

def lookupkey(the_list, the_key, the_default):
  try:
    return the_list[the_key]
  except KeyError:
    return the_default

Next a function to retrieve the Facebook username for a given Facebook user. Given that we want to back-up messages into Gmail, we have to make them look like email. So, each message will have to appear to come from a unique email address belonging to the relevant Facebook user. Since Facebook generally provides all their users with email addresses at the facebook.com domain based on their usernames, I’ve used these. However, to make it a bit more efficient, I cache the usernames in a list so that I don’t have to query Facebook again when the same person appears in the feed multiple times.

def getusername(id, friendlist):
  uname = lookupkey(friendlist, id, '')
  if '' == uname:
    uname = lookupkey(graph.get(str(id)), 'username', id)
    friendlist[id] = uname # Add the entry to the dictionary for next time
  return uname

The email standards expect times and dates to appear in particular formats, so now a function to achieve this based on whatever date format Facebook gives us:

def getnormaldate(funnydate):
  dt = dateutil.parser.parse(funnydate)
  tz = long(dt.utcoffset().total_seconds()) / 60
  tzHH = str(tz / 60).zfill(2)
  if 0 <= tz:
    tzHH = '+' + tzHH
  tzMM = str(tz % 60).zfill(2)
  return dt.strftime("%a, %d %b %Y %I:%M:%S") + ' ' + tzHH + tzMM

Next, a function to find the relevant bit of a URL to help travel back and forth in the Facebook feed. Given that the feed is returned to use from the Graph API in small chunks, we need to know how to query the next or previous chunk in order to get it all. Facebook uses a URL format to give us this information, but I want to unpack it to allow for more targeted navigation.

def getpagingpart(urlstring, part):
  url = urlparse.urlsplit(urlstring)
  qs = urlparse.parse_qs(url.query)
  return qs[part][0]

Now a function to construct the headers and body of the email from a range of information gleaned from processing the Facebook Graph API results.

def message2str(fromname, fromaddr, toname, toaddr, date, subj1, subj2, msgid, msg1, msg2, inreplyto=''):
  if '' == inreplyto:
    header = ''
  else:
    header = 'In-Reply-To: <' + inreplyto + '>\n'
  utcdate = dateutil.parser.parse(date).astimezone(dateutil.tz.tzutc()).strftime("%a %b %d %I:%M:%S %Y")
  return "From nobody {}\nFrom: {} <{}>\nTo: {} <{}>\nDate: {}\nSubject: {} - {}\nMessage-ID: <{}>\n{}Content-Type: text/html\n\n<p>{}{}</p>".format(utcdate, fromname, fromaddr, toname, toaddr, date, subj1, subj2, msgid, header, msg1, msg2)

Okay, now we’ve gotten all that out of the way, here’s the main function to process a message obtained from the Graph API and place it in an IMAP message folder. The Facebook message is in the form of a dictionary, so we can look up the relevant parts by using keys. In particular, any comments to a message will appear in the same format, so we recurse over those as well using the same function.

Note that in a couple of places I call encode("ascii", "ignore"). This is an ugly hack that strips out all of the unicode goodness that was in the original Facebook message (which allows foreign language characters and symbols), dropping anything exotic to leave plain ASCII characters behind. However, for some reason, the Python installation on my Raspberry Pi would crash the program whenever it came across unusual characters. To ensure that everything works smoothly, I ensure that these aren’t present when the text is processed later.

def printdata(data, friendlist, replytoid='', replytosub='', max=MAX_ITEMS, conn=None):
  c = 0
  for d in data:
    id = lookupkey(d, 'id', '') # get the id of the post
    msgid = id + ID_SUFFIX
    try: # get the name (and id) of the friend who posted it
      f = d['from']
      n = f['name'].encode("ascii", "ignore")
      fid = f['id']
      uname = getusername(fid, friendlist) + "@facebook.com"
    except KeyError:
      n = ''
      fid = ''
      uname = ''
    try: # get the recipient (eg. if a wall post)
      dest = d['to']
      destn = dest['name']
      destid = dest['id']
      destname = getusername(destid, friendlist) + "@facebook.com"
    except KeyError:
      destn = ''
      destid = ''
      destname = DEFAULT_TO
    t = lookupkey(d, 'type', '') # get the type of this post
    try:
      st = d['status_type']
      t += " " + st
    except KeyError:
      pass
    try: # get the message they posted
      msg = d['message'].encode("ascii", "ignore")
    except KeyError:
      msg = ''
    try: # there may also be a description
      desc = d['description'].encode("ascii", "ignore")
      if '' == msg:
        msg = desc
      else:
        msg = msg + "<br />\n" + desc
    except KeyError:
      pass
    try: # get an associated image
      img = d['picture']
      msg = msg + '<br />\n<img src="' + img + '" />'
    except KeyError:
      img = ''
    try: # get link details if they exist
      ln = d['link']
      ln = '<br />\n<a href="' + ln + '">link</a>'
    except KeyError:
      ln = ''
    try: # get the date
      date = d['created_time']
      date = getnormaldate(date)
    except KeyError:
        date = ''
    if '' == msg:
      continue
    if '' == replytoid:
      email = message2str(n, uname, destn, destname, date, t, id, msgid, msg, ln)
    else:
      email = message2str(n, uname, destn, destname, date, 'Re: ' + replytosub, replytoid, msgid, msg, ln, replytoid + ID_SUFFIX)
    if conn:
      conn.append(GMAIL_FOLDER, "", time.time(), email)
    else:
      print email
      print "----------"
    try: # process comments if there are any
      comments = d['comments']
      commentdata = comments['data']
      printdata(commentdata, friendlist, replytoid=id, replytosub=t, conn=conn)
    except KeyError:
      pass
    c += 1
    if c == max:
      break
  return c

The last bit of the program uses these functions to perform the backup and to set up a cron job to run the program again every hour. Here’s how it works..

First, I grab the Facebook Graph API token that the previous program (setupfbtoken.py) provided, and initialise the module that will be used to query it.

# Initialise the Graph API with a valid access token
try:
  with open("fbtoken.txt", "r") as f:
    oauth_access_token = f.read()
except IOError:
  print 'Run setupfbtoken.py first'
  exit(-1)

# See https://developers.facebook.com/docs/reference/api/user/
graph = GraphAPI(oauth_access_token)

Next, I set up the connection to Gmail that will be used to store the messages using the credentials from before.

# Setup mail connection
mailconnection = imaplib.IMAP4_SSL('imap.gmail.com')
mailconnection.login(GMAIL_USER, GMAIL_PASS)
mailconnection.create(GMAIL_FOLDER)

Now we just need to initialise some things that will be used in the main loop: the cache of the Facebook usernames, the count of the number of status updates to read, and the timestamp that marks the point in time to begin reading status from. This last one is to ensure that we don’t keep uploading the same messages again and again, and the timestamp is kept in the file fbtimestamp.txt.

friendlist = {}

countdown = MAX_ITEMS
try:
  with open("fbtimestamp.txt", "r") as f:
    since = '&since=' + f.read()
except IOError:
  since = ''

Now we do the actual work, reading the status feed and processing them:

stream = graph.get('me/home?limit=' + str(REQUEST_ITEMS) + since)
newsince = ''
while stream and 0 < countdown:
  streamdata = stream['data']
  numitems = printdata(streamdata, friendlist, max=countdown, conn=mailconnection)
  if 0 == numitems:
    break;
  countdown -= numitems
  try: # get the link to ask for next (going back in time another step)
    p = stream['paging']
    next = p['next']
    if '' == newsince:
      try:
        prev = p['previous']
        newsince = getpagingpart(prev, 'since')
      except KeyError:
        pass
  except KeyError:
    break
  until = '&until=' + getpagingpart(next, 'until')
  stream = graph.get('me/home?limit=' + str(REQUEST_ITEMS) + since + until)

Now we clean things up: record the new timestamp and close the connection to Gmail.

if '' != newsince:
  with open("fbtimestamp.txt", "w") as f:
    f.write(newsince) # Record the new timestamp for next time

mailconnection.logout()

Finally, we set up a cron job to keep the status updates flowing. As you can probably guess from this code snippet, this all is meant to be saved in a file called exportfbfeed.py.

cron = CronTab() # get crontab for the current user
if [] == cron.find_comment("exportfbfeed"):
  job = cron.new(command="python ~/exportfbfeed.py", comment="exportfbfeed")
  job.minute.on(0) # run this script @hourly, on the hour
  cron.write()

Alright. Well, that was a little longer than I thought it would be. However, the bit that does the actual work is not very big. (No sniggering, people. This is a family show.)

It’s been interesting to see how stable the Raspberry Pi has been. While it wasn’t designed to be a home server, it’s been running fine for me for weeks.

There was an additional benefit to this backup service that I hadn’t expected. Since all my email and Facebook messages are now in the one place, I can easily search the lot of them from a single query. In fact, the Facebook search feature isn’t very extensive, so it’s great that I can now do Google searches to look for things people have sent me via Facebook. It’s been a pretty successful project for me and I’m glad I got the chance to play with a Raspberry Pi.

For those that want the original source code files, rather than cut-and-pasting from this blog, you can download them here:

• setupfbtoken.py
• exportfbfeed.py

If you end up using this for something, let me know!

Tags: code, facebook, gmail, python, raspberry pi, software

I’ve been on Facebook for almost six years now, and active for almost five. This is a long time in Internet time.

Facebook has, captured within it, the majority of my interactions with my friends. Many of them have stopped blogging and just share via Facebook, now. (Although, at least two have started blogging actively in the last year or so, and perhaps all is not lost.) At the start, I wasn’t completely convinced it would still be around – these things tended to grow and then fade within just a few years. So, I wasn’t too concerned about all the *stuff* that Facebook would accumulate and control. I don’t expect them to do anything nefarious with it, but I don’t expect them to look after it, either.

However, I’ve had a slowly building sense that I should do something about it. What if Facebook glitched, and accidentally deleted everything? There’s nothing essential in there, but there are plenty of memories I’d like to preserve. I really wanted my own backup of my interactions with my friends, in the same way I have my own copies of emails that I’ve exchanged with people over the years. (Although, fewer people seem to email these days, and again they just share via Facebook.)

The trigger to finally do something about this was when every geek I knew seemed to have got themselves a Raspberry Pi. I tried to think of an excuse to get one myself, and didn’t have to think too hard. I could finally sort out this Facebook backup issue.

Part of the terms of my web host are that I can’t run any “robots” – it’s purely meant to handle incoming web requests. Also, none of the computers at home are on all the time, as we only have tablets, laptops and phones. I didn’t have a server that I could run backup software on.. but a Raspberry Pi could be that server.

For those who came in late, the Raspberry Pi is a tiny, single-board computer that came out last year, is designed and built in the UK, and (above all) is really, really cheap. I ordered mine from the local distributor, Element14, whose prices start at just under $30 for the Model A. To make it work, you need to at least provide a micro-USB power supply ($5 if you just want to plug it into your car, but more like $20 if you want to plug it into the wall) and a Micro SD card ($5-$10) to provide the disk, so it’s close to $60, unless you already have those to hand. You can get the Model B, which is about $12 more and gets you both more memory and an Ethernet port, which is what I did. You’ll need to find an Ethernet cable as well, in that case ($4).

When a computer comes that cheap, you can afford to get one for projects that would otherwise be too expensive to justify. You can give them to kids to tinker with and there’s no huge financial loss if they brick them. Also, while cheap, they can do decent graphics through an HDMI port, and have been compared to a Microsoft Xbox. No wonder they managed to sell a million units in their first year. Really, I’m a bit slow on the uptake with the Raspberry Pi, but I got there in the end.

While you can run other operating systems onto it, if you get a pre-configured SD card, it comes with a form of Linux called Raspbian and has a programming language called Python set up ready to go. Hence, I figured as well as getting my Facebook backup going, I could use this as an excuse to teach myself Python. I’d looked at it briefly a few years back, but this would be the first time I’d used it in anger. I’ll document here the steps I went through to implement my project, in case anyone else wants to do something similar or just wants to learn from this (if only to learn how simple it is).

The first thing to do is to head over to developers.facebook.com and create a new “App” that will have the permissions that I’ll use to read my Facebook  feed. Once I logged in, I chose “Apps” from the toolbar at the top and clicked on “Create New App”. I gave my app a cool name (like “Awesome Backup Thing”) and clicked on “Continue”, passed the security check to keep out robots, and the app was created. The App ID and App secret are important and should be recorded somewhere for later.

Now I just needed to give it the right permissions. Under the Settings menu, I clicked on “Permissions”, then added in the ones needed into the relevant fields. For what I want, I needed: user_about_me, user_status, friends_about_me, friends_status, and read_stream. “Save Changes” and this step is done. Actually, I’m not sure if this is technically needed, given the next step.

Now I needed to get a token that can be used by the software on the server to query Facebook from time to time. The easiest way is to go to the Graph API Explorer, accessible under the “Tools” menu in the toolbar.

I changed the Application specified in the top right corner to Awesome Backup Thing (insert your name here), then clicked on “Get access token”. Now I need to specify the same permissions as before, across the three tabs of User Data Permissions (user_about_me, user_status), Friends Data Permissions (friends_about_me, friends_status) and Extended Permissions (read_stream). Lastly, I clicked on “Get Access Token”, clicked “OK” to the Facebook confirmation page that appeared, and returned to the Graph API explorer where there was a new token waiting for me in the “Access token” textbox. It’ll be needed later, but it’s valid for about two hours. If you need to generate another one, just click “Get access token” again.

Now it’s time to return to the Pi. Once I logged in, I needed to set up some additional Python packages like this:

$ sudo pip install facepy
$ sudo pip install python-dateutil
$ sudo pip install python-crontab

And then I was ready to write some code. The first thing was to write the code that will keep my access token valid. The one that Facebook provides via the Graph API Explorer expires too quickly and can’t be renewed, so it needs to be turned into a renewable access token with a longer life. This new token then needs to be recorded somewhere so that we can use it for the backing-up. Luckily, this is pretty easy to do with those Python packages. The code looks like this (you’ll need to put in the App ID, App Secret, and Access Token that Facebook gave you):

# Write a long-lived Facebook token to a file and setup cron job to maintain it
import facepy
from crontab import CronTab
import datetime

APP_ID = '1234567890' # Replace with yours
APP_SECRET = 'abcdef123456' # Replace with yours

try:
  with open("fbtoken.txt", "r") as f:
  old_token = f.read()
except IOError:
  old_token = ''
if '' == old_token:
  # Need to get old_token from https://developers.facebook.com/tools/explorer/
  old_token = 'FooBarBaz' # Replace with yours

new_token, expires_on = facepy.utils.get_extended_access_token(old_token, APP_ID, APP_SECRET)

with open("fbtoken.txt", "w") as f:
  f.write(new_token)

cron = CronTab() # get crontab for the current user
for oldjob in cron.find_comment("fbtokenrenew"):
  cron.remove(oldjob)
job = cron.new(command="python ~/setupfbtoken.py", comment="fbtokenrenew")
renew_date = expires_on - datetime.timedelta(1)
job.minute.on(0)
job.hour.on(1) # 1:00am
job.dom.on(renew_date.day)
job.month.on(renew_date.month) # on the day before it's meant to expire
cron.write()

Apologies for the pretty rudimentary Python coding, but it was my first program. The only other things to explain are that the program sits in the home directory as the file “setupfbtoken.py” and when it runs, it writes the long-lived token to “fbtoken.txt” then sets up a cron-job to refresh the token before it expires, by running itself again.

I’ll finish off the rest of the code in the next post.

Tags: code, facebook, python, raspberry pi, software, source, technology

Yale Theatre

I have been trying out iTunes U by doing the Open Yale subject ECON252 Financial Markets. What attracted me to the subject was that the lecturer was Robert Shiller, one of the people responsible for the main residential property index in the US and an innovator in that area. Also, it was free. :)

I was interested in seeing what the iTunes U learning experience was like, and I was encouraged by what I found. While it was free, given the amount of enjoyment I got out of doing the subject, I think I’d happily have paid around the cost of a paperback book for it. I could see video recordings of all the lectures, or alternatively, read transcripts of them, plus access reading lists and assessment tasks.

The experience wasn’t exactly what you’d get if you sat the subject as a real student at Yale. Aside from the general campus experience, also missing were the tutorial sessions, professional grading of the assessments (available as self-assessment in iTunes U), an ability to borrow set texts from the library, and an official statement of grading and completion at the end. Also, the material dated from April 2011, so wasn’t as current as if I’d been doing the real subject today.

Of these, the only thing I really missed was access to the texts. I suppose I could’ve bought my own copies, but given I was trying this because it was free, I wasn’t really inclined to. Also, for this subject, the main text (priced at over $180) was actually a complementary learning experience with seemingly little overlap with the lectures.

While I tried both the video and transcript forms of the lectures, and while the video recordings were professionally done, in the end I greatly preferred the transcripts. The transcripts didn’t capture blackboard writing/diagrams well, and I sometimes went back and watched the videos to see them, but the lecturer had checked over the transcripts and they had additions and corrections in them that went beyond what was in the video. Also, I could get through a 1hr lecture in a lot less than an hour if I was reading the transcript.

Putting aside the form of delivery, the content of the subject turned out to be much more interesting that I expected at the beginning. Shiller provided a social context for developments in finance through history, explained the relationships between the major American financial organisations, and provided persuasive arguments for the civilising force of financial innovations (e.g. for resource allocation, risk management and incentive creation), positioning finance as an engineering discipline rather than (say) a tool for clever individuals to make buckets of cash under sometimes somewhat dubious circumstances. I’ll never think of tax or financial markets or insurance in quite the same way again.

I will quote a chunk from one of his lectures (Lecture 22) that illustrates his approach, but also talks about how technology changes resulted in the creation of government pension schemes. I like the idea that technology shifts have resulted in the creation of many things that we wouldn’t ordinarily associate with “technology”. By copying his words in here, I’ll be able to find them more easily in the future (since this is a theme I’d like to pick up again).

In any case, while I didn’t find the iTunes U technology to be a good alternative for university education, I think it’s a good alternative to reading a typical e-book on the subject. Of course, both e-books and online education will continue to evolve, and maybe there wont be a clear distinction in the future. But for now, it’s an enjoyable way to access some non-fiction material in areas of interest.

The German government set up a plan, whereby people would contribute over their working lives to a social security system, and the system would then years later, 30, 40 years later, keep a tab, about how much they’ve contributed, and then pay them a pension for the rest of their lives. So, the Times wondered aloud, are they going to mess this up? They’ve got to keep records for 40 years. They were talking about the government keeping records, and they thought, nobody can really manage to do this, and that it will collapse in ruin. But it didn’t. The Germans managed to do this in the 1880s for the first time, and actually it was an idea that was copied all over the world.

So, why is it that Germany was able to do something like this in the 1880s, when it was not doable anywhere else? It had never been done until that time. I think this has to do ultimately with technology. Technology, particularly information technology, was advancing rapidly in the 19th century. Not as rapidly as in the 20th, but rapidly advancing.

So, what happened in Europe that made it possible to institute these radical new ideas? I just give a list of some things.

Paper. This is information technology, but you don’t think – in the 18th century, paper, ordinary paper was very expensive, because it was made from cloth in those days. They didn’t know how to make paper from wood, and it had to be hand-made. As a result, if you bought a newspaper in, say, 1790, it would be just one page, and it would be printed on the smallest print, because it was just so expensive. It would cost you like $20 in today’s prices to buy one newspaper. Then, they invented the paper machine that made it mechanically, and they made it out of wood pulp, and suddenly the cost of paper went down. …

There was a fundamental economic difference, and so, paper was one of the things.

And you never got a receipt for anything, when you bought something. You go to the store and buy something, you think you get a receipt? Absolutely not, because it’s too – well, they wouldn’t know why, but that’s the ultimate reason – too expensive. And so, they invented paper.

Two, carbon paper. Do you people even know what this is? Anyone here heard of carbon paper? Maybe, I don’t know. It used to be, that, when you wanted to make a copy of something, you didn’t have any copying machines. You would buy this special paper, which was – do you know what – do I have to explain this to you? You know what carbon paper is? You put it between two sheets of paper, and you write on the upper one, and it comes through on the lower one.

This was never invented until the 19th century. Nobody had carbon paper. You couldn’t make copies of anything. There was no way to make a copy. They hadn’t invented photography, yet. They had no way to make a copy. You had to just hand-copy everything. The first copying machine – maybe I mentioned that – didn’t come until the 20th century, and they were photographic.

And the typewriter. That was invented in the 1870s. Now, it may seem like a small thing, but it was a very important thing, because you could make accurate documents, and they were not subject to misinterpretation because of sloppy handwriting. … And you could also make many copies. You could make six copies at once with carbon paper. And they’re all exactly the same. You can file each one in a different filing cabinet.

Four, standardized forms. These were forms that had fill-in-the-blank with a typewriter.

They had filing cabinets.

And finally, bureaucracy developed. They had management school. Particularly in Germany, it was famous for its management schools and its business schools.

Oh, I should add, also, postal service. If you wanted to mail a letter in 1790, you’d have trouble, and it would cost you a lot. Most people in 1790 got maybe one letter a year, or two letters a year. That was it. But in the 19th century, they started setting up post offices all over the world, and the Germans were particularly good at this kind of bureaucratic thing. So, there were post offices in every town, and the social security system operated through the post offices. Because once you have post offices in every town, you would go to make your payments on social security at the post office, and they would give you stamps, and you’d paste them on a card, and that’s how you could show that you had paid.

- Robert Shiller, ECON252 Financial Markets, 2011

Tags: econ252, education, financial markets, itunes u, online, open yale, robert shiller, shiller, technology, yale

If you’re under ten years old, stop reading now. Spoilers are coming.

There’s a community of atheists who all teach their children to believe in God. They enjoy seeing the comfort that this brings their kids, and the kids enjoy hearing about Jesus and the various Saints. As the children get older though, they question their parents whether God is real, and the atheist parents go to some trouble to persuade their children that it is so, because they want to keep the beliefs going as long as possible. However, inevitably one of the children discovers that the parents don’t really believe, and then tell all their friends.

Except, there is no such community – I just made it up. It would be absurd. It would also be absurd if a creationist community brought up their children with stories about evolution, or an Islamic community taught their children to believe in the Norse Pantheon.

I found myself reflecting on this over the Easter weekend, as I was caught up in the exercise of teaching children about the Easter Bunny. Are kids really better off with me telling them it’s real when I don’t believe it in myself? I have previously found myself conflicted over the Christmas-time story of Father Christmas / Santa / St Nick, and I expect I’ll find it troubling to get involved in the Tooth Fairy when our kids get older.

An article over at Parenting Science states that one researcher found there was no anger when children found out that their parents were lying to them. But on the other hand, that researcher didn’t interview me, and I recall being angry at the time I found out Santa wasn’t real.

Just like most caring parents, mine took extra effort to build up evidence of Santa’s existence: presents mysteriously appeared under the tree in the dead of night, food left for Santa was eaten, and sometimes Santa even left a note. I stayed up late to try to catch Santa in the act or spot a reindeer, but I never did. One year I did suspect the truth, and confronted my parents, but they denied it and talked me around to believing again. In the end, it was my younger brother who forced the situation, later getting my parents to admit it. I was absolutely distraught. Not really so much because Santa wasn’t real but because I’d been deceived, and (if I’m being honest) that my younger brother managed to discover the truth before me.

However, even if I am an exception (although some other people’s recollections suggest otherwise), and in fact no children are at all distressed by discovering the truth, then why should parents be anxious about their children finding out?

If anything, this is one of the things worrying me about being truthful with my own children: how other parents will react. There is the unspoken basic rule of parenting that no-one else should interfere with how you raise your kids, and others’ children finding out the truth from my own children could be seen as interfering. Unfortunately, it’s not clear how I could tell the truth to my own children and yet prevent them from telling this to their friends.

Still, learning the truth didn’t prevent me from continuing to enjoy Christmas and Easter traditions. An easter egg hunt is still fun even if the eggs were hidden by adults rather than a mystical rabbit. Receiving presents is still a delight even if it is adults giving them. I don’t feel I’ve lost anything important by gaining the truth about what is really going on. All the good stuff keeps happening, despite what Virginia was told.

One strategy I’ve heard is to share the truth but engage in some kind of doublethink where children are told that if they stop believing, then the good stuff  will stop happening, eg. “if you don’t believe in Santa, he won’t bring you a present”. This doesn’t sit well with me, as the solution to a lie from an adult appears to be to invite lies from children: even if they don’t believe, they have to say that they do.

Another strategy I’ve heard is make the truth the answer to a puzzle. For example, if a child works it out, let them know they have been a clever-clogs but keep it a secret so as to not spoil their young friends’ and relatives’ efforts to work it out also. However, surely there’s no quicker way to encourage a child to share the secret than to tell them that?

A final strategy I’ve heard is to answer children’s questions truthfully, but position the belief in Santa et al as a game. For example, adults typically don’t have to explain to their children that Peter Pan (or Shrek or Cinderella) isn’t real, and acting out parts of the story in play-time isn’t engaging in deception.  I feel that this strategy is probably a good one, but I’m not sure how easy it will be to implement in practice. It must be possible, since there are a couple of discussions of this approach on the Gransnet forum, including this gorgeous story from “veronica”:

I could not bring myself to lie to my children but they just grew up knowing that FC was a traditional thing that it was fun to keep up. My daughter when she was was about two had a red coat and she dressed up as FC with a beard and distributed presents to those present.

I’m aware that I’m not yet ready for The Question. However, with Easter successfully navigated and Christmas eight months away yet, the need to find The Answer is not an urgent one. But it would be great if Santa could bring it to me as a present.

Tags: belief, easter bunny, father christmas, real, santa claus, tooth fairy

Just before Christmas, a friend brought me a new pair of headphones back from the US. I still haven’t quite decided yet whether they are the future of personal audio or just a step in the right direction, but I am finding them a bit of a revelation.

The headphones are the AfterShokz Sportz M2, which are relatively cheap, bone conduction headphones. Bone conduction means that instead of the headphones sending sound into your ear canal (like in-ear or full size headphones), they sit against the bones of your skull and send vibrations along them to your inner ear. The main advantage is that while listening to audio from these headphones, you can still hear all the environmental sound around you. The main disadvantage is that, of course, you can still hear all the environmental sound around you.

Clearly, this is not desirable for an audiophile. Obviously, you don’t get these sorts of headphones for their audio quality, and while I find them perfectly decent for listening to music or podcasts, the bass is not as good as typical headphones either. That said, if I want to hear the sound better, I can pop a finger in my ear to block out external noise. Sometimes I use the headphones for telephone calls on my mobile when traveling on the tram, and it probably looks a little odd to the other travelers that I am wearing headphones and putting my finger to my ear, but it is very effective.

For the first week or so that I was wearing them, I had strange sensations in my head, very much like when I first get new frames for my glasses. They push on my head in a way that I’m not used to, and it takes a little bit to get used to. The fact that I can hear music playing in my “ears” and yet hear everything around me was also initially a bit surreal – a bit like I was in a movie with a soundtrack – but the strangeness here diminished very quickly and now it is just a delight.

While they are marketed to cyclists or people who need to be able to hear environmental sound for safety reason (like, well, pedestrians crossing roads, so almost everyone I guess), it’s not the safety angle that really enthuses me. I am delighted by being able to fully participate in the world around me while concurrently having access to digital audio. When the announcer at a train station explains that a train is going to be cancelled, I still hear it. When a barista calls out that my coffee is ready, I still hear it. When my wife asks me a question while I’m doing something on the computer, I still hear it.

A couple of years ago, I yearned for this sort of experience:

For example, if I want to watch a TV program on my laptop, while my wife watches some video on the iPod on the couch next to me, we are going to interfere with each other, making it difficult for either of us to listen to our shows.

Being able to engage with people in my physical environment and yet access audio content without interfering with others is very liberating. I had hoped that highly directional speakers were the solution, but bone conduction headphones are a possible alternative.

Initially I had tried headphones that sat in only one ear, leaving the other one free. They were also very light and comfortable. One issue was that these were Bluetooth headphones and had trouble staying paired with several of the devices I had. However, and more importantly, I looked a bit like a real estate agent when I wore them, and was extremely self-conscious. Even trying to go overboard and wear them constantly for a month wasn’t enough to rid me of the sense of embarrassment I felt. Additionally, others would make a similar association and always seemed to assume that I must be on a phone call. If I did interact with others, I always had to explain first that I wasn’t on a call. What should’ve been a highly convenient solution turned out to be quite inconvenient.

The AfterShokz have none of these issues. I did try coupling them with a Bluetooth adaptor, but it had similar Bluetooth pairing issues. I see that AfterShokz have since released headphones with Bluetooth built in, but I haven’t tested these.

One potential new issue with the AfterShokz that I should discuss relates to the ability for others to hear what I’m listening to – this had been mentioned by some other online reviewers. While at higher volumes, others can hear sounds coming from the headphones (although this is not unique to AfterShokz’ headphones), at lower volumes it is actually very private. In any case, I’ve got a niggling sense of a higher risk of damage to my inner ear from listening to music at higher volumes: bone conduction headphones surely need to send sound-waves at higher energy levels than normal headphones because the signal probably attenuates more through bone than through air, and this is coupled with the fact that it needs to be operated at higher levels in order to be heard over background noise that would be otherwise blocked out by normal headphones. So, I try to set it at as low a volume as I can get away with, and block my ear with my finger if I need to hear better. In quiet environments, it’s not an issue.

Perhaps I am worrying about something that isn’t a problem, since I note that some medical professionals who specialise in hearing loss are advocating them. For that matter, the local group that specialises in vision loss is also promoting them. Although, I guess the long term effects of this technology are still unclear.

In any case, I find using this technology to be quite wonderful. I feel that I’ve finally found stereo headphones that aren’t anti-social. I hope if you have the chance to try it, you will also agree.

Tags: aftershokz, audio, bone conduction, future, headphones, review, technology

Our old laptop entered its death spiral a few months ago, but instead of replacing it immediately, we borrowed a stand-in laptop from a kind friend and decided to wait for the slew of Windows 8 compatible laptops that we expected to come in November. Not only would waiting mean that the machines available then be more “future proof”, and cheaper due to competition from other Windows 8 laptops, but we’d be able to pick up (I hoped) a decently-priced touch screen laptop.

Having had the iPad for a couple of years now, and experienced using it with a Bluetooth keyboard, I was completely sold on the idea of a keyboard-enabled device with a touch-screen. The combination of decent keyboard to type with and pleasant touch-based interface is a winner. It also doesn’t hurt that you double the screen real estate by moving from a soft keyboard to a hard keyboard.

So, November came, I put the plan into action, and bought a Sony VAIO E-Series touch laptop for a little over $1,000. It runs a standard Intel i5 processor and comes with a 750 GB hard disk drive. And of course, it had Windows 8.

This post is about sharing my thoughts on Windows 8, having used it now for almost two months. Initially, I was pretty excited with it, but I have since discovered some limitations, so I feel I have a reasonably balanced view of it now.

Major Changes

There are two really big changes that I’ve encountered, coming from Windows 7 up to Windows 8. Many of the other changes stem from these. The changes I’m referring to are: (i) you can now touch the screen to do things, and (ii) the Start Menu has become a Start Screen.

Realistically, Microsoft could’ve introduced native touch screen support in earlier versions of Windows. For example, HP has had this capability on their TouchSmart series of machines. However, it’s not enough for the operating system to be designed around touch if none of the applications are, since controls designed for mouse-based interaction are typically too small to easily manipulate using fingers. So, to introduce touch required Microsoft to push their entire developer community to redesign their applications, and this is logically done together with a major new operating system release.

This may have also spurred Microsoft to redesign their Start Menu. A feature of Windows since Windows 95, it was really a bit too compact for touch and required multiple clicks to navigate which becomes annoying with touch. They could’ve just made the Start Menu bigger and supported scrolling rather than clicking, but instead they took the pretty risky decision to replace the menu with an entirely new screen with a different user interface and its very own app store. Perhaps this understates the level of change. It’s almost as if they decided to replace the humble Start Menu with the entirety of Windows Phone 7.

The Touch Experience

I really love being able to touch the screen. Yesterday I used another laptop without touch, and kept having to pull myself back from touching its screen. It’s not that I use touch at every opportunity: it’s just one way I interact with the interface, along with the keyboard, trackpad and mouse. Some things are best done with a mouse, sometimes the keyboard is best, and some of the time touch is best. This is why I know that eventually touch-screen laptops will become common as those with trackpads.

Windows 8 enables this, but it’s not 100% there yet. Let me tell you about some of the gaps.

When you use touch to control the interface, the mouse pointer disappears. However, since the mouse pointer also used to indicate that the operating system is doing something (a little circular animation appears on it, although in previous versions of Windows it was a sand-timer), having the pointer disappear also leaves me in the dark as to whether that icon I just touched is really launching the program I wanted or whether I was a few pixels out and should really touch it again. Unsurprisingly, sometimes I launch things multiple times. This can get annoying.

It’s not just when launching programs, but any time I try to take an action where there may be a delay. Normally applications rely on the mouse pointer to communicate activity back to the users, so they don’t provide any other indication that things are happening (web browsers are a significant exception). Such applications will need to be rewritten to have an application specific activity indication. Or Microsoft will need to fix this, perhaps in Windows 9.

This tells me that the touch experience was not foremost in the mind of the designers of Window 8. On the contrary, it seems more to be designed around a “keyboard first” principle. Power users are given a range of handy key combinations, and it appears that some of these have been turned into useful gestures, but the whole touch thing isn’t totally elegant.

I find one of the handiest key combinations to be alt-tab, allowing me to quickly switch between applications/windows without having to use the mouse. As this is so useful, this has been converted to a touch gesture: place finger on the left-side bevel outside the screen, swipe to the right onto the screen, then without lifting your finger swipe back to the left. As well as being a clumsy gesture, it doesn’t actually list all the applications since all desktop applications are grouped together.

Another thing is the on-screen “Touch Keyboard”. Despite it being completely unnecessary because this machine is a laptop, ie. it has a keyboard, the Touch Keyboard keeps popping up. It slides up onto the screen when I am logging in, when I’m using Google Chrome, and at other random times. As soon as I touch a key on the real keyboard, the on-screen Touch Keyboard slides away, but I can’t prevent it appearing in the first place. Unchecking the Touch Keyboard Toolbar in the Task Bar properties is a temporary fix, but this resets after rebooting.

Apps and the Start Screen

Despite the Start Screen having the old Start Menu as its heritage, there are two types of application you can start from the Start Screen: (i) Windows desktop applications that we’re all familiar with, and (ii) “apps”. These apps can appear as “live tiles” on the start screen (showing a snippet of content from the full application), a full-screen application with a new touch-centric user interface, or a version of that full-screen application but adapted to fit just a fraction of the screen to allow multiple apps to be on the screen at the same time (not every app necessarily supports this though).  These two types of application live in different worlds.

To get new apps, most users will need to use the Windows Store app to discover and download them. Using the Windows Store is like using Apple iTunes or Google Play, and a Windows Live account needs to be set up with Microsoft before you can download anything, even free apps. This was a pain, since I’d set up one of the computer accounts as a local account for our 4 year old and I didn’t want to set up a Windows Live account for them. Another aspect to apps is that they are associated only with one user. Desktop applications can be installed system-wide for anyone to use, but not these apps. So, it also meant that I couldn’t install apps from the Windows Store under my log-in for my 4 year old to use.

This is not a problem on our iPad, where there is no concept of multiple accounts, so I can easily download apps from the Apple App Store and then my 4 year old can get to them. I guess she’s just going to have to stick to desktop applications for now.

There are a range of built-in apps that are available from the Start Screen, eg. Photos, Music, Video. These are similar to Windows Photo Viewer or Windows Media Player, except they are much simpler and have fewer features, so you might be inclined to just ignore them. Unfortunately, they are the default applications assigned to a large variety of file types. I’ve had to go into the Control Panel and change the defaults back to what they’ve been in previous versions of Windows so that I can actually get things done.

I have downloaded a few useful apps from the Windows Store, such as Skype, a couple of games, and a good internet banking app. However, there are strange omissions, such as no official Facebook or Twitter apps, no iView app, and no YouTube app. Given that Microsoft released their operating systems to developers a long time before they made the final version public for sale, it tells me that it wasn’t for lack of opportunity: these major developers have had absolutely no interest in making their services available as apps on Windows 8.

Developers have generally been pretty slow at updating their desktop applications for Windows 8, also. For example, iTunes 11 was the first version of iTunes that officially supported Windows 8 and it came out well after the public version had shipped (let alone when the original developer versions of Windows 8 were available). Google’s Picasa still doesn’t officially support Windows 8.

Concluding Remarks

Windows 8 is a big change from Windows 7, and users are going to go through a learning curve. However, the rapid uptake of Apple iPads by Windows users has shown that they’re quite happy to learn a completely different interface if there’s enough value in it.

For me, the experience of doing tasks on a Windows 8 touch-screen laptop is better than doing them on an iPad. For example, the freedom of using a powerful and modern web browser like Chrome that also has Adobe Flash support means I can get to all the content on the Internet that I’d ever want to visit – there’s little risk that I’ll come across a site that won’t load or for some reason corrupts my form data when I hit submit – and yet I can tap and swipe to my heart’s content so it is a pleasure to browse. When the experience falls down, it is usually when doing things that can’t be done on an iPad, eg. managing multiple accounts, using desktop applications, or multi-tasking.

Yet it is glaringly obvious that the experience must improve. Both application developers and Microsoft will need to update their software to work properly in this brave new touch-enabled world of Windows. Still, what’s available right now is both fun and useful (notwithstanding several annoyances) and gives me confidence that this world is achievable.

That said, if I didn’t have a touch screen laptop, I’d stay away from Windows 8, and if I didn’t have a high pain threshold when it comes to tinkering with my PC (or have someone in my household like this), I’d hold off on Windows 8 until there was more widespread application support, but for me it was worth the wait.

Tags: laptop, review, sony, touch, vaio, win8, windows, windows 8

Over the last year, I’ve been in a number of discussions where the concept of Winner Takes All was raised, and it’s now starting to annoy me. In a Winner Takes All market, there is a dominant competitor who takes a very large share of the profits. An example at the moment is the mobile phone manufacturing market, where it seems Apple is the winner who is taking all (or, at least, most). However, there may be a widespread view that any market relating to the Internet is Winner Takes All, and that would be a problem.

Winner Takes All is typically put forward to justify either betting big (eg. intentionally making multi-year losses in order to get the scale of users/customers needed to be dominant in a market) or not doing anything (eg. because only one can win anything, and the likelihood is that it won’t be you). In other words, Winner Takes All markets are for only the bravest of the brave. But if anything relating to the Internet is Winner Takes All, then unless you’re pretty special, you should stay off the Internet. Or so the thinking goes.

You might expect that I disagree – and you’d be right. Let me break down why.

Firstly, there are mature businesses on the Internet that have multiple big players, and yet not a single winner. Web mail is a good example, with the biggest services (from Microsoft, Yahoo and Google) having similar sized user-bases.

And while one counter-example is enough to disprove a hypothesis, here’s another one to show the first wasn’t merely an exception. Desktop web browser share is largely split between three big players (Microsoft, Mozilla and Google). Another one is Internet-connected game consoles. I’d say this myth is busted.

A response to the above is granting that not every market relating to the Internet is applicable to Winner Takes All, but that there are some important ones that are. For example, Internet services with “network effects” (those where the more users that adopt it, the more value they are to those users) are in such a market, and Facebook’s dominance in social networking illustrates this.

While this watered-down Winner Takes All view appears more reasonable, there are two lines of evidence that discount it also. The first is the historical record of all the previous social networking services where it appeared there was a winner, but then they lost to a subsequent service that rapidly took over. Back in 2007, MySpace was considered dominant over Facebook, and before MySpace were other services like GeoCities which, according to Wikipedia, in 1999 was the third most visited site on the Internet. If a winner can be displaced so quickly, can they really be said to have “won”?

The second line of evidence is the active competition still occurring in the social networking market. There are both alternative services such as Twitter, LinkedIn and Yammer, and also similar services operating in specific (yet still sizeable) markets such as Qzone, Renren or Sina Weibo in the Chinese language market. If a service isn’t dominant everywhere, can they really be said to have taken it “all”?

But wait, I hear someone say. Cyworld was dominant in the South Korea language market, and yet now Facebook has displaced Cyworld over the course of a year. Doesn’t this show that the same could happen in China and Facebook is operating in a Winner Takes All market? Well, yes it could happen in China, but no, all this shows is that Facebook is a good competitor. There’s no need to explain away Facebook’s appeal through claiming their rise in South Korea was an inevitable consequence of the market structure.

So, I don’t find Winner Takes All convincing, but the danger is that some people believe it and choose not to attempt to launch valuable Internet-based ideas. We users of the Internet would end up deprived of those services as a result. But, it seems the good news is that plenty of people do not believe in the pessimistic world view of Winner Takes All and are happily putting their products and services on the Internet.

Tags: business, internet, market, philosophy, winner takes all

I’m unlike 99% of other Melbourne residents because I’ve used the city’s bike share scheme. In fact, I’ve used it four times now, and I’m a happy customer.

The scheme’s mission is to establish an alternative transport system to things like taxis and trams, but it hasn’t exactly been a raging success. A recent article in The New York Times on bike sharing reported that the Melbourne Bike Share gets about 150 rides a day, and since the scheme offers 600 bikes, at least 75% remain stationary on a given day. That article blames Australia’s helmet laws, but I think bike sharing just hasn’t found the killer application yet.

I don’t use Melbourne Bike Share for transport reasons – I use it for exercise. Instead of paying, say, $19 / week to go to the gym and use a treadmill, I jump on a bike for $2.60 a session, whenever I want. There is an additional cost for a helmet, but that’s just $5 from the nearby 7-Eleven convenience store, so hardly counts in the scheme of things. The whole arrangement seems to be fantastic value.

However, when I’ve been out riding at lunchtime, I haven’t seen another soul using Bike Share bikes. The idea of using them for casual exercise doesn’t seem to have caught on yet. I’m thinking it’ll be like boot-camps in the parks – suddenly all the exercise junkies will realise the value of this public resource and it will be difficult to get in for normal uses.

I know that no-one reads my blog, so I’m not worried about tipping off the world, and hence preventing my own use of the bikes. This is more establishing the evidence of my own use now, so later I can prove that I was in front of this trend. Yes, it’s all about me.

But in case someone is reading this, I thought I’d share some other observations.

Firstly, the 30 min “free” trip length that you get with a $2.60 day pass is not really enough for a good exercise, since down-hill and flat roads plus traffic lights mean I get, at best, two-thirds of the time doing productive up-hill riding. Also, half the time I’ve been a couple of minutes late, meaning I’ve been charged an extra $2 as a result. So, I’m thinking of paying the $52 for an annual pass, which gives you 45 min “free”  trips. (I note that the corporate plan at $100 provides for 60 min “free” trips along with the ability to share it between people, so this option may be better still.) In theory, you could also plan an exercise route that took you past multiple bike stations, where you could swap your bike, starting the clock again.

There are some annoying glitches in the set-up that suggest to me hardly anyone uses this, or they would’ve gotten fixed. For instance, I found when I reached a bike station for the first time that the helmet I got from 7-Eleven had packaging that needs scissors to remove. Also it turned out I’d gotten the wrong size helmet and had to exchange it. The instructions at the bike station terminal when you enter your credit card for a day pass has the message “insert card” on the screen when in fact you need to remove the card to continue. You need to accept a long legal agreement (80 screens at seven lines per screen) each time you rent the bike. My bike code print-out has gotten stuck inside the machine, and I’ve needed to get it to print me another receipt in order to push it out. As a result, people who plan to use the Bike Share just the once, like tourists, may find this all too hard.

As someone who hasn’t owned a bike for something like 20 years, the irony isn’t lost on me that I’m enthusing about bike riding, but as I’ve given the Melbourne Bike Share a go over the last couple of weeks, I’ve come to appreciate it. Melbourne has some rather pretty streets and paths, and my experience riding around in the scenic outdoors sure beats the time I spent pounding on treadmills.

Tags: bicycle, bike share, fitness, gym, melbourne, treadmill

Now the Games of the XXX Olympiad (that’s London 2012 Olympics for us mere mortals) have come to a close, there will be much hand-wringing, soul-searching and other colourful metaphors used within the offices of the Australian Olympic Committee. The AOC forecast fifteen Gold Medals for Australia at this Olympics, not quite the seven we actually received, and at times we ranked on the Gold Medal Tally behind well-known Olympic super-powers like Cuba and New Zealand (g’day neighbours!). This is not to say that our athletes didn’t perform well, that it’s only about bringing home The Gold, or that I could somehow qualify for more than sweat-wiping duty of athletes at the games. However, the AOC is all about exchanging cash for gold and they will now be looking for more cash given that there will be a demand for more gold.

In some ways, it’s an Australian success story. The Australian Institute of Sport was established in 1980, four years after our zero-gold result at the Montreal Olympics. In the years that followed, Australia averaged more than nine gold per Summer games (including London 2012). What is sometimes overlooked is that this has been a scientific triumph as much as a sporting one. While it is the athletes standing on the podium receiving the medals for all to see, it would be justified to also see up there the army of sports scientists and specialists in sports medicine standing there that Australia has willed into being to create our international success. It is probably the Australian scientific organisation with the most influence on the international scene in recent years, but it’s disguised as sport to keep the masses happy. Thanks to the AIS, a country of around 20 million people was able to rank fourth at the Olympics as recently as 2004.

But eventually the limitations of our population size and ability to fund elite sport was going to catch up with us. We had a temporary advantage due to the skills and knowledge in the AIS and the system around it, but information wants to be free. Australian coaches, scientists and even athletes are now helping out other nations. Even Australia’s first individual gold medalist at the London 2012 games, Tom Slingsby, wasted little time after the games to head to the US to help the team defend their America’s Cup title. This is not a slight on the patriotism of those individuals, but recognition that information advantages last for only so long. We now need to find another approach to win a disproportionate share of the Olympic gold.

Our new competitive advantage could be our women. While women made up 46% of the Australian team, wins in their sports accounted for 57% of our Olympic medals (note also that only 46% of the opportunities for a medal are eligible to women). Our first gold medal of these games was from women, and it stood as the only one for nine days. However, while wins in women’s events were responsible for 57% of the medals, they were responsible for only 43% of the gold. Something is not quite right.

Even before the games started, there was some contention that Australia’s female athletes were treated differently from the male athletes. Media reports revealed our women’s basketball team flew to London in economy class while the men’s team flew business. This may be just be the rumblings of a couple of disgruntled athletes, but let’s see if there are some stats that could shine some light on a systemic difference between the men’s and women’s performance at the games.

Despite winning more medals, our team’s women’s medals converted to gold less often than the men’s medals in London. Just 15% of the women’s medals were gold, while 27% of the men’s medals were gold. This difference of 12% between their ability to convert medals to gold, despite coming from the same country, being selected by the same process, and being supported by the same institutions, turns out to be pretty poor compared with other nations. The following table compares all countries who won at least a dozen medals (just so we can have some level of statistical validity):

Australia comes 5th worst, out of 22 countries, right behind Belarus, Iran, Jamaica and Ukraine. Not exactly the company we’d chose to validate our support for women athletes.

If Australia had lifted women’s events conversion to parity with men’s events, we would’ve gotten another two gold on top of our other seven, and we would’ve ranked eighth rather than tenth. So, if the AOC is looking for somewhere to spend to increase our performance in the face of the declining benefits of our leading science, I suggest ploughing money into women’s sport.

NB. For the purposes of the above analysis, I’ve looked at whether sports events are male-only events (like Men’s Marathon), female-only events (like Rhythmic Gynamistics) or support both (like Equestrian or Mixed Doubles in Tennis). When determining if an event is a women’s event, I’ve included the two latter categories, and similarly men’s events include the first and last category.

For those that are interested, my analysis spreadsheet is also available. A more thorough analysis would look at previous summer and winter Olympics to see if this was just a one-off, but I leave that as an exercise for the interested reader.

Tags: 2012, Australia, london, olympics, sport, statistics, women

Legend tells us…

A young Isaac Newton was sitting quietly in his garden when an errant apple fell from a tree above, striking his head. Shocked (and presumably a bit sore), Newton found that he also found himself in possession of the idea of the Universal Law of Gravitation. This law was that any two objects feel a gravitational force towards each other related to their masses and the distance of separation, presumably explaining why the apple hit his head as hard as it did.

Unfortunately, the bit about being struck on the head is fabricated, but the rest of the story is basically true. (And here in Melbourne, there’s an apple tree grown from a cutting taken from Newton’s tree, which I think is pretty cool.) Still, in this story I find the three basic ingredients that I’ve also always found whenever I’ve had a good idea: (1) deep knowledge of a domain, (2) a reflective state of mind, and (3) a trigger.

Ideas may also be called discoveries or inventions, but all are open to being tackled with this same process. Essentially, it is about finding a solution to a known problem.

Deep Knowledge of a Domain

The first part of having deep knowledge is having a grip on what that problem is, but it is also important to know why previous solutions didn’t work. According to Wikipedia’s article on benzene, German chemist Kekule  had the idea that the benzene molecule is arranged in a ring after “years of studying the nature of carbon-carbon bonds” and was a thought-leader in the field, having solved previous carbon bonding questions earlier.

While cramming is generally considered to be a poor study technique, I’ve found it consistently very helpful in exams. One benefit for me is that going into an exam with a deep background in the topic allows me to adapt to the questions on the fly and occasionally come up with a cunning answer.

Reflective State of Mind

The next ingredient is being in the right mental state for the idea to occur. For me, it could be in the shower, at the train stop, or even just before nodding off. In general, I am not doing anything mentally demanding, but waiting for something to happen.

The archetypal example here is that of Archimedes, who realised how to measure the volume of an irregular shape – a crown – when he was relaxing in a bath. In his excitement, he ran from the bath naked, shouting “Eureka!”

This sort of relaxing-in-the-tub, contemplative state is typically not what is produced in a traditional brainstorming workshop, with participants encouraged to shout out ideas as they occur to them. Or, for that matter, in answering a problem-solving question in a job interview. That said, a reflective state of mind doesn’t need to be a meditative state of mind.

I’ve found that in practice, a reflective state of mind can be assumed at will. Perhaps I get practice at this as part of my job, but in most conversations that I have with people, I am looking out for something surprising that doesn’t fit my mental model of the world. I am always looking to be pleasantly disrupted.

A Trigger

I sat on this post for several weeks because I’d initially written this part as “an external trigger” but it wasn’t sitting right with me. External triggers are the stuff of inspiration legend, something external to the problem domain that triggers thinking in the right direction. But another type of trigger is important too.

I don’t mean chocolate, although I do find this very helpful. I’m thinking of thoughts triggered through the application of methodical process, a problem-solving technique, or creative framing/structuring of the problem. These approaches tend to be learned over time, and most people probably have several trusty ones in their problem-solving kit bag. I don’t remember being taught these things formally, but I have adopted some that I liked from seeing their use by experienced people in a field. I’ll call these process triggers, for want of a name, and they can be replicated and re-used. Recently, I used a scenario planning approach to identify an idea for a new service at work.

Compared with process triggers, it is probably impossible to replicate successful external triggers. In the previous examples, it was Newton’s apple falling, Kekule thinking of the symbol of a snake biting its tail (hence the ring of benzene), or Archimedes overflow of bath water. Other people grappling with other problems would likely not find these helpful at all.

However, both types of triggers overcome a problem inherent in having deep domain knowledge – that of being locked within the universe of previous thinking on a topic. A example of an everyday trigger for me was recently when I was listening to a radio interview of someone explaining the evolution of printed media. I was previously trying to identify an appropriate analogy for the evolution of telephony, and wanted something that would go beyond a VoIP story. As the interview progressed, I suddenly realised that the story of print is what I needed.

So…

The ingredients needed to come up with ideas are developing deep domain knowledge, adopting a reflective mental state, and finding a good trigger. Perhaps there’s nothing earth-shattering here, but I’ve found it useful to write it out and convince myself that this makes sense.

While the ingredients are identified here, the trigger may very well not be under personal control, and there may be no useful idea to add to a domain, so it’s not likely to form the basis for a scientifically testable hypothesis. Doing some more work to find ways  to maximise the chance of an external trigger occurring would be valuable, though. Until then, I’ll just have to rely on chocolate.

Tags: creativity, discovery, idea, ingredients, invention, process