Windows 10 on Raspberry Pi 2

Windows 10 IoT Core on Raspberry Pi 2I was one of those who ordered the Raspberry Pi 2, when it was announced back in February 2015, off the back of the claims that it would run Windows 10. Not the full desktop version of Windows 10 of course, but a version for simpler devices. Still, it impressed me that here was a $36 computer that could run the latest version of Microsoft Windows.

Unfortunately, while the Pi 2 became available back then, the required version of Windows was not. It’s only been in the last month that Microsoft launched Windows 10 IoT Core, so I’ve finally had a chance to try it out.

For those that also are interested in this option, I thought I’d note down my experiences on installing it, connecting to it and running software on it.

Installing Windows 10 IoT Core

There are some official instructions provided by Microsoft on how to do this. However, they require that you are running Windows 10 on a PC, and none of my computers have Windows 10 yet. I also didn’t want to use up the hard disk space that would be needed if I had set up a Windows 10 virtual machine. I was more interested in unofficial options.

What didn’t work:

  • Using the Python ffu2img tool to convert the official Windows 10 IoT SD card image to something that could be loaded onto the SD card with something like Win32 Disk Imager. The ffu2img developer admits that they are pretty sure that there’s something wrong.
  • Downloading the official Windows 10 Home edition ISO and using the version of DISM in the sources directory there to load the SD card image.

What did work:

  • I got the official Windows 10 IoT Core for Raspberry Pi 2 ISO from Microsoft, opened it, ran the installer, and it put the flash.ffu file in C:\Program Files (x86)\Microsoft IoT\FFU\RaspberryPi2\
  • Next, I got the Windows ADK for Windows 10 installer from Microsoft, and it loaded a suitable version of DISM into C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Deployment Tools\x86\DISM\
  • Then I formatted my SD card using SD Formatter
  • I copied the flash.ffu file into the DISM directory and used it (following the instructions on the Raspberry Pi forums) in an Administrator Command Prompt to copy it onto my SD card
  • I safely ejected the SD card, and popped into the Pi and it booted up fine.

Connecting to Windows 10 IoT Core

Once the Pi got going, I needed to tell it what language to use. I had plugged a decent quality USB keyboard in, but it was extremely finnicky: key presses were seemingly ignored. In the end, I plugged a USB mouse in and it was much more responsive to mouse clicks.

Windows 10 IoT is really designed to run a single GUI application. It boots into one that shows the hostname and IP address for the Pi, as well as displaying some simple tutorial instructions. It’s designed to connect to Visual Studio 2015, and allow a developer to push their application straight to the Pi. However, I don’t work with my Pi that way – I connect into it and configure/run it via a remote shell.

It’s possible to SSH straight into the Pi (as user Administrator, initially, until you set up some other users). You basically get a DOS prompt. Cool! What was less straightforward was getting files onto it.

What didn’t work:

  • SCP – I kept getting an “exec request failed on channel 0” error
  • Trying to get the Pi to download files using an Invoke-WebRequest via PowerShell running on the Pi. The version of PowerShell seems to be missing some modules.

What did work:

  • The Pi appears on the LAN as a Windows network share. You can use a Windows PC and put in \\192.168.1.10\c$ (or whatever your IP address is) and then login as minwinpc\Administrator with your password. Voila!
  • Similarly, on a mac, you can access it via the Finder using Go > Connect to Server smb://192.168.1.14/c$ (or whatever your IP address is). The Pi will also then appear under /Volumes/c$/
  • Once the share has been opened, it’s straightforward to copy files to and from the Pi.

Running Software on Windows IoT Core

As mentioned above, the standard way to get software running on Windows IoT is for Visual Studio to load it onto the Pi over the network. However, I’m more interested in running standard server apps that don’t rely on the Microsoft ecosystem, so I focussed my efforts on getting Node.js to run on the Pi.

Microsoft is doing some very cool stuff around supporting platforms like Node.js and even Python on Windows IoT. It’s still very much in its early days, but shows promise.

Here’s what I did:

  • I downloaded and installed the Node.js Tools for Windows IoT (v1.1) from GitHub. These were installed into C:\Program Files (x86)\Node.js (chakra)\
  • I copied the whole Node.js (chakra)\ installation directory over to the Pi into C:\Node.js\
  • I downloaded the ARM version of node.exe from the same GitHub page as above, which I copied over the top of the previous (Intel version of) node.exe in C:\Node.js\
  • Set up the APPDATA environment variable to be somewhere useful (it wasn’t set for me): set APPDATA=C:\Users\Public
  • Set up other useful environment variables for Node by running: C:\Node.js\nodevars.bat
  • Now commands like “npm install -s express” and “node test.js” work.

While I could run simple Hello World style programs with Node that wrote text out to the screen, I was unable to get working a slightly more advanced Node program that ran a basic webserver.

Conclusions

It was fun to see Windows 10 boot up on the Raspberry Pi. However, I was a little disappointed how limited it was, given how powerful a Pi is with the default Linux-based OS.

Microsoft’s approach to developing for the Raspberry Pi brings something new to the space, and may make the platform more accessible to developers who are already adept with Microsoft tools. Still, it would’ve been nice to see the basic image come with something immediately useful, if only the new Edge web browser (this would’ve make super-cheap Internet Explorer based kiosks really simple to create).

There’s the old saying that you should always wait for the third version of a Microsoft OS. I don’t know if we’ll need to wait that long for a compelling Microsoft OS on Raspberry Pi, but I am excited to see what Microsoft does with this in future.

XBMC on Raspberry Pi 2

Raspberry Pi running OpenELEC with XBMCI got a Raspberry Pi 2 on the first day they were available in Australia. It has twice the memory and is up to six times faster than the old Raspberry Pi, and at some point in the future it will be able to run Windows 10. But in the mean-time, I thought it would be cool to see what sort of media centre appliance I could get going on a $36 computer. This post is for posterity, but also in case it helps others who are trying to get this working.

The default media centre platform is called XBMC, but the first thing I learnt was that it was now called Kodi. According to the Kodi Wiki, there are just two versions that work on the Pi 2. The first one I tried was OSMC, but it is still in Alpha release and not so stable. The other is OpenELEC and v5.0.3 supports the Pi 2.

Following the installation instructions didn’t work for me, perhaps running Windows 7 64-bit caused problems for the Win32 Disk Imager program. So, I tried using WinFLASHTool instead, and it worked for me perfectly.

This got me a media centre on the Pi, but what I really wanted was to be able to control it from the TV remote control – this requires HDMI CEC to work. I have an LG 42LN5710 television, and LG calls their implementation of CEC “Simplink”. There are two ways to turn it on: press the Simplink button on the remote, or press the Input source button on the remote and then the green button on the remote. Neither worked for me.

After a lot of stuffing around, I learned two things that got me on the right track.

Firstly, not every HDMI cable supports CEC. I had a cheap HDMI 1.3 cable that was fine for delivering A/V from the Pi to the TV, but I needed to replace it with a new cable. CEC is implemented in a single wire in the cable, and is apparently mandatory, but not mandatory enough.

Secondly, any HDMI device can communicate with any other HDMI device connected on any HDMI cable using CEC. I had three HDMI devices (including the Pi) plugged into my TV. One of them was misbehaving, and stopping CEC on the Pi from working. I had to unplug the rogue device and reboot the Pi.

After this, I was able to turn on Simplink and the TV identified the Pi as a Simplink device. Excellent!

Pi, Python and I (part 2)

Raspberry PiIn my previous post, I talked about how I’m using a Raspberry Pi to run a Facebook backup service and provided the Python code needed to get (and maintain) a valid Facebook token to do this. This post will be discussing the actual Facebook backup service and the Python code to do that. It will be my second Python program ever (the first was in the previous post), so there will likely be better ways to do what I’ve done, although you’ll see it’s still a pretty simple exercise. (I’m happy to hear about possible improvements.)

The first thing I need to do is pull in all the Python modules that will be useful. The extra packages should’ve been installed from before. Also, because the Facebook messages will be backed-up to Gmail using its IMAP interface, the Google credentials are specified here, too. Given that those credentials are likely to be something you want to keep secret at all costs, all the more reason to run this on a home server rather than on a publicly hosted server.

from facepy import GraphAPI
import urlparse
import dateutil.parser
from crontab import CronTab
import imaplib
import time

# How many status updates to go back in time (first time, and between runs)
MAX_ITEMS = 5000
# How many items to ask for in each request
REQUEST_ITEMS = 25
# Default recipient
DEFAULT_TO = "my_gmail_acct@gmail.com" # Replace with yours
# Suffix to turn Facebook message IDs into email IDs
ID_SUFFIX = "@exportfbfeed.facebook.com"
# Gmail account
GMAIL_USER = "my_gmail_acct@gmail.com" # Replace with yours
# and its secret password
GMAIL_PASS = "S3CR3TC0D3" # Replace with yours
# Gmail folder to use (will be created if necessary)
GMAIL_FOLDER = "Facebook"

Before we get into the guts of the backup service, I first need to create a few basic functions to simplify the code that comes later. Initially, there’s a function that is used to make it easy to pull a value from the results of a call to the Facebook Graph API:

def lookupkey(the_list, the_key, the_default):
  try:
    return the_list[the_key]
  except KeyError:
    return the_default

Next a function to retrieve the Facebook username for a given Facebook user. Given that we want to back-up messages into Gmail, we have to make them look like email. So, each message will have to appear to come from a unique email address belonging to the relevant Facebook user. Since Facebook generally provides all their users with email addresses at the facebook.com domain based on their usernames, I’ve used these. However, to make it a bit more efficient, I cache the usernames in a list so that I don’t have to query Facebook again when the same person appears in the feed multiple times.

def getusername(id, friendlist):
  uname = lookupkey(friendlist, id, '')
  if '' == uname:
    uname = lookupkey(graph.get(str(id)), 'username', id)
    friendlist[id] = uname # Add the entry to the dictionary for next time
  return uname

The email standards expect times and dates to appear in particular formats, so now a function to achieve this based on whatever date format Facebook gives us:

def getnormaldate(funnydate):
  dt = dateutil.parser.parse(funnydate)
  tz = long(dt.utcoffset().total_seconds()) / 60
  tzHH = str(tz / 60).zfill(2)
  if 0 <= tz:
    tzHH = '+' + tzHH
  tzMM = str(tz % 60).zfill(2)
  return dt.strftime("%a, %d %b %Y %I:%M:%S") + ' ' + tzHH + tzMM

Next, a function to find the relevant bit of a URL to help travel back and forth in the Facebook feed. Given that the feed is returned to use from the Graph API in small chunks, we need to know how to query the next or previous chunk in order to get it all. Facebook uses a URL format to give us this information, but I want to unpack it to allow for more targeted navigation.

def getpagingpart(urlstring, part):
  url = urlparse.urlsplit(urlstring)
  qs = urlparse.parse_qs(url.query)
  return qs[part][0]

Now a function to construct the headers and body of the email from a range of information gleaned from processing the Facebook Graph API results.

def message2str(fromname, fromaddr, toname, toaddr, date, subj1, subj2, msgid, msg1, msg2, inreplyto=''):
  if '' == inreplyto:
    header = ''
  else:
    header = 'In-Reply-To: <' + inreplyto + '>\n'
  utcdate = dateutil.parser.parse(date).astimezone(dateutil.tz.tzutc()).strftime("%a %b %d %I:%M:%S %Y")
  return "From nobody {}\nFrom: {} <{}>\nTo: {} <{}>\nDate: {}\nSubject: {} - {}\nMessage-ID: <{}>\n{}Content-Type: text/html\n\n<p>{}{}</p>".format(utcdate, fromname, fromaddr, toname, toaddr, date, subj1, subj2, msgid, header, msg1, msg2)

Okay, now we’ve gotten all that out of the way, here’s the main function to process a message obtained from the Graph API and place it in an IMAP message folder. The Facebook message is in the form of a dictionary, so we can look up the relevant parts by using keys. In particular, any comments to a message will appear in the same format, so we recurse over those as well using the same function.

Note that in a couple of places I call encode("ascii", "ignore"). This is an ugly hack that strips out all of the unicode goodness that was in the original Facebook message (which allows foreign language characters and symbols), dropping anything exotic to leave plain ASCII characters behind. However, for some reason, the Python installation on my Raspberry Pi would crash the program whenever it came across unusual characters. To ensure that everything works smoothly, I ensure that these aren’t present when the text is processed later.

def printdata(data, friendlist, replytoid='', replytosub='', max=MAX_ITEMS, conn=None):
  c = 0
  for d in data:
    id = lookupkey(d, 'id', '') # get the id of the post
    msgid = id + ID_SUFFIX
    try: # get the name (and id) of the friend who posted it
      f = d['from']
      n = f['name'].encode("ascii", "ignore")
      fid = f['id']
      uname = getusername(fid, friendlist) + "@facebook.com"
    except KeyError:
      n = ''
      fid = ''
      uname = ''
    try: # get the recipient (eg. if a wall post)
      dest = d['to']
      destn = dest['name']
      destid = dest['id']
      destname = getusername(destid, friendlist) + "@facebook.com"
    except KeyError:
      destn = ''
      destid = ''
      destname = DEFAULT_TO
    t = lookupkey(d, 'type', '') # get the type of this post
    try:
      st = d['status_type']
      t += " " + st
    except KeyError:
      pass
    try: # get the message they posted
      msg = d['message'].encode("ascii", "ignore")
    except KeyError:
      msg = ''
    try: # there may also be a description
      desc = d['description'].encode("ascii", "ignore")
      if '' == msg:
        msg = desc
      else:
        msg = msg + "<br />\n" + desc
    except KeyError:
      pass
    try: # get an associated image
      img = d['picture']
      msg = msg + '<br />\n<img src="' + img + '" />'
    except KeyError:
      img = ''
    try: # get link details if they exist
      ln = d['link']
      ln = '<br />\n<a href="' + ln + '">link</a>'
    except KeyError:
      ln = ''
    try: # get the date
      date = d['created_time']
      date = getnormaldate(date)
    except KeyError:
        date = ''
    if '' == msg:
      continue
    if '' == replytoid:
      email = message2str(n, uname, destn, destname, date, t, id, msgid, msg, ln)
    else:
      email = message2str(n, uname, destn, destname, date, 'Re: ' + replytosub, replytoid, msgid, msg, ln, replytoid + ID_SUFFIX)
    if conn:
      conn.append(GMAIL_FOLDER, "", time.time(), email)
    else:
      print email
      print "----------"
    try: # process comments if there are any
      comments = d['comments']
      commentdata = comments['data']
      printdata(commentdata, friendlist, replytoid=id, replytosub=t, conn=conn)
    except KeyError:
      pass
    c += 1
    if c == max:
      break
  return c

The last bit of the program uses these functions to perform the backup and to set up a cron job to run the program again every hour. Here’s how it works..

First, I grab the Facebook Graph API token that the previous program (setupfbtoken.py) provided, and initialise the module that will be used to query it.

# Initialise the Graph API with a valid access token
try:
  with open("fbtoken.txt", "r") as f:
    oauth_access_token = f.read()
except IOError:
  print 'Run setupfbtoken.py first'
  exit(-1)

# See https://developers.facebook.com/docs/reference/api/user/
graph = GraphAPI(oauth_access_token)

Next, I set up the connection to Gmail that will be used to store the messages using the credentials from before.

# Setup mail connection
mailconnection = imaplib.IMAP4_SSL('imap.gmail.com')
mailconnection.login(GMAIL_USER, GMAIL_PASS)
mailconnection.create(GMAIL_FOLDER)

Now we just need to initialise some things that will be used in the main loop: the cache of the Facebook usernames, the count of the number of status updates to read, and the timestamp that marks the point in time to begin reading status from. This last one is to ensure that we don’t keep uploading the same messages again and again, and the timestamp is kept in the file fbtimestamp.txt.

friendlist = {}

countdown = MAX_ITEMS
try:
  with open("fbtimestamp.txt", "r") as f:
    since = '&since=' + f.read()
except IOError:
  since = ''

Now we do the actual work, reading the status feed and processing them:

stream = graph.get('me/home?limit=' + str(REQUEST_ITEMS) + since)
newsince = ''
while stream and 0 < countdown:
  streamdata = stream['data']
  numitems = printdata(streamdata, friendlist, max=countdown, conn=mailconnection)
  if 0 == numitems:
    break;
  countdown -= numitems
  try: # get the link to ask for next (going back in time another step)
    p = stream['paging']
    next = p['next']
    if '' == newsince:
      try:
        prev = p['previous']
        newsince = getpagingpart(prev, 'since')
      except KeyError:
        pass
  except KeyError:
    break
  until = '&until=' + getpagingpart(next, 'until')
  stream = graph.get('me/home?limit=' + str(REQUEST_ITEMS) + since + until)

Now we clean things up: record the new timestamp and close the connection to Gmail.

if '' != newsince:
  with open("fbtimestamp.txt", "w") as f:
    f.write(newsince) # Record the new timestamp for next time

mailconnection.logout()

Finally, we set up a cron job to keep the status updates flowing. As you can probably guess from this code snippet, this all is meant to be saved in a file called exportfbfeed.py.

cron = CronTab() # get crontab for the current user
if [] == cron.find_comment("exportfbfeed"):
  job = cron.new(command="python ~/exportfbfeed.py", comment="exportfbfeed")
  job.minute.on(0) # run this script @hourly, on the hour
  cron.write()

Alright. Well, that was a little longer than I thought it would be. However, the bit that does the actual work is not very big. (No sniggering, people. This is a family show.)

It’s been interesting to see how stable the Raspberry Pi has been. While it wasn’t designed to be a home server, it’s been running fine for me for weeks.

There was an additional benefit to this backup service that I hadn’t expected. Since all my email and Facebook messages are now in the one place, I can easily search the lot of them from a single query. In fact, the Facebook search feature isn’t very extensive, so it’s great that I can now do Google searches to look for things people have sent me via Facebook. It’s been a pretty successful project for me and I’m glad I got the chance to play with a Raspberry Pi.

For those that want the original source code files, rather than cut-and-pasting from this blog, you can download them here:

If you end up using this for something, let me know!

Pi, Python and I (part 1)

Raspberry PiI’ve been on Facebook for almost six years now, and active for almost five. This is a long time in Internet time.

Facebook has, captured within it, the majority of my interactions with my friends. Many of them have stopped blogging and just share via Facebook, now. (Although, at least two have started blogging actively in the last year or so, and perhaps all is not lost.) At the start, I wasn’t completely convinced it would still be around – these things tended to grow and then fade within just a few years. So, I wasn’t too concerned about all the *stuff* that Facebook would accumulate and control. I don’t expect them to do anything nefarious with it, but I don’t expect them to look after it, either.

However, I’ve had a slowly building sense that I should do something about it. What if Facebook glitched, and accidentally deleted everything? There’s nothing essential in there, but there are plenty of memories I’d like to preserve. I really wanted my own backup of my interactions with my friends, in the same way I have my own copies of emails that I’ve exchanged with people over the years. (Although, fewer people seem to email these days, and again they just share via Facebook.)

The trigger to finally do something about this was when every geek I knew seemed to have got themselves a Raspberry Pi. I tried to think of an excuse to get one myself, and didn’t have to think too hard. I could finally sort out this Facebook backup issue.

Part of the terms of my web host are that I can’t run any “robots” – it’s purely meant to handle incoming web requests. Also, none of the computers at home are on all the time, as we only have tablets, laptops and phones. I didn’t have a server that I could run backup software on.. but a Raspberry Pi could be that server.

For those who came in late, the Raspberry Pi is a tiny, single-board computer that came out last year, is designed and built in the UK, and (above all) is really, really cheap. I ordered mine from the local distributor, Element14, whose prices start at just under $30 for the Model A. To make it work, you need to at least provide a micro-USB power supply ($5 if you just want to plug it into your car, but more like $20 if you want to plug it into the wall) and a Micro SD card ($5-$10) to provide the disk, so it’s close to $60, unless you already have those to hand. You can get the Model B, which is about $12 more and gets you both more memory and an Ethernet port, which is what I did. You’ll need to find an Ethernet cable as well, in that case ($4).

When a computer comes that cheap, you can afford to get one for projects that would otherwise be too expensive to justify. You can give them to kids to tinker with and there’s no huge financial loss if they brick them. Also, while cheap, they can do decent graphics through an HDMI port, and have been compared to a Microsoft Xbox. No wonder they managed to sell a million units in their first year. Really, I’m a bit slow on the uptake with the Raspberry Pi, but I got there in the end.

While you can run other operating systems onto it, if you get a pre-configured SD card, it comes with a form of Linux called Raspbian and has a programming language called Python set up ready to go. Hence, I figured as well as getting my Facebook backup going, I could use this as an excuse to teach myself Python. I’d looked at it briefly a few years back, but this would be the first time I’d used it in anger. I’ll document here the steps I went through to implement my project, in case anyone else wants to do something similar or just wants to learn from this (if only to learn how simple it is).

The first thing to do is to head over to developers.facebook.com and create a new “App” that will have the permissions that I’ll use to read my Facebook  feed. Once I logged in, I chose “Apps” from the toolbar at the top and clicked on “Create New App”. I gave my app a cool name (like “Awesome Backup Thing”) and clicked on “Continue”, passed the security check to keep out robots, and the app was created. The App ID and App secret are important and should be recorded somewhere for later.

Now I just needed to give it the right permissions. Under the Settings menu, I clicked on “Permissions”, then added in the ones needed into the relevant fields. For what I want, I needed: user_about_me, user_status, friends_about_me, friends_status, and read_stream. “Save Changes” and this step is done. Actually, I’m not sure if this is technically needed, given the next step.

Now I needed to get a token that can be used by the software on the server to query Facebook from time to time. The easiest way is to go to the Graph API Explorer, accessible under the “Tools” menu in the toolbar.

I changed the Application specified in the top right corner to Awesome Backup Thing (insert your name here), then clicked on “Get access token”. Now I need to specify the same permissions as before, across the three tabs of User Data Permissions (user_about_me, user_status), Friends Data Permissions (friends_about_me, friends_status) and Extended Permissions (read_stream). Lastly, I clicked on “Get Access Token”, clicked “OK” to the Facebook confirmation page that appeared, and returned to the Graph API explorer where there was a new token waiting for me in the “Access token” textbox. It’ll be needed later, but it’s valid for about two hours. If you need to generate another one, just click “Get access token” again.

Now it’s time to return to the Pi. Once I logged in, I needed to set up some additional Python packages like this:

$ sudo pip install facepy
$ sudo pip install python-dateutil
$ sudo pip install python-crontab

And then I was ready to write some code. The first thing was to write the code that will keep my access token valid. The one that Facebook provides via the Graph API Explorer expires too quickly and can’t be renewed, so it needs to be turned into a renewable access token with a longer life. This new token then needs to be recorded somewhere so that we can use it for the backing-up. Luckily, this is pretty easy to do with those Python packages. The code looks like this (you’ll need to put in the App ID, App Secret, and Access Token that Facebook gave you):

# Write a long-lived Facebook token to a file and setup cron job to maintain it
import facepy
from crontab import CronTab
import datetime

APP_ID = '1234567890' # Replace with yours
APP_SECRET = 'abcdef123456' # Replace with yours

try:
  with open("fbtoken.txt", "r") as f:
  old_token = f.read()
except IOError:
  old_token = ''
if '' == old_token:
  # Need to get old_token from https://developers.facebook.com/tools/explorer/
  old_token = 'FooBarBaz' # Replace with yours

new_token, expires_on = facepy.utils.get_extended_access_token(old_token, APP_ID, APP_SECRET)

with open("fbtoken.txt", "w") as f:
  f.write(new_token)

cron = CronTab() # get crontab for the current user
for oldjob in cron.find_comment("fbtokenrenew"):
  cron.remove(oldjob)
job = cron.new(command="python ~/setupfbtoken.py", comment="fbtokenrenew")
renew_date = expires_on - datetime.timedelta(1)
job.minute.on(0)
job.hour.on(1) # 1:00am
job.dom.on(renew_date.day)
job.month.on(renew_date.month) # on the day before it's meant to expire
cron.write()

Apologies for the pretty rudimentary Python coding, but it was my first program. The only other things to explain are that the program sits in the home directory as the file “setupfbtoken.py” and when it runs, it writes the long-lived token to “fbtoken.txt” then sets up a cron-job to refresh the token before it expires, by running itself again.

I’ll finish off the rest of the code in the next post.