Project 365 – week 4

This is now the fourth week of programming 365. I have spent a large amount of time so far on Python and I’m starting to fall a little out of love with it. Although that is quite unfair, I think it would be fairer to say that I am falling out of love with the python challenges. Initially, they were about the right size for a Github post. This week (no doubt due to my lack of knowledge in the area), I have started to find them a little obscure. My main complaint is that they seem to slanted more towards riddles than towards coding. It’s not so much a Python challenge as it is a “let’s decipher what this is trying to get us to do”. This is of course a perfectly valid approach, but feels quite frustrating if you have a daily coding deadline. (either you have an inspiration or you don’t in which case you are kind of stuck). So now – going forward – I will instead focus a little on recapping some of the books I have been reading and using this as an opportunity to actively code my way through some of them…

Day 22

What? Python imaging library as preparation for Python challenge

What did I learn? A little bit PIL – how to open pictures, print information on them, save them in a different format.

Day 23

What? Python imaging library – a little more work

What did I learn? How to rotate images, manipulate individual channels, crop images, filter them

Day 24

What? Python challenge – level 7 (labelled 8 in github)

What did I learn? Application of PIL in the Python challenge

Day 25

What? Python challenge – level 8 (…)

What did I learn? How to use bz2, decompress

Day 26

What? More PIL – contrast, anti-aliassing

What did I learn? …

Day 27

What? More PIL – blur and contours

What did I learn? …

Day 28

What? Started to re-read C++ concurrency in action. Added some sample code from the book.


It’s all about APIs – 22/01/14

On Wednesday, I went to a meetup hosted by IBM about APIs and I thought I should jot down a couple of notes. The meetup was extremely well organised by Laura. It’s the only meetup with an initial detailed Agenda, a reminder a couple of days after (just before the event), the slides posted on the day after the event as well as a summary of the evening writing up Laura’s thoughts on the meetup. (If you are looking for the slides from the evening, you can find them at the meetup group itself.)

The meetup itself was a mix of an initial broad brush introduction to various common concerns API developers share as well as a slightly deeper dive into a particular aspect (API security). The evening was very much a group discussion facilitated through the initial slides given by the presenters.

Some particularly interesting issues raised by the group & slides were:

  • A conceptually clean rest API vs APIs optimised for particular data interactions (or even particular devices)
  • The move away from completely open APIs towards APIs more closely managed with partners
  • Versioning
  • Three legged oauth authentications

Most of the issues are described in a fair amount of detail in the slides with references to further sources so I won’t go into more detail here.

An interesting evening. Hopefully there will be an opportunity to dive deeper into some of these topics at future meetups!

Javascript on devices #2 – 21/01/14

On Tuesday, I went to my second JS on devices meetup. Like the first one, it was full of really useful information and cool tech. It also really showed off the ease of using javascript across a range of devices. Alex did a really great job organising this. It’s also nice to see companies (like Badoo in this case) hosting these kinds of events and enabling great discussions.

Enough of the fluff though. Now for the meat. There were three separate talks – about quite different devices.

Google glass

Radek started off the evening by talking about Google glass.

Interestingly, this was the least technical of the talks. A fair amount of it was focussed on introducing us – the audience – to the device and its capabilities. My guess is that this is because Google glass is in a sense the device most foreign to a tech audience. The user experience is clearly completely different to normal computers, mobile phones, other (sensor based) embedded devices. It also felt like the USB connection-based video feed to Google glass did not manage to fully convey the actual experience of using it. Radek had to work quite hard to bridge the gap to the actual experience!
I’d really love to take one of these for a spin, but at $1500 to join the developer community, it is quite steep and a big investment. Still, some of the things which he showed were extremely cool.

On the tech side, Radek briefly introduced two different ways of developing for Glass: A mirror rest API and the more traditional Glass SDK, GDK. The latter is a close sibling of the Android SDK, but still very much under active development with new releases happening every month.

Radek also had to spend some time justifying glass as a separate device (as opposed to, say, a mobile). His argument was that Google Glass solves the attention problem that many people struggle with. I.e. glasses notify you about important events without distracting you too much from your day to day activities. Working with Google Glass is not as disruptive as getting your phone out. Personally, I have to admit that I still have to be convinced, but – still – I would love to try one!

Run Javascript on Arduino – what, why and how

In the second talk, Alex spoke about how to run javascript on the Arduino. You can find his slides here. He first spent some time giving us some Arduino context which, as an Arduino noob, I found very helpful: He talked about things like the different types of Arduinos and the different extensions (“shields”) you can get for an Arduino. It made me very much want to get one – especially because I don’t know nearly enough about the hardware/circuit side of things and this would really force me to catch up with it.

In terms of running javascript on the Arduino (the title of the talk), the approach Alex presented was that of running a javascript wrapper library on a computer which allows you to abstract the actual hardware instructions. I believe the words “Johny five” and “Noduino” might be appropriate here!

The downside, of course, is that in this setup the Arduino needs to be connected to a computer. However, Alex pointed out that this means that you can leverage the computing power and other in-built facilities (e.g. wifi) from the machine.


Finally, Andrew talked a little bit about Nodecopter. You can find his slides here. Nodecopter is a project which managed to run NodeJS on Parrot’s AR drone 2.0. This drone has got some pretty cool facilities: Front and bottom camera, 4 rotors, ultrasound sensors, 1ghz ARM cpu with 512mb of storage. It is also a flying router – it shows up as an unsecured hotspot.

Andrew showed some pretty awesome hacks for this drone:

  • Controlling the drone via a dance mat
  • Controlling the drone via a Kinect
  • Virus copter

He also talked a little about two different ways of operating the drone: one loading a node cross compiler on to the drone and then loading javascript files via ftp or USB stick so that these can be executed on the drone without needing to interact with the drone over its wifi network, the other of loading different javascript files on to the device and interacting with it that way.

Altogether a pretty awesome evening with exciting talks about three very different, but very cool devices!

Project 365 – Week 3

Third week of programming 365, third post on it – same procedure as usual. This week I have spent almost exclusively on python challenges. It’s been pretty good fun and each level is a nice small, self-contained amount of work. Beyond that, I also wrote up some code about C++ unit testing (in particular the use of std::function). Let’s see what the next week brings: Man, I looked at the “to-do list” from last week’s post and pretty much all of that is still to be done! Maybe this week – who knows?

PS: On the upside, Lynn is starting to get used to the project (the number of threats to break up over it have gone from something than can only be stored in an unsigned 64-bit int to something that can be stored in the signed part of a short).
PPS: On the downside, a lot of the python challenge code is quite rough and could definitely do with a tidy. At some point the focus just became to get through the challenges rather than code them well (booo!)

Day 15

What? Python tutorial – standard library part 2

What did I learn? Some more standard library. I learnt a little about python threads and logging. Not quite as cool as the content of standard library part one, but still pretty useful.

Day 16

What? Python challenge – levels 0 to 2 (labelled 1 to 3 in github)

What did I learn? How to use string translation with translation table and how to filter strings for rare characters.

Day 17

What? Python challenge – level 3 (labelled 4 in github)

What did I learn? Some basic regular expressions. Which is great because I am a total noob at them!

Day 18

What? Python challenge – level 4 (…)

What did I learn? How to make a series of web requests and read the responses

Day 19

What? C++ std::function and unit testing code samples

What did I learn? Wrote an illustration of a technique that I came across earlier in the work to enable unit testing with mininmal refactoring in code bases ruled by C/C++ functions. See earlier post on the blog

Day 20

What? Python challenge – level 5 (…)

What did I learn? Pickling as a way of serializing/deserializing python objects

Day 21

What? Python challenge – level 6 (…)

What did I learn? How to work with zip files in Python

Using std::function to make unit testing easier in C/C++ systems

If your day job is in a brown field C/C++ system, you can sometimes feel a little unloved. On the one hand, the Java guys seem to have all of these cool libraries for enterprise software. On the other hand, you have the web guys with their javascript (“hyyyyyyypeeee”). Finally, you have all of these “high productivity” high-level gluing languages like python.

Luckily, C++ has plenty of tools and lots of exciting things going on in the language. So it can definitely hold its own. Unfortunately, C/C++ shops can sometimes be a little slow to catch on. This is partly because you have a fair amount of existing code base which was written before all of this cool stuff was dreamt up.

In particular, a London-school (mocking) approach to TDD can be quite hard, when a large part of your codebase is ruled by various super-functions which execute lots of other functions. Those other functions may access the database, some other server etc. etc. Here, dependency injection and mocking isolation without a big refactor has traditionally not been very easy.

Say, e.g. you have a function that you want to test, but it calls another function which deals with database interaction, global memory, a request to a web service or something else:

Header file declaration

extern Project365::Widget ProductionCodeLoadWidget(const std::string& rstrWidgetRef);

Cpp file definition

extern Project365::Widget ProductionCodeLoadWidget(const std::string& rstrWidgetRef)
  if (rstrWidgetRef.find(“Database”) != std::string::npos)
    return LoadWidgetFromDatabase(rstrWidgetRef);
  else if (rstrWidgetRef.find(“Web Request”) != std::string::npos)
    return LoadWidgetFromWebRequest(rstrWidgetRef);
    return LoadWidgetFromMemory(rstrWidgetRef);

If this function had been a member of the interface of an object, you would have delegated the function call inside of the interface to another member object in a strategy pattern. To unit test, you would then have mocked that object and passed it in as part of the constructor:

Project365::Widget WidgetLoader::Load(const std::string& rstrWidgetRef)
  if (rstrWidgetRef.find(“Database”) != std::string::npos)
    return m_pdbLoader->Load(rstrWidgetRef);
  else if (rstrWidgetRef.find(“Web Request”) != std::string::npos)
    return m_pwebLoader->Load( (rstrWidgetRef);
    return m_pmemoryLoader->Load(rstrWidgetRef);

WidgetLoader::WidgetLoader(IDatabaseLoader* pdbLoader, IWebLoader* pwebLoader, IMemoryLoader* pmemoryLoader)
: m_pdbLoader(dbLoader), m_pwebLoader(pwebLoader), m_pmemoryLoader(pmemoryLoader) {}

This kind of change would isolate the particular function you want to test. However, it is also a fair amount of effort with some potentially intrusive refactoring – especially if you then want to make sure that your object’s overall responsibilities are coherent and not just a transformation of this function. (To get this to work, you would probably want to use the technique described in “Growing object-oriented software guided by tests”: You’d have an acceptance test layer which seals a larger section of code and so enables this bigger refactoring at smallish risk).

The following is a “little refactor” alternative which enables you to get some initial unit testing coverage so that you can be more confident if you are looking for a larger refactor later on: The essence of it is to supply a std::function object to the function with a default argument of the function which you want to call in production. 

Right, so what does that mean in code. You’d change the function in production code to look something like the following:

Header file declaration

extern Project365::Widget ProductionCodeLoadWidget(const std::string& rstrWidgetRef,
                          std::function<Project365::Widget(const std::string&)> loadWidgetFromDatabase = LoadWidgetFromDatabase,
                          std::function<Project365::Widget(const std::string&)> loadWidgetFromMemory = LoadWidgetFromMemory,
                          std::function<Project365::Widget(const std::string&)> loadWidgetFromWebRequest = LoadWidgetFromWebRequest);

Cpp file definition

extern Project365::Widget ProductionCodeLoadWidget(const std::string& rstrWidgetRef,
                                                         std::function<Project365::Widget(const std::string&)> loadWidgetFromDatabase,
                                                         std::function<Project365::Widget(const std::string&)> loadWidgetFromMemory,
                                                         std::function<Project365::Widget(const std::string&)> loadWidgetFromWebRequest)
  if (rstrWidgetRef.find(“Database”) != std::string::npos)
    return loadWidgetFromDatabase(rstrWidgetRef);
  else if (rstrWidgetRef.find(“Web Request”) != std::string::npos)
    return loadWidgetFromWebRequest(rstrWidgetRef);
    return loadWidgetFromMemory(rstrWidgetRef);

The nice thing about this is that the production calling code can remain unchanged because of the default argument. However, in the unit testing code, we now have the ability to provide an alternative implementation and inject this into the production code. This function that we inject could be a lambda, a more traditional functor or just plain old function). This is what renders the original function that I want to test unit testable.

An implementation in the Google test framework of a test could look something like this:

// Dummy / mock loaders so that we can test which have been called

auto databaseLoader = [&fDatabaseRoutingCalled, &aWidget] (const std::string&) -> Widget { fDatabaseRoutingCalled = true; return aWidget; };
auto webLoader = [&fWebRoutingCalled, &aWidget] (const std::string&) -> Widget { fWebRoutingCalled = true; return aWidget; };
auto memoryLoader = [&fMemoryRoutingCalled, &aWidget] (const std::string&) -> Widget { fMemoryRoutingCalled = true; return aWidget; };

// Hook into production code
ProductionCodeLoadWidget(“DatabaseRef12345” , databaseLoader, webLoader, memoryLoader);

// Expectations
EXPECT_EQ(fDatabaseRoutingCalled, true);
EXPECT_EQ(fWebRoutingCalled, false);
EXPECT_EQ(fMemoryRoutingCalled, false);

Hope you find this technique useful. The full code example can be found in my programming 365 github repo (“Day19”).

Project 365 – Week 2

I am now in the second week of the programming 365 thingy . This week I have put a lot less effort into the project. There just simply has not been the time to do any real “work”! Still, even though the week has been a lot slower, I have really appreciated how the project has forced me into doing at least a little bit of extra learning (20 minutes or half an hour) every day.

The theme of the week has been to finish the Python tutorial that I started last week. I am now almost there. (This shows how slowly the week went. Normally, I would maybe have done this kind of stuff in a day or day and a half.) Next week, I hope to make use of some of the stuff I learnt: maybe interact with Quantlib, maybe try out some NumPy and some of the json/http libraries and give the Python challenge a go too.

PS: I also need to spend a little bit of time to finish off / refactor some of the C++ code I did as well as clean the “Hello Python” solution.

Day 8

What? The Python tutorial – modules part

What did I learn? I learnt all about modules and how to import other python files and make use of functions defined in those files.

Day 9

What? Python tutorial – input and output

What did I learn? How to format strings and use keyword arguments in string formatting (useful way of organising arguments!). As well as file reading and writing (has really nice iteration facilities!)

Day 10

What? Python tutorial – exceptions

What did I learn? All about exceptions. How to raise and catch them. Inbuilt-ones. How to derive new ones. The try, else and finally statements (like else and finally – they are quite neat!)

Day 11

What? Python tutorial – classes

What did I learn? Really this was all about scope rules. Nested functions. What kind of variables you have access to where and how variables change when you modify them (Local vs non-local vs global)

Day 12

What? Python tutorial – classes

What did I learn? Class syntax. Writing a constructor, add methods & attributes. Instantiate objects, call member functions

Day 13

What? Python tutorial – classes

What did I learn? How member functions are bound to particular objects (or not as the case may be). Adding data members dynamically. Attributes are always public. Functions always “virtual”. How inheritance works. How classes really make use of nested scope in achieving what they achieve!

Day 14

What? Python tutorial – standard library – part 1

What did I learn? Some pretty standard stuff (files, path, os things and regexes) and some quite neat stuff. Http requests & responses. Deserializing json into python objects. Really neat acceptance testing. Some unit testing.

Project 365 – Week 1

This is just a quick more humanly readable update of what I have been up to in the programming 365 thingy this week. For a more code based version, have a look at the github repo.

Initially, I was going to write a big post once a week and explain what I had done, learnt, how the mood was for the day, how it contributed to the overall project… To do that, I started keeping some notes every day about the key points of the day. Having gotten a bit lazy (can’t try and find time to figure out the big picture), I will just post these notes instead. I don’t think the stand-alone-blog-post format is ultimately the right format for this. Maybe a separate page for it with some categories etc.? In the spirit of Agile, I’ll implement the content now and refactor it later. So without further ado here are the first week’s p365 notes.

Day 1

What did I do? Implemented a semaphore class. Semaphore class supports indeterminate, filo and fifo release. Wrote some sample code to show the semaphore behaviour.

How did the day go? Pretty successful day. Worked quite hard on different semaphores. Then pulled them together into one class with various impls.

What did I learn? How neatly one can work with condition variables. Overhead required for filo (notify_all) and fifo classes (notify_all and memory footprint).

Day 2

What? Unit test for the semaphore class. Check that a semaphore does not block when it shouldn’t. Tried out a new unit testing framework – catch. In particular liked the BDD style test facility.

How did it go? Okish – didn’t spend enough time in the morning / during lunch. Had a talk in the evening. So it was grab a couple of minutes whenever possible.

What did I learn? How to add catch to a project. How catch behaves with sections (multiple executions). How much better documented our unit tests would be with BDD. How catch compiles (slowly in the one compilation unit that has the main defined). How catch doesn’t like funky destructor behaviour (like e.g. the thread class has).

Day 3

Topic: Migrated tests to google. Added tests about blocked behaviour and filo ordering

Mood: Didn’t feel like much progress. A lot of familiar technologies, but learnt some stuff about futures.

What did I learn? Mutex undefined behaviour when it’s locked when it’s being destroyed (urgh). How to write “an infinite loop” type test in Google: Set off a thread and use the future status “status_not_ready”.

Had my first near miss: My sync failed and only my local commit worked. Luckily committing later on still seemed to work and the local commit set the time stamp for the github commit!

Day 4

Topic: Implemented some topics from chapter three of the Little Book of Semaphores by Downey.

Mood: Mixed – didn’t really learn much, but it felt good to revise the concepts from the book and find that I’m pretty comfortable with them.

What did I learn? That writing a rubbish implementation first is definitely something I do a fair amount of. Today I didn’t get round to nice-fying my implementation so that will have to be done tomorrow. Maybe TDD can help with that?

Day 5

Topic: Re-organised existing code into three separate projects. Much closer to the structure I want to eventually have.

Mood: Pretty good: I realised that I wouldn’t be able to do that much today (busy day otherwise) so I’m feeling happy with some tidy up even though I didn’t learn or do that much.

What did I learn? Am fairly happy with setting projects and solutions up. Learnt a little more about the “needs to have dll interface” warning. Also learnt that I still need to refer to Joshi’s book to write a self registering factory (argh!).

Day 6

Topic: Did a complete topic switch to learn a little bit of python.

Mood: Felt fairly ill (too much waiting in an empty, cold house) so decided to treat myself with working through (some of) the python tutorial.

What did I learn? PTVS is an awesome ide for python. How to install python on windows. Lots of basic stuff like variable assignments, control statements, imports, python interactive etc etc.

Day 7

Topic: Got a little bit further with the python tutorial

Mood: Took it fairly easy today. Running only through a little bit of the python tutorial. Starting to feel a little like this running through a tutorial has done all it can for now. Maybe another day of this and then I’ll try to actually write a little program with python?

What did I learn? Python has got some really neat features. Keyword arguments, arbitrary argument lists and arbitrary list of keyword arguments. Also had some really nice features for working lists and tuples.