People think that “it’s showtime” by building their own cryptosystem and end up driving their users off a cliff – killing them in a firey inferno.

To illustrate how not following standards breaks your security I will show an example brought to me by one of my (unnamed) colleagues just the other day. The problem: we have some values A, B, and C we want to hide from prying eyes. These values can either be -1 or +1. Their proposed scheme was to choose another three values X, Y, and Z that can also only be -1 or +1 rather store the values AX, BY and CZ. Up to this point what they are effectively describing is a One-time Pad, which is an unbreakable encryption scheme. If they have this unbreakable scheme, what can go wrong?

The thing is, they didn’t just want to store A, B, and C they want to hide D, E, and F as well. By using the same values X, Y, and Z. And this is where it breaks down. With a One-time Pad you cannot use the same X, Y, and Z more than once. Let’s see why:

Say we get

AX = 1

BY = 1

CZ = -1

For any choices of A, B and C we can find X, Y and Z that will satisfy these equations. For example:

A = 1, B = -1, C = -1 means that X = 1, Y = -1, Z = 1.

A = -1, B = 1, C = 1 means X = -1, Y = 1, Z = -1.

And you can do this for all 8 possible combinations of A, B, and C. There is no way of knowing which values of A, B and C gave us the output (1, 1, -1). But look what happens as soon as we learn the following:

DX = -1

EY = 1

FZ = -1

Since our variables can only take the values -1 and +1 we know that the square of any variable is equal to 1. Multiplying the corresponding terms we get:

ADXX = AD = -1

BEYY = BE = 1

CFZZ = CF = 1

From this we learn that if A = 1 then D = -1, B = E and C = F. If the One-time Pad was with different pads for each message then learning one of the messages would tell you nothing of the other one, but now if we find out what one message is we can immediately figure out the other one. Another way of leaking information is if we know that these are encryptions of bids for an auction (binary encoded with the 0s set to -1) and A, B, C was sent first we know the second person just bid 4 more than the first guy.

There exists and interesting interchange between the One-time Pad and ASCII encoding. I found out about this from the excellent online cryptography course I am following  Below I have encrypted the two phrases ‘a cat mat’ and ‘THE LOW Q’ with a One-time Pad using the same random pad:

c1 = `Rí!UX}

c2 = Uptﾬ#NoX

Now from these ciphertexts it doesn’t seem like you can figure out anything about the phrases and if you look at each in isolation you can’t. But if you take the two ciphertexts and XOR them together byte by byte you get 5h&A8o:A% and when you compare them to the original phrases:

a cat mat

THE LOW Q

5h&A8o:A%

I’m pretty sure you can see the pattern here. We now know where the spaces occur in the strings as well as some of the letters of the original phrases! Which is quite a lot of information already that we did not want to give away.

Just remember kids, save your users from the inferno by following the standards!

For illustrative purposes, let us consider the age old question “Will it rain tomorrow?”. In statistics we refer to this as a binary classification problem. Without any knowledge one can of course simply guess “yes” or “no” and still be correct approximately 50% of the time. Of course this was not satisfactory. The human thirst for knowledge stepped in and some of the early thinkers realized that there is a correlation between today’s weather and tomorrow’s weather. Predicating that tomorrow’s weather will be the similar to that of today’s is very intuitive concept that improved the accuracy of prediction to about 65%. Once again, the human thirst for knowledge demanded more accuracy. As the field progressed, more complex models were suggested. The demand for a more accurate answer on such questions caused enormous growth in the field of probability theory. Questions were answered, but were the answers accurate?

Let us now assume that we have a fully qualified statistician with full knowledge on all possible methodologies. Such an expert can then choose the appropriate algorithm for the analysis and create a model that will predict with up to about 90% accuracy. Theoretically.

It has been said that statisticians live in asymptopia. This, in my opinion, is where the first flaw in statistics arises. Most of these statistical models makes assumptions on properties of both the features as well as the outcome. These assumptions can either consist of accepting some functional form of the relationship between input and output, accepting some distribution of the error or randomness involved or accepting that some factors are negligible. More often, the assumptions include all of the above; some of which might be relatively accurate if we were to work with an infinite amount of data, but asymptopia is fictional. In reality, these assumptions are at most 90% accurate. One could argue that these models together with their assumptions had to be proven statistically significant with vigorous testing. Yes, but statistical significance only requires 90% confidence.

Furthermore, this ideal model is of course entirely useless unless it can be applied to data. A statistician is then confronted with another age old problem and in general a term dreaded by most statisticians – Data cleaning. In practice, the data we obtain is unprocessed and cannot be used directly for analysis. We have missing values that has to be disregarded. Also outlying values are often ignored. Removing these values often means simply throwing away up to 5% of the obtained data. Furthermore, when measuring the values of the features, we find that there is some inevitable amount measurement error. Out of the remaining 95% of the data, we are only in fact 95% sure of the accuracy of the measured values.

Finally, this is of course all based on the assumption that the person performing the analysis is all-knowing in the field of statistics. Highly unlikely. In Stellenbosch, we find that roughly 60% of courses include a one year module in statistics. This grants degree holders in different professions sufficient knowledge on the field to believe that they are qualified, even though they on average obtain a 70% passing grade for the one year module. This exacerbates all of the before mentioned problems. Such a person will apply an incorrect model to the data which will only theoretically be 80% accurate in prediction. He will not have sufficient knowledge to be able to perform the analysis manually and will make use of software, while probably incorrectly applying the model and incorrectly interpreting the results, causing a 10% reduction in effectiveness of the model.

Summarizing, we find that a under qualified statistician uses a model that we are only 90% sure works, with a theoretical 80% prediction accuracy, based on assumptions that are only 90% accurate and incorrectly applying software resulting in a model being only 90% effective. He will do so using 95% of obtained data even though only being 95% sure of the accuracy of the measurements. When taking all of this into consideration, we end up with a 52% prediction accuracy. Slightly more accurate than guessing.

In conclusion, quoting a personal role model:
“90% of statistics are made up on the spot” – Homer Simpson

We realize, statistics are deemed irrelevant. We in fact know nothing.

For those of you who are not well versed in game achievement systems, an achievement is some form of meta-objective defined outside the parameters of the game. This meta-objective, sometimes known to the player and sometimes left unknown, has no direct effect on the gameplay. Whereas completing a game’s quest or goal will further your progression, improve your character, or make the game easier in some way – achievements are merely an accolade for completing the meta-objectives. Some examples of these meta-objectives might be fully exploring an in-game area, killing an excessive amount of an enemy type, etc.

You could probably group all achievements into one of the following 7 groups:

1. Tutorial: Encouraging players to try new features or learn different skills.
2. Completion: Finishing a section of the game.
3. Collection: Finding every hidden or secret item.
4. Virtuosity: Playing with exceptional skill.
5. Special: Bending the rules of the game by encouraging creative play styles.
6. Dedication: Encouraging many hours of play.
7. Social: Encouraging players to interact with others.

So why do game creators implement achievements into their games? In a lecture given by Chris Hecker at the 2010 Game Developers Conference, he identifies two possible reasons. Before we talk about the two reasons we should define two terms: intrinsic motivators and extrinsic motivators. As Wikipedia puts it:

“Intrinsic motivation refers to motivation that is driven by an interest or enjoyment in the task itself, and exists within the individual rather than relying on any external pressure.”

“Extrinsic motivation refers to the performance of an activity in order to attain an outcome, which then contradicts intrinsic motivation.”

Putting these into context, intrinsic motivation would be playing a game because it is fun to do. Extrinsic motivation would be achievements in the game that drive you to do something you probably wouldn’t do otherwise. Why would you do them? To be rewarded, to feel as though you have accomplished something, or so that you can show off your achievement to other players. Who wouldn’t want to tell all their friends that they have killed 53 595 infected in Left for Dead!

So back to the two possible reasons for achievements in game design. The first reason could be that the developer is trying to provide a dull game with extrinsic motivations. Why would a game developer be making a dull game in the first place? Probably to encourage players to part with their money. Think of Farmville – that game certainly lacks intrinsic motivation, unless I’m the only person who finds watching crops grow boring. Regardless, Zynga are laughing all the way to the bank. Shame on them – no appreciation for the art of game development.

Only 4 more hours to go…

Alternatively, developers are making intrinsically interesting games and want to make them even better by adding some extrinsic motivators. However, this is where things can get dangerous very quickly.

There are plenty of interesting debates in the field of psychology over the effects of rewards on motivation. For as many who are in favour of promoting reward systems, there are just as many who feel such a strategy does more bad than good (see Dan Pink’s TED Talk). However, most of the research shows that extrinsic motivators decrease intrinsic motivation on interesting tasks. In addition to this, it shows that intrinsic motivation is superior to its extrinsic counterpart in promoting creativity, problem solving, quality and speed. I’d even go so far as to say that intrinsic motivations can produce a lot more happiness than extrinsic ones.

So with this in mind, it is tragic when a fun, enjoyable game has its intrinsic motivation sabotaged by extrinsic achievements causing the focus to be shifted away from the game and onto the rewards. This also potentially means that when the achievements are all complete, the game holds no more value to the player.

However, that’s not to say that achievements should be omitted completely, because there are some benefits to including them in a game. Many articles have been produced in an attempt to lay a guideline for successful achievement design, such as the 3 part series written by Lucas Blair.

Regardless of the motivations behind implementing achievements and whichever side of the fence you find yourself, the bottom line is that it motivates people to do things that they wouldn’t do otherwise. Like waking up on Christmas morning to log into World of Warcraft.

However, achievements don’t end at software games. Moving out of the sphere of computing, the term gamification refers to the use of game design techniques, game thinking and game mechanics to enhance non-game contexts. Introducing achievement systems into real-life situations could provide a person with extrinsic motivation to perform ordinarily dull tasks.

The possibilities for this are endless. A talk from DICE 2010 by game designer Jesse Schell gives some interesting, albeit disturbing, insight into what the future might hold. Providing achievement points and an online leaderboard could motivate people to complete exercise routines – promoting health and fitness. Similarly, achievements for completing class assignments or attending lectures could promote education. The obvious counter-argument to this is it shifts the focus away from learning and onto the reward, but could that be overlooked if the number of students succeeding increases?

The field of advertising and marketing could also be influenced immensely by achievements – would people be motivated to buy fifty boxes of matches in the month to unlock the “Firestarter” achievement? Would they then be compelled to go on to unlock the corresponding yearly “Pyromaniac” achievement by purchasing six hundred of them? And that begs the question, where is the line when motivation becomes manipulation?

Achievement systems in real-life definitely present a number of serious questions that need to be answered before we even begin to look at the practical challenge of dishing out accolades for smiling at seventeen people in a day. Just figuring out if extrinsic motivation is a sustainable way to encourage activities is open for debate. Finally, and more importantly, if games are inherently better with intrinsic motivation, shouldn’t the same rule apply to real-life?

Disclaimer: These opinions are my own and do not necessarily represent those of Naspers, MIH or the Media Lab.

Technology has been used all throughout the ages to enhance political structures and the functioning of governments. There is an ever-flowing line of inventions, from writing to print, the telegraph to the telephone, the television and the internet, which all had immeasurable effects on the functioning of governments around the world. In today’s world I believe that the new media platform of gaming is becoming more and more relevant to society and could provide an enhanced platform for global political participation.

If you adhere to, as most humans tend to do, some form of Hobbesian social contract; you would probably agree that subjecting yourself absolutely to certain sovereign powers can be quite frustrating. This is just a roundabout way of saying that politics tends to suck. Now, I would like to believe that the level of discomfort within civil society today is caused by the rigidity of the political system as a whole.

This rigidity, I believe, comes from a system that is based on outdated command and control hierarchies. These hierarchies worked relatively well in the past, but needs to be updated to fit with today’s rapidly changing world.  A potential update to civil governance can be found in the phenomena of serious games and/or gamification of society (discussed in Gaming for a better tomorrow).

Serious games refer to all games that aim to solve bigger problems than merely entertainment. How I see it, is that there are three main archetypes of serious games, namely:

1. Educational/informative games – teaches you something or informs you about a specific issue (Evoke, Obama STEM challenge, Freedom HIV/Aids)
2. Simulations and scenarios – using games to simulate reality and prepare for potential futures (Conflict: Middle East Political Simulator, Government Simulation Games, Cyber Nation)
3. Practical application and/or Crowdsourcing – using the “fun” aspects of gaming to engage people more fully into a certain task (British MP’s, ETERNA)

Most research on the link between gaming and political science has been based on improving learning through gaming, or creating government simulation games (Archetype 1&2). I believe however, that the biggest potential for improving political management lies within the third category of practical application games.

The key strength of this category lies  with its potential to include the assets of gamification into normal life. Gamification believes in the notion of homo ludens, man the player, and that we can improve life by including fun elements from games into life.

Playing politics

One area of politics that serious games and gamification would be able to improve is the realm of political participation. In today’s world we as citizens are quite far removed from the governmental decision making processes.  This, in the recent past, has been a matter of necessity as it would have been impossible to conduct a Greek forum-like assembly with millions or even billions of people participating. Not anymore!

With the rise of games and it’s increased capability to bring more and more people together  (think of World of Warcraft or Second Life) it becomes possible to create a type of forum where a deeper (more inclusive) form of political participation can take place. If millions of players can, as Jane McGonigal proposes in Reality is Broken, be harnessed to participate in real world political decision making, it would vastly increase the legitimacy of the decisions that are taken.

In political science we have a radical model of democracy that is a bit idyllic, but could become possible with this new level of participation. This model is called discursive democracy and calls for an extremely high level of citizen participation. According to this model, it is unfair that billions of people are indirectly represented by a small number of leaders that might even have gained their position via illicit means.

Discursive democrats believe in a process of collective reasoning. In this process citizens  address  public  problems  collaboratively  by collectively  reasoning to  find  the best solution to  solve specific problems. This necessitates a type of forum where discussion can take place and I believe online games could offer such a forum. Discursive democrats believe inclusive discussions would lead to more rational and fair decisions and would therefore increase the legitimacy of the decision and ultimately the functioning of the world.

How to achieve this

This is just one idea where games might aid governance structures and work towards a social contract that is more friendly to us, the humble subjects. Research into this field is still in its infancy, but I believe that it has great potential. There are leaders such as Jane McGonigal that allow us to imagine what great possibilities lie in our future. What is needed now however, is the good old scientific method and novel research examining and developing the potential of homo ludens, man the player.

The reverse of the problem is not true. Even with the most advanced hardware and algorithms, you will be hard pressed to extract any useful information from a sequence of images using a computer. Even simple problems which we as humans perceive as trivial can not always be solved satisfactory by modern systems. Here is a short list of the current areas actively being researched:

• Background and foreground separation
• Face detection and recognition
• Object tracking
• Texture classification
• Environment mapping

These are problems that we are faced with daily, yet we solve them autonomously in split seconds. So why are computers not able to deal with these problems in a satisfactory manner?

There are a few answers to this question. For one, the conversion of a 3D scene into a 2D image results in the loss of one dimension, which results in a problem with an infinite number of solutions. Another reason for the poor performance of computer vision systems compared to our own is the digitization process, where a large amount of the data is lost due to sampling. While these answers can explain why we should not be too concerned about the rise of the machines just yet, it still does not explain why the human brain is able to extract volumes of useful information from the same sequence of digital images.

The real reason why computer vision is still in its infancy is because we simply do not know how to effectively model the tasks associated with vision. The modelling problem can be traced back to another problem, which is we simply do not know how human vision works. It is true that the physical side is already well documented and can be emulated by a normal camera. The mental side, however, is still very much a mystery.

A Simple Vision Test

In order to create efficient algorithms to address the vision problem, it might be beneficial to try and figure out how the brain solves a simple vision problem, namely object detection. Consider the following image of a mouse:

Now try to find the mouse in the following images.

The first two images should not have presented much of a problem, but the third image should have been slightly more challenging. What did you look for in the above images when searching for the mouse? The answer will likely differ from person to person, but the simplest approach for the first two images would be to look for areas in the image which have the same colour tone as that of the original mouse. This approach will obviously not work for the third image, in which the image consists solely of black lines and artefacts against a white background. A more appropriate approach would be to search for a shape similar to that of the mouse.

The above test demonstrates two characteristics of human vision. Firstly, we look for similarities between previous experiences and that of the current situation to extract important information from what we perceive. Secondly, we are able to use more than one rule of comparison to extract information from the world. This highlights one of the important considerations for any computer vision algorithm: What are the important features to extract from a known model and look for in an unknown environment. In the case of the first image, the important feature to extract from the model of the mouse would be the colour, while for the third image the shape of the mouse would be more appropriate.

Lets consider the shape of the mouse. One question we have to asks ourselves is how can we effectively model the shape of the mouse in such a way that we will always be able to identify similar mice if present? Consider the next two images:

In the first image a large portion of the mouse is blended into the background. From this image it can be seen that the mouse cannot be found using the shape of the mouse as a whole. Instead, we can find the mouse by looking for certain key parts of the mouse, such as the nose, ears, tail and legs. The more parts we can find, the more certain we are that we found a similar mouse. In the second image we can also identify various mice by their individual parts, even though large portions are occluded by objects.

This brings us to another important characteristic of human vision: We can identify objects by their parts. We can also identify the position of smaller parts by first detecting the larger object. Detection by parts is one of the relatively newer methods currently being researched within the field of Computer Vision and has shown good results.

The Results

The one characteristic of human vision that both these tests shows is our ability to adapt our problem solving process to the given environment. This highlights one of the biggest problems in computer vision, the fact that many algorithms use only one method to solve a specific vision problem. While this might be satisfactory for constrained conditions, it simply cannot solve the general vision problem. Unfortunately, it is not possible to combine various algorithms to create one super algorithm, as this will require enormous amounts of processing power while not necessarily providing better results. Still, the key to solving the vision problem might lie in a combination of various systems.

While modern computer systems might be impressive when compared to the humble abacus and might be one of the few entities which can win a chess match by brute force, it still is nothing when compared to the amazing power of the human brain. We need to look into how we as humans perceive and extract information from the world using our visual senses in order to create competent vision systems in the future.

First. If I get my numbers right, there are only 250 million people currently connected to your internet. By the end of the decade, a massive 2 billion people will become connected. This scale is unprecedented! Next year, at the turn of the century, the Nokia 3210 and its successor 3310 will be released. These cellphones, together will sell almost 280 million units! This is only the tip of the iceberg.

I would suggest not investing in any technology companies right now. It is a bubble that will burst very soon! Buy Apple shares. And if you can, get involved with Google.

In 2002/2003, two websites will be launched called Friendster and MySpace. If you are young, you will probably go on it. Enjoy it while it lasts, because, it will overshadowed by a new online social network called Facebook (2004). What do these sites do? Well, you create a profile of yourself and connect with other people. Yes, it will be strange and weird at first. Facebook, however, comes from the premise of using your real name, online. And no, it won’t be weird at all. Most, if not everyone will be comfortable with that! But listen. Get this. By the end of 2011, 800 million will be on this website, connecting, interacting and sharing information. It will be much bigger than you can ever imagine!

The strangest part is yet to come. I bet you won’t believe me. In 2005, a website will be launched, called YouTube that will make it so easy to upload any videos online. Celebrities will be created through it, new revolutions will be sparked, and we will be able to see what is happening around the world through the eyes of others without having to rely on any of the current traditional media you have now. In the year 2012, a cat with a poptart for a body, flying through space will be viewed 70 million times! Look look!

There is even a 100 hour version that has been viewed 6 million times. I won’t even start on the other memes, and culture that developed around the internet. It is probably difficult to understand (some traditional media in 2012 still don’t understand concepts such as the global chaotic neutral collective called Anonymous). Okay, but lets get away from flying cats in space.

In 2006, a website called Twitter will be launched, that will allow people to stay in touch with short bursts of information. You can say anything, and people will follow you and read what you say. Once again, bear with me. It won’t be weird. You will want to have people following you! This short form information spreading will allow some amazing emergent behaviour! Using an add-on site called TwitPic, a guy will break the news that a plane landed in the Hudson River, before any of the mainstream media could break the news. Here is the tweet (yes, it is called a tweet).

http://twitpic.com/135xa – There’s a plane in the Hudson. I’m on the ferry going to pick up the people. Crazy.

— Janis Krums (@jkrums) January 15, 2009

This will be a watershed (no pun intended) moment for the rise of real-time networks. What usually took a few hours to spread (bound by time and space), everyone is now a potential part in larger system for spreading news (instantly). This powerful way of sharing information (along with Facebook) will become an incredibly powerful for political use. The so-called Arab Spring will start at the end of 2010 and topple many authoritarian regimes. Be prepared!

Closer to the end of the decade, Apple will introduce, the iPhone (yes, a phone), that will spark a new trend. The latest version of the phone (the 4S) will have a dual-core 800mhz core, 512mb of RAM and can have up to 64gb of storage space (all in your hand). You know how I said you should look to get involved with Google as well? Well, they are rolling out their smartphone operating system called, Android (it’s not a real android). A massive 850 000 new android smartphones will be activated each day in the beginning of 2012, mostly being activated in territories that did not have any access to the internet!

I’d like to tell you another awesome thing! Google will drive around with cars taking pictures of popular places around the world. This will allow anyone (for free) to view what the world looks like. With some imagination, you can pretend you are a frenchman walking the streets of Paris!

Oh. Wait. It seems I have little time left! In 2012, we still can’t manage to use our time wisely. I just got an e-mail on my iPad. Look. These tablets will completely change how users interact with digital media. No longer will you have to resort to having that massive tower and heavy laptop. You can easily enjoy all the above I mentioned (YouTube, Twitter, Facebook) all in the palm of your hand. Remember. This new power you will achieve is incredible. Use it for good!

The e-mail is from the year 2022. This guy wants me to meet with him. He mentions self-driving cars, africa, mobile wallets, 3d printing, ambient awareness and augmented reality. The future sounds exciting!

I’m a big fan of the Python programming language and to be quite honest, the majority of my masters work is written in Python. Along with Python, I also spend a lot of time working with multiple workstations on the network. This is because my masters is focused on securing media in a peer-to-peer media distribution platform. Now anyone with experience in working with multiple workstations knows you spend plenty of time SSHing into those workstations. Today I’m going to make your life easier with the help of Fabric.

What is Fabric?

“Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks”. You could have read that on the website so let me elaborate.

“It provides …” Ok, seriously now. Fabric helps you execute commands remotely without you having to individually SSH into each machine. What’s more, Fabrics allows you to define roles and associate hosts to these roles. With the roles in place, functions can then be created to execute on a role.

How I use it

In the scenario of a peer-to-peer media distribution platform, I have multiple peers sharing information with each other. The tracker maintains the availability of media in the platform and provides this information to the peers.

Now in a production environment Fabric is of no use to me, as the end user will be operating the peer but as my masters is still a work in progress, I am not in a production environment. Instead I have to test the system without asking colleagues to install and run the peer software.

What Fabric allows me to do, is too install the peer software on any devices on the network that I have SSH access to and then execute commands. In order to do this I create two roles, namely peer and tracker. For the tracker role I only specify the tracker as a host, but for the peer roles, I specify all hosts that will function as peers. I then define the folder structure for synchronization through rsync.

With all the configurations set up, I am then able to modify code on my local machine and deploy the code straight to the tracker and peers. I can then with another command, start the tracker and run all the peers on the network in uploading mode. All of this without having to manually SSH into each and every machine.

Take note

Something to take note of with Fabric is that is does not release from a host until the command on the host has finished executing. This is a problem if you need to start multiple peers in uploading mode and leave them running until your tests are complete. In this case Fabric will SSH into the first host, run the command and wait until execution has finished before moving on to the next host.

This is where a very clever application called Screen comes into the equation. Screen is included in most *nix systems. According to the Wikipedia page for Screen, “it is a software application that can be used to multiplex several virtual consoles”. This is especially useful if you only have one session available on a host.

The issue of a stuck session can be solved with Screen, if you include the following steps in your Fabric file:

1. SSHing into a remote host
2. Creating a detached Screen session and attaching
3. Executing a bash script and then detaching

Give it a try

I use Fabric everyday now and would be completely stuck without it. Give it a try and even if you don’t need to make use of Fabric, then you should definitely have a look at GNU Screen.

PS: A big thank you to an old labber Jacques for introducing Fabric to me.

• it uncovers requirements that have been misinterpreted or overlooked;
• it confirms that your logic and experimental methodology are correct; and
• it lessens the workload when you are writing your report.

Now, in this post, I shall look at a project the SINAD group at the MIH Media Lab is working on, which uses these ideas to formalise our data management so that data generated by members can be combined into a digital library.

A digital library can be defined as: a potentially virtual organisation, that comprehensively collects, manages and preserves rich digital content, and offers to its targeted user communities specialised functionality on that content, based on codified policies. More information can be found here and here.

The research done on digital libraries and scientific data management has focused on large scale systems, such as the those described in this article on scientific data management. The data they are interested in usually has well structured metadata schemas such as those found in the Metadata Encoding and Transmission Standard, and is often also of the same type, for example, the DNA databases of the NCBI. However, we’re interested in running many different kinds of experiments on projects spanning a wide range of fields in natural language processing (NLP). Therefore, it stands to reason that the data that are recorded for different experiments can vary considerably. Also, we are not aiming to create a virtual organisation as the definition implies, but only create a simple set of tools to facilitate research in the lab (and hopefully to others outside of the lab eventually).

After some initial requirements analysis we started to look for an appropriate database system. The following are some of our database requirements:

• the database must store large volumes of data;
• it must handle large traffic volumes;
• you must be able to easily publish data from a local private database to a global shared database in order to share data with colleagues;
• it must be convenient to store metadata describing the data for documentation purposes, since it is easy to forget important information about why and how the data was generated;
• it must be easy to insert new fields or remove existing fields from an existing schema; and
• it must be easy to remove unwanted or erroneous data.

It became apparent that MongoDB covers our requirements quite well, and it was therefore chosen as our database of choice over relational databases and good old text files; the latter, worryingly, are still the go to method for many experimenters.

MongoDB’s collections are in some ways comparable to SQL tables, and will be used to group data from the same experiment together. The data generated by a subsequent run of this experiment can then also be stored in this collection, thus forming a larger data set. Each data point is stored as a BSON object. BSON is a binary-encoded serialisation of a JSON document, the structure (or schema) of which are easily changed and existing objects updated, if alterations need to be made at some later stage. This allows us to change the database easily when compared to relational databases and text files, even if data of the type we are changing is already present in the database. As an added bonus, APIs that make it easy to create and manipulate JSON objects in your code are available for almost every language you can shake a stick at.

Let us look at what a data point represented in a JSON object might look like. Take a simple example of a coordinate in Cartesian space, where the object only stores its own unique identifier, and the x and y values of the coordinate. Here we have the coordinate (4, 2):

{
"_id" : ObjectId(8fcc3ebc1277000304506170""),
"x-axis" : "4",
"y-axis" : "2"
}


Multiple objects such as this can be observed in some experiment and stored in a collection created for that experiment.

We create a special object for each collection to distinguish and describe the collection, which we shall call the experiment description object. This object is given a specific schema:

{
"_id" : ObjectId("4dcd3ebc9278000000005158"),
"author" : "Peter Hayward",
"overview" : "Data recorded for ExperimentTypeA",
"motivation" : "The reasons of why you run the experiment(s) that generated the data in this collection.",
"program" : "pipeline1",
"parameters" : "-a 10 -b 12",
"repo_name" : "git@github.com:MIH-Media-Lab/pjhayward.git/projectx",
"git_commit_hash" : "9969247ce1f9693d9573b797bdb516f2c35acd32",
"object_vars": "x-axis, y-axis"
}

The aim of this schema is to capture all the important metadata of the experiment, such as the experimenter’s/author’s name, overview, motivation, the information needed to replicate the experiment, and the attributes the other objects in this collections must have (in our example the x and y axis values).

We can then use the experiment description object to validate the data that the author (or at a later stage some other user) wants to upload to the collection. For example, if I run an experiment with exactly the same parameters again, and want to add the coordinates to the collection, then the coordinate objects must have the x and y values as specified in the experiment description object.

Arguably, the most important reason for collecting the metadata stored in the experiment description object is to be able to search the digital library for useful datasets. We aim to develop not only search tools, but also a web interface that makes it easy to browse datasets and visualise them as well. For example, you might be able to pull up a plot showing all or a sample of the coordinates we’ve been talking about thus far.

If you want to run the same type of experiment with different parameters, you can simply use sub-collections in the collection designated for the experiment type. For example, say we have an experiment type called ExperimentTypeA (as described in the JSON given above), and we want to run the experiment with two sets of parameter values. First, a=10 and b=12, and then a=20 and b=24. Logically, providing different parameter values would influence the results, so it would be wrong to clump the results of these two experiments together. Therefore, we create two sub-collections and store the data points of each experiment separately. Each of these sub-collections will also have its own experiment description object explaining exactly the difference in the parameter values.

We tick all our requirement boxes with a simple system like the one I just described. Such a system makes exploring large amounts of data more plausible, and, hopefully, makes working with experimental data less of a pain. My hope is that it will help our research group goal of working on more intergroup projects; and that other readers might get some ideas for tackling the data management problems they face.

Android Samsung Galaxy Tab

Despite the incredible market penetration of smartphones and exponential growth of their apps market, their utility has been and will remain severely limited by their battery life. Unlike the microprocessor industry or the communication hardware industry, where computation capability or the line rate has been continuously improved (regularly doubled every 18 months), battery technology has been relatively unchanged for many years. There is need for more investments in sustainable battery storage technologies and energy efficient design for smart phones.

An “energy bug” or “ebug” is defined as an error in the system, either application, OS, hardware, firmware or external that causes an unexpected amount of high energy consumption by the system as a whole. In fact, “ebugs” have become so prominent that already they have led to significant user frustrations. 70% of phones returned to Motorola last year were related to energy problems [2]. Unlike traditional software bugs, such errors could happen in any of the wide variety of entities in a phone (e.g., hardware components, app processes, or the OS), and due to a wide variety of root causes such as programming errors, inappropriate API usage, flaws in the design of applications or the OS (device drivers), complicated interactions between hardware components of smartphones, changing external conditions (e.g., remote server crash, or weakened wireless signal strength), and faulty hardware itself from daily “wear and tear” due to the portable nature of smartphones (14% of all technical support calls for Android is related to faulty hardware [3], 9% for WP7 and 8% for iPhone).

Also different from traditional software bugs, ebugs do not lead to an application crash, or OS screen of death. In most cases of ebugs, the applications and the whole system continue to run, providing their intended services normally, with the exception of consuming unexpectedly large amount of energy. This makes bug detection and root cause tracing much more difficult. The end result of an ebug is that mobile devices run out of battery sooner or much sooner than expected, requiring frequent recharging or resulting in unplanned device battery outage. Together, these two unique characteristics of ebugs, diversity in root causes and stealth nature, suggest that they can be much harder to detect and pinpoint the root causes when compared to traditional software bugs.

Currently, systematic treatment of ebugs on smartphones does not exist, and users cope with ebugs in an ad-hoc manner. An ebug is first detected by a user through the obvious symptom: the battery drain rate suddenly becomes very high for no apparent reason. This triggers the user to hunt for the cause. Typically, the user employs task killers to kill a suspicious app, hoping to stop the energy drain. If the symptom persists, the process is repeated. A few tools  exist today that help users narrow down the suspicious app, either by providing information such as the fraction of the total system energy consumed per process and by the OS (e.g., the battery tool in Android), or by specialized monitoring (e.g., spareparts keeps tracks of CPU wake locks [4]). The forum posts suggest many users try to use these tools in bug hunting. However, these tools fall far short in dealing with the entire diverse spectrum of ebugs (as evident from the posts from a number of blogs), and additionally cannot pinpoint the root cause of ebugs.

Charging batteries in Africa

As I wait to hear from my network provider regarding the problem with my Galaxy Tab, my thoughts are with the large number of mobile phone users on the African continent who are or might encounter a similar problem. Over the last few months African countries have seen the introduction of affordable smart phones from different manufacturers some of which do not have warranties or technical support presence in Africa. Most of the mobile phone repair shops which are present in the developing countries are involved in selling mobile phones and accessories, charging mobile phones, replacing faulty mobile phone parts and batteries. There is need for the cell phone technicians to be trained in order to differentiate the energy problems smart phones will face especially the energy bugs because they will not be able to fix some of the problems. Mobile users will be asked to continuously replace batteries for their mobile devices in order to address the energy challenges until they dispose of the mobile devices since they will fail to solve the problems. There is an need for device manufacturers and researchers to address the energy bug problems and communicate the knowledge to the cell phone technicians and mobile users.

References
[1]  Android – an open handset alliance project.
[2]  The bulk of returned motorola phones are the result of applications that cause performance/energy problems.
[3]  Android phones more prone to hardware problems.
[4] Maemo community.

Wikipedia’s structure is built around articles. Currently, the English Wikipedia consists of 3.8 million articles of which 1.3 million articles are acknowledged as good articles, since they contain more than 200 readable characters (~33 words) and have at least one incoming link from a different page within Wikipedia. The rate at which the English Wikipedia grows has decreased over the last couple of years, nevertheless about 900 new articles are added each day (1).

Wikipedia’s articles are written and organized by following editorial and structural guidelines. For example, an article may only describe a single concept and there is only one article for each concept. Also, the title of an article has to be similar to the corresponding definition found in traditional thesauri. If a title has to be further qualified an expression, enclosed in parenthesis, is added to the title. For example, Car_(function) is the title of the article about the operation on linked lists used in the LISP programming language. Furthermore, the title of Wikipedia articles are case sensitive such that Optic_nerve describes the nerve that transmits information from the retina to the brain, while Optic_Nerve is the title of an article about the same named comic book.

Wikipedia redirects play an important role in navigating through articles, since equivalent terms are linked to an article using redirects. Usually the singular term that is the most general is chosen to define an article. For example, if you type http://en.wikipedia.org/wiki/Cars into the browser you are automatically redirected to the article with the title “Automobile”. Currently, Wikipedia contains 5.2 million redirects. Furthermore, if the desired article cannot be automatically determined, disambiguation pages are created that describe all possible meanings of an ambiguous term to help the reader find the desired article. The English Wikipedia contains about 100.000 disambiguation pages.

Another important property of Wikipedia is that it contains nearly 80 million internal links connecting articles with each other. Not only is the destination of a link important when analysing the link structure but also the anchor text which is used for a link. This information, for example, can be used for word sense disambiguation.

Articles in Wikipedia are also organized into a category structure. For example, the article Car belongs to the category “Wheeled vehicles” which is a subcategory of “Land Vehicles” which in turn belongs to the category “Land Transport”. Currently, Wikipedia contains around 800.000 categories. The purpose of categories is to organize articles into a hierarchical structure. Interestingly, this structure is not merely a tree structure but also contains cycles. For example, “Education” belongs to “Social Science”, which belongs to “Academic disciplines”, which belongs to “Academia” which again falls under the category “Education”. This cycle indicates that people can be educated about education.

There are more aspects about Wikipedia that all have been used as a basis for research but that I am not going to discuss further. These are info boxes, templates, discussion pages and edit histories.

The accuracy of Wikipedia articles is always under scrutiny and has also been a major topic for research. Giles (2) compares randomly chosen articles from Wikipedia with their associated articles from Encyclopedia Britannica. It turns out that both encyclopedias had equally many significant errors while Wikipedia contained more subtle errors such as misleading statements and fact omissions. Other approaches to assess the quality of Wikipedia articles use metrics based on the number of authors that worked on an article, total number of edits of the article, the size of an article, the number of internal and external links or the stability of an article (3, 4, 5). Wikipedia’s articles have also been used for creating language models to capture particular characteristics of a language (6).

Wikipedia’s internal link and category structure is an ideal starting point for creating thesauri. Terms in a thesaurus are related using four different kinds of relations:
1) the terms are equivalent
2) one term is broader or
3) narrower than the other
4) any other type of semantic relation
Wikipedia’s redirect pages provide precisely the information that is needed to define equivalent terms. Furthermore, the category structure can be used to define broader or narrower relationships between terms. Lastly, the internal links of articles can be mined for defining semantic relations between two terms. The most important aspect is that Wikipedia contains articles in more than 270 languages. Therefore, Wikipedia is an ideal source to translate thesauri into other languages.

Wikipedia articles can also be used to create ontologies where informative relations, such as “Tree consists-of Wood”, between articles are created by mining redirects, internal links, category links, category names etc. (7, 8).

Wikipedia’s links structure has also been used to create a network structure on which algorithms such as the HITS algorithm and Google’s PageRank algorithm were applied (9). Interestingly, the results of PageRanking Wikipedia articles indicate that articles that are closely related to religion achieve high scores, while the HITS algorithm prefers articles about famous people, common words, animals and abstract concepts such as music, philosophy and religion. As such, the articles “Pope”, “God” and “Priest” where the highest ranking articles for PageRank compared to “Television”, “Scientific classification” and “Animal” for the HITS algorithm.

(1) Wikipedia Statistics

(2a) or (2b pdf) Internet encyclopedias go head to head.

(3) Wikipedia as participatory journalism: reliable sources? Metrics for evaluating collaborative media as a news source.

(4) Cooperation and quality in Wikipedia.

(5) Extracting trust from domain analysis: a case study on the Wikipedia Project.

(6) N-Gram-based text categorization.

(7) Extracting semantic relationships between Wikipedia categories

(8) A thesaurus construction method from large scale web dictionaries

(9) Network analysis for Wikipedia.