Planet WUGLUG

February 13, 2010

Mulletron

Welcome

I’ve decided to move away from Warwick Blogs, on the grounds that I’ll soon be finishing my time at Warwick. Consequently True Contradictions has reached the end of the line. A new blog demands a new name, but I’m sticking with my theme of referencing Para Consistent Logic. Thats Marketing! Its worth linking directly back to my first blog post on True Contradictions.

Quite a lot has happened since then – I’ve graduated, Started a PhD, nearly finished a PhD, Friends have gotten married … but one thing remains the same. My commitment to intermittently writing overly long blog posts and about things that only I care about. I’d buy that for a dollar!

by admin at February 13, 2010 01:26 PM

Bucko

Adventures with DVDs

So I did some analysis on my evil MVM (UK) anime DVDs. I've worked out that they're basically unrepairable. They're run up to 25FPS from 24FPS without just speeding up the content (which is apparently usual), by interpolating fields. The idea is to make 25 ticks per 24 frames, and on tick n add n/25 of the "next" frame to 1-n/25 of the current one. In fact, this is done on fields, though the bottom field is on the opposite phase to the top field. The formula doesn't appear to be exactly linear in this way, but the principle is the same at least. The idea is presumably that on CRT TVs, you won't notice the preghost and postghost artefacts which are quickly covered up by the real content.

The overall result is that on average there's a 1/4 visible ghost on each field, and about 2/25 or more frames look heavily interlaced. Unlike the 3:2 telecine process used on US (NTSC/R1) DVDs, this process is not reversible (not strictly true; a perfect field of frame n can be subtracted from the field in frame n+1 in the correct ratio, then the resulting field multiplied -- but since it's encoded, any artefacts will end up getting added on, then multiplied up and used to derive the next frame). However, you can at least make it appear something other than awful.

Solution (for playback): Use a bob deinterlacer. With mplayer, this is -vf yadif=1 or -vf yadif=3.

Solution (for encoding): Either encode interlaced using -x264encopts interlaced (have fun with your increased bitrate), encode with a bob deinterlacer as above (have fun with your doubled framerate and increased bitrate), or use some other deinterlacer and suffer the drop in quality.

Solution (for screenshots): There is no solution. Sorry, if you own an MVM anime DVD you will never be able to post a full-resolution screenshot. Your best bet is to pick the field which is purest and obliterate the other one, giving a half-res image. You could also try working out the ratios and subtracting fields from the previous and next frames (example: if your frame is 0.7 * n + 0.3 * n+1, then 0.25 * n+1 + 0.75 * n+2 is next, you can remove 6/5 the next image, then add on some of the next one and so on; eventually you'll hit a full image but by that time I don't think you'll be enjoying the result).


Summary: If faced with a choice, and you want to take screenshots of a DVD, I would advise steering clear of any release encoded in this way. I've no idea how to check, but I'll make a list of all the DVDs I find like this. I imagine it's not just limited to anime.

February 13, 2010 11:34 AM

Faux

Catchlogger

Post seriousness: 70%.

Catchlogger [jar] [git src] checks that exceptions are being logged properly.

It’s entirely fallible, but, if your codebase is prone to catching and ignoring exception, and you use log4j, it’s Very Helpful.

For example:


try {
foo();
} catch (IOException e) {
logger.error("bar: failed to foo", e);
}

This is fine; an error has occurred and it’s full stacktrace will be placed in the log4j configured log.


try {
foo();
} catch (IOException e) {
logger.error(e);
logger.info("oh noes", e);
e.printStackTrace();
}

None of these, however, will do anything useful. The first logs just the toString() to the error log, info is too low-level to log exceptions at, and printStackTrace() doesn’t necessarily go anywhere at all.

For this block, catchlogger will issue:
IOException e unused at (Test.java:15) in public main(String[] args) in path.

The JAR is huge as it pulls in the entire Eclipse compiler to parse the source. BSD/MIT licensed.

by Faux at February 13, 2010 11:06 AM

Sinjo

Google are stealing my privacies!

I had my attention brought to another blog post earlier today (which for the purposes of this post, I shall assume you have read). Initially, my thoughts were that Google had gone and committed a rather large error, but I didn’t look too far into it as it wasn’t going to affect me and really had to set off towards home. When I arrived back I started talking about it to a friend, who immediately put me right. It turns out that Google Buzz is not exposing any new data to anyone without being allowed to. It is in fact aggregating already public data into, in my opinion, a pretty nice format.

What has happened here is a complete misunderstanding. Following the main example from that post (though it’s completely true of other services which Buzz can import from) Google Buzz will only display shared Google Reader items and comments that you publicly share. If these are private, then Google Buzz isn’t going to ruin your day and wave them around publicly.

To top this off, within Buzz there is actually a link (under Connected Sites, select Edit next to the Sharing With column) which takes you straight to the privacy options page, where you can go back and fix your earlier mistake of not correctly choosing your privacy settings. I’d actually say that, far from revealing your private data, Google have done a reasonable job of letting you protect your privacy in this case.

The hype that this has generated is rather saddening, and a reminder that people are all too quick to jump on the bandwagon. The author has posted a follow up, with a bit more explanation and as it turns out, a confirmation from Google of what I have focused on here (though this doesn’t have too much attention brought to it).

by Sinjo at February 13, 2010 01:14 AM

February 08, 2010

Blood God

Personal Projects Versus Work

Since I started work, I’ve found that I’ve done less and less work on any of the personal projects that I’ve picked up over time. As I write this, I can think of at least four different projects that I was going to do some work on when I next got an opportunity (such as today – as I’ve had the day off), but I’ve not touched in weeks at best. Of course, these opportunities don’t really come round all that often, as I don’t often have time after work (at least, not if I want to eat and get some sleep), and I always seem to end up otherwise engaged at weekends.

However, when I do have a day where I don’t really need to do anything and have the time to spare (like today) I inevitably end up wasting it through watching TV/films, playing on the XBox, or writing blogs like this. I’ve convinced myself that the main problem is that I spend 40+ hours a week in the office looking at one project or another there. This might involve writing actual code (as most of last week did), integrating various components to solve problems that way, performing analysis on what we need to do or any number of other activities, most of which are things I’d need to do on the personal projects. Given this, I guess I’ve been subconsciously avoiding doing anything on them as it seems too much like work.

Of course, by being apathetic in regard to these projects, they’re ever growing in number as I come up with an idea for something that I (or others) might find useful, and therefore add it to the list. A prime example is the Choob functionality I mentioned in passing in my previous entry about Code Style last month (which was a lot of hot air, and no real action), but there are also a whole load of other things that I’ve been thinking about for a while and should probably do something about.

So, what do I do about this? Well, I guess it all comes down to forcing myself to sit down and write some code, rather than wandering off and parking myself in front of the TV for an entire day. With this in mind I’m going to try and set aside a couple of hours each week (be it at the weekend, or one evening), where I can get something written. This will mean that I need to actually think about what needs doing, and break it up into suitable chunks – but that’s something that would need doing anyway if the projects are to avoid spiralling out of control.

I guess if I can manage that, then I should also be able to keep this blog a bit more up to date, especially with progress, so I may even start updating this more often. Of course, it could all fail miserably, so I guess we’ll just have to see!

by Chris Hawley at February 08, 2010 08:00 PM

February 04, 2010

Alp

HTML5 Theora Video Codec for Silverlight

I’m glad to announce the first release of our fully managed Theora audio / video decoder implementation for the Silverlight platform! The Highgate media suite will bring installation-free support for HTML5 streaming video to an additional ~40% of web users overnight.

Theora

So, a few drinks will be in order to celebrate the release at the FOSDEM beer event, Friday — drop by! And of course, I’d like to invite anyone excited about making open codecs a first class citizen of the Silverlight / Moonlight ecosystem to visit the Mono dev room over the weekend for source code and some frivolous demos between sessions.

Technology

We’ll be releasing a high-performance decoder for Theora video / Ogg Vorbis audio streams that plugs into the Silverlight 3 streaming media abstraction, as well as a reference front-end player interface and JavaScript bridge layer providing basic compatibility with standard HTML5 media tags, adding support for the standard to Internet Explorer and extending the capabilities of WebKit-based browsers like Safari and Epiphany. A cunning plan, one might say!

by alp at February 04, 2010 11:02 PM

February 01, 2010

Sinjo

Mirror’s Edge (PC) Review

I picked up Mirror’s Edge in the recent Steam sales (along with far too many other games, which saw off a little more money than I’d hoped). I didn’t get it with any of my own expectations, only the knowledge that people had told me it was a must buy. Unfortunately the game left me wondering what had deluded these people into thinking something like that.

If you care about the story not being spoiled you may want to avert your eyes. The game starts with you being plonked down on a roof somewhere in some city where the graphical bloom goes up to 11 and stays there at all times. You’re taught your repertoire of jumping and fighting techniques in one go, god forbid you should forget them, and then launched into the game. The story is set in your regulation future dystopia, where the police are evil, the government are evil and concrete blocks are extremely springy. You’re not really given too much motivation to hate the government; you just take the game’s word for it and set about acting like a royal prick with a poorly explained briefcase fetish.

Sadly, the game play doesn’t even come close to making up for the plot. There is one route, and only one route, and any deviation from it will result in crunchy death as you faceplant into the pavement. Most of the time this route is helpfully painted red, but the game occasionally decides you’re more competent than that and lets you decide what to do. Don’t be fooled though, there’s one path, and any attempt at creativity will be duly rewarded with death.

Linearity aside, the game still manages to produce an unnecessarily frustrating experience. Every time you grab an object or ledge, you’re treated to a face full of concrete and have to stop and move the camera around, breaking the flow of the game. This, teamed up with the loss of momentum every time you jump over anything higher than a cinder block, caused a steady feeling of rage which didn’t really subside at any point in the game.

The lack of choice in paths reared its ugly head more than once during the game, but one incident stuck out more than the others. In a section in chapter 7 the character is running on top of some ventilation shafts and some pipes are highlighted red on the other side of the room. I spent a good half hour wall running along the wall that headed directly to them, only to be left disappointed at the game’s edge detection and plummeting to an inevitable death. It turned out that I was meant to run along a wall parallel to what I was aiming at and make possibly the most hilarious jump ever to reach the bars.

The edge detection was a constant annoyance. At times it seemed like the protagonist simply wasn’t trying, but as soon as a bit of concrete turned red it may as well have been a fucking spring board. The net result is that you only go where the game wants you to go, no matter what incredible leap that may involve. Occasionally it will even help you when you don’t jump far enough, and you’ll end up performing a mid air vault for a pipe or ledge.

Overall, I’m confounded as to why people recommended this game so highly. While it’s a nice concept, it’s totally marred by a lack of polish (no, not bloom, there’s enough of that) in the execution. Bring on the release of Assassin’s Creed II on a real platform.

by Sinjo at February 01, 2010 08:58 PM

January 28, 2010

Sinjo

iPad: Why?

Following the usual excess of hype that lead up to the launch of a new Apple product, I can’t help but feel completely underwhelmed. I’ll be honest, I’m not a user of any Apple products; however, I’d be lying if I said I didn’t like any of them. The iPod is a nice music player. Having used a few friends’ I can say that I like the UI and that there was no huge learning curve to using the device. The iPhone was certainly responsible for pushing the idea of smart phones to a greater audience and once again did so in a nice tidy package that was very usable.

The iPad though, leaves a lot to be desired. It seems like Apple have produced an enlarged version of the iPhone (but not a phone, so really more like an iPod Touch), without applying significant work to it. It suffers the same issues as the iPhone, those being:

  • A total lack of multitasking. Running applications in the background is not a new concept, and failing to provide such functionality seems like a massive step backwards. I’ve felt this about the iPhone for a while and it seems even more crippling for a device which truthfully should provide more.
  • No camera included. While it’s obvious that this device isn’t for taking photos, I don’t understand why it doesn’t have a small, integrated webcam on the front. Being able to make video calls on, for instance, Skype seems like a reasonably sensible use-case for this device.
  • Lack of a hardware keyboard. This is a more debatable point, but to me it makes little sense. With a device of this size I don’t envisage that an on screen keyboard would be pleasant to use. Selling a keyboard dock isn’t really a solution as you’re just increasing the amount of clutter that you have to carry with you to use the device as you want to.

I’m left not really knowing who this product is aimed at. Which group of people want a larger iPhone whose main advantages are negated by dodgy design decisions? It seems more  like they’re simply riding the wave of success which means that people will end up buying them simply because it’s the latest thing to emerge from the shinies factory.

by Sinjo at January 28, 2010 01:10 AM

January 20, 2010

Fred

Multiple Sites with One SSL Certificate

There's two ways of doing this that I use; one is fairly clean:

  • Get a wildcard certificate (*.example.com)
  • Put all of your sites as subdomains (eg foo.example.com, bar.example.com)

While it "just works", it restricts your choice of domains; an alternative solution, which isn't really suitable for your users, but fine if you just want secure access to an admin panel (your blog, PhpBB, etc) on several different domains, you can hack something together with Apache, mod_proxy, mod_rewrite, and some cookies:

Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
Header set Pragma "no-cache"
Header set Expires "Thu, 01 Jan 1970 00:00:00 +0000"

RewriteEngine on

RewriteRule ^unset$ https://example.com/ [L,R=301,CO=site:example.com:example.com]

# foo.example.com
RewriteRule ^foo$ https://example.com/ [L,R=301,CO=site:foo:example.com]

RewriteCond %{HTTP_COOKIE} site=foo
RewriteRule ^(.*)$ http://foo.example.com/$1 [L,P]

# bar.example.com
RewriteRule ^bar$ https://example.com/ [L,R=301,CO=site:bar:example.com]

RewriteCond %{HTTP_COOKIE} site=bar
RewriteRule ^(.*)$ http://bar.example.com/$1 [L,P]

If you install the above as ".htaccess" in the root of https://example.com, then https://example.com/foo will make https://example.com/ act as a proxy for http://foo.example.com, https://example.com/bar will make https://example.com/ act as a proxy for http://bar.example.com, and https://example.com/unset will make it display whatever it normally displays.

I hope someone finds this useful.

by Fred Emmott at January 20, 2010 08:38 PM

Bucko

Holy balls!

So, I read an observation that the closed ball in a metric space might be strictly larger than the open ball. Is it possible to construct a subset of a real Euclidean space so that this is true for every ball? What about if the centre is fixed? If not, what about rational Euclidean space? With rational radii? What if the dimension is countable? (It's pretty easy in the uncountable dimension case).

Some background: A metric space is a space in which between every two points x and y there is a distance d(x,y). The distance obeys some obvious properties: It's zero if, and only if, x=y. It's never negative. It's symmetric (that is, d(x,y)=d(y,x) always). Finally, it obeys the triangle inequality, that d(x,z) cannot be more than d(x,y)+d(y,z), for any y. That is to say, there are no "short-cuts".

The open ball Bo(x,r) is the set of points y with d(x,y)<r (that is, anything strictly less than distance r from x). The closed ball is the same but using less than or equal to.

The closure of a set X is the set of points y for which you can find a point in X which is arbitrarily close (for instance, if X is the set of real numbers strictly less than 1, and d(x,y)=|x-y| is just the absolute value then the closure of X is the set of real numbers less than or equal to 1, because you can find numbers in X which are as close as you like to 1).

The easy example of the ball above is if you remove the half-open interval [1,2) from the real numbers, the closure of Bo(0,2)=(-2,1) is [-2,1] but the closed ball Bc(0,2) also includes 2.

My thinking is that removing chunks of a 2-dimensional space might be able to result in a similar construction. Somehow.

January 20, 2010 04:38 PM

January 18, 2010

Fred

Qt4 GUI Testing with Hooq

One of the things I've had the pleasure of working with for Mendeley is Hooq; Hooq is a GPL/LGPL-licensed system for testing Qt4 GUI applications, on multiple platforms (Linux and Windows being where most of the focus has been at the moment). It's mostly GPL, but an LGPL library is loaded into applications being tested, at runtime.

With apologies in advance for the incredibly unfriendly UI at the moment (patches welcome :p), here's a lots-of-pictures demonstration on using it to test a trivial application.

After building the demo application and Hooq, and running hooq, you'll get a screen like the following:

Screenshot of a fresh Hooq installation

For organisation, Hooq uses the concept of a 'test set' - this can be one per application, tests for different areas of the same application, or whatever you choose. First, you need to create one - select "New Test Set..." from the file menu, and you'll get another ugly dialog:

Screenshot of Hooq's Test Set dialog

After filling it in and clicking OK, you'll be back to the main screen, except that 'Test set' will be filled in (screenshot); type in 'Foo' in the box at the bottom, then click 'Record' (screenshot). The demo application will then start - if you type in "Hello, World" you should see something like this:

QTextEdit with

Click the close button in the window decorations, and the Hooq window will now have a test entry, with run and edit buttons:

Hooq main window with a test in it

After clicking on the edit button highlighted above, you'll get a script editor (Hooq uses QtScript for test scripts):

Hooq's script editor

If you then click in the margin on the line after "QTextEdit_0.sendText(string0)", there'll then be a breakpoint marker on that line:

Hooq script editor with a breakpoint set

If you then click the run button in the toolbar, the script will stop running at that line:

Hooq after triggering a breakpoint

At this point, if you click 'pick property' in the toolbar (new icons welcome), you can then click on any widget in the application being tested, and get a list of its Q_PROPERTYs:

Q_PROPERTY list

For this example, scroll down to "plainText", and click compare; this will insert a check into the JavaScript code in the editor:

Script editor with added compare()

Remove the breakpoint, and run it a few times; it should repeat your input and quit each time, without an error; however, if you change the "string0" variable at the top from "Hello, World" to something else, like "Hello, World!", then click run, the test will error, if you don't also change the compare statement:

Thrown exception in Hooq

This is more useful when the test hasn't changed, but when the application itself has (i.e. a regression). Hooq also has a "Run All" feature, to run all the tests in a set unattended, and show a summary at the end.

As some more detailed information:

Hooq loads a shared library into the application being tested (via gdb under Linux, and a very ugly hack under Windows) which recieves every QEvent in the application, and sends user input events, encoded in XML, to a hooq client (the GUI client is shown above, but there's also a CLI one that just records and plays the XML data). In the GUI client, the XML is converted into a tree of sorts, and then a few modifications are optionally made (selectable via the "Options" menu):

Hooq options menu

These options are:

Ignore Mouse Movements
Don't create JavaScript code for mouse events, except where they end up creating drag and drop events.
Simplify Mouse Movements
Only available if the above is off; instead of creating code for every single 1-pixel mouse movement made, create simplified code, like "move in a straight line from (x1, y1) to (x2, y2) over 300 milliseconds"
Simplify Strings
Repeated keyPress/keyRelease events are composed into "sendText" events, which take a string parameter. Like the two above options, this greatly decreases the length, and increases the readability of the generated code.
Use Variables For Strings
Any string used by sendText is stored in a JavaScript variable, instead of being hardcoded every time it is used; this makes it easier to modify the script in the future.
Use Variables For Object References
This doesn't change the meaning of the code at all, but generally makes the code shorter.
Include Raw XML as Comment
Mainly useful for debugging; includes the raw XML code received over the socket as a JavaScript comment
Ignore Qt Internal Widgets
Qt in some circumstances (such as drag and drop under X11) automatically creates magic widgets; this option makes Hooq ignore them.

by Fred Emmott at January 18, 2010 09:30 PM

Blood God

Code Style

One of the biggest annoyances I have with dealing with other people’s code is when they don’t use a sensible style. Obviously there have been many arguments over time about certain aspects of code style (brace placement, indentation, etc.) – but this isn’t really a problem as far as I’m concerned. I couldn’t care less where a given project has the braces placed, or if spaces are used for indentation, as long as it’s clear and consistent. This is something that recently irked me when I came to do some changes on Choob for the first time in what’s probably years.

From what I can tell (I wasn’t on the original team – so only have the code and documentation to go from) there was never a set of style guidelines drawn up when the project started. This, along with people like myself submitting code later through the project, has lead to a mix of code styles being used – and as a result the code is a bit of a mess in places. I’m pretty sure I’m to blame for some of this, so I’d love to fix it, but that’s a bigger problem than it first seems for a couple of reasons; the sheer number of lines of code it would touch, and getting everyone to use the new format going forward.

The first of these problems is less about the logistics of making the change than the problems posed afterwards. If I change the format of every class in the project, I’m going to have my name against pretty much every line of code when viewed in the source control system. Sure, my change was only superficial, but this doesn’t matter when it comes to running “svn blame” to work out who broke what. This is a tricky one, and one I’m not sure there’s a good solution to, other than to make sure that the commit with the format changes in only has those, such that people can then look at the previous revision, which hopefully has the name of the person responsible in it.

Looking at the second problem there’s a whole other set of challenges faced. Choob doesn’t have too many active developers, which I’m not sure would make it easier or more difficult. The numbers being small certainly makes it easier to get the word out, but because of the infrequency with which work is done – it’s entirely possible that they’ll forget that there is now a proper style and eventually the code base ends up back where it started. This can be made easier through IDEs with formatting features, and the ability to distribute a set of rules (I certainly plan to do this with Eclipse if I do reformat the code), but there’s no doubt at least one developer who doesn’t use an IDE, or uses one that hasn’t had the settings created for.

These are both great reasons why you not only should, but need to set out some code style rules at the start of the project. At work we’ve got projects that have strict rules as to how code should be laid out (with the Java ones having approved formatter settings ship with the internal build of Eclipse), and as a result the code base is consistent. Admittedly there are also measures in place, such as code reviews, that wouldn’t be practical in the case of something like Choob (where all the developers are working on it in their spare time around work/university), but having had an agreed style from the start is a huge bonus.

So, what makes a good code style? Well most languages have a set of conventions or even a set of “standards” specified by the creators of the language (for example Sun’s for Java), so in my opinion it makes sense to use these. This way it’s consistent with what developers are likely to find in other code bases (at least, I hope so) and therefore reduces the barrier to entry. There is obviously the problem of ending up with a lot of idiomatic code, but as long as it’s a widely known way of doing things – you should be able to get away with it (Perl’s returning undef for failure for example).

Unfortunately even with this there’s bound to be some argument that comes down to the braces – and I have to admit to there being one element about coding that frequently irritates me with them. Unlike the majority of arguments, however, it’s not about where to put them – but the lack of them when the language makes the optional. Sure for a one line if statement the additional braces can seem like a lot of wasted code, but what happens when a year or two down the line someone comes along and needs to add something? If they’re not careful they can very easily fall into the trap of not adding in the now required braces, and all of a sudden some of the block isn’t conditional. Equally the other side of the argument exists, where a conditional statement is followed by another that is expected to be run regardless, but is indented to the same level – making it impossible to spot at a cursory glance. Having seen both of these it’s never pretty, and for all its sins Perl manages this well be mandating braces in these very situations (obviously Python’s use of whitespace also avoids this problem).

Of course, part of that problem stems from indentation, and I’d hope that requiring sensible indentation goes without saying when discussing good code style. This, when combined with clear code and written in a consistent manner should make it easily maintainable – which is very important in any project that’s actually going to last.

With all of this in mind, I’m currently trying to find a sensible set of format guidelines that work well on the Choob project and can be automated within Eclipse, which I’m hoping I’ll be able to get into the code base and improve the experience for everyone. This is likely to be a bit of effort, and involve looking for the most common code style at the moment, but hopefully I’ll get something sorted this month (and then get nothing but complaints for the months afterwards!)

by Chris Hawley at January 18, 2010 07:00 PM

January 16, 2010

Antz

2010 Presidential Election in Ukraine, Round 1

First round of the 2010 presidential election is due to take place in Ukraine tomorrow. Five years after the Orange Revolution changed the outcome of a falsified election the political picture has changed a lot. Many of the faces are still the same, but there is now a feeling of a wide disillusionment among the voters. For most Ukrainians not a great deal has changed for the better in the last five years: the global recession has hit Ukraine hard, very little has been done to fight corruption that is sadly still the way of life around here, and there is very little to show for the supposedly pro-western stance Ukraine was supposed to have over the last five years.

Yushenko, the current president, is running again. He only has around 4% support in the opinion polls, and has no chance whatsoever of getting into the second round of the election (which takes place if no candidate secures 50% in the first round). With the complete failure of living up to expectations his political career is quickly speeding toward an end, which is just as well.

The candidate leading the opinion polls is Yanukovich, the candidate that nearly won at the previous election. With 33.3% support in the opinion polls he seems sure to have a strong first place in the first round, and has a real chance of winning in the second round. He is the reason the Orange Revolution happened at all, and I am saddened to see him poised to get elected properly this time around. He still remains a puppet candidate for the old regime, which still unofficially controls much of the country. I have a strong dislike for him as a person (as I would for anyone who thinks taking part in gang rapes and beating people to within an inch of their lives is a good way to pass time), in terms of his mental abilities (inability to spell "Prime Minister" without making four spelling mistakes is a huge minus), and in the sense of where him being the president will take the country.

The runner up in the polls is the current Prime Minister, Timoshenko. She is an excellent speaker (to the point of every other candidate being too afraid of sharing the same platform with her), and has charmed much of the electorate into securing herself 16.6% in the opinion polls. Her actual policies are however very much self-serving, and her term as the Prime Minister has revealed that she has very little interest for the direction of Ukraine, as long as it ends up with her pockets becoming fuller. For that reason many voters feel uncomfortable voting for Timoshenko, despite the main opponent being Yanukovich, instead preferring to cast their votes for one of the minor candidates, effectively splitting the non-Yanukovich votes into smaller fractions. In a two-round election that is not very dangerous, as whoever comes second will have a one on one run-off.

There are three more candidates that each have around 5% in the polls, but none of them have any real chance of getting through into the second round, and so discussing them here would be a waste of time.

An interesting aspect of the Ukrainian election process is the "Against All" vote. It currently has 9.6% in the opinion polls, making it third most popular option. Unfortunately the implementation makes this option uninviting: votes cast "Against All" do not count towards or against any of the candidates, and even if it does get the majority of the votes it will not in any way effect the outcome of the election. To further confuse the matter one of the candidates has changed his name to "Against All".

In conclusion, the first round is coming up, the outcome of it is already known, the two candidates in the second round will be Yanukovich and Timoshenko. Neither option spells anything good for Ukraine, so a rather depressing election the outcome of it will invariably hold Ukraine from turning into a real country for at least another five years. Leigh Turner, the UK ambassador to Ukraine seems more optimistic about the election, but in the absence of any pro-western candidates every choice is a bad choice.

January 16, 2010 09:37 PM

January 01, 2010

Sinjo

Seeing in 2010

As 2009 is drawing to a close I figured I’d do what I failed to do just before Christmas and write a blog post (hope you all had a good time). I don’t know exactly how to categorise this year. Overall I’d say it’s gone pretty well, I’ve had a nice time of it and quite look forward to going on into the new year without changing anything too drastically. I’m not going to label anything as a new year’s resolution as that’s a sure fire way to ensure you’ll keep it up for exactly two weeks. For me, the main thing is to keep a straight head in the next term, keep on top of work and see what comes out of it (this may look like a new year’s resolution, but it’s not, honestly).

With that spiel out of the way, there were a few things from 2009 which stick out in my mind. In no particular order, other than what seemed like a logical grouping at the time:

  • Nick Griffin’s appearance on Question time: I’d never been a fan of his politics, and as expected this only reaffirmed my opinion. What did come out of this for me, was the showcasing of the ludicrous views that he promoted. If anyone was in any doubt, this displayed the truly regressive stance he holds on such a wide range of issues. All I can hope for is that people saw the same thing and went away realising that perhaps, as far as “alternative” parties go, there are better choices.
  • America got a dose of change with the inauguration of Barack Obama: I’m not qualified to expand too much on this as I’ve not followed his actions especially closely this year. I’ll pick one thing out though; his policies on health care and the reform he wants to bring are a good thing. In Britain we’re accustomed to being entitled to health care, and it seems crazy that it would be denied to those less well off. Any policy which helps turn this around is a good thing. The Nobel Peace prize was a little premature though, seems like it’d be better to see how he does in his term before handing it out.
  • Swine Flu: I nearly left this one out. The worst part of this (okay, not the worst part really, any death caused by it is of course above this) was the reaction of the general populace. People were panicked by what boiled down to regular flu with a higher chance of being caught. People don’t get like this over your average flu which goes round in winter and there was no need to worry so much over it. Go get your extra flu jab if it applies to you, other than that, go have fun and stop worrying.
  • Jenson Button wins the F1 World Drivers’ Championship: This was a great season of Formula 1. It was thrilling all the way through and at the end we got the icing on the cake with a British champion. I hope we have this kind of close racing again next year as producing cars to this specification is no longer a new challenge. The introduction of the new points system should heat things up by heavily rewarding challenging for a win (new points are: 25, 20, 15, 10, 8, 6, 5, 3, 2, 1).
  • Muse album release: Perhaps not too much of a landmark for the year, but something I’d been looking forward to and therefore I’ll sneak it in here. They departed once again from the precedent they had set, experimenting on the new album with some stadium rock (which has drawn comparisons to Queen), a true R&B track in the form of “Undisclosed Desires”, a few tunes in the kick ass style we’re used to and a 3 part symphony. I initially didn’t know what to make of the album. A part of me had wanted it to simply be a continuation of their earlier music, but it’s grown on me and I appreciate it more for being a different experience.
  • Rage Against the Machine make Christmas Number 1: This was the first time I listened to the chart show on BBC Radio 1 and I was genuinely on edge throughout. That is until the internet took over and leaked the result part way through. It wasn’t totally certain, but as more of the order it listed was confirmed correct it seemed like Rage were in for the victory, and so they were. Having bought the song myself I was thrilled to see the pop machine dethroned and the smile wiped off Simon Cowell’s face. It only lasted the one week, and McElderry made it to the top a week later (congratulations to him, this wasn’t done to spite him), but the point was made.
  • Joe McElderry was well gutted: Sorry, couldn’t resist. It’s not too bad anyway, apparently he’ll make some money out of all this music lark.

Let’s see what happens next year anyway. Overall try to keep a positive vibe going on into 2010. All the best to you from here!

by Sinjo at January 01, 2010 12:59 AM

December 27, 2009

lamby

Readline tab completion using Django models

The following snippet allows readline-style tab completion backed onto Django models. I am using this in management commands in preference to specifying primary keys as arguments.

class QuerySetCompleter(object):
    def __init__(self, qs, field):
        self.qs = qs
        self.field = field
        self.prefix = None

    def __call__(self, prefix, index):
        if prefix != self.prefix:
            qs = self.qs.values_list(self.field, flat=True)

            if prefix:
                qs = qs.filter(**{
                    '%s__startswith' % self.field: prefix,
                })

            self.matching = list(qs)
            self.prefix = prefix

        try:
            return self.matching[index]
        except IndexError:
            return None

Example usage:

import readline

from mymodels import MyModel

completer = QuerySetCompleter(MyModel.objects.all(), 'name')
readline.set_completer(completer)

readline.parse_and_bind("tab: complete")
readline.set_completer_delims('')

while 1:
    print repr(raw_input(">>> "))

Pressing <tab> at this >>> prompt will complete from the name field of the MyModel model just like a regular shell would.

We call readline.set_completer_delims as our items may contain word delimeters - otherwise readline would attempt a seperate completion of (eg) "Bar" if one of our items was "Foo Bar".

December 27, 2009 07:22 PM

November 23, 2009

Bucko

Target Practice, redux

This is a response to Warwick Maths Challenge, 9 Nov 2009. The problem (should it be gone at the time of reading) sets the scene where Robin Hood and Friar Tuck have been shooting at a circular target. The target is painted with consecutive rings, each one the same width as the last. They observe that while the smallest ring Tuck hit was bigger than the biggest ring Hood hit, the total area of Tuck's and Hood's rings are equal. It then asks which rings they could have hit.

Reading through and trying to understand what I've done near to the end will help a great deal in solving a number of problems on Project Euler, namely those where you are required to solve certain types of 3 variable quadratic integer equations.

The problem asks a variety of minimisation questions about sums of odd numbers. Let's first work out why that is. We want to find the area of those rings. A ring is made by removing a small circle from a big circle, so we actually need to find the area of the circles. The circle numbered n is n times as big in all directions as the circle numbered 1 and we'll pretend for the sake of argument that the target is picked so that the smallest circle has area equal to 1.

Early on in school, you learn that multiplying the lengths of something by n will multiply the area by n×n. This is obvious for a square as its area is the side lengths multiplied together, so a square with side length 1 has area 1 and side length 2 has area 2×2=4, so the big circle for ring number n has area . The area of the rings is the area of the big circle, minus the area of the little circle, and the little circle is just the previous big one, so it's n×n - (n-1)×(n-1). A bit of algebra tells us that the answer is 2n-1 or the nth odd number.


So, first we have to find the first answer where the sum of some small rings equals the sum of some bigger ones. A few facts aren't too hard to see: First, Tuck, hitting the outer rings must have hit less rings than Hood, hitting the inner ones (as all of his are worth more). Second, since odd+odd=even and odd+even=odd, we find that Hood actually hit a multiple of two more rings than Tuck (otherwise one of them hit an odd total area and the other hit an even total area). So the smallest example likely has Hood hitting 3 and Tuck hitting 1. In fact, rings 1, 2 and 3 have total area 1+3+5=9, which is the area of ring 5, so this is the smallest example.

We've definitely picked the smallest example here, and so it's the only one using 5 rings. (If Hood hits a different ring, his total will be more, so Tuck has to hit something further out.) What's the next one? Well, if Hood's smallest ring is increased, so must the rest, giving rings 2, 3 and 4. This has area 3+5+7=15 which is a lot higher. It adds much less area to only make one ring bigger, so we have to change ring 3. It turns out that rings 1, 2 and 4 have total area 1+3+7=11, which is the area of ring 6, so with 6 rings there are two ways they could have hit the target as described. Actually, this works in general: If you make Hood hit a ring that's further out than his original set, you can balance it by moving one of Tuck's rings out by the same amount. Again, it turns out that there are no more examples with 6 or less rings.

With seven rings, things are different. The method described above gives two more examples: Hood can hit 1, 2 and 5 or 1, 3 and 4 for a total area of 1+3+9=1+5+7=13 while Tuck hits ring 7 which also has area 14. As well as this, we can bump up the number of rings they hit. Hood can hit 2, 3, 4 and 5 with total area 3+5+7+9=24 while Tuck hits 6 and 7 with total area 11+13=24. This also must be the first example where Hood hits 4 rings and Tuck hits 2 (if you move Hood's smallest ring in you find there's a ring that both Tuck and Hood must have hit). It's also the first example where Tuck and Hood hit a completely consecutive run of rings (with 3 rings for Hood, you always miss at least one before getting to a Tuck ring) and the second example where Hood and Tuck's rings are consecutive but a gap between is allowed.

The answers to the three questions are, then, as follows. What's the smallest number of rings required for
  1. There to be a solution? 5
  2. There to be 2 solutions? 6
  3. There to be two solutions where both Hood and Tuck hit a run of rings, missing none inbetween? 7
I found this question pretty interesting, not for the above solutions, but for the following, other question: What are the examples where both Tuck and Hood's rings are all consecutive, like with 2, 3, 4, 5 and 6, 7?

In this case, there's four runs of rings on the target: Small ones that Hood and Tuck both miss, ones Hood hits, ones Tuck hits and big ones that they both miss. Let's say that there's i small ones that they both miss, that Hood hits j and that Tuck hits k. The total areas here are pretty easy to work out, since they're just the areas of big circles, minus the areas of smaller ones. The area that get missed is i×i, Hood hits (i+j)×(i+j)-i×i and Tuck hits (i+j+k)×(i+j+k)-(i+j)×(i+j). We want Hood and Tuck to total the same, so we want:

(i+j)×(i+j)-i×i = (i+j+k)×(i+j+k)-(i+j)×(i+j)

In fact, notice that i+j is the biggest ring Hood hits and i+j+k is the biggest ring Tuck hits, so let's make the equation simpler by setting m=i+j and n=i+j+k. Notice that we can re-arrange these as j=m-i and k=n-m, so we can work out j and k if we know i, m and n. Anyway, we now want:

m×m - i×i = n×n - m×m

Or in other words:

2×m×m = i×i + n×n

And we need i, m and n to be whole numbers with 0≤i<m<n. We'll not worry about the ineqations for now, so some of the values we get might not make sense in the context of the question at hand.

I'm going to switch to squared symbols now, so a2 = a×a and ab=a×b. An interesting observation is that we can divide both sides by m2 to get:

(i÷m)2 + (n÷m)2 = 2

If you know your geometry, you'll know that x2+y2=2 is the equation you use to find if the co-ordinates (x, y) represent a point on the circle of radius the square root of 2, centre (0, 0). This works in reverse, too! If x=p÷q and y=r÷s then p2+r2=2(q×s)2. That's pretty useful, as we now know that if we can find a point on the circle with co-ordinates (p÷q, r÷s) then we can set m=qs, i=ps and n=rq to get a solution to what we want to solve, and vice versa. So now we want a list of these p, q, r and s.

Notice that if we multiply or divide p and q by some number, we will multiply or divide i, m and n by the same number. We can therefore assume that p÷q and r÷s are in their lowest terms (ie. you can't do any cancellation).

There's one pretty obvious point on the circle, (1,1). It corresponds to i=m=n, in other words where neither Tuck nor Hood hit anything at all. If you think about the situation a bit, it's pretty clear that lines coming out of (1,1) (except vertical ones) are described by their gradient, and each line hits exactly one point on the circle other than (1,1) (except for the tangent at that point). Let's calculate the gradient of a line from (1,1) to (p÷q, r÷s). It's equal to (1-r÷s)÷(1-p÷q). Notice that this is a fraction whenever the point's co-ordinates are fractions and vice versa.

Let a÷b be the gradient in its lowest terms so that the line hits the point (p÷q,r÷s)=(1-bt,1-at). At what value of t do we find this? In other words, can we write p, q, r and s in terms of a and b? The answer is when (1-bt)2+(1-at)2=2, in other words when (tb)2+(ta)2=2×(a+b)t. A bit of re-arrangement gives t=2(a+b)÷(a2+b2). Notice that p÷q=1-t=1-2b(a+b)÷(a2+b2)=(a2+b2-2b(a+b))÷(a2+b2) and similarly r÷s=(a2+b2-2a(a+b))÷(a2+b2), so we can set

p=a2+b2-2b(a+b)
r=a2+b2-2a(a+b)
q=s=a2+b2

A bit of fiddling shows that the two cases we ignored above (vertical line and tangent line) also fit into this schema, so all solutions come from values a and b in this way. I'm not going to explain the following analysis so deeply as it involves some number theory that I don't want to explain.

Suppose that p÷q is not in lowest terms. Then 2b(a+b) has a prime common with q=a2+b2. Let's assume a+b=1 mod 2, so that this prime is not 2. Then gcd(a(a+b),q) is nontrivial. Computing the greatest common denominator, we find this means gcd(b(a-b),a(a+b)) is nontrivial. We know gcd(a,b)=1, so this means gcd(a,a-b) or gcd(b,a+b) is nontrivial, both of which are contradictions. So the additional constraint of a+b=1 mod 2 ensures the results are in lowest terms. Retrieving i, m and n, we find:

i=ps=(a2+b2-2b(a+b))(a2+b2)
m=qs=(a2+b2)2
n=qr=(a2+b2-2a(a+b))(a2+b2)

These are clearly not in their lowest terms; the reason is that we've multiplied through by qs instead of lcm(q,s). Doing this, we find:

i=p=a2+b2-2b(a+b)
m=q=s=a2+b2
n=r=a2+b2-2a(a+b)

Back to those inequalities, we now need a2+b2 ≥ 2b(a+b) and 2b(a+b) > 0 > 2a(a+b) to ensure the solution makes sense. If a+b > 0 then -b < a < 0 < b, otherwise the inequality is reversed. The first condition doesn't seem to simplify too well. To summarise everything, the complete set of distinct solutions is:

i=k(a2+b2-2b(a+b))
m=k(a2+b2)
n=k(a2+b2-2a(a+b))

Where k is a positive integer, and a and b satisfy:

gcd(a,b)=1,
a+b=1 mod 2,
-b < a < 0 < b or -b > a > 0 > b, and
a2 ≥ 2ba+b2

As a quick check, setting a=1 and b=-2 gives us the original example with i=1, m=5 and n=7!


If you were paying attention, this should help you solve certain Project Euler problems, for example those that ask you for a list of Pythagorean triples.

November 23, 2009 03:42 PM

November 19, 2009

lamby

Transparent directory replacement with zsh

Under UNIX, it is possible to get into a position where your shell's working directory is pointing to an inode that no longer exists, but a "newer" directory with the same name exists.

The easiest way of reproducing this scenario is as follows:

$ cd $(mktemp -d)
$ rm -rf $PWD
$ mkdir $PWD

However, as concrete example, imagine you are debugging something deep inside a generated code tree — if you ran make clean and then make again from another shell, your debugging shell would be left pointing to an inode that does not exist (as it was removed by make clean), yet the make call would have made an identically named directory in its place.

Shells behave rather unintuitively in this situation; ls will (correctly) return no results, but attempting to do anything useful results in curious error messages.

I use the following zsh precmd hook to transparently fix this for me:

precmd () {
  # "DOS emulation mode"
  if [ "$(stat -c %i . 2>/dev/null)" != \
                 "$(stat -c %i -- "${PWD}" 2>/dev/null)" ]
  then
    OLDOLDPWD="${OLDPWD}"
    if ! cd -- "${PWD}" >/dev/null 2>&1
    then
      echo "W: ${PWD} does not exist anymore"
      return 1
    fi
    OLDPWD="${OLDOLDPWD}"
  fi
}

The cost of the stat calls seem justified by the convenience and are in any case insignificant compared to the other things I am doing in my precmd. If you are using zmodload zsh/stat, then replace stat -c %i with stat +inode.

(My previous experience with zsh tricks is that the functionality invariably exists already in some obscurely named builtin. Here's hoping.)

November 19, 2009 12:57 PM

November 18, 2009

Blood God

KDE 4 on Kubuntu – Part II

It’s been a while since I first tried KDE 4 on Kubuntu, but as I recently bought a new machine, I figured I’d give it another whirl. Admittedly, I didn’t have much choice in this matter if I wanted to install a recent version of Kubuntu, as they moved to using KDE 4 as the default quite a while ago, but Kubuntu Karmic Koala (9.10) comes with KDE 4.3, the latest full version that was released not too long ago.

The install process was nice and smooth, with everything just working out of the box, which was certainly a nice feature. The one thing that needed a bit of manual intervention was the installation of some non-free components, such as the nVidia graphics drivers so as to get 3D acceleration, but this wasn’t too much trouble (although editing the XOrg.conf file is probably beyond a real beginner).

Given that almost two years has passed, I was hopeful that there’d been progress, and I wasn’t disappointed. The niggles that I had previously noticed with KDE 4 were gone, and I can now have a sensibly sized panel, displaying only the current desktop’s activities in the task bar, along with a couple of useful widget on the desktop. The settings are all there, and applications that were missing last time, like Kontact, are now back and ready to go.

One of the other applications that has been updated since I last used KDE is Amarok. Now on version 2.2, a whole load of changes have occurred – including some rather drastic ones to the UI. Unfortunately, it seems to me that performance has dropped considerably – with it taking minutes and 100% CPU in order to do basic tasks such as adding a track to the play list. I’m currently wondering if this performance (or lack thereof) is down to the database, as previously I’d migrated to using MySQL rather than the default SQLite. I’m tempted to look into this again, as the performance is a bit of a joke at the moment. One other thing I’ve noticed, is that the on screen display doesn’t always work – instead only appearing intermittently and with no real consistent cause for it (not) appearing.

All in all the experience is a lot better than where it was, and it’s great to see that it’s come on in such leaps and bounds since the original release almost two years ago – and I’m certainly not looking at other desktop environments to move to!

by Chris Hawley at November 18, 2009 06:00 PM

Bucko

Independent's Best 50 Board Games

The Independent has produced a list of the Best 50 Board Games

Having used a few quick and dirty bash scripts to pull the list from their extremely laggy web site, I'm reproducing the list here for your speedy reading pleasure. There's information on how to get each game on The Independent's article.

1STRATEGYOthello
2ALL THE FAMILYPictureka! Flipper Game
3UNDER-EIGHTSSnakes and Ladders
4OLD FAVOURITESMake Your Own Opoly
5STRATEGYWar on Terror
6ALL THE FAMILYThe Logo Game
7UNDER-EIGHTSSnail's Pace Race
8OLD FAVOURITESTrivial Pursuit Team
9STRATEGYLabyrinth
10ALL THE FAMILYMurder Mystery Mansion
11UNDER-EIGHTSEnchanted Forest
12OLD FAVOURITESRisk
13STRATEGYPentago
14ALL THE FAMILYBeat the Parents
15UNDER-EIGHTSThe Bad-Tempered Ladybird
16OLD FAVOURITESNo Stress Chess
17STRATEGYBackgammon
18ALL THE FAMILYLast Word
19UNDER-EIGHTSCastle Knights
20OLD FAVOURITESArticulate
21STRATEGYSequence
22ALL THE FAMILYSmart Ass
23UNDER-EIGHTSSuitcase Detectives
24OLD FAVOURITESRummikub
25STRATEGYPerplexCity
26OLD FAVOURITESSyl-la-bles
27UNDER-EIGHTSLudo
28OLD FAVOURITESCranium Wow
29STRATEGYIngenious
30ALL THE FAMILYFamily Fortunes
31OLD FAVOURITESGuess Who?
32OLD FAVOURITESScrabble Deluxe
33STRATEGYBrainmaster
34ALL THE FAMILYBig Brain Academy
35UNDER-EIGHTSShut the Box
36OLD FAVOURITESGame of Life
37STRATEGYBlokus
38ALL THE FAMILYScattergories
39UNDER-EIGHTSQwirkle
40OLD FAVOURITESTaboo
41OLD FAVOURITESQuoridor
42ALL THE FAMILYPass the Bomb
43UNDER-EIGHTSOne Banana Two Banana
44OLD FAVOURITESHexago Continuo
45STRATEGYCarcassonne
46ALL THE FAMILYWhere Is Moldova?
47UNDER-EIGHTSCreationary
48OLD FAVOURITESPictionary
49STRATEGYTicket to Ride
50?The Really Nasty Motor Racing Game


So, here's the thing. Ignoring the rather inconsistent naming (why is Othello strategy while Hexago Continuo is an Old Favourite?), this list seems to be rather unbalanced.

Firstly, it ignores a few obvious candidates, like Chess, Go, and Dominoes. I could just about forgive go if it were a popularity contest, as its only recently started to gain popularity in the UK, but the other two? What, are you saying Hexago Continuo supersedes dominoes and No Stress Chess supersedes chess? Give me a break.

The thing that really bugs me, though, is how few of these games I've even heard of. I have to dig all the way to #45 before I find a game that the Warwick Sci-Fi and Fantasy Society (the local board game geeks) play. Excepting Old Favourites, I've only heard of about 8 of those games, most of which belong in 'old favourites' anyway.

Board Game Geek has a take on many of the entries, and many of the posters seem to have reached the common conclusion: It's a huge wasted opportunity.

(For those interested, I did or N in `seq 1 50`; do wget -O - 'http://www.independent.co.uk/extras/indybest/outdoor-activity/the-50-best-board-games-1815441.html?action=Popup&ino='$N -q|grep -A1 '<p><i>'|sed 's/<[^>]*>//g'|sed 's/^[ ]*\| *$//g'|(read I; read J; echo "$N,$I,$J"); done|tee bestgames.csv then edited the CSV a bit)

November 18, 2009 12:52 AM

November 09, 2009

Brad

The AVR32 Debian port

So, Debian have kindly funded some new hardware for the AVR32 port, it is the €300 ICnova board, probably one of the best AVR32 boards you can buy to date. The first thing I noticed after finally taking delivery of this a few days ago, it how small it is. I was expecting it to be pretty large, especially seeing as how much stuff it has on it. In fact, it's just smaller than the current NGW100 board I have.

So why then new board you ask? Well firstly, I don't own nor have physical access to a board with a display, so this makes testing the all important X server rather difficult [1]. Secondly, the board I currently have is pretty slow due to its lack of RAM, and the fact that it's only 16bit.. This makes using it for porting things rather painful. Finally, the port is getting to the stage where it really needs a porter machine, so that's what this machine will become eventually.

Whilst I'm here, I'm going to take this opportunity to prod people about the AVR32 port. ;) The port is currently at a complete standstill since I've run out of buildable packages. This is solely due to the GCC port being buggy (I'm currently fighting with a GCC internal compiler error whilst building Qt4, did I mention that delta is a godsend?) and lots and lots of packages with outdated config.{guess,sub} files. I currently have around 120 bugs filed for this, so if one of your packages is one of them please fix it. Also, if one of your packages has an Architecture field which avr32 should be in (it should be in most since it's just another linux architecture) please add it.

I'd also like to point people at AVR32 dependency status. This is a summary of unbuildable packages and what other packages are stopping them from building (thanks to KiBi for writing this). If you see one of your packages at the bottom of one of the graphs, that means your package is blocking a load of packages from building, and that's bad, m'kay :) I'd very much appreciate help from other people with this, especially for the silly build failures that are trivial to fix.

For those who are interested, the port currently stands like this:

  • BD-Uninstallable: 2213
  • Failed: 495
  • Installed: 5336

Now for some useful and interesting AVR32 port related links..

Finally, a picture of the new board, and a big thanks to Debian for funding this!

[1] Unfortunately X doesn't seem to work at all yet, it just segfaults. :(

November 09, 2009 10:15 PM

November 05, 2009

lamby

Confusing bugs

I love bugs that are just confusing and/or break your mental model of how you think the software works.

Here is a good example: Given the following HTML in Gecko, what happens when a user clicks the button?

<form method="POST" action="/form-action">
  <a href="/anchor-target">
    <input type="button" value="harro">
  </a>
</form>
GET /anchor-target HTTP/1.1
GET /anchor-target HTTP/1.1

2 GET requests? Okay, perhaps some sort of event bubbling problem; it's relatively common in GUI internals. But what about:

<form method="POST" action="/form-action">
  <a href="/anchor-target">
    <input type="image" src="http://upload.wikimedia.org/wikipedia/en/7/75/LHC5.jpg">
  </a>
</form>
POST /form-action HTTP/1.1
GET /anchor-target HTTP/1.1
GET /anchor-target HTTP/1.1

Wait, this time it POSTs the form and makes 2 GET requests?

Answers in a Bugzilla report, thanks.

November 05, 2009 01:27 PM

October 23, 2009

Brad

Brand new website, almost....

So I've decided to join the django train, and redo my website in django. As you can probably see it's basically the same as it was before, apart from the obvious addition of a blog, mainly motived by Sledge wanting a blog post about the AVR32 hardware Debian are going to fund for me, and me hating wordpress^Wphp... ;)

I should probably thank lamby somewhat at this point since the code for the blog part of this was mainly stolen^Wborrowed from him ¬_¬

October 23, 2009 11:28 PM

October 12, 2009

Laurence

Catalyst, Catalyst::Authentication::Credential::Remote and mod_perl

Mod_perl does not pass REMOTE_USER as an environment variable to Catalyst, so the Catalyst::Authentication::Credential::Remote plugin which allows convenient offloading of authentication to apache will not work.

This is easily solved by adding this code to the root auto function:

$c->req->remote_user($c->apache->user) if $c->can('apache');

Now I can offload authentication to apache’s mod_kerb whilst using catalyst for authorisation.

by Laurence at October 12, 2009 09:29 AM

September 30, 2009

lamby

GeoDjango and the UK postcode database

This post describes how to play with the UK postcode database that recently turned up on WikiLeaks using GeoDjango.

You will need the following pieces of free software:

  1. PostgreSQL (MySQL does not provide sufficient geospacial features) — I will be using Postgres 8.4.1; please adjust your paths for your version and/or distributor.
  2. PostGIS spacial database extensions — I will be using version 1.4.0.
  3. Django — I will be using version 1.1. This version includes the GeoDjango framework.
  4. GEOS geometry engine — I will be using version 3.0.0 from the libgeos-c1 package in Debian unstable.

You will also need a copy of the leaked postcode data. Usage of this data is almost certainly a copyright violation.

Getting started

First, create a Django project and application for this example:

$ django-admin startproject ukpostcodes
$ cd ukpostcodes
$ ./manage.py startapp postcodes

Next, edit settings.py:

  • Configure DATABASE_*. You must use the postgresql_psycopg2 PostgreSQL database driver.
  • Add django.contrib.gis & ukpostcodes.postcodes to INSTALLED_APPS.

If you have not already done so, you must install PostGIS in your database:

$ sudo -u postgres createlang plpgsql -d <databasename>
$ sudo -u postgres psql <databasename> -f /usr/share/postgresql/8.4/contrib/postgis.sql
$ sudo -u postgres psql <databasename> -f /usr/share/postgresql/8.4/contrib/spatial_ref_sys.sql

Modelling a postcode

Now define a simple model to store imported postcodes:

from django.contrib.gis.db import models

class Postcode(models.Model):
    name = models.CharField(max_length=8, db_index=True)
    location = models.PointField()

    objects = models.GeoManager()
  • We import models from django.contrib.gis.db instead of django.db.
  • We override the default objects manager with GeoDjango's GeoManager() so we can perform spacial queries.

Save this model in postcodes/models.py. We can now create this model in our database using:

$ ./manage syncdb

Importing postcode data

We will use a management command to import the data. First, create the necessary modules with:

$ mkdir -p postcodes/management/commands
$ touch postcodes/management/commands/__init__.py
$ touch postcodes/management/__init__.py

Next, save the following code in postcodes/management/commands/import_postcodes.py:

import sys
import csv

from django.contrib.gis.geos import Point
from django.core.management.base import BaseCommand

from postcodes.models import Postcode

class Command(BaseCommand):
    def handle(self, \*args, \*\*options):
        Postcode.objects.all().delete()
        count = 0

        for row in csv.reader(sys.stdin):
            try:
                name = row[0].upper().strip().replace('  ', ' ')
                location = Point(map(float, row[13:15]))
            except ValueError:
                print "I: skipping %r" % row
                continue

            Postcode.objects.create(name=name, location=location)

            count += 1
            if count % 10000 == 0:
                print "Imported %d" % count

Finally, we can import the data using:

$ bunzip2 /path/to/uk-post-codes-2009.bz2 | ./manage.py import_postcodes

This takes approximately one hour on my computer. If you plan to import the data multiple times, consult the Django documentation for faster methods of importing initial data.

Quering the postcode database

We can now query our postcode database. First, let's find the postcode for Buckingham Palace:

>>> from postcodes.models import Postcode
>>> palace = Postcode.objects.get(name='SW1A 1AA')
>>> palace
<Postcode: SW1A 1AA>

Next, let's find all postcodes within 1 mile. For this, we will need to construct a tuple representing the area we want to search in.

>>> from django.contrib.gis.measure import Distance
>>> area = (palace.location, Distance(mi=1))
>>> Postcode.objects.filter(location__distance_lte=area)
[<Postcode: SE1 7BF>, <Postcode: SE1 7SG>, <Postcode: SE1 7JA>, <Postcode: SE1 7PB>,
 <Postcode: SE1 7PD>, <Postcode: SW1A 1DW>, <Postcode: SW1A 1EA>, <Postcode: SW1A 1EE>,
 <Postcode: SW1A 1EF>, <Postcode: SW1A 1EG>, '...(remaining elements truncated)...']
>>> Postcode.objects.filter(location__distance_lte=area)
7024

To find "neighbour" postcodes, we can pass a Point object to the .distance(..) method on our queryset – this adds a field we can then sort on:

>>> Postcode.objects.distance(palace.location).order_by('distance')[:4]
[<Postcode: SW1A 1AA>, <Postcode: SW1E 6JP>, <Postcode: SW1E 6LA>, <Postcode: SW1E 6WG>]

Putting it all together

Let's pretend we are making an app to find pizza from an arbitrary user-submitted postcode. (We will assume we have parsed the user's postcode and have found its Postcode object, but we will continue to use Buckingham Palace here.)

First, define the following model in postcodes/models.py and re-run ./manage.py syncdb:

class PizzaJoint(models.Model):
    name = models.CharField(max_length=200)
    postcode = models.ForeignKey(Postcode)

    objects = models.GeoManager()

    def __unicode__(self):
         return self.name

Now, add some local pizza places:

>>> def add_pizza_joint(name, loc):
...     code = Postcode.objects.get(name=loc)
...     PizzaJoint.objects.create(name=name, postcode=code)
...
>>> add_pizza_joint('Pizza Hut', 'SW1E 6SQ')
>>> add_pizza_joint('Pizza Express', 'SW1E 5NA')
>>> add_pizza_joint('Pizza On The Park', 'SW1X 7LY')

If we were not using GeoDjango, we would have to be content with simply matching the postcode:

>>> PizzaJoint.objects.filter(postcode=palace)
[]

However, we can now find the cloest pizza places:

>>> PizzaJoint.objects.distance(palace.location,
            field_name='postcode__location').order_by('distance')
[<PizzaJoint: Pizza Hut>, <PizzaJoint: Pizza Express>, <PizzaJoint: Pizza On The Park>]

The .distance(..) method also sets a distance attribute on all the objects returned; this makes it easy to show the actual distance to the user:

>>> for x in PizzaJoint.objects.distance(palace.location,
                     field_name='postcode__location').order_by('distance'):
...    print "%s (%d metres away)" % (x, x.distance.m)
...
Pizza Hut (499 metres away)
Pizza Express (509 metres away)
Pizza On The Park (902 metres away)

For more details, see the GeoDjango documentation.

September 30, 2009 01:10 AM

September 20, 2009

Tim

Escape Velocity

I am no longer going to tag my blog posts with Planet {WUGLUG,UWCS}.

As I don't even lurk in the IRC channels any more (or read the mailing lists), and as of recently don't even live near enough to campus to visit, I think it's time to stop imposing my ramblings on the respective groups. :)

You can still read my blog directly, or on Facebook, if you are so inclined.

September 20, 2009 01:28 PM

September 18, 2009

Benji

openSUSE conference – Software Portal Talk

Here are my notes from the software portal talk Pascal and I gave yesterday at the openSUSE conference.

The slides we used are also available.

What is the Software Portal?

Software Portal is a free-software web-application that runs as a service on on opensuse-community.org. It provides both a web-ui and an API. It is possible you are already using the software portal if you use the webpin search page or the command line webpin programme written by Pascal. For the last few months this has been backing on to the software portal.

The software portal indexes package repositories, both those discoverable with the openSUSE build service API and those users add manually through the web interface. It also allows users to contribute additional metadata such as ratings, comments, tags, and screenshots. These two sources of information are both associated against “applications”. The web user-interface is primarily a way of finding, installing, and adding information about applications.

Applications

So, what is an application? We define applications within software portal to be a higher level concept than packages. Applications are what users perceive software as – A product they are looking for and may wish to install to perform a task. Packages are the mechanism of how applications are distributed to users. Packages are platform-specific, they require different packages for different operating systems, and different architectures. The split between packages is also defined by pragmatic issues related to distribution more than conceptual boundaries. Some applications may require multiple packages installed in order to use. Conversely some packages contain more than one application. For example the application GIMP on openSUSE on a 32bit platform might consist of the main gimp package, a gimp-banding package, and some core plugins in another package. Another opposite example is the kdenetwork3 package that contains multiple chat applications.

The web-ui provides application-level search, tag-browsing, and installation.

In the software portal we strive to map packages to the appropriate applications. There are three mechanisms we currently use to do this:

  1. A set of rules stored in the database that map common package naming conventions to application names.
  2. For example: currently kde4 applications tend to have the packagename kde4-packagename. Many non-desktop specific packages start with the application name. Rules are assigned priorities so that more-specific rules can override general rules.

  3. The presence of .desktop files in packages.
  4. User-defined mappings.
  5. Through the web-ui, users can define additional applications and edit existing applications. While editing, users can define rules for which source-packages should be mapped to which applications during repository indexing.

One of the main aims of abstracting away from packages to applications is that we want to allow users to just click install at an application level, without having to know what their hardware/software platform is or choose which package vendor to use. To this end we need some way to choose a preferred vendor for each application in the case where there are multiple vendors of that application. We do this by

  1. Associating a priority with repositories based upon their support status and stable/experimental status.
  2. So the highest priority would be the update and static distribution repositories, and the lowest would be random home: repositories in the openSUSE build service.

  3. Allowing power-users to customise the preferred vendor and selection of binary packages to install by default for an application. Via the web-ui.

The operating system agnosticism is provided by one-click-install and the system package management can choose the appropriate architecture packages.

Why is it needed?

openSUSE embraces third party vendors of packages. Which means that there’s a low barrier to entry and anyone can start packaging and publishing software to users. However, this freedom leads to a confusing situation for users.

Many repositories

(Software Portal is currently indexing around 4000 and this is not exhaustive.)

Package duplication.

Some applications such as amarok has been packaged dozens of times for openSUSE by different people for various reasons. In this case how does a user choose which vendor to choose? We also encourage duplication by not making it easy to find existing packages.

Lack of user-feedback

One packager may package dozens of packages but may not extensively use many of them. We’re not currently good at collecting feedback of package and software quality from users. It’s also not clear for users where to file bugs for the software they’ve just installed.

Confusing concepts

It should not be necessary for users to understand concepts such as packages and package repositories just to install software. It’s also unnecessary for users to know what operating system (version) they’re running or on what hardware.

Insufficient data

We currently rely heavily upon a few package maintainers to source and supply metadata for packages. Packages come with descriptions and summaries but lack more rich metadata such as screenshots. There are many more end users than packagers so it makes sense to let them contribute this information.

Implementation Platform

Software portal project is implemented using Java 6, making heavy use of Spring Framework, Apache Wicket, Jersey, Maven, and MySQL.

Future Plans

We are currently working on finalising a “1.0″ release, fixing some of the major bugs, performance issues, and polishing the user interface to get to the point that it is safe for end users to start using and we can publicise it. We need help right now in testing and finding bugs.

The biggest performance issues I am currently working on are:

  • Search of all 20+ million unique file names every time someone performs a search.
  • Calculation of the best binary packages to install for applications with dozens of different vendors.

Challenges arise from the frequency of change of the package repositories (thousands of packages may change every day). Index formats are a balance between query performance and update performance.

After first release one of the biggest issues we want to resolve is that people cannot use their existing login that they use for all other openSUSE services to log into the software portal. Due to Novell control of the authentication system. We have a dependency on Novell here, but hope to implement openID support and that Novell will be able to implement an openID provider.

Further ahead we would like to work on things like:

A recommendation engine

We could do things like recommend you plugins for software you have installed.

User-specific search.

When we know information about the user, their preferences, and what operating system they have installed we can tailor search results to match them, prioritise applications that are available for their platform, and give higher priority to applications for their preferred desktop.

Translation

The web-ui is already translatable. Application descriptions are stored in a translatable manner but the source of this information (package repositories) generally does not have translations. We could potentially build an interface to allow people to translate the content itself if there were a demand for it.

Use Build-Service remoting or Hermes notification framework

Move towards a push model where the software portal is notified of new software, rather than the current pull system where we have to poll package repositories to see if they’re changed, then compute the differences from the previous version if they have.

Integration with bugzilla

Report bugs directly from software portal.

Integration with upstream

Perhaps via the DOAP (Description of a Project) format. We could both consume DOAP files provided by upstream projects, and expose DOAP for software we have indexed.

The Master Plan

Our original aim was to supplement the web-service with a rich desktop client that utilises the information indexed by the software portal and combines that with information available to it as an application running on the local system, such as the currently installed packages, and the repositories the user subscribes to.

Perhaps a YaST module? We already have a package-search yast module that does use the software-portal API to search for packages, but much more powerful things are possible. Unfortunately we do not currently have the time to work on a desktop client on top of the web app itself so if anyone is interested in working on this side of things please get in touch.

Demo

We gave a demonstration of some of the features mentioned above during our talk. You can play with the current version online. Though please be aware there are lots of known bugs and performance issues, it’s not ready for end-users yet.

That said, please do play with it, and let us know what’s broken by filing bugs under the openSUSE.org->Software Portal bug component.

If you want to try out more of the administrator features such as adding repositories, adding applications, customising package to application mapping etc then create an account and drop me an email so I can grant you permissions.

How can you help?

  • Start testing it
  • File bug reports
  • Help create and organise content
  • Developers who know Java and want to contribute code would be welcome. We would be happy to help people get started with the codebase.
  • Designers can help us improve the look and feel and interaction. Some of the processes such as mapping packages to applications are somewhat klunky. It would be good to get some suggestions on how to make this functionality easier to use.

by benji at September 18, 2009 04:39 PM

September 14, 2009

Fred

Server Issues

phoenix.slamd64.com, which normally runs this site, slamd64.com, and several other sites is currently down due to what looks like a hardware failure - it'll be back up again as soon as possible. I've not imported the old blog data in case it spams planet - if it happens, I'd rather only do that once (when it's back up).

In the mean time, my various projects live on Gitorious - let me know if I've missed one you're interested in. Here's a quick summary (sorry, my normal screenshot-dumping-ground is on the server that's down):

Jerboa
A Qt4/Phonon-based cross-platform music player. Actually, the code I've pushed is a nearly-from-scratch rewrite I started a few months back, with much better performance, and nicer code.
FastCgiQt
A library for writing dynamic websites with Qt. This is especially nice combined with QtSql, and the XSLT2 support that entered Qt 4.5. I prefer this to the widget-based approaches, as I like writing my own [X]HTML, and it makes it easy to output RSS/other XML as well.
StatusBoard
A small application for displaying big red/green squares - for example, at Mendeley, we use it to have an at-a-glance indication of if there's any Nagios warnings, or if any of our buildbot processes are failing.
TypoFail
Think typeracer/typespeed, with LAN multiplayer, and written in Qt.
fredemmott-www
The code for this site - uses FastCgiQt
paincc
A small shell script to put in your path, that forces -Wall -Wextra -Werror -Wnon-virtual-dtor -Wold-style-cast -Woverloaded-virtual -fdiagnostics-show-option if you don't want to spend the time figuring out how to do that on whatever build system you're using right now.
Hooq
More on this later; it's a GPLv2+ GUI testing tool from Mendeley for Qt4 applications.

by Fred Emmott at September 14, 2009 09:14 PM

Twinklefeet

Muse - The Resistance.

Ever since I first heard Feeling Good while watching some random music channel I’ve been in love with Muse. I’ve eagerly awaited every album they’ve released and their new offering The Resistance is no exception. The album opens with the heavy, rolling sounds of Uprising; The droning vocals of Bellamy and strong beats combined with familiar muse sounds should leave the old fans feeling happy. The album continues with some new and interesting sounds with songs such as Undisclosed Desires, with a beat-heavy electronic backing track, or the Queen-esque United States of Eurasia combined with a little Chopin in the form of Collateral Damage. Similar to the band’s two previous albums these tracks feel like a natural evolution of their sound while remaining distinctly “Muse”. Going into the album’s second half with the 7 minute Unnatural Selection throws the album into overdrive with a tune that starts as reminiscent of some of their classic songs such as Plug in Baby or Hysteria. The album slows down again with the jazzy I Belong To You, a piano heavy tune that wouldn’t sound out of place in a 1930s feelgood film. Again, a slightly different direction for the band but one that still manages to sound like noone else. I think I even made out a clarinet in there somewhere. The album finishes off with the three part Exogenesis Symphony totalling just under 13 minutes, starting off with the epic and eerie Overture. Bellamy makes a return to the soft, drawn out vocals of previous albums working seamlessly into the music. Building up in part two with a haunting piano it strikes familiar to the bands previous work, especially Origin of Symmetry. The queen style guitar makes a subtle return, but soon disappears as the song winds down into part three; the mellow and peaceful Redemption. A simple, soft piece that winds the album down perfectly.

All in all, I love this album. Again the band have managed to try out a few new things, go in a few new directions while still remaining the same band and would definitely recommend it to a fan of their previous stuff. Well worth the time.

by admin at September 14, 2009 12:08 AM

September 09, 2009

Tim

Unit testing

I spent the last day and a half writing a vaguely interesting Perl module for testing some code which gives a subtly different answer each time (i.e. incorporates data from time() and /dev/urandom) and has side effects (i.e. writes to the file system).

By overriding Perl's built-in 'open' function, it is possible to prefix each filename with the location of a temporary directory, effectively emulating chroot(). I also replaced Perl's time() with one that always returned the same answer. This meant that the login code I was testing would return a reliable result.

You have to be careful with prototypes. Spot the difference:

my $result = gmtime(time+$seconds);
my $result = gmtime(time()+$seconds);

Without adding a prototype to the new time() function, these will give different answers. I now have to go back to work tomorrow and close a bug I mistakenly filed. :)

I'm hoping to finish off my evil hacky overriding module and release it to CPAN. I want to add some routines to set up and tear down temporary chroot directories. Obviously there are some limitations to my approach; I'm not currently handling relative paths very well, and system() calls will not be "chrooted". But it should be quite handy and reusable in any case.

September 09, 2009 06:16 PM

August 31, 2009

Bucko

Automating repository uploads

A few of you may know I run an image repository which just grabs all links on IRC and puts them onto an archive. For those who don't, it's probably not safe for work, and lives at abhor.co.uk or disillusionment.co.uk for #compsoc only.

Anyway, I wanted to make it so I could upload images directly without too much hassle, and without storing an insecure SSH key (so I could in principle give the key to someone else).

Turns out it's pretty fun. I now even got it to support scp with multiple files. Here's an example:

[12:41:02] bucko(tank) ~$ scp test.jpg repo_up:
test.jpg                                    100%  110KB 110.3KB
Uploaded test.jpg (112971 bytes)

Start with:

bucko@tank$ ssh-keygen -f ~bucko/.ssh/abhor_upload

Make a 755 script abhor:~repo/bin/ssh_incoming_upload:

#!/bin/sh
if [ "$SSH_ORIGINAL_COMMAND" != "scp -t ." -a "$SSH_ORIGINAL_COMMAND" != "scp -d -t ." -a "$SSH_ORIGINAL_COMMAND" != "scp -r -t ." -a "$SSH_ORIGINAL_COMMAND" != "scp -r -d -t ." ]; then
    printf "Can't login, noob. Command was: $SSH_ORIGINAL_COMMAND\r\n" >&2
    exit
fi
cd /home/repo/incoming
printf '\000'
while read MODE SIZE NAME; do
    case "$MODE" in
        C*)
            printf '\000'
            MODE=C0644
            SIZE16=$(echo $SIZE 16384 / p|dc)
            SIZEREMAIN=$(echo $SIZE $SIZE16 16384 \* - p|dc)
            FILENAME=$(basename $(tempfile .))

            if dd of=$FILENAME count=$SIZE16 ibs=16K 2> /dev/null && dd of=$FILENAME oflag=append conv=notrunc count=$SIZEREMAIN ibs=1 2> /dev/null; then
                echo "Uploaded $NAME ($SIZE bytes)" >&2
                dd ibs=1 of=/dev/null count=1 2> /dev/null # null char
                # Magic to queue the image into the archiver
                printf '\000'
            else
                rm "$FILENAME"
                printf '\002Failure!\n'
            fi
        ;;
        *) # Ignore D, E and T messages, as we don't care!
            printf "\000"
        ;;
    esac
done

In abhor:~repo/.ssh/authorized_keys2:

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/home/repo/bin/ssh_incoming_upload" ssh-rsa KEYFINGERPRINTHERE bucko@tank

In tank:~bucko/.ssh/config:

Host repo_up
HostName abhor.co.uk
User repo
IdentityFile ~/.ssh/abhor_upload

And that's it!

August 31, 2009 11:54 AM

August 27, 2009

Tim

Southampton

I have changed jobs, and now live in Southampton, working for SmoothWall.

This week I have got married on Facebook, and went out to an "alternative" club until 1am on a weeknight. (Normally it plays heavy metal, but Wednesday night is cheese night.)

August 27, 2009 05:58 PM

August 25, 2009

lamby

No-one expects the string literal exception

Every Python programmer knows that:

try:
    # .. code ..
except:
    print "Caught exception"

is semantically equivalent to:

try:
    # .. code ..
except Exception, e:
    print "Caught exception"

Or is it? Bam:

try:
    raise "No-one expects the string literal exception"
except Exception, e:
    print "Oops, didn't catch this exception"

You'll actually need Python ≤ 2.5 to reproduce that behaviour, but what's actually neat about all this is that the exception will be caught—without warnings—as you originally expected under Python 2.6, dispite string literals being "truly and utterly dead". See if you can work out why.

August 25, 2009 01:05 AM

August 19, 2009

lamby

Ambigram animations using shell and Imagemagick

$ wget http://chris-lamb.co.uk/site_media/lambigram.png
$ convert lambigram.png \
   -delay 5 -gravity center -compose Src-over -background white -dispose previous \
   $(for X in $(seq 0 10 180); do echo \( -clone 0 -rotate $X \); done) \
   -extent 90x90 out.gif
$ sensible-browser out.gif
http://chris-lamb.co.uk/wp-content/2009/lambigram_anim.gif

August 19, 2009 08:36 PM

Antz

New pidgin has voice and video support

Pidgin 2.6.1 has been released in the early hours of this morning with a surprising new feature: voice and video support. It is only supported in XMPP, and only on operating systems that are not Windows, but it is a good start.

Those who want to try it in Fedora 11 can do so by running the following:

As root:

yum-builddep pidgin
yum install libidn-devel farsight2-devel libnice-devel

As user:

wget http://sourceforge.net/projects/pidgin/files/Pidgin/pidgin-2.6.1.tar.bz2
tar -xvjf pidgin-2.6.1.tar.bz2
cd pidgin-2.6.1
./configure --prefix=$HOME/opt/pidgin
make
make install

Run

$HOME/opt/pidin/bin/pidgin

There is currently no UI in pidgin to configure the microphone or the webcam, so just run

gstreamer-properties
to do so.

August 19, 2009 09:15 AM

August 16, 2009

Benji

Switzerland

I got back this week from holiday cycling and walking in Switzerland. I cycled down from Basel in the north via Bern and Interlaken to Fiescheralp. Then back via Luzern and Zürich.

Fiescheralp is next to the beautiful Aletsch Glacier

Aletsch Glacier

I stayed in a chalet here for a few days with family and did some climbing and walking


Eggishorn to Bettmerhorn

Eggishorn to Bettmerhorn

The highlight of the trip was cycling through the mountains. Especially Grimselpass.

 


GrimselPass from Gletch

GrimselPass From Gletch


GrimselPass Towards Meiringen

GrimselPass Towards Meiringen


Brienz

Brienz


Brunigpass

Brunigpass

by benji at August 16, 2009 04:30 PM

August 11, 2009

lamby

No citations please, we&#39;re British

import re
import urllib

from lxml import etree
from BeautifulSoup import BeautifulSoup

tree = etree.parse(urllib.urlopen('http://etbe.coker.com.au/feed/'))

for item in tree.xpath('//*[name()="content:encoded"]'):
    soup = BeautifulSoup(item.text)

    for link in soup.findAll('a'):
        for x in link.contents:
            # Kids and their damn hypertext
            x.replaceWith(re.sub(r' \[\d+\]$', '', x))

    try:
        soup.findAll('ul')[-1].extract()
    except IndexError:
        pass

    item.text = unicode(soup)

print etree.tostring(tree)

August 11, 2009 11:55 PM

August 03, 2009

Blood God

360 Woes

Towards the end of June my Xbox 360 decided it didn’t like me, or the weather, or just life in general and died with the ever maligned red ring of death (RROD). This was most annoying, as I use it for both gaming and as my DVD player. I suseqently filled in the form on the Microsoft site and got my return stuff. For me this was a pre-paid UPS label to print myself, and then had to sort out a box and whatnot to ship it back in.

A month later I actually got round to having the console collected, and off it went on a jaunt to Germany. Procrastination is seemingly still rife for me when it comes to some things, despite having a job that should supposedly make me get stuff done.

Still, just over a week later and it was returned to me – complete with a month’s free XBL gold and a sheet explaining that not only had they replaced the motherboard (which was expected given the RROD) but also the DVD drive, which was unexpected. This has seemingly caused my Xbox to get quieter, as the DVD drive was never particularly subtle and the fans make one hell of a noise, and I’m certainly not going to complain about that.

All in all, excepting the fact that some shoddy design has been involved and has lead to it failing, I’m quite happy with how the process has worked. I’d have probably had it back working within 10 days had I been more efficient with getting it sent off, and Microsoft have been quite good with the communications as to the repair status. Of course, I know people who have had multiple failures, and I’m going to be wary of that happening again – but touch wood I’ll avoid that problem.

by Chris Hawley at August 03, 2009 08:00 PM

July 29, 2009

Tim

Conclusions

Well, I'm nearly at the end of DebConf - I'm missing the last day of talks, so travelling home tomorrow. It's been a really good experience - I mean, DebConf is always fun, but this time I feel like I'm getting more involved myself, and a bit closer to the heart of the community than I did before.

In practical terms, the two-week holiday has helped me feel more laid-back - I even got complimented on how I always look relaxed, which was amusing. I'm starting to get to know even more people, and that's helped me be more confident with contributing back. And it's given me some time to finish a few outstanding tasks, although my todo list remains perpetually long.

The time has also helped in less tangible ways - I've been able to brainstorm a few general ideas about what big problems I am trying to solve, and I've been able to look at my workflow and identify problem areas. I hope that I can take some solutions back to the rest of my life.

July 29, 2009 11:38 AM

July 28, 2009

Silver

Empty Windows' System File Cache

Reading Startup, part 1, where Vladimir Vukićević is measuring warm and cold start times for Firefox, he mentions that he is doing the work on Mac OS X rather than Windows because there is a simple tool to empty the system's file cache to test cold starts without actually restarting the OS. I like challenges.

There isn't, it seems, a nicely documented API for doing this on Windows - it certainly isn't something the majority of applications would want to do, though, either. However, there is an undocumented API for it: NtSetSystemInformation(SYSTEM_INFORMATION_CLASS, PVOID, ULONG) (it has a slightly documented sibling for getting information, NtQuerySystemInformation(SYSTEM_INFORMATION_CLASS, PVOID, ULONG, PULONG)). These are actually fairly generic wrappers around a whole load of kernel things; the undocumented information class of interest here is "SystemCacheInformation" (value 0x15), which lets you get and set information about the system file cache.

Each information class for NtSetSystemInformation/NtQuerySystemInformation has a single structure, which goes into the 2nd argument (and its size into the 3rd). For 32bit (applications; not OS) the cache information structure can be defined like this:

typedef struct _SYSTEM_CACHE_INFORMATION {
unsigned long CurrentSize;
unsigned long PeakSize;
unsigned long PageFaultCount;
unsigned long MinimumWorkingSet;
unsigned long MaximumWorkingSet;
unsigned long Unused1;
unsigned long Unused2;
unsigned long Unused3;
unsigned long Unused4;
} SYSTEM_CACHE_INFORMATION, *PSYSTEM_CACHE_INFORMATION;

Along very similar lines to SetProcessWorkingSetSize, specifying the minimum and maximum working set values as 0xFFFFFFFF signals to flush as much as possible out of the system file cache, which gets it down to less than 100KB by the time I can check it afterwards.

Simple, really.

The rest of the code I wrote is to enable the SE_INCREASE_QUOTA privilege Administrator users have, which is necessary to call the API. I believe the executable will work on anything from Windows 2000 and up, with no external dependencies; I've tested it on Windows XP 32bit and Windows 7 64bit. You will obviously need to run it elevated (run as Administrator).

Executable and Code: purge.exe and purge.cpp.

by Silver at July 28, 2009 11:28 PM

July 27, 2009

lamby

July 26, 2009

Bucko

Receiving donations

I have a (not charity-eligible because the rules are pretty strict) website which runs go tournaments. I'd like a public facing website which details all incoming and outgoing transactions in order to ensure that I'm demonstrably not stealing money.

Unfortunately, the only two donation services I know of are PayPal and JustGiving - the former charges fees and has no public interface like what I want, and the latter only donates to charities. Does anybody know of a service that will allow me to do what I want?

July 26, 2009 11:20 AM

July 23, 2009

Fred

Accessing joysticks with Qt

I got another email asking for the code for my interface between SDL's joystick stuff and Qt that I wrote for my final year university project; I've been meaning to tidy this up a little and publish it for some time, but never quite got around to it - well, I finally have:

http://files.fredemmott.co.uk/SDL-joystick-qt.tar.bz2 - just a .h and a .cpp, ISC-licensed.

by Fred Emmott at July 23, 2009 06:44 PM

July 22, 2009

Tim

DebCamp

This week, I've more or less finished my NM questions; I looked again at a problem in dak which turns out mainly to be caused by some edge cases in debconf (the package), so filed a bug there. Then looked at conglomerate, and came up with patches for a couple of bugs there as well. I'm quite looking forward to just being able to NMU these kind of things.

With all this hacking, I've stayed up until around 4:30 for the past few nights - I really didn't mean to, it just happened. I'm trying to fix that now (so that I can make it to breakfast).

Debian now has its own wine - there are 1800 bottles in total. Last night we tried some...

Also, there's a small cat here which seems quite friendly.

July 22, 2009 02:56 PM

July 19, 2009

Tim

DebConf9 continues

Yesterday I almost updated mdbtools. Then I went to a debian perl team meeting, and promptly spent 24 hours trying to improve the speed of working with 1300+ git repositories. It's a bit tricky.

This afternoon, I looked at some pkg-perl RC bugs... but they're tricky as well. I then updated the postgresql-autodoc man page, and sent it back upstream. Hopefully, next release the Debian package will not need any patches.

The vegetarians seem to have been having a rough time - apparently in Spanish-speaking countries, ham is not considered meat. Things picked up today for them, though, with some mushrooms at lunch and a nice-looking salad this evening.

July 19, 2009 08:40 PM

July 18, 2009

Bucko

Re-reading Tesuji

I've been rereading Tesuji by James Davies, and finding some things in problems that I hadn't seen before. Whether it's an omission or a subtlety left to entertain stronger readers, I don't know, but I find them fun, so I don't mind. Here's an example.



As stated, this is the problem. Black to capture the cutting stone. It's in the cross-cut chapter, so a simple amount of reading reveals the book answer:



White is left short of liberties, so black gets to eat 2 and the stone to its right. B has connected.

The book also lists a standard response which sometimes troubles the cross-cut:



Of course, it doesn't trouble this one and is dispatched with ease...



So that's not where the fun lies. But what if white does this?



Trying the same refutation a second time isn't so good. Black is cut:



Black can scrape by with life:



But this isn't satisfactory at all! Black was meant to be connecting out into the centre! There must be another resource. I came up with my own idea, but first, here's the idea of a 5 dan I accosted on KGS:



I can't find any way for white to get anything back now (though there's a number of close calls), so here's an example continuation which looks complex:



White has 2 liberties; black has 3: so white will die. There's no worry about the black cut around the right of 5, incidentally, because black always has enough liberties.

So, here's my idea:



It looks a bit silly - the stone at 5 looks like it'll get captured. In fact, it does, as the continuations show:



I just realised that there's a misplaced black move in the second diagram; 13 should be one to the right and one down from where it is; that move was played earlier on! The sequence is correct, though. White dies because he only has 2 liberties.

White has another resource, though:



The above-mentioned 5 dan suggested his move here, too, claiming no problems. But it seems he was wrong:



Both of the final results are seki - both black and white are alive. Black has a few opportunities to turn the whole corner into a huge ko, but that seems a bit daft. It's particularly disappointing to find that black also has horrible cut problems - if black does not have the ladder, this is utterly unplayable.

Luckily, my original move comes to the rescue:



This time, black finds himself ahead. White dies.


Pretty impressive for a small area of the board, no? Now you know why I love this game.

July 18, 2009 03:15 PM

July 17, 2009

Tim

DebConf9

DebConf is in Spain this year, in Cáceres. Getting here involved a plane to Madrid, navigating the metro system, a 3.5 hour train journey and then a walk to the accommodation at 2am. This year I had actually looked at a map before arriving in the city, although hadn't bothered to bring one with me. Or note the exact address I was heading for...

At night, it is still pleasantly warm here. The streets were quiet except for the "pfft, pfft, pfft" of the sprinklers. Any directions I'd read which involved street names were useless - but I remembered enough of the map to get to where the buildings were, and then wandered around until I found the entrance.

I was greeted at the front desk by the security guard, who didn't speak much English... but I understood enough, and got shown where the room was. The student residences are not luxurious! It's quite a contrast to Mar del Plata (or even the hostels at Edinburgh), but not a lot of time is spent in the rooms anyway. The shower is worryingly powerful - think 'firehose'. Tomorrow I hope to find a way not to flood the whole room while using it.

The building seems designed to stay cool, fortunately. Meals are later in the day to avoid the worst of the heat, but I've still been feeling tired. People seem to buy large (1.5l) bottles of water. The weather's supposedly going to get even hotter - I did go out at 6pm to buy an ice cream, but I'm wondering whether my sunblock is going to last two weeks. I'm already in shorts and sandals - I might need to find more T-shirts.

Today has been fairly productive, actually - I updated postgresql-autodoc, fixed my blog, and updated my laptop. Since dinner I've been looking at the other packaging work I need to do; I'm hoping to get another few uploads out of the way first, and then work on NM questions and fixing bugs.

July 17, 2009 11:26 PM

Blog back

I have restored all my previous blog entries. I couldn't be bothered to think of a way to restore the post times, but I got the dates out of the path and used hardcodedates.py to overrride the timestamps. If this floods planets, I'm sorry.

July 17, 2009 01:36 PM

July 14, 2009

Antz

Google Latitude is finally available on Android G1 from T-Mobile UK

Go to Marketplace, search for Google Maps

This has been an inconvenience for Android G1 users using T-Mobile phones in UK for a while now - Google Maps application did not have Latitude enabled even after the update that was supposed to unlock it. Other major mobile platforms (Symbian, iPhone, Windows, Blackberry) all had working clients in UK, but of course, not on Google's own flagship phone. T-Mobile did not provide an explanation for this omission, redirecting all queries to Google, and Google were silent for over a month, eventually releasing a short statement placing blame on "legal issues".

As I discovered yesterday it is now possible to get Latitude working on the T-Mobile UK G1 without needing to root the phone or overwrite the OS shipped by T-Mobile. In the Marketplace there is an installable entry for Google Maps. Although the new application replaces the existing Google Maps application it is not marked as an update, and so will not show up on the list of available updates. It needs to be found and installed manually by the user.

Once the new Google Maps app is installed the Latitude dialogue comes up as expected and Latitude starts working. The update has been available for over a month now, but either no one has noticed, or I have been living in a cave.

July 14, 2009 12:38 PM

July 13, 2009

Fred

A couple of small helper scripts for git-svn

While using git-svn, I frequently need to either look at the diff of a specific commit in trunk, or, to cherry-pick a commit from trunk to the current branch; here's two helpful scripts:

  • git-svn-revision - show the commit message and full diff for an svn revision number, assuming that 'master' follows trunk; usage: git-svn-revision 123
  • git-svn-merge - cherry pick a commit from master to the current branch, by revision number; usage: git-svn-merge 123

If you chuck these in the same directory as the other git commands (eg /usr/libexec/git-core on Slackware/Slamd64), you can run these like the others; eg "git svn-revision123" and "git svn-merge 123".

by Fred Emmott at July 13, 2009 12:54 PM

July 08, 2009

Laurence

Migrating from fastcgi to cgi

I’ve just migrated all of the sites on my VMS (including this blog) from running on a fastcgi backend to running on a cgi backend. Why, you may ask. Well, although fastcgi is substantially more responsive than plain old cgi (since it keeps the processes running between requests so the process start-up and take-down times are removed) it consumes much more memory (due to keeping the processes around). Nowadays this is not normally a problem but on a virtual machine with only 128MB available and no swap memory usage becomes a big issue.

By moving from fastcgi to “normal” cgi and tweaking my mysql config I have increased the free memory when the machine is idle from 4MB to 80MB and now have lots of headroom if a single process (e.g. DenyHosts, which is idling at 20MB) decides it wants loads of ram before the Linux low memory reaper starts killing processes.

As a slight aside, the bigest memory hogs on my box are Python programs (DenyHosts [~20M] and rss2email [~45M]). I don’t know it this is a fault with Python or the way the scripts are written (or both, not being intimately familiar with Python I don’t know if it encourages conservative use of memory or not).

by Laurence at July 08, 2009 11:34 AM

July 06, 2009

Bucko

When do people start uploading images from their cameras?

I noticed (well, I've known for years!) that my camera does not seem to reset its image number counter no matter what I do - change memory card, format card, send it in for repairs, and it seems like this is the norm.

So I got to thinking. When I first got it, I took a large number of "scrap" images, just playing with its settings. I wonder if that's normal? How many do people take?

Luckily, with the power of the web, we can attempt to find this out!

On a number of search engines, I searched for various (exp-distributed) searches of the form 'dscfNNNN', then tabulated the results. You can see the resulting graph below (Y axis is percent of maximum from each source).



As you can see, the prevailing opinion seems to be that the first 5 images are binned, at the most, with the majority of people scrapping relatively few initial photos. The speed with which the graph tails off also suggests that either not many people take many photos (ie. most cameras are just chucked in cupboard), that people sort their images more effectively, or that they just don't bother to upload as many once they get through their first memory stick. It may also suggest that some cameras do reset their counter, but then what causes the bulge at around 2-3 photos?

Both results from Google are odd in some way - the web one initially seems to fluctuate a lot, and the images one does it near the end. I presume for images the jump indicates that there's just a cap on the number which results in a second algorithm being used (crossover is at about 35).

July 06, 2009 09:22 AM

June 30, 2009

Tim

The Techteam Suck

(But so does PyBlosxom for using file timestamps.)

June 30, 2009 08:17 AM

June 29, 2009

Bucko

Cider

I also cleared out all my old cider bottles today (just under 3 years worth; about 56 ciders and one limoncello which happened to be around). Photos here.

June 29, 2009 05:31 PM

Taxis

On the way to Morrison's today I spied a number of taxis, so I left home with my camera once I got back to photograph them. They'd unfortunately started to leave once I got about halfway down the road, though. For reference, the road is about a half mile long and was lined from one end to the other (plus some side roads, plus a bit of the other side) with taxis.

Here's some photos.

This is the lowest angle of incidence one I took:



Apparently they were protesting the number of taxis in Coventry - seems it's hard for them to make a living with so many drivers. Of course, some of them stand to lose their jobs if they get their way, but I guess they're the longer-employed drivers who'll be out last.

June 29, 2009 05:21 PM

June 22, 2009

Tim

Thinking

I spend (and have spent) a lot of time thinking. I also spent a lot of time reading. Every so often I reach conclusions... sometimes I remember them, sometimes not. A lot of them just got incorporated into my general world-view, I suppose. A long time ago it was thoughts about philosophy... these days I suppose it's mainly psychology.

I've been thinking a lot about cognitive dissonance - I think there have been differences between where I am and how I think of myself. People tend to act to reduce dissonance; it is a motivator, which I find very interesting - for a given task, it does not seem to fit neatly into either intrinsic or extrinsic motivation - it comes from how we feel about ourselves, not the task itself.

June 22, 2009 11:00 PM

June 21, 2009

Tim

Paper

I used to have a pile of paper, with notes/letters/junk going back several years. Piles like this have been sitting around in various corners of my room for as long as I remember.

Now it's filed away - all in square-cut beige folders with labels on, in alphabetical order in a metal box. (Most of it got recycled.)

I'm not sure what this means.

Is "organization" an end in itself, or a means to an end? And what end?

June 21, 2009 11:00 PM

June 20, 2009

Antz

June 19, 2009

Fred

StatusBoard

It's been a long time since I posted anything; I've been working on a few slackware, slamd64, and jerboa-related things, but I'll hopefully get into those in detail in another blog post.

As a quick post, someone at work mentioned putting an at-a-glance "is something broken" display on a dedicated monitor or few at work; that led to me writing StatusBoard yesterday/today:

StatusBoard screenshot

If a test doesn't pass, the relevant item turns red. It's configured via JSON (example included), and supports full-screen mode, and has an optional system tray icon. It basically fetches a web address. It supports:

  • A different refetch interval for each item
  • A pattern for a revision number
  • A pattern that must be matched to count as a 'pass'
  • A pattern that must be matched to count as a 'failure'
  • Another pattern to limit the scope of the 'pass'/'fail' patterns
  • HTTP + HTTPS (well, anything QNetworkAccessManager supports)
  • Ignoring SSL errors, such as self-signed certificates
  • Linux + OSX + (probably, but untested) Windows

The source is available via git, under the GPLv3.

git clone http://git.fredemmott.co.uk/repo/statusboard/

See the README file for build/configuration instructions.

by Fred Emmott at June 19, 2009 06:55 PM

June 18, 2009

Benji

Java Abuse – Currency Pattern

Here’s a sillier one from last night…

People often complain about not being able to return multiple values from a method in Java. I can’t see a good reason for wanting to do this, but some do.

The example I was given was wanting to do:

int foo, bar, baz; list(foo, bar, bar) = func(ponies);

Where func(ponies) returns 3 values which are then assigned to the local variables foo, bar, and baz.

There are plenty of sensible ways of achieving this, but what if we’re trying to get as close as possible to the above method? We can only return one thing from a method, but that could be a chain. We also can’t modify the local foo,bar,baz from inside the list method, but we could if we wrap them in references.

If we abuse the fact that currency characters are valid identifiers in Java, for the ultimate in unreadability; We can use $(x,y) to be a pair of x and y, $$ to terminate a list, and £(x) to either reference or dereference x.

Then we can do something like this ¬_¬

/** OUTPUT
Foo = 1
Bar = 2
Bingo = Badgers
Baz = 3
**/
 
package scratchpad;
 
import static scratchpad.$.$;
import static scratchpad.$$.$$;
import static scratchpad.$.expand;
import static scratchpad.£.£;
 
public class MultipleReturn
{
	public static void main(String... args)
	{
		new MultipleReturn().demo();
	}
 
	public void demo()
	{
		£<Integer> foo = £(-1), bar = £(-1), baz = £(-1);
		£<String> bingo = £("");
 
		expand(foo()).into(foo,bar,bingo,baz);
 
		System.out.println("Foo = " + £(foo));
		System.out.println("Bar = " + £(bar));
		System.out.println("Bingo = " + £(bingo));
		System.out.println("Baz = " + £(baz));
	}
 
	public $<Integer,$<Integer,$<String,$<Integer,$$>>>> foo()
	{
		return $(1,$(2,$("Badgers",$(3,$$))));
	}
}
 
interface Tuple {}
 
class $$ implements Tuple
{
	private $$() { }
	public static $$ $$ = new $$();
}
 
class $<T,U extends Tuple> implements Tuple
{
	public T _1;
	public U _2;
 
	protected $(T t,U u)
	{
		this._1 = t;
		this._2 = u;
	}
 
	public static <T> $<T,$$> $(T t)
	{
		return new $<T,$$>(t,$$);
	}
 
	public static <T,U extends Tuple> $<T,U> $(T t, U u)
	{
		return new $<T,U>(t,u);
	}
 
	public interface Expander
	{
		public void into(£... refs);
	}
 
	public static Expander expand(final $<?,? extends Tuple> vals)
	{
		return new Expander()
		{
			public void into(scratchpad.£... refs)
			{
				if (refs.length < 1)
					return;
 
				$ current = vals;
				refs[0]._1 = current._1;
				if (current._2 instanceof $$)
					return;
				int i = 1;
				while (!(current._2 instanceof $$))
				{
					if (i >= refs.length)
						return;
					current = ($)current._2;
					refs[i]._1 = current._1;
					i++;
				}
			}
 
		};
 
	}
}
 
class £<T> extends $<T,$$>
{
	protected £(T t)
	{
		super(t,$$);
	}
 
	public static <T> £<T> £(T t)
	{
		return new £<T>(t);
	}
 
	public static <T> T £(£<T> t)
	{
		return t._1;
	}
}

by benji at June 18, 2009 05:26 PM

June 14, 2009

lamby

DebConf9

http://chris-lamb.co.uk/wp-content/2009/debconf9.png

I will be attending Debconf9 in Extremadura, Spain.

June 14, 2009 11:13 PM