Linux packages – an update

It’s been pointed out to me that many people aren’t aware of the current status of Linux packages on, so I’m here’s a summary:

Stable packages

Mono 3.10.0, MonoDevelop, NuGet 2.8.1 and F# packages are available. Plus related bits. MonoDevelop on Linux does not currently include the F# addin (there are a lot of pieces to get in place for this to work).

These are built for x86-64 CentOS 7, and should be compatible with RHEL 7, openSUSE 12.3, and derivatives. I haven’t set up a SUSE 1-click install file yet, but I’ll do it next week if someone reminds me.

They are also built for Debian 7 – on i386, x86-64, and IBM zSeries processors. The same packages ought to work on Ubuntu 12.04 and above, and any derivatives of Debian or Ubuntu. Due to ABI changes, you need to add a second compatibility extension repository for Ubuntu 12.04 or 12.10 to get anything to work, and a different compatibility extension repository for Debian derivatives with Apache 2.4 if you want the mod-mono ASP.NET Apache module (Debian 8+, Ubuntu 13.10+, and derivatives, will need this).

MonoDevelop 5.5 on Ubuntu 14.04

MonoDevelop 5.5 on Ubuntu 14.04

In general, see the install guide to get these going.


You may have seen Microsoft recently posting a guide to using ASP.NET 5 on Docker. Close inspection would show that this Docker image is based on our shiny new Xamarin Mono docker image, which is based on Debian 7.The full details are on Docker Hub, but the short version is “docker pull mono:latest” gets you an image with the very latest Mono.

directhex@desire:~$ docker pull mono:latest
Pulling repository mono
9da8fc8d2ff5: Download complete 
511136ea3c5a: Download complete 
f10807909bc5: Download complete 
f6fab3b798be: Download complete 
3c43ebb7883b: Download complete 
7a1f8e485667: Download complete 
a342319da8ea: Download complete 
3774d7ea06a6: Download complete 
directhex@desire:~$ docker run -i -t mono:latest mono --version 
Mono JIT compiler version 3.10.0 (tarball Wed Nov  5 12:50:04 UTC 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors.
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

The Dockerfiles are on GitHub.

The unstoppable march of mobile technology

It’s been more than 2 years since my last post about my smartphone. In the time after that post I upgraded my much loved Windows Phone 7 device to Windows Phone 8 (which I got rid of within months, for sucking), briefly used Firefox OS, then eventually used a Nexus 4 for at least a year.

After years of terrible service provision and pricing, I decided I would not stay with my network Orange a moment longer – and in getting a new contract, I would get a new phone too. So on Friday, I signed up to a new £15 per month contract with Three, including 200 minutes, unlimited data, and 25GB of data roaming in the USA and other countries (a saving of £200,000 per month versus Orange). Giffgaff is similarly competitive for data, but not roaming. No other network in the UK is competitive.

For the phone, I had a shortlist of three: Apple iPhone 6, Sony Xperia Z3 Compact, and Samsung Galaxy Alpha. These are all “small” phones by 2014 standards, with a screen about the same size as the Nexus 4. I didn’t consider any Windows Phone devices because they still haven’t shipped a functional music player app on Windows Phone 8. Other more “fringe” OSes weren’t considered, as I insist on trying out a real device in person before purchase, and no other comparable devices are testable on the high street.

iPhone 6

This was the weakest offering, for me. £120 more than the Samsung, and almost £200 more than the Sony, a much lower hardware specification, physically larger, less attractive, and worst of all – mandatory use of iTunes for Windows for music syncing.

Apple iPhone 6, press shot from, all rights reserved

The only real selling point for me would be for access to iPhone apps. And, I guess, decreased chance of mockery by co-workers.

Galaxy Alpha

Now on to the real choices. I’ve long felt that Samsung’s phones are ugly plasticy tat – the Galaxy S5 is popular, well-marketed, but looks and feels cheap compared to HTC’s unibody aluminium One. They’ve also committed the cardinal sin of gimping the specifications of their “mini” (normal-sized) phones, compared to the “normal” (gargantuan) versions. The newly released S5 Mini is about the same spec as early 2012’s S3, the S4 Mini was mostly an S2 internally, and so on.

However, whilst HTC have continued along these lines, Samsung have finally released a proper phone under 5″, in the Alpha.

Samsung Galaxy Alpha press shot from, all rights reserved

Samsung Galaxy Alpha press shot from, all rights reserved

The Alpha combines a 4.7″ AMOLED screen, a plastic back, metal edges, 8-core big.LITTLE processor, and 2GB RAM. It is a PRETTY device – the screen really dazzles (as is the nature of OLED). It feels like a mix of design cues from an iPhone and Samsung’s own, keeping the angular feel of iPhone 4->5S rather than the curved edges on the iPhone 6.

The Galaxy Alpha was one of the two devices I seriously considered.

Xperia Z3 Compact

The other Android device I considered was the Compact version of Sony’s new Xperia Z3. Unlike other Android vendors, Sony decided that “mini” shouldn’t mean “low end” when they released the Z1 compact earlier this year. The Z3 follows suit, where the same CPU and storage are found on both the big and little versions.

Sony Xperia Z3 Compact press shot from Sony Xperia Picasa album. CC BY-NC-SA 3.0

Sony Xperia Z3 Compact press shot from Sony Xperia Picasa album. CC BY-NC-SA 3.0

The Z3C has a similar construction to the Nexus 4, with glass front and back, and plastic rim. The specification is similar to the Galaxy Alpha (with a quadcore 2.5GHz Qualcomm processor about 15% faster than the big.LITTLE Exynos in the Galaxy Alpha). It differs in a few places – LCD rather than AMOLED (bad); a non-removable (bad) 2600 mAh battery (good) compared to the removable 1860 mAh in the Samsung; waterproofing (good); A less hateful Android shell (Xperia on Android vs Samsung Touchwiz).

For those considering a Nexus-4-replacement class device (yes, rjek, that means you), both the Samsung and the Sony are worth a look. They both have good points and bad points. In the end, both need to be tested to form a proper opinion. But for me, the chunky battery and tasteful green were enough to swing it for the Sony. So let’s see where I stand in a few months’ time. Every phone I’ve owned, I’ve ended up hating it for one reason or another. My usual measure for whether a phone is good or not is how long it takes me to hit the “I can’t use this” limit. The Nokia N900 took me about 30 minutes, the Lumia 800 lasted months. How will the Z3 Compact do? Time will tell.


Once upon a time, a large number of angry video game players were up in arms about perceived corruption in games journalism. They took their outrage to social networks, joining in with the existing outrage going on under the hashtag #gamergate. And yet every time they tried to discuss the injustices demonstrated to them in infographics and YouTube videos, they were dismissed as harassers or misogynists.

This confused and angered the gamers further. Why were they on trial for something they had not done? They had serious concerns about cozy relationships between developers and the press, yet that means they hate women? WTF?

The thing is, they got played. If you’re a gamer, and recognise any part of yourself in the first two paragraphs, you got played.

Good doggie!

There’s a term in politics called “dog-whistle“. Dog-whistle politics are where you want to achieve A, but can’t tell the electorate “let’s achieve A” as they would not approve of it. So you talk about B instead. B has the side-effect of achieving A. You enlist all sorts of people who truly believe in the promise of B, and who reject the concept of A – but their support of B just happens to achieve A anyway.

The famous quote about dog-whistle politics comes from a former senior Regan administration staffer:

You start out in 1954 by saying, “N*****, n*****, n*****.” By 1968, you can’t say “n*****” — that hurts you. Backfires. So you say stuff like forced busing, states’ rights and all that stuff. You’re getting so abstract now [that] you’re talking about cutting taxes, and all these things you’re talking about are totally economic things and a byproduct of them is [that] blacks get hurt worse than whites. And subconsciously maybe that is part of it. I’m not saying that. But I’m saying that if it is getting that abstract, and that coded, that we are doing away with the racial problem one way or the other. You follow me — because obviously sitting around saying, “We want to cut this,” is much more abstract than even the busing thing, and a hell of a lot more abstract than “N*****, n*****.”

Those who go along with B and inadvertently help achieve A are known as – and this is the term, not a personal attack, so don’t freak out – useful idiots.

In this instance, B is the discussion of games journalism. There are a whole lot of people who truly believe, in their hearts, that there are causes for concern in the games journalism industry, and they’re right – but those people are being orchestrated, without their consent, to help achieve A. And in this case, A is stamping out “social justice warriors” from all areas of the games industry – ending their careers, or invoking suicide, or both. The main targets singled out for attention by #gamergate are not selected at random, nor are they especially corrupt – they’ve been selected because attacking them serves aim A, and attacking others would not.

This was never your pool in the first place

You stand in a swimming pool. Many others are here, splashing about. You try to have a serious discussion with someone on the edge of the pool, as a turd floats by.

“Dude, what the fuck, there’s shit in that pool”

You don’t know what they’re talking about. You didn’t shit in the pool, none of your friends shat in the pool, why are they changing the subject, and attacking you?

Wispy, misshapen scraps of excrement float past, knocking your arm.


“Seriously, that pool is literally filled with poo. It’s those guys in the corner, with their trunks down. They’re just squatting and shitting, non-stop. You’re getting covered in shit.”

“I really have no idea what you’re talking about, why are you attacking me?” – you recruit some friends to join you in the pool, to prove that it’s not full of shit.

Small, rabbit-pellet crap squelches underfoot.

“Okay, really, I’m not talking to you now, you stink of shit” – and they walk off.

You turn to your other friends in the pool outraged. “How DARE they imply these things? We have *nothing* to do with those guys in the corner, they have *nothing* to do with this pool!”

They nod in agreement, as a wave of brown water splashes over you.

Not your personal army

So who’s orchestrating things? And why? 4chan, basically. This all started in an effort to destroy the career and/or life of indie developer Zoe Quinn. Here’s some of the original planning work – trigger warning, for those that applies to. From there, it moved on to anyone else labelled as a “social justice warrior” by the 4chan crowd – anyone who brings up issues of discrimination, race, gender, sexuality, etc. The 4channers and their allies (hey, a few people independently agree with cause A) are terrified of a world where there might be a reduction in the number of eviscerated prostitutes in a game – kicking out the people who might cause that shift is an important step in saving the future of “vidya”

And, sweet gamer, you may insist that you have nothing to do with 4chan – but the evidence shows that whether you know about it or not, it’s their tune you’re singing to. That’s why you’ve been directed at so many “corrupt” journalists who happen to be women.You’re not the one sending rape/murder threats to the few women journalists, you’re just accusing them of corrupt practices. It’s those *other* guys you know nothing about, sharing the same banner and discussion space, sending those threats.

As a thought exercise, listen to this podcast. It’s quite long, but it’s a good example. You’ll know when you get to the relevant part – unless you’re a sociopath, incapable of forming human emotions. And you’re not – you’ve just been played. This is the result of you being played. People who have devoted decades of their working lives to the hobby you both love, driven out by unbridled hate.

The truth about games journalism

A number of articles popped up in the opinions pages of various games (and non-games) sites, in the early days of #gamergate, saying things like “gamers are over”. This certainly helped convince plenty of those who believe in cause B that games journalists are not only corrupt – but hold them in contempt too. Fuelling anger. Justifying angry responses.

The problem with an era where attention spans don’t go past 6 seconds or 140 characters is nobody really understood the point any of these articles made. In 2014, what does “gamer” even mean? Every retirement home has a Wii, every smartphone has Angry Birds, everyone on Facebook plays Candy Crush Saga. What does it mean to be a “gamer” when you spend fewer hours a day playing Rust than a businessman spends playing smartphone games on the train? The “gamer” identity meant something when playing games was something unusual, but now? Would you describe yourself as “food eater” or “TV watcher”? Because it’s about as meaningful.

And you know what? All those “corrupt” games journalists there’s been work to evict? Those guys are gamers too. You know why a journalist might back a Kickstarter or Patreon? The same reason you might, as an individual gamer. You know why they’re getting paid $50 an article and barely keeping the lights on? Because they love gaming, because they’re you. Yes, even the “social justice warriors”, even the women. They’re gamers like anybody else.

“Games journalism” comes from a sector known as “enthusiast press” – the same space that has, say, car magazines. Do car journos and car company PR folk know each other? Of course – that’s where most of the magazine content comes from! Does the journo have bias? Of course! They love cars, and drive one too! You don’t want unbiased reviews – you want to read reviews from a reviewer whose biases match your own. The same goes for games journalism. Game reviews don’t need to start every page with “video games are totally fun and worth your time, weird as it may seem”, the way the occasional review in the mainstream press & newspapers do – because you, and the reviewer, are on the same page.They don’t hold you in contempt. You speak a common language about a common interest. That’s their entire value to you – they know the right people to get access to the info you want, and you don’t.

A great example of why this has nothing to do with ethics? Part of #gamergate called “Operation Disrespectful Nod” (everything is an Operation to 4chan). This aims to put pressure on “corrupt” (employs a hated “social justice warrior”) sites by putting pressure on that site’s advertisers. Advertisers like EA. The idea is that EA has enough sway over a site to get disliked journalists fired and that’s fine and a source of allies – but the true corruption is over who is backing whose Patreon campaign for the price of a beer a month.

The idea that there’s a problem about the relationship between the press and their sources isn’t new in the enthusiast press. Which is why most outlets have a strict division between editorial and advertising departments. Lapses happen – but they’re rare, and heavily covered by other sources, because those lapses are news!

In conclusion, I’m always right

Everything I say needs to be taken with a pinch of salt – but so does everything anyone else says. And the “I don’t support harassment” gamers of #gamergate need to ask themselves why they’ve swallowed every word wholesale from meaningless infographics and videos, but insist that any evidence of wrongdoing on the part of their movement is fabricated.

Most gamers, including those caught up in #gamergate, are good, ordinary, decent folk. At worst, gullible. And what a good, ordinary, decent gamer should want is for interesting people to make interesting content – be it games or articles or art or mods or soundtrack remixes or Let’s Plays or cosplays or game-themed bakery or whatever. How does driving interesting people out of the industry achieve that?

Image courtesy of BrandonSigma at

An exception to the sitewide license is used for this post – it is under Creative Commons Attribution-NoDerivs 2.0 UK: England & Wales. The main difference from the site license is that this article may be re-posted, with attribution, as long as the text is unaltered, even on commercial for-profit sites.

Xamarin Apt and Yum repos now open for testing

Howdy y’all

Two of the main things I’ve been working on since I started at Xamarin are making it easier for people to try out the latest bleeding-edge Mono, and making it easier for people on older distributions to upgrade Mono without upgrading their entire OS.

Public Jenkins packages

Every time anyone commits to Mono git master or MonoDevelop git master, our public Jenkins will try and turn those into packages, and add them to repositories. There’s a garbage collection policy – currently the 20 most recent builds are always kept, then the first build of the month for everything older than 20 builds.

Because we’re talking potentially broken packages here, I wrote a simple environment mangling script called mono-snapshot. When you install a Jenkins package, mono-snapshot will also be installed and configured. This allows you to have multiple Mono versions installed at once, for easy bug bisecting.

directhex@marceline:~$ mono --version
Mono JIT compiler version 3.6.0 (tarball Wed Aug 20 13:05:36 UTC 2014)
directhex@marceline:~$ . mono-snapshot mono
[mono-20140828234844]directhex@marceline:~$ mono --version
Mono JIT compiler version 3.8.1 (tarball Fri Aug 29 07:11:20 UTC 2014)

The instructions for setting up the Jenkins packages are on the new Mono web site, specifically here. The packages are built on CentOS 7 x64, Debian 7 x64, and Debian 7 i386 – they should work on most newer distributions or derivatives.

Stable release packages

This has taken a bit longer to get working. The aim is to offer packages in our Apt/Yum repositories for every Mono release, in a timely fashion, more or less around the same time as the Mac installers are released. Info for setting this up is, again, on the new website.

Like the Jenkins packages, they are designed as far as I am able to cleanly integrate with different versions of major popular distributions – though there are a few instances of ABI breakage in there which I have opted to fix using one evil method rather than another evil method.

Please note that these are still at “preview” or “beta” quality, and shouldn’t be considered usable in major production environments until I get a bit more user feedback. The RPM packages especially are super new, and I haven’t tested them exhaustively at this point – I’d welcome feedback.

I hope to remove the “testing!!!” warning labels from these packages soon, but that relies on user feedback to my account preferably (jo.shields@)

Transition tracker

Friday was my last day at Collabora, the awesome Open Source consultancy in Cambridge. I’d been there more than three years, and it was time for a change.

As luck would have it, that change came in the form of a job offer 3 months ago from my long-time friend in Open Source, Miguel de Icaza. Monday morning, I fly out to Xamarin’s main office in Boston, for just over a week of induction and face time with my new co workers, as I take on the title of Release Engineer.

My job is to make sure Mono on Linux is a first-class citizen, rather than the best-effort it’s been since Xamarin was formed from the ashes of the Attachmate/Novell deal. I’m thrilled to work full-time on what I do already as community work – including making Mono great on Debian/Ubuntu – and hope to form new links with the packer communities in other major distributions. And I’m delighted that Xamarin has chosen to put its money where its mouth is and fund continued Open Source development surrounding Mono.

If you’re in the Boston area next week or the week after, ping me via the usual methods!

Stephenson’s Rocket – the new name for Ye Olde SteamOSe

I’ve made a new release of my curiously popular SteamOS derivative, and given it a new name: Stephenson’s Rocket.

You can download the new release from here.

Release highlights:

  • Updated to alchemist_beta 93
  • Support for pre-HD5000 Radeon cards
  • Support for motherboard-based “FakeRAID”
  • Video tutorial series – about an hour of instructional material for all competency levels

Dear Fake Debian Developers, shoo.

Another post about the Valve/Collabora free games thing. This time, the bad bit – people trying to scam free games from us.

Before I start, I want to make one thing clear – there are people who have requested keys who don’t meet the criteria, but are honest and legitimate in their requests. This blogspam is not about them at all. If you’re in that category, you’re not being complained about.

So. Some numbers. At time of writing, I’ve assigned keys to 279 Debian Developers or Debian Maintainers – almost 25% of the total eligible pool of about 1200.

I’ve denied 22 requests. Of these 10 were polite requests from people who didn’t meet the conditions stated (e.g. Ubuntu developers rather than Debian). These folks weren’t at all a problem for us, and I explained politely that they didn’t meet the terms we had agreed at the time with Valve. No problem at all with those folks.

Next, we have the chancers, 7 of them, who pretended to be eligible when they clearly weren’t. For example, two people sent me signed requests pointing to their entry on the Debian New Maintainers page when challenged over the key not being in the keyring. The NM page showed that they had registered as non-uploading Debian Contributors a couple of hours previously. A few just claimed “I am a DD, here is my signature” when they weren’t DDs at all. Those requests were also binned.

Papers, Please screenshot - denied entry application


And then we move onto the final category. These people tried identity theft, but did a terrible job of it. There were 5 people in this category:

From: Xxxxxxxx Xxxxxx <>
Subject: free subscription to Debian Developer
8217 A205 5E57 043B 2883 054E 7F55 BB12 A40F 862E

This is not a signature, it’s a fingerprint. Amusingly, it’s not the fingerprint for the person who sent my mail, but that of Neil McGovern – a co-worker at Collabora. Neil assured me he knew how to sign a mail properly, so I shitcanned that entry.

From: "Xxxxx, Xxxxxxxxx" <>
Subject: Incoming!
Hey dude,

I want to have the redemption code you are offering for the Valve Games


Wat? Learn to GPG!

From: Xxxxxx-Xxxx Le Xxxxxxx Xxxx <>
Subject: pass steam
Hey me voila

Merci beaucoup


2069 1DFC C2C9 8C47 9529 84EE 0001 8C22 381A 7594

Like the first one, a fingerprint. This one is for Sébastien Villemot. Don’t scammers know how to GPG sign?

From: "Xxxxxxxxx Xxxxxxx" <>
Subject: thanks /DD/Steam gifts us finally something back

Yet again, a fingerprint. This one is for Marco Nenciarini. I found this request particularly offensive due to the subject line – the haughty tone from an identity thief struck me as astonishingly impertinent. Still, when will scammers learn to GPG?

From: Sven Hoexter <>
Subject: Valve produced games
I'm would like to get the valve produced games
My keyring: 0xA6DC24D9DA2493D1 Sven Hoexter <hoexter> sig:6

Easily the best scam effort, since this is the only one which both a) registered an email address under the name of a DD, and b) used a fingerprint which actually corresponds to that human. Sadly for the scammer, I’m a suspicious kind of person, so my instinct was to verify the claim via IRC.

31-01-2014 16:52:48 > directhex: Hoaxter, have you started using gmail without updating your GPG key? (note: significantly more likely is someone trying to steal your identity a little to steal valve keys from collabora)
31-01-2014 16:54:51 < Hoaxter! directhex: I do not use any Google services and did not change my key

So… yeah. Nice try, scammer.

I’m not listing, in all of this, the mails which Neil received from people who didn’t actually read his mail to d-d-a.

I’m also not listing a story which I’ve only heard second ha… actually no, this one is too good not to share. Someone went onto, did a search for every DD in France, and emailed every Jabber JID (since they look like email addresses) asking them to forward unwanted keys.

All in all, the number of evildoers is quite low, relative to the number of legitimate claims – 12 baddies to 279 legitimate keys issued. But still, this is why the whole key issuing thing has been taking me so long – and why I have the convoluted signature-based validation system in place.

Enjoy your keys, all 279 of you (or more by the time some of you read this). The offer has no explicit expiry on it – Valve will keep issuing keys as long as there is reason to, and Collabora will continue to administer their allocation as long as they remain our clients. It’s a joint gift to the community – thousands of dollars’ worth of games from Valve, and a significant amount of my time to administer them from Collabora.

Dear Debian Developers, lrn2gpg

For some strange reason, I’ve been receiving a lot of GPG-signed mail from Debian Developers and Maintainers lately. In response to each of these mails, I need to send a GPG-encrypted reply. The rate at which I’m able to send replies has been significantly hampered by the poor state in which many DD/DM’s maintain their GPG keys. Here are a few common mistakes, so you can consider correcting them.

Ensure you have a UID for the email address(es) you use

When I send an encrypted mail, I need to be sure that the recipient is legit. This means any decent mail client should refuse to send an encrypted message to unless that email address is known somehow to GPG. In many cases, someone with a valid key for would send their signed mail from without that being a valid UID. In some cases, isn’t even a valid email address anymore (i.e. the mail server says no such mailbox).

You should have a UID for each address you use.

Signatures are per-UID

You may well have a valid UID for,, and – but the PGP trust model doesn’t automatically trust every UID as much as its peers. Each individual UID needs to be trusted (i.e. signed/uploaded) by others. What if you added as a UID – should that automatically be trusted? Clearly not. Just because you have doesn’t mean it’s trusted for encryption without some signatures.

Make sure you actually have an encrypting subkey

GPG sucks, and as a result, it reports “Skipping unusable pubkey” when the issue is a lack of valid encrypting subkeys. If you have revoked all encrypting subkeys, or allowed them to expire, then I cannot send you encrypted mail.

Exact naming matters

“Bob Bobbertson <>” and “Böb Böbbërtsön <>” are different people. Check your mail client’s “From:” setting, to ensure it matches your UID. If not, fix one of them.

Check your webmail plugin isn’t shit

Some people use third party plugins to integrate GPG into their webmail client (e.g. Hotmail or GMail). Make sure this actually works.

Don’t use Enigmail

Enigmail is a popular plugin to integrate GPG into Mozilla Thunderbird. It doesn’t work, in most cases. Almost every single BADSIG in my inbox is due to Enigmail. Thunderbird will insert spurious line wraps and escape characters into your mail after signing, resulting in invalid signatures.

It’s mostly okay if you never quote mail, and restrict messages to about 70 characters.

I know plenty of Debian Developers don’t care about GPG other than for package signing – but please, for the sanity of the rest of us, take an occasional moment to care a little.

I should note that the worst offenders for keys which don’t “just work” were Developers with 1024D keys – the best behaved were Maintainers of all stripes.

Here Ye, Here Ye

Valve Software’s Steam is the number one digital game distribution service, with more than 65 million registered accounts. Steam runs on Windows, Mac OS, and Linux x86/amd64 computers, and provides access to several thousand games, at varying price points – an enormous growth from less than a dozen games for Windows only about a decade ago. Valve’s latest endeavour has been to bring their storefront into the living room, with a three-pronged attack: a new game controller, a programme of licensed x86 “consoles” to plug into the family TV, and an OS to run on the “Steam Machines” to tie it all together.

December saw the first public release of their “SteamOS” – which, as it turns out, is basically just a preconfigured desktop Linux. Specifically, it’s a Debian Wheezy derivative, comprising a subset of 502 source packages from Wheezy; 8 of Valve’s own source packages; and 51 source packages which have been either patched compared to Wheezy, backported from post-Wheezy, or both. For example, the compiler used by default is gcc-4.7 (rather than Wheezy’s 4.6) and the libc version postdates Wheezy too.

Valve’s official instructions and installer release concerned quite a few people who had planned to try SteamOS on an older PC, by mandating a large (500GB) hard disk and a PC with UEFI firmware. Very quickly a number of instructions started appearing from people trying to fix what people felt were real issues – specifically provision for BIOS-based computers, and installation from optical media.

After being assured that redistribution of derivatives of SteamOS were entirely authorized (and, in fact, encouraged) I decided to produce my own variant, calling upon my own experience with debian-installer modification from past and present jobs, as well as calling upon the skills and experience of the UK Debian community as needed.

The end result is Ye Olde SteamOSe.

SteamOS on VMware Workstation 10

This weekend saw the third release of Ye Olde SteamOSe, a derivative designed to greatly widen the pool of computers capable of running Valve’s OS. And unlike the first two releases, the public response this time has been crazy. Like, totally crazy. Combined score across several subreddits totals about 1700. 7 pages of Google search results. Hundreds of tweets. Mentions in dozens of blogs around the world. Coverage on the Linux Action Show. Unilaterally added to Softpedia’s list of distributions. Almost 2000 views of a video installation walkthrough I posted on YouTube. Crazy.

Sadly the idea to try and track visitors to the page didn’t occur to me until long after the initial rush subsided, but 1000 visitors on Tuesday for a news story which landed on Sunday is still pretty hot in my book.

It’s also interesting to observe the demographics of site visitors – the #1 referrer on Tuesday was Dutch PC site, and StumbleUpon outranks reddit for referrals. About 66% of visitors interested in installing my Debian Wheezy derivative derivative came using Windows (20% using Linux), which suggests there’s a lot of potential Linux users out amongst the Windows gamer masses.

So what’s the purpose of this self-congratulatory blog post? Just dick-waving? Well, there’s an element of that (I’m only human), but I think it might be nice to alert the audience on Planet Debian/Ubuntu, many of whom are not big gamers, to the “next big thing” in “embedded” Linux – except this time a real GNU/ distribution, not some NIH thing like Android.

So now you know!

The directhex! In an Adventure with MIPS

We’ve been trying to build Mono on MIPS in Debian for a long time. Just under a decade, in fact. Mono was the first attempt, back when the Mono source was 10 meg, not today’s ~80.

It never worked though. Not once. At the end of 2004, 10 upstream versions later, we gave up, and turned off mipsel as a target architecture in the package.

When I became a Debian Developer in 2009, one of the first things I did was try again with Mono on various previously unsupported architectures, by running a build on the Debian porterboxes. Supposedly MIPS support was there, as evidenced by all the MIPS-related files in the source tree. But I’d try it, and it’d fail, and upstream would say “works for me”. I repeat the process every major release or so, to see what’s changed.

The breakthrough

Spurred by the removal of two bitrotted architectures (IA64 and SPARC), I decided to try again, this time investigating more deeply into the reason for the failures, with the help of upstream developer Alex Rønne Petersen. A couple of hours of IRC-driven gdb and test program disassembly later, a seemingly innocent comment flagged something in my brain:

09-07-2013 15:10:11 < alexrp! TIL that mul and mult are not the same thing on mips

Why is this notable? Well, MIPS processors lack a whole bunch of instructions which are commonly used in assembler. MUL is one of them – it’s valid MIPS assembler, which is expanded to MULT/MFLO when compiling. Call it a macro, or a mnemonic, or shorthand – the preferred term is “pseudoinstructions”. So what’s the issue?

09-07-2013 15:18:27 > directhex: mono isn't trying to use a mul instruction, right? i mean, that instruction doesn't exist as far as the cpu is concerned, it's a macro the compiler does things with

See where this is headed?

09-07-2013 15:23:59 > directhex: mini/mini-mips.c:#define USE_MUL 1 /* use mul instead of mult/mflo for multiply */


See, the thing about MIPS pseudoinstructions is they may be real instructions on a given CPU implementation. Strictly speaking MUL isn’t a standard instruction, but a given CPU might have it anyway, to make multiplication a little faster (by using only one instruction, not two, for multiplication). In this case, the Debian MIPS infrastructure is based around ICT Loongson-2E processors which don’t have that extension – but the upstream Mono developers were building and testing on an extended CPU, never seeing the issue themselves.

Flipping that define to 0 (and amending the instruction length setting in another file) fixed the build. Mono was running on MIPS for me for the first time ever.

Digging through the history in git showed just how annoying this implementation quirk was. USE_MUL was added in late 2008 – replacing a previously used “#if 1”. The mult/mflo version of the code existed in the Mono source since the first time the full MIPS port was committed in 2006, but we never saw it.

The breakage

So, with that patched to work, I added mipsel to the Experimental build of Mono… which still failed. The runtime would build fine, but the class library build would fail at random times, with random meaningless stack traces. Unrepeatable. Some kind of race condition. The build would eventually succeed if I hammered “make” a few times, but that’s no good for the Debian build daemons. Back to square one…

… except I had an epiphany yesterday. I have heard more than once that Loongson processors are missing a few instructions. What if one of those was being hit, intermittently? I started doing a search in places that might need to work around that kind of issue, and found this. A patch to binutils in 2009, replacing one no-op instruction with another, when /usr/bin/as is fed the -mfix-loongson2f-nop flag.

Turns out NOP is another pseudoinstruction on MIPS. Well, more of an alias. The opcode 0x000000 is “Shift Left Logical” with 0 registers and 0 data, which is a no-op. But on all but the latest generation of Loongson-2F chips, that opcode can, under heavy load, fail – causing inconsistent state in the CPU registers. The flag to “as” replaces “sll 0,0,0” with “or $at,$at,0”, which is also a no-op instruction, but doesn’t trigger the failure on Loongson-2F chips (and 2E chips, although that’s not stated in the documentation).

As long as ALL your programs get fed through “as”, you don’t have a problem, since it uses the replacement opcode… but what if you use a JITter to generate your own opcodes? Oh fuck, it couldn’t be…

diff --git a/mono/arch/mips/mips-codegen.h b/mono/arch/mips/mips-codegen.h
index dc4df7d..1dbd1c6 100644
--- a/mono/arch/mips/mips-codegen.h
+++ b/mono/arch/mips/mips-codegen.h
@@ -334,7 +334,7 @@ enum {
 /* misc and coprocessor ops */
 #define mips_move(c,dest,src) mips_addu(c,dest,src,mips_zero)
 #define mips_dmove(c,dest,src) mips_daddu(c,dest,src,mips_zero)
-#define mips_nop(c) mips_sll(c,0,0,0)
+#define mips_nop(c) mips_or(c,mips_at,mips_at,0)
 #define mips_break(c,code) mips_emit32(c, ((code)<<6)|13)
 #define mips_mfhi(c,dest) mips_format_r(c,0,0,0,dest,0,16)
 #define mips_mflo(c,dest) mips_format_r(c,0,0,0,dest,0,18)

Oh yes it could! Mono was using sll 0,0,0 (the recommended no-op instruction from the MIPS instruction reference manual), causing failures in my tests, because Debian’s build and test infrastructure just happens to use defective silicon. And, again, upstream were unable to reproduce a problem because they use better silicon than we do.

So what now?

Well, last night I uploaded mono_3.2.3+dfsg-3, which includes the above patch to force the replacement no-op instruction. It test built fine on the porterbox, and it should (when the damn experimental buildd gets around to it), just work.


After just under a decade, Mono packages will be available on MIPS in Debian.

And after all this time, all we had to change was 4 lines to work around 7 year old Chinese knock-off processors.

The edit

So, things are finally built.

It turns out that despite everything, the replacement NOP opcode is not enough.

If you re-read the post to the binutils list, pay close attention to:

In theory this is still not enough to fully eliminate possible hangs, but the possiblity is extremely low now and hard to be hit in real code.

It’s a filthy lie. It’s easy to hit the issue in real code: just do a from-source build of the whole Mono class library. With the replacement instruction it builds .NET 2.0, 3.0, 3.5, 4.0, and most of 4.5, before dying in the same way as before – an improvement on failing early in the 2.0 build, but not enough.

Thankfully, 2 out of the 5 Debian mipsel build servers are not Loongson 2 – they’re 11 year old Broadcom SWARM developer boards. Not fast – but also not broken. Luck smiled on me, and caused my build to go to one of these Broadcom machines. As a result…

(experimental_mipsel-dchroot)directhex@eder:~$ mono --version | head -1
Mono JIT compiler version 3.2.3 (Debian 3.2.3+dfsg-3)

It’s been a long time coming.