Packaging is hard. Packager-friendly is harder.

Releasing software is no small feat, especially in 2018. You could just upload your source code somewhere (a Git, Subversion, CVS, etc, repo – or tarballs on Sourceforge, or whatever), but it matters what that source looks like and how easy it is to consume. What does the required build environment look like? Are there any dependencies on other software, and if so, which versions? What if the versions don’t match exactly?

Most languages feature solutions to the build environment dependency – Ruby has Gems, Perl has CPAN, Java has Maven. You distribute a manifest with your source, detailing the versions of the dependencies which work, and users who download your source can just use those.

Then, however, we have distributions. If openSUSE or Debian wants to include your software, then it’s not just a case of calling into CPAN during the packaging process – distribution builds need to be repeatable, and work offline. And it’s not feasible for packagers to look after 30 versions of every library – generally a distribution will contain 1-3 versions of a given library, and all software in the distribution will be altered one way or another to build against their version of things. It’s a long, slow, arduous process.

Life is easier for distribution packagers, the more the software released adheres to their perfect model – no non-source files in the distribution, minimal or well-formed dependencies on third parties, swathes of #ifdefs to handle changes in dependency APIs between versions, etc.

Problem is, this can actively work against upstream development.

Developers love npm or NuGet because it’s so easy to consume – asking them to abandon those tools is a significant impediment to developer flow. And it doesn’t scale – maybe a friendly upstream can drop one or two dependencies. But 10? 100? If you’re consuming a LOT of packages via the language package manager, as a developer, being told “stop doing that” isn’t just going to slow you down – it’s going to require a monumental engineering effort. And there’s the other side effect – moving from Yarn or Pip to a series of separate download/build/install steps will slow down CI significantly – and if your project takes hours to build as-is, slowing it down is not going to improve the project.

Therein lies the rub. When a project has limited developer time allocated to it, spending that time on an effort which will literally make development harder and worse, for the benefit of distribution maintainers, is a hard sell.

So, a concrete example: MonoDevelop. MD in Debian is pretty old. Why isn’t it newer? Well, because the build system moved away from a packager ideal so far it’s basically impossible at current community & company staffing levels to claw it back. Build-time dependency downloads went from a half dozen in the 5.x era (somewhat easily patched away in distributions) to over 110 today. The underlying build system changed from XBuild (Mono’s reimplementation of Microsoft MSBuild, a build system for Visual Studio projects) to real MSbuild (now FOSS, but an enormous shipping container of worms of its own when it comes to distribution-shippable releases, for all the same reasons & worse). It’s significant work for the MonoDevelop team to spend time on ensuring all their project files work on XBuild with Mono’s compiler, in addition to MSBuild with Microsoft’s compiler (and any mix thereof). It’s significant work to strip out the use of NuGet and Paket packages – especially when their primary OS, macOS, doesn’t have “distribution packages” to depend on.

And then there’s the integration testing problem. When a distribution starts messing with your dependencies, all your QA goes out the window – users are getting a combination of literally hundreds of pieces of software which might carry your app’s label, but you have no idea what the end result of that combination is. My usual anecdote here is when Ubuntu shipped Banshee built against a new, not-regression-tested version of SQLite, which caused a huge performance regression in random playback. When a distribution ships a broken version of an app with your name on it – broken by their actions, because you invested significant engineering resources in enabling them to do so – users won’t blame the distribution, they’ll blame you.

Releasing software is hard.

Long-term distribution support?

A question: how long is reasonable for an ISV to keep releasing software for an older distribution? When is it fair for them to say “look, we can’t feasibly support this old thing any more”.

For example, Debian 7 is still considered supported, via the Debian LTS project. Should ISV app vendors keep producing builds built for Debian 7, with its ancient versions of GCC or CMake, rudimentary C++11 support, ARM64 bugs, etc? How long is it fair to expect an ISV to keep spitting out builds on top of obsolete toolchains?

Let’s take Mono as an example, since, well, that’s what I’m paid to care about. Right now, we do builds for:

  • Debian 7 (oldoldstable, supported until May 2018)
  • Debian 8 (oldstable, supported until April 2020)
  • Debian 9 (stable, supported until June 2022)
  • Raspbian 8 (oldstable, supported until June 2018)
  • Raspbian 9 (stable, supported until June 2020)
  • Ubuntu 12.04 (EOL unless you pay big bucks to Canonical – but was used by TravisCI long after it was EOL)
  • Ubuntu 14.04 (LTS, supported until April 2019)
  • Ubuntu 16.04 (LTS, supported until April 2021)
  • CentOS 6 (LTS, supported until November 2020)
  • CentOS 7 (LTS, supported until June 2024)

Supporting just these is a problem already. CentOS 6 builds lack support for TLS 1.2+, as that requires GCC 4.7+ – but I can’t just drop it, since Amazon Linux (used by a surprising number of people on AWS) is based on CentOS 6. Ubuntu 12.04 support requires build-dependencies on a secret Mozilla-team maintained copy of GCC 4.7 in the archive, used to keep building Firefox releases.

Why not just use the CDN analytics to form my opinion? Well, it seems most people didn’t update their sources.list after we switched to producing per-distribution binaries some time around May 2017 – so they’re still hardcoding wheezy in their sources. And I can’t go by user agent to determine their OS, as Azure CDN helpfully aggregates all of them into “Debian APT-HTTP/1.x” rather than giving me the exact version numbers I’d need to cross-reference to determine OS release.

So, with the next set of releases coming on the horizon (e.g. Ubuntu 18.04), at what point is it okay to say “no more, sorry” to an old version?

Answers on a postcard. Or the blog comments. Or Twitter. Or Gitter.

Hello PGO

Assuming the Planet configuration change was correct, this should be my first post aggregated on Planet GNOME.

Hello!

I’m Jo.

I used to work on Free Software at Collabora, until I sold out, and now I work on Free Software at Microsoft. Specifically, I divide my time between administration of various Xamarin engineering services (primarily the public Jenkins server and its build agents); develop and manage the release of the Mono framework on Windows/Linux and MonoDevelop IDE on Linux; and occasionally work on internal proprietary projects which definitely don’t include Visual Studio Enterprise for Linux. I’m based in the Microsoft office in Cambridge, Mass, along with the Xamarin Release Engineering team, and most of the Xamarin engineering team.

Whilst it hasn’t had the highest profile in the GNOME community for a while, Mono is still out there, in its current niches – in 2018 that would primarily be on smartphones in a wider context, and for games (either via Unity3D or MonoGame/FNA) on the Linux desktop. But hey, it’s still there for desktop apps on Linux if you want it to be! I still use Smuxi as my IRC client. Totally still a thing. And there’s the MonoDevelop IDE, which nowadays I’m trying to release on Linux via Flatpak.

So, um, hi. You’ll see blog posts from me occasionally about Linux software releasing from an ISV perspective, packaging, etc. It’ll be fun for all concerned.

Mono repository changes, beginning Mono vNext

Up to now, Linux packages on mono-project.com have come in two flavours – RPM built for CentOS 7 (and RHEL 7), and .deb built for Debian 7. Universal packages that work on the named distributions, and anything newer.

Except that’s not entirely true.

Firstly, there have been “compatibility repositories” users need to add, to deal with ABI changes in libtiff, libjpeg, and Apache, since Debian 7. Then there’s the packages for ARM64 and PPC64el – neither of those architectures is available in Debian 7, so they’re published in the 7 repo but actually built on 8.

A large reason for this is difficulty in our package publishing pipeline – apt only allows one version-architecture mix in the repository at once, so I can’t have, say, 4.8.0.520-0xamarin1 built on AMD64 on both Debian 7 and Ubuntu 16.04.

We’ve been working hard on a new package build/publish pipeline, which can properly support multiple distributions, based on Jenkins Pipeline. This new packaging system also resolves longstanding issues such as “can’t really build anything except Mono” and “Architecture: All packages still get built on Jo’s laptop, with no public build logs”

So, here’s the old build matrix:

Distribution Architectures
Debian 7 ARM hard float, ARM soft float, ARM64 (actually Debian 8), AMD64, i386, PPC64el (actually Debian 8)
CentOS 7 AMD64

And here’s the new one:

Distribution Architectures
Debian 7 ARM hard float (v7), ARM soft float, AMD64, i386
Debian 8 ARM hard float (v7), ARM soft float, ARM64, AMD64, i386, PPC64el
Raspbian 8 ARM hard float (v6)
Ubuntu 14.04 ARM hard float (v7), ARM64, AMD64, i386, PPC64el
Ubuntu 16.04 ARM hard float (v7), ARM64, AMD64, i386, PPC64el
CentOS 6 AMD64, i386
CentOS 7 AMD64

The compatibility repositories will no longer be needed on recent Ubuntu or Debian – just use the right repository for your system. If your distribution isn’t listed… sorry, but we need to draw a line somewhere on support, and the distributions listed here are based on heavy analysis of our web server logs and bug requests.

You’ll want to change your package manager repositories to reflect your system more accurately, once Mono vNext is published. We’re debating some kind of automated handling of this, but I’m loathe to touch users’ sources.list without their knowledge.

CentOS builds are going to be late – I’ve been doing all my prototyping against the Debian builds, as I have better command of the tooling. Hopefully no worse than a week or two.

edit I guess Ubuntu 12.04 is coming back too, despite being EOL, for TravisCI support.

A quick introduction to Flatpak

Releasing ISV applications on Linux is often hard. The ABI of all the libraries you need changes seemingly weekly. Hence you have the option of bundling the world, or building a thousand releases to cover a thousand distribution versions. As a case in point, when MonoDevelop started bundling a C Git library instead of using a C# git implementation, it gained dependencies on all sorts of fairly weak ABI libraries whose exact ABI mix was not consistent across any given pair of distro releases. This broke our policy of releasing “works on anything” .deb and .rpm packages. As a result, I pretty much gave up on packaging MonoDevelop upstream with version 5.10.

Around the 6.1 release window, I decided to take re-evaluate question. I took a closer look at some of the fancy-pants new distribution methods that get a lot of coverage in the Linux press: Snap, AppImage, and Flatpak.

I started with AppImage. It’s very good and appealing for its specialist areas (no external requirements for end users), but it’s kinda useless at solving some of our big areas (the ABI-vs-bundling problem, updating in general).

Next, I looked at Flatpak (once xdg-app). I liked the concept a whole lot. There’s a simple 3-tier dependency hierarchy: Applications, Runtimes, and Extensions. An application depends on exactly one runtime.  Runtimes are root-level images with no dependencies of their own. Extensions are optional add-ons for applications. Anything not provided in your target runtime, you bundle. And an integrated updates mechanism allows for multiple branches and multiple releases parallel-installed (e.g. alpha & stable, easily switched).

There’s also security-related sandboxing features, but my main concerns on a first examination were with the dependency and distribution questions. That said, some users might be happier running Microsoft software on their Linux desktop if that software is locked up inside a sandbox, so I’ve decided to embrace that functionality rather than seek to avoid it.

I basically stopped looking at this point (sorry Snap!). Flatpak provided me with all the functionality I wanted, with an extremely helpful and responsive upstream. I got to work on trying to package up MonoDevelop.

Flatpak (optionally!) uses a JSON manifest for building stuff. Because Mono is still largely stuck in a Gtk+2 world, I opted for the simplest runtime, org.freedesktop.Runtime, and bundled stuff like Gtk+ into the application itself.

Some gentle patching here & there resulted in this repository. Every time I came up with an exciting new edge case, upstream would suggest a workaround within hours – or failing that, added new features to Flatpak just to support my needs (e.g. allowing /dev/kvm to optionally pass through the sandbox).

The end result is, as of the upcoming 0.8.0 release of Flatpak, from a clean install of the flatpak package to having a working MonoDevelop is a single command: flatpak install --user --from https://download.mono-project.com/repo/monodevelop.flatpakref 

For the current 0.6.x versions of Flatpak, the user also needs to flatpak remote-add --user --from gnome https://sdk.gnome.org/gnome.flatpakrepo first – this step will be automated in 0.8.0. This will download org.freedesktop.Runtime, then com.xamarin.MonoDevelop; export icons ‘n’ stuff into your user environment so you can just click to start.

There’s some lingering experience issues due the sandbox which are on my radar. “Run on external console” doesn’t work, for example, or “open containing folder”. There are people working on that (a missing DBus# feature to allow breaking out of the sandbox). But overall, I’m pretty happy. I won’t be entirely satisfied until I have something approximating feature equivalence to the old .debs.  I don’t think that will ever quite be there, since there’s just no rational way to allow arbitrary /usr stuff into the sandbox, but it should provide a decent basis for a QA-able, supportable Linux MonoDevelop. And we can use this work as a starting point for any further fancy features on Linux.

Gtk# app development in Flatpak MonoDevelop

Editing MonoDevelop in MonoDevelop. *Inception noise*

mono-project.com Linux packages, June 2015 edition

The latest stable release of Mono has happened, the first bugfix update to our 4.0 branch. Here are the release highlights, and some other goodies.

Stable Packages

This release covers Mono 4.0.1, and MonoDevelop 5.9. As promised last time, this includes builds for RPM-based x64 systems (CentOS 7 minimum), Debian-based x64, i386, ARMv5 Soft Float, and ARMv7 Hard Float systems (Debian 7/Ubuntu 12.04 minimum).

Version numbering

From now on, we’re going to be clearer with our version numbering scheme. Historically, we’ve shipped, say, “4.0.0” to the public – internally, there have been a lot of builds on this target branch, all of which get an internal revision number. “4.0.0” as-shipped was in fact 4.0.0.143 internally – that was the first 4.0.0 branch release approved of for stable release.

This release is the first service release on the 4.0.0 branch, numbered 4.0.1.44 – it’ll be officially referred to as “4.0.1” in some places, but isn’t the same as 4.0.1.0, which already released on Linux/Windows a while back, to include an emergency bugfix for those platforms.

That was sorta a screwup really. Using the 4-part version removes the ambiguity, rather than having 44 different “4.0.1”‘s in existence. And we’ll aim to be clearer in future about what is alpha, what is beta, and what is final (and what is a random emergency snapshot).

Alpha Linux packages

Want to see things earlier? We’ve now got the structure in place to provide Linux packages (and source releases) to mirror what we do on Mac. When we upload a prospective package to our Mac customers, we will automatically trigger builds for Linux too. See http://www.mono-project.com/download/alpha/

Beta Linux packages

See above. s/alpha/beta/.

Weekly git Master snapshots

We already have packages in place for every git commit, which parallel-install Mono into /opt. This is different.

Weekly (or, right now, when I manually run the requisite Jenkins job), the latest Mac build of Mono git master from our internal CI system will be copied to a public location just for you, a source tarball generated, and packages built. See here for info on making use of that.
directhex@marceline:~$ mono --version
Mono JIT compiler version 4.3.0 (Nightly 4.3.0.21/88d2b9d Thu May 28 10:54:32 UTC 2015)

mono-project.com Linux packages, January 2015 edition

The latest version of Mono has released (actually, it happened a week ago, but it took me a while to get all sorts of exciting new features bug-checked and shipshape).

Stable packages

This release covers Mono 3.12, and MonoDevelop 5.7. These are built for all the same targets as last time, with a few caveats (MonoDevelop does not include F# or ASP.NET MVC 4 support). ARM packages will be added in a few weeks’ time, when I get the new ARM build farm working at Xamarin’s Boston office.

Ahead-of-time support

This probably seems silly since upstream Mono has included it for years, but Mono on Debian has never shipped with AOT’d mscorlib.dll or mcs.exe, for awkward package-management reasons. Mono 3.12 fixes this, and will AOT these assemblies – optimized for your computer – on installation. If you can suggest any other assemblies to add to the list, we now support a simple manifest structure so any assembly can be arbitrarily AOT’d on installation.

Goodbye Mozroots!

I am very pleased to announce that as of this release, Mono users on Linux no longer need to run “mozroots” to get SSL working. A new command, “cert-sync”, has been added to this release, which synchronizes the Mono SSL certificate store against your OS certificate store – and this tool has been integrated into the packaging system for all mono-project.com packages, so it is automatically used. Just make sure the ca-certificates-mono package is installed on Debian/Ubuntu (it’s always bundled on RPM-based) to take advantage! It should be installed on fresh installs by default. If you want to invoke the tool manually (e.g. you installed via make install, not packages) use

cert-sync /path/to/ca-bundle.crt

On Debian systems, that’s

cert-sync /etc/ssl/certs/ca-certificates.crt

and on Red Hat derivatives it’s

cert-sync /etc/pki/tls/certs/ca-bundle.crt

Your distribution might use a different path, if it’s not derived from one of those.

Windows installer back from the dead

Thanks to help from Alex Koeplinger, I’ve brought the Windows installer back from the dead. The last release on the website was for 3.2.3 (it’s actually not this version at all – it’s complicated…), so now the Windows installer has parity with the Linux and OSX versions. The Windows installer (should!) bundles everything the Mac version does – F#, PCL facades, IronWhatever, etc, along with Boehm and SGen builds of the Mono runtime done with Visual Studio 2013.

An EXPERIMENTAL OH MY GOD DON’T USE THIS IN PRODUCTION 64-bit installer is in the works, when I have the time to try and make a 64-build of Gtk#.

mono-project.com Linux packages – an update

It’s been pointed out to me that many people aren’t aware of the current status of Linux packages on mono-project.com, so I’m here’s a summary:

Stable packages

Mono 3.10.0, MonoDevelop 5.5.0.227, NuGet 2.8.1 and F# 3.1.1.26 packages are available. Plus related bits. MonoDevelop on Linux does not currently include the F# addin (there are a lot of pieces to get in place for this to work).

These are built for x86-64 CentOS 7, and should be compatible with RHEL 7, openSUSE 12.3, and derivatives. I haven’t set up a SUSE 1-click install file yet, but I’ll do it next week if someone reminds me.

They are also built for Debian 7 – on i386, x86-64, and IBM zSeries processors. The same packages ought to work on Ubuntu 12.04 and above, and any derivatives of Debian or Ubuntu. Due to ABI changes, you need to add a second compatibility extension repository for Ubuntu 12.04 or 12.10 to get anything to work, and a different compatibility extension repository for Debian derivatives with Apache 2.4 if you want the mod-mono ASP.NET Apache module (Debian 8+, Ubuntu 13.10+, and derivatives, will need this).

MonoDevelop 5.5 on Ubuntu 14.04

MonoDevelop 5.5 on Ubuntu 14.04

In general, see the install guide to get these going.

Docker

You may have seen Microsoft recently posting a guide to using ASP.NET 5 on Docker. Close inspection would show that this Docker image is based on our shiny new Xamarin Mono docker image, which is based on Debian 7.The full details are on Docker Hub, but the short version is “docker pull mono:latest” gets you an image with the very latest Mono.

directhex@desire:~$ docker pull mono:latest
Pulling repository mono
9da8fc8d2ff5: Download complete 
511136ea3c5a: Download complete 
f10807909bc5: Download complete 
f6fab3b798be: Download complete 
3c43ebb7883b: Download complete 
7a1f8e485667: Download complete 
a342319da8ea: Download complete 
3774d7ea06a6: Download complete 
directhex@desire:~$ docker run -i -t mono:latest mono --version 
Mono JIT compiler version 3.10.0 (tarball Wed Nov  5 12:50:04 UTC 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

The Dockerfiles are on GitHub.

The unstoppable march of mobile technology

It’s been more than 2 years since my last post about my smartphone. In the time after that post I upgraded my much loved Windows Phone 7 device to Windows Phone 8 (which I got rid of within months, for sucking), briefly used Firefox OS, then eventually used a Nexus 4 for at least a year.

After years of terrible service provision and pricing, I decided I would not stay with my network Orange a moment longer – and in getting a new contract, I would get a new phone too. So on Friday, I signed up to a new £15 per month contract with Three, including 200 minutes, unlimited data, and 25GB of data roaming in the USA and other countries (a saving of £200,000 per month versus Orange). Giffgaff is similarly competitive for data, but not roaming. No other network in the UK is competitive.

For the phone, I had a shortlist of three: Apple iPhone 6, Sony Xperia Z3 Compact, and Samsung Galaxy Alpha. These are all “small” phones by 2014 standards, with a screen about the same size as the Nexus 4. I didn’t consider any Windows Phone devices because they still haven’t shipped a functional music player app on Windows Phone 8. Other more “fringe” OSes weren’t considered, as I insist on trying out a real device in person before purchase, and no other comparable devices are testable on the high street.

iPhone 6

This was the weakest offering, for me. £120 more than the Samsung, and almost £200 more than the Sony, a much lower hardware specification, physically larger, less attractive, and worst of all – mandatory use of iTunes for Windows for music syncing.

Apple iPhone 6, press shot from apple.com, all rights reserved

The only real selling point for me would be for access to iPhone apps. And, I guess, decreased chance of mockery by co-workers.

Galaxy Alpha

Now on to the real choices. I’ve long felt that Samsung’s phones are ugly plasticy tat – the Galaxy S5 is popular, well-marketed, but looks and feels cheap compared to HTC’s unibody aluminium One. They’ve also committed the cardinal sin of gimping the specifications of their “mini” (normal-sized) phones, compared to the “normal” (gargantuan) versions. The newly released S5 Mini is about the same spec as early 2012’s S3, the S4 Mini was mostly an S2 internally, and so on.

However, whilst HTC have continued along these lines, Samsung have finally released a proper phone under 5″, in the Alpha.

Samsung Galaxy Alpha press shot from samsungmobile.com, all rights reserved

Samsung Galaxy Alpha press shot from samsungmobile.com, all rights reserved

The Alpha combines a 4.7″ AMOLED screen, a plastic back, metal edges, 8-core big.LITTLE processor, and 2GB RAM. It is a PRETTY device – the screen really dazzles (as is the nature of OLED). It feels like a mix of design cues from an iPhone and Samsung’s own, keeping the angular feel of iPhone 4->5S rather than the curved edges on the iPhone 6.

The Galaxy Alpha was one of the two devices I seriously considered.

Xperia Z3 Compact

The other Android device I considered was the Compact version of Sony’s new Xperia Z3. Unlike other Android vendors, Sony decided that “mini” shouldn’t mean “low end” when they released the Z1 compact earlier this year. The Z3 follows suit, where the same CPU and storage are found on both the big and little versions.

Sony Xperia Z3 Compact press shot from Sony Xperia Picasa album. CC BY-NC-SA 3.0

Sony Xperia Z3 Compact press shot from Sony Xperia Picasa album. CC BY-NC-SA 3.0

The Z3C has a similar construction to the Nexus 4, with glass front and back, and plastic rim. The specification is similar to the Galaxy Alpha (with a quadcore 2.5GHz Qualcomm processor about 15% faster than the big.LITTLE Exynos in the Galaxy Alpha). It differs in a few places – LCD rather than AMOLED (bad); a non-removable (bad) 2600 mAh battery (good) compared to the removable 1860 mAh in the Samsung; waterproofing (good); A less hateful Android shell (Xperia on Android vs Samsung Touchwiz).

For those considering a Nexus-4-replacement class device (yes, rjek, that means you), both the Samsung and the Sony are worth a look. They both have good points and bad points. In the end, both need to be tested to form a proper opinion. But for me, the chunky battery and tasteful green were enough to swing it for the Sony. So let’s see where I stand in a few months’ time. Every phone I’ve owned, I’ve ended up hating it for one reason or another. My usual measure for whether a phone is good or not is how long it takes me to hit the “I can’t use this” limit. The Nokia N900 took me about 30 minutes, the Lumia 800 lasted months. How will the Z3 Compact do? Time will tell.

#GamerHate

Once upon a time, a large number of angry video game players were up in arms about perceived corruption in games journalism. They took their outrage to social networks, joining in with the existing outrage going on under the hashtag #gamergate. And yet every time they tried to discuss the injustices demonstrated to them in infographics and YouTube videos, they were dismissed as harassers or misogynists.

This confused and angered the gamers further. Why were they on trial for something they had not done? They had serious concerns about cozy relationships between developers and the press, yet that means they hate women? WTF?

The thing is, they got played. If you’re a gamer, and recognise any part of yourself in the first two paragraphs, you got played.

Good doggie!

There’s a term in politics called “dog-whistle“. Dog-whistle politics are where you want to achieve A, but can’t tell the electorate “let’s achieve A” as they would not approve of it. So you talk about B instead. B has the side-effect of achieving A. You enlist all sorts of people who truly believe in the promise of B, and who reject the concept of A – but their support of B just happens to achieve A anyway.

The famous quote about dog-whistle politics comes from a former senior Regan administration staffer:

You start out in 1954 by saying, “Nigger, nigger, nigger.” By 1968, you can’t say “nigger” — that hurts you. Backfires. So you say stuff like forced busing, states’ rights and all that stuff. You’re getting so abstract now [that] you’re talking about cutting taxes, and all these things you’re talking about are totally economic things and a byproduct of them is [that] blacks get hurt worse than whites. And subconsciously maybe that is part of it. I’m not saying that. But I’m saying that if it is getting that abstract, and that coded, that we are doing away with the racial problem one way or the other. You follow me — because obviously sitting around saying, “We want to cut this,” is much more abstract than even the busing thing, and a hell of a lot more abstract than “Nigger, nigger.”

Those who go along with B and inadvertently help achieve A are known as – and this is the term, not a personal attack, so don’t freak out – useful idiots.

In this instance, B is the discussion of games journalism. There are a whole lot of people who truly believe, in their hearts, that there are causes for concern in the games journalism industry, and they’re right – but those people are being orchestrated, without their consent, to help achieve A. And in this case, A is stamping out “social justice warriors” from all areas of the games industry – ending their careers, or invoking suicide, or both. The main targets singled out for attention by #gamergate are not selected at random, nor are they especially corrupt – they’ve been selected because attacking them serves aim A, and attacking others would not.

This was never your pool in the first place

You stand in a swimming pool. Many others are here, splashing about. You try to have a serious discussion with someone on the edge of the pool, as a turd floats by.

“Dude, what the fuck, there’s shit in that pool”

You don’t know what they’re talking about. You didn’t shit in the pool, none of your friends shat in the pool, why are they changing the subject, and attacking you?

Wispy, misshapen scraps of excrement float past, knocking your arm.

ID-10078723

“Seriously, that pool is literally filled with poo. It’s those guys in the corner, with their trunks down. They’re just squatting and shitting, non-stop. You’re getting covered in shit.”

“I really have no idea what you’re talking about, why are you attacking me?” – you recruit some friends to join you in the pool, to prove that it’s not full of shit.

Small, rabbit-pellet crap squelches underfoot.

“Okay, really, I’m not talking to you now, you stink of shit” – and they walk off.

You turn to your other friends in the pool outraged. “How DARE they imply these things? We have *nothing* to do with those guys in the corner, they have *nothing* to do with this pool!”

They nod in agreement, as a wave of brown water splashes over you.

Not your personal army

So who’s orchestrating things? And why? 4chan, basically. This all started in an effort to destroy the career and/or life of indie developer Zoe Quinn. Here’s some of the original planning work – trigger warning, for those that applies to. From there, it moved on to anyone else labelled as a “social justice warrior” by the 4chan crowd – anyone who brings up issues of discrimination, race, gender, sexuality, etc. The 4channers and their allies (hey, a few people independently agree with cause A) are terrified of a world where there might be a reduction in the number of eviscerated prostitutes in a game – kicking out the people who might cause that shift is an important step in saving the future of “vidya”

And, sweet gamer, you may insist that you have nothing to do with 4chan – but the evidence shows that whether you know about it or not, it’s their tune you’re singing to. That’s why you’ve been directed at so many “corrupt” journalists who happen to be women.You’re not the one sending rape/murder threats to the few women journalists, you’re just accusing them of corrupt practices. It’s those *other* guys you know nothing about, sharing the same banner and discussion space, sending those threats.

As a thought exercise, listen to this podcast. It’s quite long, but it’s a good example. You’ll know when you get to the relevant part – unless you’re a sociopath, incapable of forming human emotions. And you’re not – you’ve just been played. This is the result of you being played. People who have devoted decades of their working lives to the hobby you both love, driven out by unbridled hate.

The truth about games journalism

A number of articles popped up in the opinions pages of various games (and non-games) sites, in the early days of #gamergate, saying things like “gamers are over”. This certainly helped convince plenty of those who believe in cause B that games journalists are not only corrupt – but hold them in contempt too. Fuelling anger. Justifying angry responses.

The problem with an era where attention spans don’t go past 6 seconds or 140 characters is nobody really understood the point any of these articles made. In 2014, what does “gamer” even mean? Every retirement home has a Wii, every smartphone has Angry Birds, everyone on Facebook plays Candy Crush Saga. What does it mean to be a “gamer” when you spend fewer hours a day playing Rust than a businessman spends playing smartphone games on the train? The “gamer” identity meant something when playing games was something unusual, but now? Would you describe yourself as “food eater” or “TV watcher”? Because it’s about as meaningful.

And you know what? All those “corrupt” games journalists there’s been work to evict? Those guys are gamers too. You know why a journalist might back a Kickstarter or Patreon? The same reason you might, as an individual gamer. You know why they’re getting paid $50 an article and barely keeping the lights on? Because they love gaming, because they’re you. Yes, even the “social justice warriors”, even the women. They’re gamers like anybody else.

“Games journalism” comes from a sector known as “enthusiast press” – the same space that has, say, car magazines. Do car journos and car company PR folk know each other? Of course – that’s where most of the magazine content comes from! Does the journo have bias? Of course! They love cars, and drive one too! You don’t want unbiased reviews – you want to read reviews from a reviewer whose biases match your own. The same goes for games journalism. Game reviews don’t need to start every page with “video games are totally fun and worth your time, weird as it may seem”, the way the occasional review in the mainstream press & newspapers do – because you, and the reviewer, are on the same page.They don’t hold you in contempt. You speak a common language about a common interest. That’s their entire value to you – they know the right people to get access to the info you want, and you don’t.

A great example of why this has nothing to do with ethics? Part of #gamergate called “Operation Disrespectful Nod” (everything is an Operation to 4chan). This aims to put pressure on “corrupt” (employs a hated “social justice warrior”) sites by putting pressure on that site’s advertisers. Advertisers like EA. The idea is that EA has enough sway over a site to get disliked journalists fired and that’s fine and a source of allies – but the true corruption is over who is backing whose Patreon campaign for the price of a beer a month.

The idea that there’s a problem about the relationship between the press and their sources isn’t new in the enthusiast press. Which is why most outlets have a strict division between editorial and advertising departments. Lapses happen – but they’re rare, and heavily covered by other sources, because those lapses are news!

In conclusion, I’m always right

Everything I say needs to be taken with a pinch of salt – but so does everything anyone else says. And the “I don’t support harassment” gamers of #gamergate need to ask themselves why they’ve swallowed every word wholesale from meaningless infographics and videos, but insist that any evidence of wrongdoing on the part of their movement is fabricated.

Most gamers, including those caught up in #gamergate, are good, ordinary, decent folk. At worst, gullible. And what a good, ordinary, decent gamer should want is for interesting people to make interesting content – be it games or articles or art or mods or soundtrack remixes or Let’s Plays or cosplays or game-themed bakery or whatever. How does driving interesting people out of the industry achieve that?

Image courtesy of BrandonSigma at FreeDigitalPhotos.net

An exception to the sitewide license is used for this post – it is under Creative Commons Attribution-NoDerivs 2.0 UK: England & Wales. The main difference from the site license is that this article may be re-posted, with attribution, as long as the text is unaltered, even on commercial for-profit sites.