Long-term distribution support?

A question: how long is reasonable for an ISV to keep releasing software for an older distribution? When is it fair for them to say “look, we can’t feasibly support this old thing any more”.

For example, Debian 7 is still considered supported, via the Debian LTS project. Should ISV app vendors keep producing builds built for Debian 7, with its ancient versions of GCC or CMake, rudimentary C++11 support, ARM64 bugs, etc? How long is it fair to expect an ISV to keep spitting out builds on top of obsolete toolchains?

Let’s take Mono as an example, since, well, that’s what I’m paid to care about. Right now, we do builds for:

  • Debian 7 (oldoldstable, supported until May 2018)
  • Debian 8 (oldstable, supported until April 2020)
  • Debian 9 (stable, supported until June 2022)
  • Raspbian 8 (oldstable, supported until June 2018)
  • Raspbian 9 (stable, supported until June 2020)
  • Ubuntu 12.04 (EOL unless you pay big bucks to Canonical – but was used by TravisCI long after it was EOL)
  • Ubuntu 14.04 (LTS, supported until April 2019)
  • Ubuntu 16.04 (LTS, supported until April 2021)
  • CentOS 6 (LTS, supported until November 2020)
  • CentOS 7 (LTS, supported until June 2024)

Supporting just these is a problem already. CentOS 6 builds lack support for TLS 1.2+, as that requires GCC 4.7+ – but I can’t just drop it, since Amazon Linux (used by a surprising number of people on AWS) is based on CentOS 6. Ubuntu 12.04 support requires build-dependencies on a secret Mozilla-team maintained copy of GCC 4.7 in the archive, used to keep building Firefox releases.

Why not just use the CDN analytics to form my opinion? Well, it seems most people didn’t update their sources.list after we switched to producing per-distribution binaries some time around May 2017 – so they’re still hardcoding wheezy in their sources. And I can’t go by user agent to determine their OS, as Azure CDN helpfully aggregates all of them into “Debian APT-HTTP/1.x” rather than giving me the exact version numbers I’d need to cross-reference to determine OS release.

So, with the next set of releases coming on the horizon (e.g. Ubuntu 18.04), at what point is it okay to say “no more, sorry” to an old version?

Answers on a postcard. Or the blog comments. Or Twitter. Or Gitter.

9 Responses to “Long-term distribution support?”

  1. Chris, on February 12th, 2018 at 9:50 pm Said:

    As a developer with a little IT security experience, I cannot take anyone seriously who still runs systems without TLS 1.2 support. Doing this is almost as irresponsible as still running Windows XP.

    As for Debian and other “conservative” distros: They don’t usually get any updates and often miss security updates. Prominent example: WebKitGtk. Debian stable has webkit2gtk4, last built on 2017-08-07 (https://packages.debian.org/stable/libs/libwebkit2gtk-4.0-37). Upstream has fixed more than 52 CVEs since then (https://webkitgtk.org/news.html). Qt5 WebKit is even worse. All web browsers except Firefox and Chrome/Chromium on these distros have 50+ security bugs open. This is rotten to the core and it doesn’t seem like anyone gives a sh*t. If it is GUI software you are talking about, I would not even try to support anything older than the most recent Ubuntu LTS and Debian stable.

    If it is non-GUI software, the situation is much better because that software tends to be more stable and released less often so maintenance and backporting bug fixes is easier.

  2. Jonathan, on February 13th, 2018 at 10:22 am Said:

    Debian stable carries libwebkit2gtk-4.0-37 but the package version is 2.16.6-0+deb9u1. That upload was in August last year and included patches for 13 CVEs (up to WSA-2017-0006)

    https://tracker.debian.org/news/862764

    I’m not certain (because the relevant Debian web service seems to be down at the time of writing) but I think all known Webkit security bugs have been fixed in the Debian packages. Or at least all known to August 2017. I would expect to see a list of open security bugs on this page, and there are none:

    https://tracker.debian.org/pkg/webkit2gtk

    So at least in the case of Debian stable, people do “give a sh*t”.

  3. Jonathan, on February 13th, 2018 at 10:23 am Said:

    Aha it finally loaded. here’s the list of known CVEs affecting this particular package across oldstable/stable/testing/unstable. There are several still open for stable.

    https://security-tracker.debian.org/tracker/source-package/webkit2gtk

  4. Chris, on February 13th, 2018 at 7:47 pm Said:

    I don’t know why, but the webkitgtk website lists about twice as much CVEs.

    Ok, saying they “don’t give a sh*t” was overreacting. They sometimes do care. Having these CVEs open since last August (that is almost half a year) makes it look like they don’t care *most of the time* though.

  5. Paulo Almeida, on February 14th, 2018 at 11:10 pm Said:

    All but one of those are fixed in buster and sid, and their urgency is “unimportant” (most are for “an issue in certain Apple products”), so my guess is they don’t want to force people to upgrade when it’s not necessary.

  6. Chris, on February 16th, 2018 at 10:22 pm Said:

    The CVEs are labeled “an issue in certain Apple products” because there is no official CVE process for WebKitGtk or QtWebKit. WebKitGtk just has a process to make sure that all CVEs patched for WebKit will land in their security updates.

    Buster and sid are just testing and alpha-grade software distributions respectively not dedicated for everyday use. They even get regular updates.

  7. Paulo Almeida, on February 17th, 2018 at 1:41 am Said:

    Chris,

    I know what buster and sid are. My point is that if something is fixed for them, it’s a choice of the security team to not issue a security update and fix it in stable. See for instance the Debian Security FAQ:

    https://www.debian.org/security/faq#cvedsa

    “Low-impact issues not qualifying for a DSA can be fixed in the next release of Debian, in a point release of the current stable or oldstable distributions, or are included in a DSA when that is being issued for a more serious vulnerability.”

  8. Luis Matos, on February 13th, 2018 at 12:05 am Said:

    Well … i write industrial control applications.
    These applications once in production are REALLY expensive to upgrade to newer technologies or development environments. So we relly on long term software releases, for example debian or ubuntu LTS. Because we can have an evoluting base, but no new software.

    Having new versions of mono in debian 8, i think is unneeded work, but maintaining a version for debian 8 should be done.

    my 0.02 !

  9. Paulo Almeida, on February 17th, 2018 at 9:56 pm Said:

    Chris,

    I stand corrected (I think). Today there was a message [1] on the debian-devel list that mentioned a similar situation with Javascript, where CVEs are not dealt with. I went to check the ones in the library you mentioned and they also have a note saying they are not covered by security. If I’m interpreting that correctly, what I said before about not issuing a security update as a choice by the security team is wrong.

    [1] https://lists.debian.org/debian-devel/2018/02/msg00368.html

Leave a Reply

« »