|This page may be outdated or contain incorrect details. Please update it if you can.|
(note that most of the content here should be considered a draft, not established practice)
How to learn about security issues
Debian Security Tracker
Debian maintains a central database for all security information publicly available (TODO: mention exceptions). Most source packages in Ubuntu are identical to Debian (sans Ubuntu-specific patches applied), so the same base information can be applied to Ubuntu as well. Some notable exceptions wrt source packages in Ubuntu compared to Debian:
- Some packages in Ubuntu share the source package name, but the content is entirely different. E.g. the Linux kernel is both packaged as src:linux, but Debian and Ubuntu maintain their kernels independently.
- In addition to src:linux, Ubuntu provides additional hardware-enablement-kernels (e.g. src:linux-lts-utopic). i.e. multiple packages might be affected
- Some packages only exist in Ubuntu. Possible reasons for that may be that a package is specific to Ubuntu infrastructure or that the package has been created by a Ubuntu contributor and hasn't been uploaded to Debian yet. In some cases it might also have happened that a package was removed from Debian; while Ubuntu follows on many removals, it doesn't necessarily follow all)
The data tracked in the Debian Security tracker comes from many sources:
- All security issues reported to the Debian security team are added in timely manner; these reports can either originate from users, Debian package maintainers or upstream developers who reach out to notify Debian
- MITRE, the organization behind the CVE mapping standard for vulnerability information provides a feed of it's vulnerability master database. This database is pulled automatically and all new additions are added to the Debian security tracker database (after some manual processing)
- External bug tracking systems of other distributions (up to now only for Red Hat) is scraped on a daily basis and new issues are added to the Debian security tracker (again, after manual processing)
- Information sent to important security mailing lists (see below) is added in a timely manner
The information in the Debian Security Tracker not limited to packages shipped by Debian, but also includes packages which are planned/requested for archive inclusion and software not packaged. TODO: elaborate
Project-specific (security) announcement mailing lists
- apache-announce: http://httpd.apache.org/lists.html
- varnish-announce: https://www.varnish-cache.org/lists/mailman/listinfo/varnish-announce
Security mailing lists to follow
- oss-security mailing list, source of many initial reports/CVE requests, previously private reports on the linux-vendors lists are posted here
- Bugtraq (general security mailing list, high volume, includes advisories and announcements for many vendors)
- US-CERT alerts mailing list
How to assess whether we're affected
So, we've learned about a new security problem, it needs to be figured out whether the component is installed anywhere: servermon only displays packages which are flagged for updating, but doesn't maintain the full installed package status (which would be too much data for each puppet run).
This can be queried manually using salt:
salt HOSTLIST cmd.run "dpkg-query -l PACKAGE"
TODO: Check possibilities to make querying the package inventory simpler; e.g. by adding it to facter. The package inventory doesn't necessarily need to be updated with every puppet run, maybe a daily cron job might do just as well
The version number of installed packages need to be checked against the three distributions in use (Ubuntu Precise/Trusty and Debian Jessie) (the sole remaining Lucid machine has a ticket to get upgraded to jessie) In many cases the installed versions will map to the standard version in either of the three distros. Possible deviations are:
- Software not available in Debian or Ubuntu which has been packaged locally and provided by apt.wikimedia.org
- Backports of more recent package versions (which usually means that distribution security updates cannot be used as-is)
- Software with local patches applied (which usually means that the patch(es) need to be rebased on the security update)
Note that this only covers software deployed in the Debian package format. Some packages are distributed through mediawiki-services-restbase-deploy
TODO: Maybe have a central list of all software deployed which is not packaged as a DEB?
Once we have established that a vulnerable software component is in use, the impact needs to be checked. One possibility is that we're not affected at all; possible reasons are
- vulnerabilities which only affect a more recent release than the versions deployed (e.g. which a feature is affected which was introduced later)
- some vulnerabilities might already be fixed, since the security impact was discovered in hindsight
- vulnerabilities specific to Linux distributions not in use (e.g. Red Hat-specific patches) or operating systems not used on servers (e.g. vulnerabilities specific to running on Windows)
Such not-affecting issues are still useful to track, e.g. with a standard Phab ticket, so that they show up when searching for the status of a CVE ID.
If a vulnerable software is present in an affected version, the impact should be checked. The following list is not exhaustive, but lists some important vulnerability classes:
- denial of service vulnerabilities, especially in public-facing services
- vulnerabilities resulting in the execution of attacker-controlled code (e.g. by exploiting vulnerabilities in servers or indirectly in libraries used by servers)
- privilege escalation through bugs in the kernel or software with elevated privileges (while local access is limited, they could be used in combination with vulnerabilities allowing the execution of arbitrary code)
- Information disclosure of sensitive information (e.g. password hashes or cryptographic keys)
If a vulnerability is present, a Phabricator ticket should be opened (TODO: in a different project/queue than "operations" to keep queues neat and tidy?) with a brief description/assessment and a CVE ID (which allows searching for past problems more easily)
How to get the vulnerability fixed
- The version number of installed packages need to be checked against the three distributions in use. If the affected package is part of the standard version set in Ubuntu Precise/Trusty or Debian Jessie and not local modifications are applied, the status of a distribution fix needs to be assessed:
- The Ubuntu archive is divided into four sections: main, restricted, universe and multiverse. Only main and restricted are supported by the Ubuntu security team while the other are covered by the wider Ubuntu community. For practical purposes this usually means that for universe/multiverse no security update will be provided. The status of a vulnerability can either be checked at http://people.canonical.com/~ubuntu-security/cve/$YEAR/$CVEID.html, e.g. http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-0160.html (which often contains a Launchpad bug reference)
- In contrast Debian supports almost all packages with security updates. The exceptions are
- Software from the non-free/contrib sections of the Debian archive
- A small list of packages are exempted from security support since they cannot be sensibly supported with backported patches. These packages are shipped in the debian-security-support package and are mostly related to browsers or desktop-related packages. E.g. QT provides classes to display web content derived from the Webkit codebase (similar to what's used in the Safari browser), but there's no security support for QTWebkit by upstream. Usually unsupportable software is excluded from Debian releases, but in selected cases exceptions are made (as in the case of QTWebkit which is mostly used to display web content from trusted sources as opposed to a web browser processing potentially malicous HTML/Javscript. The status of upcoming security fixes can be queried in the Debian security tracker or (if not obvious) by asking for the status on #debian-security at irc.debian.org
- If the software not coming from the Debian or Ubuntu archive and has been packaged locally or is used with a patch, it's most certainly managed in operations/debs/FOO.git and the fix should be made here (with the standard git workflow in place). In most cases isolating the security fix and merging it is preferred to updating to a new upstream release (which might introduce behavioural changes). (In some cases isolating the fix might not be possible, though)
How to get the fix deployed
Now that we have a fix available, we need to get it rolled out.
Updated packages can be installed using salt (from the salt-master (palladium)):
salt $HOSTS cmd.run "apt-get install PACKAGE=version"
- TODO: Test salt's behaviour in case of packages using debconf
What services/programs require a restart depends very much on the updated software:
- Most daemons perform an automatic restart in postinst
- If libraries are updated, make sure to also restart applications/daemons using that software. The way the dynamic linker works on Linux is to keep older versions of a library loaded as long as existing programs use it. So, e.g. if daemon foo uses libbar 1.0-1 and the security fix 1.0-2 is installed on the system, foo will continue to use libbar 1.0-1 until it's also restarted.
- For some very low level libraries (e.g. glibc) a reboot will usually be required. The same applies to the Linux kernel.
- Contacts when updating some classes of systems
- elastic* cluster node restarts: Erik Bernhardson/David Causse
- analytics* hadoop restarts: Andrew Otto
- TODO: Add notes for updating specific systems etc, which systems to pick as canaries, run which initial tests
- TODO: Add Icinga checks to detect services/programs which need to be restarted (src:checkrestart can be used as the base tool)