Sign up to save your podcasts
Or
Share Episode 224
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Episode 224
Overview
It’s been an absolutely manic week in the Linux security community as the news
and reaction to the recent announcement of a backdoor in the xz-utils project
was announced late last week, so we dive deep into this issue and discuss how it
impacts Ubuntu and give some insights for what this means for the open source
and Linux communities in the future.
This week in Ubuntu Security Updates
20 unique CVEs addressed
[USN-6718-2] curl vulnerability
[USN-6719-1] util-linux vulnerability
[USN-6686-5] Linux kernel (Intel IoTG) vulnerabilities
[USN-6715-1] unixODBC vulnerability
[USN-6704-4] Linux kernel (Intel IoTG) vulnerabilities
[USN-6707-4] Linux kernel (Azure) vulnerabilities
[USN-6720-1] Cacti vulnerability
Goings on in Ubuntu Security Community
xz-utils backdoor and Ubuntu
disclosed to the open source community via oss-security mailing list - this
was in the recent 5.6.0/5.6.1 releases from late Feb/early March this year
impact only xz-utils and so only in handling of xz compressed data / files -
but even with that assumption that perhaps it could then infect anything that
got compressed/decompressed
xz-utils/liblzma itself but openssh - and the affect was to provide a backdoor
into openssh that would allow the attacker to get remote access to any machine
running ssh server with this backdoor liblzma installed
Easter break, so lots of folks were either EOD or on leave etc - and trying to
grapple with a threat that we knew could possibly impact the impedending 24.04
LTS release
currently in-development 24.04 release - not in any other Ubuntu versions -
and was removed as soon as we became aware, so unless you are running the
devel release AND you had manually opted to install this version from the
-proposed pocket, you would not be affected - very lucky
(and speculation) since it was announced, both at the social side of things
and the technical aspects of the backdoor itself
and cover the salient details
long and patient campaign by the attacker, who slowly gained the trust of the
upstream project over the last 2 years and likely pressured the maintainer via
sock-puppet accounts to then get themselves added as an additional maintainer
adding new features and bug fixes etc over the past 2 years, but then suddenly
introduced the backdoor into the most recent 2 releases
required 2 parts - the binary containing the backdoor and the code to get this
compiled into the liblzma library at build time
archive itself used as part of the test suite
instead was just in the tarball prepared by the maintainer for the official
release
fake test xz archive
not just multiple levels of obfuscation in the build process but the backdoor
code itself contained many elements to try and make it harder to recognise and
reverse engineer, presumably to allow it to hide in plain sight
was what gave it away eventually
community, which is summarised as Linus’ Law - with enough eyeballs all bugs
are shallow - but sadly this wasn’t proven out in this case
found by anyone doing review of the changes upstream or by the various distros
like Debian / Fedora / OpenSUSE / Arch or even Ubuntu when incorporating this
new version into their repos - but instead was found by Andres Freund, one of
the maintainers of PostgreSQL, when they were looking to benchmark some new
changes scheduled for the next PostgreSQL release
this new version into unstable a few weeks ago, and wanted to get the
performance noise floor of the system as low as possible before doing
benchmarking of PostgreSQL - noticed large transient CPU spikes in sshd and
then eventually weird memory errors in sshd due to bugs in the initial version
of the backdoor
culprit and appeared to contain some very strange code to hook into the
authentication process of sshd when it was launched via systemd
of xz-utils on Github
lookup symbol names at runtime, rather than directly encoding them in the
exploit binary, presumably to try and make reverse engineering of the binary
harder (since you can’t just run strings on it and get any real sensible output)
the backdoor itself, as well as the process taken by the attacker to
incorporate this into the xz-utils project - both from a community point of
view and a technical point of view
attacker to get pre-authentication remote-code execution in sshd via a
specific private key when connecting to the server - NOBUS
particular matching private key from the client - and if found would then
proceed to execute arbitrary commands specified by the client without
requiring usual authentication
they don’t have the matching private key - so the impact of the backdoor is
somewhat limited
huge number of distros, means they essentially wanted a backdoor into any
Linux machine in the future that only they could use
what their intended purpose was especially given this wide reaching goal of
getting this into all the major Linux distros - but it would be just
speculation
to look at the timeline as it concerned Ubuntu
publicly disclosed, and as soon as we heard of the possible backdoor, and
realised that it only affected the version in-development noble-proposed we
quickly notified the Archive Admin team who then deleted it from
noble-proposed on 29th March, neutralising the main threat
approach - not only have we removed this version from noble-proposed as soon
as we became aware, we are then rebuilding every binary package that got built
since that compromised version was in noble-proposed originally - out of an
abundance of caution - we don’t have any information that says this backdoor
was doing anything other than what the various writeups have found so far, BUT
we also can’t be certain that it didn’t have other functionality either - so
being very cautious and rebuilding everything that was itself built since 27th
Feb
by security researchers or maintainers but by a developer from Microsoft who
happened to be looking for the right things at the right time and decided to
be curious
packages to be stuck in noble-proposed and not in the release pocket, else
many Ubuntu users and developers would have been impacted if this had migrated
to the noble release pocket
OSS development practices and was quite persistent in trying to get this
incorporated into distros - even urging for this new version to be synced to
Ubuntu so that it would land in the upcoming noble release as recently as
Thursday last week, just hours before the public disclosure
and technical work to develop and hide the backdoor, but could then
interface with distros via their established practices to try and get them
to incorporate their new backdoored version faster than they may have
otherwise
well as the technical details of both the code used to incorporate the
backdoored code into the final liblzma binary during the build process, as
well as the details of the backdoor itself and how it operates at runtime
projects have long dependency chains - in this case openssh when integrated
with libsystemd which in turn used liblzma - and it is unclear who the
maintainers and authors are or what procedures are in place for vetting and
transferring ownership of OSS projects - all present significant challenges
for OSS
to get involved, which is what we saw in the aftermath - despite all the
advanced obfuscation techniques employed was able to be analysed in a matter
of days by the community working together - and to analyse it in a huge amount
of depth and in such an open way that it leaves little room for questioning
the validity of the assessment - anyone can double check the work and come to
the same conclusions
first against an OSS project but it is a wake-up call to the OSS and Linux
ecosystem
Get in contact
View all episodes
By Ubuntu Security Team
4.8
1010 ratings
Overview
It’s been an absolutely manic week in the Linux security community as the news
and reaction to the recent announcement of a backdoor in the xz-utils project
was announced late last week, so we dive deep into this issue and discuss how it
impacts Ubuntu and give some insights for what this means for the open source
and Linux communities in the future.
This week in Ubuntu Security Updates
20 unique CVEs addressed
[USN-6718-2] curl vulnerability
[USN-6719-1] util-linux vulnerability
[USN-6686-5] Linux kernel (Intel IoTG) vulnerabilities
[USN-6715-1] unixODBC vulnerability
[USN-6704-4] Linux kernel (Intel IoTG) vulnerabilities
[USN-6707-4] Linux kernel (Azure) vulnerabilities
[USN-6720-1] Cacti vulnerability
Goings on in Ubuntu Security Community
xz-utils backdoor and Ubuntu
disclosed to the open source community via oss-security mailing list - this
was in the recent 5.6.0/5.6.1 releases from late Feb/early March this year
impact only xz-utils and so only in handling of xz compressed data / files -
but even with that assumption that perhaps it could then infect anything that
got compressed/decompressed
xz-utils/liblzma itself but openssh - and the affect was to provide a backdoor
into openssh that would allow the attacker to get remote access to any machine
running ssh server with this backdoor liblzma installed
Easter break, so lots of folks were either EOD or on leave etc - and trying to
grapple with a threat that we knew could possibly impact the impedending 24.04
LTS release
currently in-development 24.04 release - not in any other Ubuntu versions -
and was removed as soon as we became aware, so unless you are running the
devel release AND you had manually opted to install this version from the
-proposed pocket, you would not be affected - very lucky
(and speculation) since it was announced, both at the social side of things
and the technical aspects of the backdoor itself
and cover the salient details
long and patient campaign by the attacker, who slowly gained the trust of the
upstream project over the last 2 years and likely pressured the maintainer via
sock-puppet accounts to then get themselves added as an additional maintainer
adding new features and bug fixes etc over the past 2 years, but then suddenly
introduced the backdoor into the most recent 2 releases
required 2 parts - the binary containing the backdoor and the code to get this
compiled into the liblzma library at build time
archive itself used as part of the test suite
instead was just in the tarball prepared by the maintainer for the official
release
fake test xz archive
not just multiple levels of obfuscation in the build process but the backdoor
code itself contained many elements to try and make it harder to recognise and
reverse engineer, presumably to allow it to hide in plain sight
was what gave it away eventually
community, which is summarised as Linus’ Law - with enough eyeballs all bugs
are shallow - but sadly this wasn’t proven out in this case
found by anyone doing review of the changes upstream or by the various distros
like Debian / Fedora / OpenSUSE / Arch or even Ubuntu when incorporating this
new version into their repos - but instead was found by Andres Freund, one of
the maintainers of PostgreSQL, when they were looking to benchmark some new
changes scheduled for the next PostgreSQL release
this new version into unstable a few weeks ago, and wanted to get the
performance noise floor of the system as low as possible before doing
benchmarking of PostgreSQL - noticed large transient CPU spikes in sshd and
then eventually weird memory errors in sshd due to bugs in the initial version
of the backdoor
culprit and appeared to contain some very strange code to hook into the
authentication process of sshd when it was launched via systemd
of xz-utils on Github
lookup symbol names at runtime, rather than directly encoding them in the
exploit binary, presumably to try and make reverse engineering of the binary
harder (since you can’t just run strings on it and get any real sensible output)
the backdoor itself, as well as the process taken by the attacker to
incorporate this into the xz-utils project - both from a community point of
view and a technical point of view
attacker to get pre-authentication remote-code execution in sshd via a
specific private key when connecting to the server - NOBUS
particular matching private key from the client - and if found would then
proceed to execute arbitrary commands specified by the client without
requiring usual authentication
they don’t have the matching private key - so the impact of the backdoor is
somewhat limited
huge number of distros, means they essentially wanted a backdoor into any
Linux machine in the future that only they could use
what their intended purpose was especially given this wide reaching goal of
getting this into all the major Linux distros - but it would be just
speculation
to look at the timeline as it concerned Ubuntu
publicly disclosed, and as soon as we heard of the possible backdoor, and
realised that it only affected the version in-development noble-proposed we
quickly notified the Archive Admin team who then deleted it from
noble-proposed on 29th March, neutralising the main threat
approach - not only have we removed this version from noble-proposed as soon
as we became aware, we are then rebuilding every binary package that got built
since that compromised version was in noble-proposed originally - out of an
abundance of caution - we don’t have any information that says this backdoor
was doing anything other than what the various writeups have found so far, BUT
we also can’t be certain that it didn’t have other functionality either - so
being very cautious and rebuilding everything that was itself built since 27th
Feb
by security researchers or maintainers but by a developer from Microsoft who
happened to be looking for the right things at the right time and decided to
be curious
packages to be stuck in noble-proposed and not in the release pocket, else
many Ubuntu users and developers would have been impacted if this had migrated
to the noble release pocket
OSS development practices and was quite persistent in trying to get this
incorporated into distros - even urging for this new version to be synced to
Ubuntu so that it would land in the upcoming noble release as recently as
Thursday last week, just hours before the public disclosure
and technical work to develop and hide the backdoor, but could then
interface with distros via their established practices to try and get them
to incorporate their new backdoored version faster than they may have
otherwise
well as the technical details of both the code used to incorporate the
backdoored code into the final liblzma binary during the build process, as
well as the details of the backdoor itself and how it operates at runtime
projects have long dependency chains - in this case openssh when integrated
with libsystemd which in turn used liblzma - and it is unclear who the
maintainers and authors are or what procedures are in place for vetting and
transferring ownership of OSS projects - all present significant challenges
for OSS
to get involved, which is what we saw in the aftermath - despite all the
advanced obfuscation techniques employed was able to be analysed in a matter
of days by the community working together - and to analyse it in a huge amount
of depth and in such an open way that it leaves little room for questioning
the validity of the assessment - anyone can double check the work and come to
the same conclusions
first against an OSS project but it is a wake-up call to the OSS and Linux
ecosystem
Get in contact