Since Linux 6.9, LUKS suspend stopped wiping disk-encryption keys from memory (mathstodon.xyz)

193 points by IngoBlechschmid 2 hours ago

84 comments:

by tombert a minute ago

I don't think this bothers me.

The only reason that I do the disk encryption is so that I don't have to worry about people going through my laptop to steal tax documents and/or credit card stuff when I sell the laptop. I of course also wipe the laptop too, but I figure that if the data is encrypted at the drive level then there's very little risk of anyone being able to use some kind of forensics tool and recover data.

by bbminner 30 minutes ago

I am far from a security expert, but from the number of "we missed a single line C check across files during refactoring" critical security bugs discovered on a regular basis these days, the whole premise of a "giant secure open source C codebase" seems questionable. It is not specific to C of course, but invariants are arguably even harder to enforce and track consistently (esp under changes to code) in C. Unsure if FP with invariants encoded in types is a practically feasible scalable solution either. Model checking? [LLM] fuzzing? Fewer primitives with clear boundaries? Is that how seLinux was "checked"?

by fsddfsdfssdf 3 minutes ago

While I can see the shortcomings of C and generally don't recommend it for new projects I don't see this particular bug as a good example of something Rust's borrow checker or some other language's type system will catch. I don't think even static analyzers can catch this.

It's basically something like this:

original: DoTheThing()

new: DoTheThingSlightlyDifferentButKeepMyCredentialsAlive()

fix: DoTheThingSlightlyDifferentButDoInFactNOTKeepMyCredentialsAlive()

In my experience a substantial portion of gnarly bugs come down to a violation of a high-level system invariant and those do not strike me as something that can be automated. Even with something like Lean you can prove your program satisfies certain properties but you need to have thought about those properties in the first place. The proof doesn't discover the invariant for you.

If you'd had thought about the relevant security property you could have written a regression test for it which is not hard. IMO the really hard part isn't expressing the implementation safely, but it's the realization that this was a property the implementation needed to preserve.

by WhitneyLand 8 minutes ago

The premise of a secure open codebase is fine.

The problem is being more auditable does not automatically make it more audited.

There have to be enough people with skill taking enough time to work on it.

by moritzwarhier 27 minutes ago

The whole premise of a "giant secure open source C codebase" seems questionable

Because code review is sometimes not much different from an idealized version of the halting problem, where you would have access to a formalized version of a specification.

In other words, there is no strict definition of what is a security issue.

by russdill 4 minutes ago

The lesson here is that if a feature (at a minimum) does not have a associated test case, it is not actually a feature.

by lazide 24 minutes ago

In open source, someone (many, many) someone’s can at least check.

Closed source…..

by Twirrim 12 minutes ago

Not sure why you're getting downvoted, this is the entire point of open source.

Does such a bug exist in Windows? OSX? Who checks? If someone finds the key in memory, can they tell what conditions might be causing it and where?

Their only recourse under those situations is to hand it off to the OS Vendor and trust that what they implement does solve the problem, and trust that it wasn't a deliberate back-door that is now being replaced by another back-door.

by charcircuit 9 minutes ago

Security researchers find security bugs in closed source operating systems all of the time.

by lazide 4 minutes ago

Yup, it’s just harder to know for sure.

by kokada an hour ago

While it is certainly an interesting bug, I kinda feel that the title is click bait? Because this `cryptsetup luksSuspend` from what I understood is not really officially supported but an extension done in Debian, so if anything this regression only affected Debian? I am not sure if you can blame the kernel for something that is not supported or even widely tested.

I still find this impressive, and it is nice that we now have a test (NixOSTests BTW are awesome, I agree with OP) to avoid this regression from coming back. But from the title it seems to be a widespread issue, not something that affects only one Distro.

by IngoBlechschmid 37 minutes ago

Sorry, aimed for a technically precise title and didn't want to bait clicks.

Yes, this does not affect people on stock configurations for the plain reason that they wouldn't expect the volume key to be safe during suspend anyway.

Debian's solution was ported to several (most?) other distributions and I guess quite a few people maintained private ports.

The thread-keyring(7) manpage promises: "A thread keyring is destroyed when the thread that refers to it terminates." For their key upload (from userspace to kernelspace) mechanism, the cryptsetup project relied on this property; but kernel 6.9 introduced a regression invalidating this property.

by bitbasher 2 hours ago

I don't see any other way? When you sleep (suspend to RAM), everything is stored in RAM and is encrypted but the master key is present in kernel memory (if I recall correctly).

However, if you hibernate (suspend to disk) the entire contents of RAM (including the master key) is written/encrypted to disk and the RAM is cleared.

When you wake the machine up you have to re-enter the passphrase to decrypt the master key to re-load disk contents back to memory.

by IngoBlechschmid an hour ago

Yes, if you simply suspend your laptop on most stock Linux distributions, then everything including the master key is still kept in memory. But Debian pioneered the (optional) cryptsetup-suspend addon. This issues a luksSuspend command which is supposed to wipe the key from memory, and on resume asks you to resupply your passphrase.

Up to kernel 6.8, this worked as described; starting with kernel 6.9, it silently didn't.

by herywort an hour ago

So you would still be asked for a passphrase, even though it's already available?

by IngoBlechschmid an hour ago

Exactly. Cryptsetup wouldn't know about the extra copy of the volume key in kernel memory. Which is why, dramatically, it appeared secure ("surely I wouldn't be asked to resupply the passphrase if the volume key is still in memory, right?").

by pedrocr 30 minutes ago

It was still more secure than the default if I understand this correctly. On resume from suspend the laptop would still be locked by the encryption key and without access to the disk even if you can somehow circumvent the lock. The only insecurity was that somewhere in the kernel memory the key still exists so if you can somehow extract that from the live system you can unlock it.

by IngoBlechschmid 9 minutes ago

Yes, you are right: LUKS encryption protests your data at rest. An attacker which steals your disk can only gain little, like the information that you have used LUKS (unless you put your LUKS headers elsewhere, separated from the disk) and perhaps disk and disk sector usage statistics.

by naturalmovement an hour ago

FYI: VeraCrypt is not the defacto encryption software for Windows.

by IngoBlechschmid an hour ago

Oh, which one is it?

(You don't mean BitLocker, right?)

by naturalmovement an hour ago

It absolutely is and they have most the enterprise market.

by IngoBlechschmid an hour ago

Okay, yes, sure. It definitely is the most-used encryption software for Windows.

But I would never trust it a second, being property and known for issues. You likely know that, but for the benefit of others:

38C3 - Windows BitLocker: Screwed without a Screwdriver https://media.ccc.de/v/38c3-windows-bitlocker-screwed-withou... https://www.youtube.com/watch?v=5eNtT2p12cM

by noinsight 23 minutes ago

If you’re at all serious about security and not user convenience, you deploy BitLocker with a PIN instead of TPM only. And then a whole class of vulnerabilities goes away.

by bri3d 25 minutes ago

The issues you linked with BitLocker are obvious properties of BitLocker-with-SecureBoot-only architecture. If you configure Linux that way, you get similar issues (for example, it's pretty easy to mis-configure TPM sealed disk encryption on Linux to still allow a recovery shell, which will run with the disk unsealed).

BitLocker with a password (the equivalent of the LUKS configuration in question) does not share these issues.

by saidnooneever 38 minutes ago

veracrypt lost their drivers license so afaik you should avoid it since it cannot update its drivers any longer. didnt see any news about them reacquiring that license

by snailmailman 29 minutes ago

Assuming this is what you are referring to, it was resolved within a few days. The incident being resolved just didn't make headlines. https://sourceforge.net/p/veracrypt/discussion/general/threa...

by nacs an hour ago

Reminder that by using Bitlocker, you're using a closed source encryption for which Microsoft will happily hand out your recovery key on request.

https://www.forbes.com/sites/thomasbrewster/2026/01/22/micro...

by andrewpiroli an hour ago

Only if you store your key with Microsoft, which is not required or the default if you're using a local account which I assume most privacy sensitive people are.

by gruez 31 minutes ago

Not to mention that unless the bitlocker activation flow changed recently, it specifically asks you how to store your backup keys, with a choice given been local options (eg. usb drive, printing it off, etc.) and saving it to your microsoft account.

by briHass an hour ago

Bitlocker can use keys that are local only, but the default for home editions of Windows was to use the online account to back it up.

'Happily' is also a stretch, as they really don't have a choice if served a valid court order.

If you want encryption that is safe from the US government, keys need to be stored in your head. Anything physical is subject to court orders.

by john_strinlai an hour ago

for enterprises, where this doesn't really matter, bitlocker is great.

by dijit an hour ago

if by "great" you really mean "fine".

It's still brittle, awkward and puzzlingly awful UX despite being the literal standard for the platform.

Compare it to any of the actively maintained alternatives, Filevault for MacOS (which is wonderful and never sends your key to be kept somewhere else) or LUKS on Linux.. heck, even Veracrypt is actually easier to understand and more robust.

by john_strinlai an hour ago

>if by "great" you really mean "fine".

no, i mean great.

managing a fleet of 100+ laptops with bitlocker is a breeze. its so seemless that the users don't even realize its enabled (i.e. no UX issues, at all).

on the other hand, i am not managing 100+ laptops that use veracrypt. sounds absolutely awful. i've never managed an apple fleet, so i can't speak to that, and will take your word on it.

for personal use, i do not recommend bitlocker (or windows, really), but for already-windows enterprises? absolutely

by dijit 31 minutes ago

Flicking a button to turn something on is not what I'm talking about, that's normally the easy part of any setup, and I judge people harshly who only take that aspect of something into consideration when discussing systems.

Brittle is what happens when you haven't logged on to the machine in 60 days, trust with AD is broken, TPM has a glitch and wipes the in device key and forces you into recovery... or god forbid you service the laptop and now you have to enter recovery mode.

Then you're in a nightmare, trying to give someone a super long passphrase over the phone is a not-too-uncommon occurance.

That's assuming you have a good policy for storing the recovery keys. Too loose and they're handed out to everyone, sort of defeating the purpose: too strict and you need the IT department (or specific members), and its still predicated on the notion that you have a policy for it... Given that Admins are a dying breed... I don't think this is workable.

If you compare with Filevault on MacOS: which tracks the credentials of the logged in user; there's no "issue" if the device loses trust because ultimately you always use the real unlock key: not something cached in a "secure storage".

by bri3d 21 minutes ago

Having dealt with FileVault in this context, it's also frustrating; it's really common to have it fail to follow the logged-in user's credentials, and if you use any kind of federated login, you will frequently get users with FileVault passwords that are either ahead of or behind their system login password.

I think both approaches are valid trade-offs and I think that the default Secure Boot BitLocker configuration, for all its architectural tradeoffs, can probably be credited for an enormous amount of data loss mitigation originating from used hard drives alone.

by john_strinlai 21 minutes ago

maybe i am missing something, but how did veracrypt solve all of the admin and policy issues you’re bringing up? (specifically for large enterprise fleets)

by dijit 17 minutes ago

If you use your key every day you tend not to forget it.

If I as an admin give you your key: it is “leaked” effectively.

by john_strinlai 15 minutes ago

>If you use your key every day you tend not to forget it.

hoping users don’t forget their password is a very weak policy.

specifically, the policy and admin points you brought up above, how does veracrypt solve them?

by dcrazy 10 minutes ago

Have you never gone on vacation and forgotten your daily-use password upon return?

by akerl_ 43 minutes ago

Managing an Apple fleet is similarly fine, and that includes using any of the MDM tooling that also does key escrow on enterprise Filevault devices.

by dcrazy 15 minutes ago

FileVault absolutely has an optional iCloud Keychain escrow. That’s how the “unlock with Apple Account” feature works. Apple doesn’t have the keys for iCloud Keychain, but it is still stored in iCloud.

by j16sdiz 29 minutes ago

> Filevault for MacOS (which is wonderful and never sends your key to be kept somewhere else)

Did you read the documentation?

https://support.apple.com/guide/mac-help/protect-data-on-you...

"iCloud account: Click “Allow my iCloud account to unlock my disk” if you already use iCloud. Click “Set up my iCloud account to reset my password” if you don’t already use iCloud."

https://developer.apple.com/documentation/devicemanagement/f...

"FileVault Full Disk Encryption (FDE) recovery keys are, by default, sent to Apple if the user requests them. Only one payload of this type is allowed per system."

by dijit 26 minutes ago

Can.

If you click "Allow my iCloud account to unlock my disk", your recovery key is escrowed to Apple, tied to your Apple Account.

If you don't select that option it never does.

I should have said "without your explicit permission", but I assumed we were all adults and understood that.

The main point is that it's using your account password to unlock, the recovery key is for if you forget your account password.

by dcrazy 13 minutes ago

No, you were just plain wrong. You said “never”, when in reality BitLocker and FileVault both have optional escrow.

by Arainach 39 minutes ago

Veracrypt is more difficult to set up - whether on one machine or a fleet. Bitlocker is a few buttons in the UI, configurable via Group Policy, and so much more.

What is brittle or awkward?

by dijit 36 minutes ago

"PLEASE ENTER YOUR BITLOCKER RECOVERY KEY"

Where is it?

A) Uploaded to microsoft

B) Somewhere in EntraID?

C) Somewhere in our onprem AD?

D) Written down on a scrap of paper when I set up the laptop

the fact that they never ask for the passphrase is a weakness of the system. Because now you have an extremely difficult situation as soon as you're off the happy path.

It's also like 64 characters alphanumeric with no capability to copy/paste.

Compare it to Vera/Filevault where the access key is the users passphrase. In MacOS it's literally your account password, which follows along with your in-OS account credentials.

by philipallstar an hour ago

Does that mean it's not the de facto standard on Windows?

by naturalmovement an hour ago

So exactly like FileVault?

by dist-epoch an hour ago

Both Intel/AMD CPUs produced in the last 5 years or so support full transparent (to the OS) memory encryption. So cold boot attacks are a thing of the past if you enable this feature (it's typically disabled because it reduces RAM speed by about 0.5%).

by johnathan101 an hour ago

This is one of those regressions that's easy to miss because everything still "works." Security bugs often don't announce themselves.

by IngoBlechschmid an hour ago

Right! Which is why integration tests for these kinds of features are all the more important.

It was also fun to write, and enabled git-bisecting to isolate the specific kernel refactoring which introduced this bug: https://github.com/NixOS/nixpkgs/pull/532499

by CodesInChaos an hour ago

I don't have to re-enter my boot password after Sleep, so obviously the encryption key is still in memory.

by wrs an hour ago

Obviously your distro isn’t using cryptsetup-luksSuspend.

by unethical_ban an hour ago

Correct.

The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?

by ksbd-pls-finish an hour ago

Because debian users with luks-suspend have to re-enter their boot password after sleep.

by weaksauce 42 minutes ago

> The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?

If that was the case for the people using the debian extra secure extension that should have wiped the memory clean then someone would have found this bug much earlier than two years. Their password was required to be re-entered even though the key was still in memory somewhere.

by akerl_ 42 minutes ago

The reason this bug is unexpected is that the user is expecting to have to enter their password (because they expect the key to be wiped on suspend), and then _they are_ asked for their password. But there was a copy of the key elsewhere in kernel memory that was never cleared.

by killerstorm 29 minutes ago

Well, potentially a key might be stored in TPM. But I don't think that's better

by fpoling 39 minutes ago

On my laptop with Fedora I just configured Linux to hibernate to disk after 15 minutes of suspend. Powering memory off ensures that bugs like this Debian-specific would not matter.

Plus what Debian extension to Linux tooling does although nice in theory, but in practice if one really worries about cold-boot attacks, then all keys and important documents has to be wiped out from memory, not only LUKS keys.

So hibernating is really the only proper way to protect against cold boot.

by IngoBlechschmid 28 minutes ago

> So hibernating is really the only proper way to protect against cold boot.

I agree; or resurrecting FridgeLock: https://www.sec.in.tum.de/i20/publications/fridgelock-preven...

by killerstorm 33 minutes ago

Hmm, where does it get a key to decrypt memory on resume?

AFAIK it's practical only if you make use of TPM. And if you do, you're basically at mercy of TPM.

by teravor 32 minutes ago

    > where does it get a key to decrypt memory on resume?
you enter it...
by teravor 35 minutes ago

on the subject of encryption keys and memory there is something you can do:

- if your CPU supports it, enable memory encryption.

- if your TPM module supports this look for MemoryOverwriteRequestControl & MemoryOverwriteRequestControlLock (/sys/firmware/efi/efivars/) and toggle them. make sure that your computer always reboots and never powers off. memory will always be wiped on boot.

by deng an hour ago

> Except that, for more than two years, the encryption key remained resident in memory across suspend, leaving it there for the taking by anyone who seized the still-powered laptop.

I don't get it. Obviously, the laptop is locked when it resumes, how is that key "for the taking by anyone"? I'm not saying it is impossible to read out RAM from a locked laptop, but surely not by "anyone".

by nicce an hour ago

Anyone with physical access. I think it is understandable from the phrase.

There is a common misconception about how lock-screens in general work - they usually just prevents using the current hardware and software as it is to access the current OS. But the disk encryption is the main thing that prevents modification and other kind of access to actual data. And if the disk encryption key is lying in the memory, then effectively, the disk encryption is bypassed if someone can access the machine physically and assuming that there are no sufficient tampering protections in place for that machine.

by acdha an hour ago

Anyone with physical access, significant tools, and experience. The FBI has people who can pull data out of memory after freezing the RAM but the average laptop thief doesn’t so how serious this is depends significantly on your threat model. If you’re not a major criminal, bitcoin whale, or intelligence target this is almost certainly academic.

by deng 40 minutes ago

> If you’re not a major criminal, bitcoin whale, or intelligence target this is almost certainly academic.

Thanks, that's what I thought.

by deng an hour ago

> Anyone with physical access. I think it is understandable from the phrase.

Sorry, I'm probably dense, I still don't get it. You steal a laptop, you open it, the screen is locked with a password/fingerprint whatever. How do you read out the RAM from that laptop?

by IngoBlechschmid an hour ago

Several options. One is you restart and boot from a live system where you are root, and then dump all memory. This is described in the paper with the witty title "Lest We Remember: Cold Boot Attacks on Encryption Keys":

https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...

Other options: DMA attacks. Also you never know what the Intel Management Engine hidden in your computer is doing. It's running a version of Minix you don't have any control over, and it has full access to memory.

by john_strinlai an hour ago

>How do you read out the RAM from that laptop?

the term to look up is "cold boot attack" (https://en.wikipedia.org/wiki/Cold_boot_attack).

tons of cool live demonstrations of how it works on youtube if you've got the 20-40 minutes to spare

by deng an hour ago

Still, this is a pretty crazy definition of "anyone".

by jakewins an hour ago

There are attacks that allow dumping RAM if the device is powered on though and you have physical access. Depending on config it may be very easy (just plug in a dumper over Thunderbolt on USB C and do direct memory access) or hard (freeze and swap physical RAM to an unlocked machine).. but the idea was defense-in-depth here; a well configured device should both be hard to dump RAM on and it should not give encryption keys if an attacker succeeds.

by saidnooneever 36 minutes ago

you dump the physical memory, then decrypt the disk offline

by naturalmovement an hour ago

Definitely not a symptom of Linux being a hodgepodge of code thrown together from a thousand different sources and no one person could tell you how it all fits.

by cevn an hour ago

Bugs happen in all code. The difference is, anyone can fix stuff in open source. Closed source bugs are out of control and must be worked around. Usually by switching to OSS

by steve918 an hour ago

I wonder if you think other OSes are any different?

TempleOS is the only thing that comes to mind that doesn't fit your description and it's not practically useful.

Any sufficiently large codebase is a mix of ideas and concepts implemented by different people with different priorities over a large timespan and if you can fit the entire thing in your head it's not very interesting or complex.

by IngoBlechschmid an hour ago

Qubes OS, the Linux distribution aspiring to offer a reasonably secure operating system, pioneering a "every app runs in a virtual machine" approach in the Linux laptop/desktop space, tracks this at the following issue:

https://github.com/QubesOS/qubes-issues/issues/2890

by saidnooneever 28 minutes ago

QubesOS is Xen based. Not Linux.

by naturalmovement an hour ago

The *BSDs, Mac, and Windows all keep critical code in the same tree as the OS.

Something like disk encryption would be immediately visible.

So you don't have this mess of 80 different distros with 60 different versions of systemd, 20 that don't use it, a million kernel versions and it's all thrown together in a Costco-sized trash bag and we call the output "Linux".

by yaris an hour ago

In my experience any software system (not just operating system) after crossing a certain limit on complexity and age looks exactly as hodgepodge of code pieces thrown together, sometimes from different sources even if developed by one org. All major OSs have long crossed those limits, I believe.

by brainwad an hour ago

Windows for ages did not really keep all the code in one repo. There were like a dozen parallel repos for e.g. the shell, kernel, IE, etc. Also every feature was developed on team-level branches; integrating all those branches often caused unexpected bugs.

by stackghost an hour ago

Of course it's (indirectly) a symptom of that.

What's the alternative? Proprietary closed-source operating systems owned by corps who can be compelled to insert covert backdoors?

If BSD was as popular as Linux it would have the exact same problems.

by dist-epoch 42 minutes ago

"Mythos, find me a bug in LUKS. I know there is one in there".

Data from: Hacker News, provided by Hacker News (unofficial) API