LUKS volumes are not recognised when "Entire disc" is checked. was:Drive appears unformatted in Windows 8.1 x64 after mounting

[ghost]

I created a whole drive LUKS container for my USB drive on Ubuntu 14.04 formatted as NTFS, there are no partitions on the drive, inside or outside the container. The container mounts successfully in LibreCrypt by going to File -> Linux container -> Open LUKS partition and checking the "entire drive" checkbox. It assigns the drive letter but windows thinks the drive is unformatted. Any ideas? When I do a LUKS info dump I get this:

Application version : v6.2.5613.42403
Driver ID : v5.00.0000
Unable to read LUKS header; this does not appear to be a LUKS container.

[linux-modder]

luks itself does not format the container or any partition with a OS readable Filesystem, You mention formatting in ntfs , may I ask how it was formatted aka like with gparted, parted, mkfs.$fs etc. and so I can attempt to Reproduce what version of windows did this occur on ?

[t-d-k]

Hi @bparker06 , thanks for reporting this.
There have been other reports of problems on Windows 8.1, but I don't have an 8.1 system to test on, so am limited in the help I can give.
I think the most likely reason for windows not detecting the file-system is if you are using LVM. There is no LVM driver in windows.
Can you test this and other filesystem problems, by reformatting in Windows, and then trying to mount on Linux?
thanks
tdk

[ghost]

I am not using LVM and I formatted the container with mkfs.ntfs -Q
/dev/mapper/, but I can try reformatting under Windows. The drive
has a GPT label and physically it is a 1TB msata ssd in an external USB 3.0
enclosure.

Yesterday after I could not see the data under Windows, I moved back to a
Linux machine and it mounted just fine.
On Aug 3, 2015 6:20 AM, "tdk" [email protected] wrote:

Hi @bparker06 https://github.com/bparker06 , thanks for reporting this.
There have been other reports of problems on Windows 8.1, but I don't have
an 8.1 system to test on, so am limited in the help I can give.
I think the most likely reason for windows not detecting the file-system
is if you are using LVM. There is no LVM driver in windows.
Can you test this and other filesystem problems, by reformatting in
Windows, and then trying to mount on Linux?
thanks
tdk

—
Reply to this email directly or view it on GitHub
#30 (comment).

[AgentOak]

I can confirm this issue. I'm on Windows 7 x64.

I have a hard drive set up the same way. I have a LUKS container on the raw drive, without partitions. Inside the LUKS container is just an NTFS filesystem.
When I try to mount it using "Mount a disk or partition based encrypted container" and select the entire disk, the menu where you type in the passphrase looks different from LUKS containers which are on an actual partition, therefore I believe LibreCrypt doesn't recognize the disk as a LUKS container.
Using File -> Linux container -> Open LUKS partition I get the same behavior @bparker06 described. It mounts "something" instantly (which is unusual because it normally takes 2-3 seconds to mount a partition for me) but it is just a raw disk and Windows says it needs to be formatted.

FreeOTFE opened the exact same LUKS container just fine.

[linux-modder]

so this is now confirmed on win 7 and 8.1 64 bit. Can one or both of you indicate what happens if a windows OS creates the ntfs formatting inside the crypt?

[AgentOak]

Interesting behavior there. I just tried it with a small USB drive. I created the LUKS container on Linux (Kernel 4.1.3-2-MANJARO + cryptsetup 1.6.7 to be precise) using the following commands:
# dd if=/dev/zero of=/dev/sdb bs=4M count=32
(to wipe the existing partition layout and headers off the device)
# cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 luksFormat /dev/sdb
(note I'm creating the LUKS container on the raw device, no partitions being used)

If I try to mount that entire device using the Mount a disk or partition based encrypted container-Button LibreCrypt shows a generic passphrase dialog (it doesn't say LUKS in the window title as it usually would). If I type in my passphrase it yields a generic error about my passphrase possibly being wrong.

Then I mounted the device using File -> Linux container -> Open LUKS partition and let Windows format the appearing drive with NTFS. I can store files on it and they keep intact after unmouting & mounting the drive using the same menu option again. However Linux cannot read the drive anymore.
# cryptsetup luksDump /dev/sdb
Device /dev/sdb is not a valid LUKS device.
I'm not sure what's going on there. Neither Windows' partition manager nor Linux can see the NTFS partition directly, so LibreCrypt seems to be doing something, but not proper LUKS, that's for sure.

I don't believe this issue is related to NTFS or the filesystem inside the container at all. It seems LUKS containers can only be mounted successfully when LibreCrypt detects them on its own. However it can only detect them if they are actual partitions and not when the LUKS container is directly on the disk.
I just tried the following on said Linux system (on the same usb drive):
# dd if=/dev/zero of=/dev/sdb bs=4M count=32
# fdisk /dev/sdb
(Created a GPT partition table, then a single partition of type 'Linux')
# cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 luksFormat /dev/sdb1
(Note I'm now creating the LUKS container on a partition)
# cryptsetup open /dev/sdb1 testcontainer
# mkntfs -Q /dev/mapper/testcontainer
A LUKS container created this way opens fine on Windows 7 x64 LibreCrypt. Doesn't even matter if you use the Mount a disk or partition based encrypted container-Button or File -> Linux container -> Open LUKS partition. In both cases the same Enter LUKS passphrase dialog pops up and sucessfully mounts the NTFS partition which was created on Linux.

[linux-modder]

@Marco01809 thanks. @t-d-k that would almost sound like a ntfs-3g issue on linux side , I will take a look tomorrow before heading off for my conference. @Marco01809 will keep you up to date on changes you can try as I personally have no windows h/w handy.

[t-d-k]

This looks like an issue recognising the LUKS header, probably introduced with the GPT feature. Likely you will find it works on whole disks without GPT.
Unfortunately LibreCrypt doesn't make it clear atm when a volume is being opened as plain dm-crypt vs. LUKS. This can be confusing as any data can be opened as dm-crypt, and it uses the same menu items for dm-crypt and LUKS.
This will be improved in 6.3, which should make these issues clearer.
Meanwhile I'll try to reproduce the problem.
Can I confirm @bparker06 and @Marco01809 , you've only had this issue with opening whole-disks which use GPT?

@linux-modder if you have no windows box available why are you assigning windows issues to yourself? How do you hope to reproduce them, let alone fix them ?:-/

[AgentOak]

No, that's a little misunderstanding. The issue only occurs when not using a partition table, i.e. when you have created the LUKS container over the whole disk device without partitioning the disk before (e.g. cryptsetup luksFormat /dev/sdb). Mouting LUKS containers from both MBR and GPT partitions works (e.g. cryptsetup luksFormat /dev/sdb1).

In case it wasn't clear: I no case did I create a GPT partition table or any kind of partitions inside the LUKS container. Inside the LUKS container I always directly have an NTFS filesystem.

To create a LUKS container with which the problem can be reproduced, just execute the commands I've used in my previous post to create a LUKS container at disk device-level:
# dd if=/dev/zero of=/dev/sdX bs=4M count=32
# cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 luksFormat /dev/sdX
Optionally, create an NTFS partition inside it (It doesn't really matter as LibreCrypt won't mount it properly either way. Windows will always complain the disk needs to be formatted after LibreCrypt mounted it):
# cryptsetup open /dev/sdX testcontainer
# mkntfs -Q /dev/mapper/testcontainer
As can be seen, no partitions are used at all. Note that FreeOTFE can handle this, only LibreCrypt fails by not recognizing it as a LUKS container, making it impossible to properly mount it.

Or put another way: The "Entire drive"-Option isn't working for LUKS containers in LibreCrypt, which is a regression compared to FreeOTFE 5.21.

[AgentOak]

So I was poking around in the LibreCrypt v6.2-beta source code for a bit but couldn't really find the problem, just wanted to share what I found out so far. All of the following are just vague guesses as I'm not familiar with either Pascal or Delphi, even less so with the internals of FreeOTFE/LibreCrypt.
I found the IsLUKSVolume() function in OTFEFreeOTFEBase_U.pas, which compares the first bytes of the selected volume with the LUKS_MAGIC "LUKS\xba\xbe". I counterchecked that with a LUKS container created the way I described in my previous post:
# hexdump --length 16 -C /dev/sdb
00000000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....|
As this seems right, I doubt IsLUKSVolume() is causing the problems itself, I believe the filename being passed into it is wrong when using the EntireDisk-Option. It is constructed using SDUDeviceNameForPartition() found in SDUGeneral.pas. This in turn creates the filename string from "\Device\Harddisk%d\Partition%d". With EntireDisk enabled the PartitionId becomes 0 (see GetSelectedDevice() in TfmeSelectPartition.pas). So for my usb drive it would return something like "\Device\Harddisk3\Partition0", which I believe could be wrong, because indexing starts at 0, so Partition0 would point to the first partition on the disk, which is impossible because there are no partitions at all. Could it be that when using EntireDisk it should be just "\Device\Harddisk3"? Note that's just a wild guess, I don't actually know anything about windows drive APIs.

[t-d-k]

Hi, I've been able to reproduce the problem in Windows 7.
Interestingly, if you create a LUKS container on the raw device (as in @Marco01809's second comment above) then although it won't open if the 'Entire Disc' checkbox is checked, it does open if it is not checked, and the disc is selected.
So, this should give you a workaround for now.
@Marco01809 - Thanks for the analysis, I think you are right: "\Device\Harddisk3\Partition0" doesn't look like a proper identifier for the whole disc - it should be something like ".\PhysicalDriveX".
Not checking 'Entire Disc' probably works because the partition information is initialised to 0, so it is opening a partition that starts at the start of the disc.

[ghost]

@t-d-k how do you open it without checking 'entire disc'? The OK button is greyed out for me in that case. I also just upgraded to Windows 10 and the issue persists there.

[AgentOak]

I'm having the same problem as @bparker06. For usb drives Windows 'finds' a RAW partition. If i select it in LibreCrypt I can indeed mount it succesfully. However for SATA HDDs Windows doesn't provide RAW partitions, so there is nothing to select.

[ghost]

Just FYI I am using a USB 3.0 enclosure for a 1TB mSATA SSD.

[t-d-k]

@bparker06 There should be a light green area, if you click it, it should turn dark green and the OK button should become enabled.
@Marco01809 Do you mean a drive connected directly to the motherboard SATA connection is different to one via a USB adapter?

[ghost]

@t-d-k The green area only appears when checking 'entire drive'... but the workaround was stated to be that you had to click OK without checking 'entire drive', however the OK button is greyed out for me in that case... so I'm still not able to mount my drive in any way on Windows.

[AgentOak]

I made screenshots to demonstrate the problem:

This is how it looks like for my usb flash drive. I cannot mount it using EntireDisk, but I can when selecting the green partition (Windows reports it as a RAW partition, LibreCrypt reports it as "Small FAT16").

However for my internal SATA HDD there is no such partition that I could select:

@bparker06 must be experiencing the same behavior for his USB3-enclosed mSATA SSD.

EDIT: Here is a screenshot of how windows displays those two devices in the Disk Management:

They we're both created using the same commands, i.e. creating the LUKS container on device-level, no partitions used. Windows actually differentiates between hard disks and removable media - removable drives are considered to have a RAW partition if Windows doesn't recognize a partition table, hard disks however don't.

[t-d-k]

@Marco01809 Thanks for that. Unfortunately all the discs I have to test with are removable ones and show a raw partition, so I can't reproduce it at the moment.

[AgentOak]

Is there any update on this or do you have an ETA? This is a major issue for me since I have to switch between FreeOTFE and LibreCrypt fairly often to be able to access all of my disks. Not to mention I have to reboot multiple times because LibreCrypt disables TESTSIGNING mode during the uninstallation even if LibreCrypt wasn't the one to enable it during the installation.

[t-d-k]

Hi, sorry I've been busy so I haven't been able to investigate very much.
Opening the harddisc as a whole works if you use direct Windows calls to access the disc, but not through the LC driver. The next step is to try LC with FreeOTFE drivers and isolate if it is a change in the drivers for LC, or something else.
So far as rebooting - LC should work with the FreeOTFE drivers, so you can install FreeOTFE and then use LC portable and shouldn't have to reboot unless you want to access a GPT disc.

[t-d-k]

Just an update:
I've tested with FreeOTFE and it seems to behave identically to LC for the 'whole disc' with removable discs. So it seems there are two bugs; one applies to removable discs and one internal discs, with the internal disc one being introduced with LC.
Using Windows API calls to access the whole disc seems to read the LUKS header OK, so this is bug dependant on #38 . If we move to using windows API calls for all disc access then that should automatically fix this as well.