Very long post ahead (sorry for the wall of text), part of a series of posts on some sysadmin topics, see post 1 and post 2. I want to show you how I set up my tiny dedicated server to have encrypted partitions, and to reinstall it from scratch. All of this without ever accessing the actual server console.

Introduction

As much as my provider may have gold standards on how to do things (they don't, there are some very bad practises in the default installation, like putting their SSH key into root's authorized_keys file), I wouldn't trust an installation done by a third party. Also, I wanted to have all my data securely encrypted.

I know this is not perfect, and there are possible attacks. But I think it is a good barrier to have to deter entities without big budgets from getting my data.

I have done this twice on my servers, and today I was reviewing each step as my friend was doing the same thing (with some slight differences) on his brand new server, so I think this is all mostly correct. Please, tell me if you find a bug in this guide.

This was done on my 12 £/month Kimsufi dedicated server, sold by OVH (see my previous post on why I chose it), and some things are specific to them. But you can do the same thing with any dedicated server that has a rescue netboot image.

The process is to boot into the rescue image (this is of course a weak link, as the image could have a keylogger, but we have to stop the paranoia at some point), manually partition the disk, set-up encryption, and LVM; and then install a Debian system with debootstrap.

To be able to unlock the encrypted disks, you will have to ssh into the server after a reboot and enter the passphrase (this is done inside the initrd phase). Once unlocked, the normal boot process continues.

If anything fails, you end up with an unreachable system: it might or might have not booted, the disk might or might not be unlocked, etc. You can always go back into the rescue netboot image, but that does not allow you to see the boot process. Some providers will give you real remote console access, OVH charges you silly money for that.

They used to offer a "virtual KVM", which was a bit of a kludge, but it worked: another netboot image that started a QEMU connected to a VNC server, so by connecting to the VNC server, you would be able to interact with the emulated boot process, but with a fake BIOS and a virtual network. For some unspecified reason they've stopped offering this, but there is a workaround available. The bottom line is, if you have some kind of rescue netboot image, you can just download and run QEMU on it and do the same trick.

Update: Online.net (which has a similar offering to OVH/Kimsufi) now provides serial console access, which is way more convenient than the virtual KVM, but note that it will be attached to the second serial port (ttyS1). This drove me crazy for hours!

The gritty details

Start by netbooting into your rescue image. For OVH, you'd go to the control panel, in the Services/Netboot section and select "rescue pro". Then reboot your server. OVH will mail you a temporary password when it finishes rebooting.

Connect to it, without saving the temporary SSH key:

$ ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no root@${IP}

Note that you will need these packages installed in your rescue image:

cryptsetup lvm2 debian-archive-keyring debootstrap

If they are not present, now it is a good time to install them.

Disk partitioning and encryption

For the rest of the text, I am assuming you have one hard drive called /dev/sda. We start by partitioning it:

# fdisk /dev/sda

Start a new partition table with o, and then create two primary partitions: a small one for /boot at the beginning (100 to 300 MB would do), and a second one with the remaining space. Set both as type 83 (Linux), and don't forget to activate the first one, as this servers refuse to boot from the hard drive without that.

Create the file system for /boot, and the encrypted device:

# mkfs.ext4 -L boot /dev/sda1
# cryptsetup -s 512 -c aes-xts-plain64 luksFormat /dev/sda2

The encryption parameters are the same as the ones used by the Debian Installer by default, so don't change them unless you really know what you are doing. You will need to type a passphrase for the encrypted device, be sure not to forget it! This passphrase can later be changed (or secondary passphrases added) with the cryptsetup tool.

Look up the crypt device's UUID, and save it for later:

# cryptsetup luksDump /dev/sda2 | grep UUID:
UUID:           xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Open the encrypted device (type the passphrase again), and set up the LVM volume group:

# cryptsetup luksOpen /dev/sda2 sda2_crypt
# pvcreate /dev/mapper/sda2_crypt
# vgcreate vg0 /dev/mapper/sda2_crypt

Create the logical volumes, this is of course a matter of personal taste and there are many possible variations. This is my current layout, note that I put most of the "big data" in /srv.

# lvcreate -L 500m -n root vg0
# lvcreate -L 1.5g -n usr vg0
# lvcreate -L 3g -n var vg0
# lvcreate -L 1g -n home vg0
# lvcreate -L 10g -n srv vg0
# lvcreate -L 500m -n swap vg0
# lvcreate -L 100m -n tmp vg0

Some possible variations:

  • You can decide to use a ramdisk for /tmp, so instead of creating a logical volume, you would add RAMTMP=yes to /etc/default/tmpfs.
  • You can merge / and /usr in one same partition, as neither of them change much.
  • You can avoid having swap if you prefer.
  • You can put /home in /srv, and bind mount it later.

Now, create the file systems, swap space, and mount them in /target. Note that I like to use human-readable labels.

# for i in home root srv tmp usr var; do 
  mkfs.ext4 -L $i /dev/mapper/vg0-$i; done
# mkswap -L swap /dev/mapper/vg0-swap

# mkdir /target
# mount /dev/mapper/vg0-root /target
# mkdir /target/{boot,home,srv,tmp,usr,var}
# mount /dev/sda1 /target/boot
# for i in home srv tmp usr var; do
  mount /dev/mapper/vg0-$i /target/$i; done
# swapon /dev/mapper/vg0-swap

Don't forget to set the right permissions for /tmp.

# chmod 1777 /target/tmp

If you want to do the /home on /srv, you'll need to do this (and then copy to /etc/fstab):

# mkdir /target/srv/home
# mount -o bind /target/srv/home /target/home

Debian installation

The disk is ready now. We will use debootstrap to install the base system. The OVH image carries it, otherwise consult the relevant section in the Install manual for details. It is important that at this point you check that you have a good GPG keyring for debootstrap to verify the installation source, by comparing it to a good one (for example, the one in your machine):

# gpg /usr/share/keyrings/debian-archive-keyring.gpg
pub  4096R/B98321F9 2010-08-07 Squeeze Stable Release Key <debian-release@lists.debian.org>
pub  4096R/473041FA 2010-08-27 Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
pub  4096R/65FFB764 2012-05-08 Wheezy Stable Release Key <debian-release@lists.debian.org>
pub  4096R/46925553 2012-04-27 Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>

Now, for the actual installation. You can use any Debian mirror, OVH has their own in the local network. In OVH's case it is critical to specify the architecture, as the rescue image is i386. I didn't notice that and had to painfully switch architectures in place (which was absolutely not possible a couple of years ago).

# debootstrap --arch amd64 stretch /target http://debian.mirrors.ovh.net/debian

Or the more general case, using the httpredir service, which chooses your nearest public mirror:

# debootstrap --arch amd64 stretch /target http://httpredir.debian.org/debian

Post-install adjustments

After a few minutes downloading and installing stuff, you almost have a Debian system ready to go. Since this is not D-I, we still need to tighten a few screws manually. Let's mount some needed file systems, and enter the brand new system with chroot:

# mount -o bind /dev /target/dev
# mount -t proc proc /target/proc
# mount -t sysfs sys /target/sys
# XTERM=xterm-color LANG=C.UTF-8 chroot /target /bin/bash

The most critical parts now are to correctly save the parameters for the encrypted device, and the partitions and logical volumes. You'll need the UUID saved before:

# echo 'sda2_crypt UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks' \
  > /etc/crypttab

Create the file systems table in /etc/fstab. Here I use labels to identify the devices:

# file system   mount point type    options             dump    pass
LABEL=root      /           ext4    errors=remount-ro   0       1
LABEL=tmp       /tmp        ext4    rw,nosuid,nodev     0       2
LABEL=var       /var        ext4    rw                  0       2
LABEL=usr       /usr        ext4    rw,nodev            0       2
LABEL=home      /home       ext4    rw,nosuid,nodev     0       2
# Alternative home in /srv:
#/srv/home      /home       auto    bind                0       0
LABEL=srv       /srv        ext4    rw,nosuid,nodev     0       2
LABEL=boot      /boot       ext4    rw,nosuid,nodev     0       2
LABEL=swap      none        swap    sw                  0       0

You can also just use the device mapper's names (/dev/mapper/<volume_group>-<logical_volume>), be sure not to use the /dev/<volume_group>/<logical_volume> naming, as there are some initrd-tools that choked on them.

# file system           mount point type    options             dump    pass
/dev/mapper/vg0-root    /           ext4    errors=remount-ro   0       1
/dev/mapper/vg0-tmp     /tmp        ext4    rw,nosuid,nodev     0       2
/dev/mapper/vg0-var     /var        ext4    rw                  0       2
/dev/mapper/vg0-usr     /usr        ext4    rw,nodev            0       2
/dev/mapper/vg0-home    /home       ext4    rw,nosuid,nodev     0       2
# Alternative home in /srv:
#/srv/home              /home       auto    bind                0   0
/dev/mapper/vg0-srv     /srv        ext4    rw,nosuid,nodev     0   2
/dev/sda1               /boot       ext4    rw,nosuid,nodev     0   2
/dev/mapper/vg0-swap    none        swap    sw                  0   0

Some tools depend on /etc/mtab, which now is just a symbolic link:

# ln -sf /proc/mounts /etc/mtab

Network configuration

Now configure the network. You most surely can use DHCP, but you might prefer static configuration, that's personal choice. For DHCP, it is very straightforward:

# cat >> /etc/network/interfaces
auto eth0
iface eth0 inet dhcp

For static configuration, first find the current valid addresses and routes as obtained by DHCP:

# ip address
# ip route

And then store them:

# cat >> /etc/network/interfaces
auto eth0
iface eth0 inet static
    address AAA.BBB.CCC.DDD/24
    gateway AAA.BBB.CCC.254
    pre-up /sbin/ip addr flush dev eth0 || true

Note that pre-up command I added; that is to remove the configuration that is to be done by the kernel during boot (more on that later), otherwise ifupdown will complain about existing addresses.

If your provider does IPv6, add it too. For OVH, the IPv6 set-up is a bit weird, so you need to add the routes in post-up. Your default gateway is going to be your /64 prefix, with the last byte replaced by ff, and then followed by :ff:ff:ff:ff. As you can see, that gateway is not in your network segment, so you need to add an explicit route to it. They have some information, but it is completely unreadable.

If your IPv6 address is 2001:41D0:1234:5678::1/64, you will add:

iface eth0 inet6 static
    address 2001:41D0:1234:5678::1/64
    post-up /sbin/ip -6 route add 2001:41D0:1234:56ff:ff:ff:ff:ff dev eth0
    post-up /sbin/ip -6 route add default via 2001:41D0:1234:56ff:ff:ff:ff:ff

You probably don't want the auto-configured IPv6 addresses, so disable them via sysctl:

# cat >> /etc/sysctl.conf
# Disable IPv6 autoconf 
net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.eth0.autoconf = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.eth0.accept_ra = 0

To have a working DNS resolver, we can use the local server (OVH in this case):

# cat > /etc/resolv.conf 
search $DOMAIN
nameserver 213.186.33.99

The most important part of a new install: choose a host name (and make the system use it).

# echo $HOSTNAME > /etc/hostname
# hostname $HOSTNAME
# echo "127.0.1.1 $HOSTNAME.$DOMAIN $HOSTNAME" >> /etc/hosts

Mirrors and timezones

If we want to speficy the BIOS clock to use UTC:

# echo -e '0.0 0 0.0\n0\nUTC' > /etc/adjtime

Set up your time zone:

# dpkg-reconfigure tzdata

Configure APT with your preferred mirrors. I also prevent APT from installing recommends by default.

# echo deb http://httpredir.debian.org/debian stretch main contrib non-free \
  >> /etc/apt/sources.list
# echo deb http://httpredir.debian.org/debian stretch-updates main contrib non-free \
  >> /etc/apt/sources.list
# echo deb http://security.debian.org/ stretch/updates main contrib non-free \
  >> /etc/apt/sources.list
# echo 'APT::Install-Recommends "False";' > /etc/apt/apt.conf.d/02recommends
# apt-get update

Preparing for remote unlock

Before installing any package, let's make sure that the initial ram disk (initrd) that is going to be created will allow us to connect. There will be no chance of using the root password during boot. Your public key is usually found in $HOME/.ssh/id_rsa.pub.

# mkdir -p /etc/initramfs-tools/root/.ssh/
# echo $(YOUR_PUB_RSA_KEY) > /etc/initramfs-tools/root/.ssh/authorized_keys

Update: Note that for dropbear versions 2016.73-1 and up, the location for this file has changed. You will get errors about the authorized_keys file when you run update-initramfs -u if this is the case. The new command should be as follows:

# echo $(YOUR_PUB_RSA_KEY) > /etc/dropbear-initramfs/authorized_keys

If you change this, or the host key stored at /etc/dropbear/dropbear_*_host_key, the /etc/crypttab, or any other critical piece of information for the booting process, you need to run update-initramfs -u.

Now we can install the missing pieces:

# apt-get install makedev cryptsetup lvm2 ssh dropbear busybox ssh \
  initramfs-tools locales linux-image-amd64 grub-pc kbd console-setup

During the installation you will have to choose where to install grub, I recommend directly on /dev/sda. Also, the magic initrd will be created. We want to double check that it has all the important pieces for a successful boot:

# zcat /boot/initrd.img-* | cpio -t conf/conf.d/cryptroot \
  etc/lvm/lvm.conf etc/dropbear/\* root\*/.ssh/authorized_keys \
  sbin/dropbear | sort
conf/conf.d/cryptroot
etc/dropbear/config
etc/dropbear/dropbear_dss_host_key
etc/dropbear/dropbear_ecdsa_host_key
etc/dropbear/dropbear_rsa_host_key
etc/lvm/lvm.conf
root-l2oUNW/.ssh/authorized_keys
sbin/dropbear

All these files need to be there. Most critically, we need to check that the cryptroot file has the right information to access the root file system:

# zcat /boot/initrd.img-* | cpio -i --to-stdout conf/conf.d/cryptroot
target=sda2_crypt,source=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx,key=none,rootdev,lvm=vg0-root

If all that was correct, now we need to tell the kernel to configure the network as soon as possible so we can connect to the initrd and unlock the disks. This is done by passing a command-line option though grub. This should match what was done in /etc/network/interfaces: either DHCP or static configuration. For DHCP, this line should be changed in /etc/default/grub:

GRUB_CMDLINE_LINUX="ip=:::::eth0:dhcp"

For static configuration:

GRUB_CMDLINE_LINUX="ip=MY_IP_ADDR::MY_DEFAULT_GW:MY_NETMASK::eth0:none"

It is also a good idea to disable the quiet boot and graphical boot splash, in case we need to use QEMU to fix some booting issue:

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL=console

If you have a serial console, you can instruct grub and the kernel to use it, use these lines in the grub configuration instead:

GRUB_CMDLINE_LINUX="ip=MY_IP_ADDR::MY_DEFAULT_GW:MY_NETMASK::eth0:none console=ttyS1,9600"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=1 --speed=9600 --word=8 --parity=no --stop=1"

Note that this assumes the serial console is attached to the second port, and disables completely the normal VGA console. It might also be a good idea to enable getty to login over the serial console, but that is outside this guide's scope.

Now make the changes effective:

# update-grub2

Final touches

Having fsck fix problems automatically can be a life-saver too:

# echo FSCKFIX=yes >> /etc/default/rcS

Get some very useful packages:

# apt-get install vim less ntpdate sudo

Create an user for yourself, possibly make it an administrator:

# adduser tincho
# adduser tincho sudo
# adduser tincho adm

This is mostly done, exit the chroot, and unmount everything.

# exit  # the chroot.
# umount /target/{dev,proc,sys,boot,home,srv,tmp,usr,var}
# umount /target
# swapoff -a
# lvchange -an /dev/mapper/vg0-*
# cryptsetup luksClose sda2_crypt

Disable the netboot option from your administration panel, reboot, and hope it all goes well.

First boot

If you followed every step carefully, a few minutes later you should be able to ping your server. Use this snippet to enter the password remotely:

$ stty -echo; ssh -o UserKnownHostsFile=$HOME/.ssh/known_hosts.initramfs \
  -o BatchMode=yes root@"$HOST" 'cat > /lib/cryptsetup/passfifo'; \
  stty echo

It is very important that you close the pipe (with control-D twice) without typing enter. For my servers, I have a script that reads the passphrase from a GPG-encrypted file and pipes it directly into the remote server. That way, I only type the GPG passphrase locally:

$ cat unlock.sh 
#!/bin/sh
BASE="$(dirname "$0")"
HOST="$1"
gpg --decrypt "$BASE"/key-"$HOST".gpg | \
    ssh -o UserKnownHostsFile="$BASE"/known_hosts.initramfs -o BatchMode=yes \
        root@"$HOST" 'cat > /lib/cryptsetup/passfifo'

It might be a good idea to create a long, impossible to guess passphrase that you can use in the GPG-encrypted file, and that you can also print and store in somewhere safe. See the luksAddKey function in the cryptsetup(8) man page.

Once again, if everything went right, a few seconds later the openSSH server will replace the tiny dropbear and you will be able to access your server normally (and with the real SSH host key).

Recovery

As it happens, many times something will go wrong, and the server won't come back. In those situations, a serial or KVM console is very useful to see what is going on. But sometimes that is not enough; for example, because you made a mistake with the initrd or grub.

Both OVH and online.net provide "rescue" boot options, which basically netboot your server with some special image for performing troubleshooting. Here are some basic steps to quickly get back to your system from the recovery/rescue image.

If needed, install the required tools:

# apt-get install lvm2 cryptsetup

Unlock the encrypted partition:

# cryptsetup luksOpen /dev/sda2 sda2_crypt

Detect and activate LVM volumes:

# vgchange -aay

Mount the root and special filesystem:

# mkdir /target
# mount /dev/mapper/vg0-root /target
# mount -o bind /dev /target/dev
# mount -t proc proc /target/proc
# mount -t sysfs sys /target/sys

Mount the remaining filesystems using /etc/fstab:

# chroot /target mount -a

If that does not work, you can try the manual approach:

# mount /dev/sda1 /target/boot
# for i in home srv tmp usr var; do
    mount /dev/mapper/vg0-$i /target/$i; done
# swapon /dev/mapper/vg0-swap

Now you are ready enter the chroot, and work on your system:

# XTERM=xterm-color LANG=C.UTF-8 chroot /target /bin/bash

After you are finished, don't forget to cleanly unmount everything, as explained earlier.

Hope you find this article helpful! I would love to hear your feedback.


Updated 2017/02/06: I fixed a few more errors, as I was using this guide to install another server. Changed APT configuration to use the httpredir service, and default to the upcoming stretch release. Fix location for the authorized_keys file. Added recovery section.

Wow, that's a lot of partitions

I've been bitten by having lots of partitions in the past, and if one starts to get full then resizing them all to create room where it's eventually needed can be a bit of a pain.

So I now have 3 (plus /tmp)

/ - unencrypted (~20-30Gb) swap - encrypted (~4Gb) /tmp - tmpfs /var - encrypted (Rest of disk)

And then I have

/home - bind-mount to /var/local/home /srv - bind-mount to /var/local/srv /root - bind-mount to /var/local/root

So all the potentially sensitive data is on one giant encrypted partition that's as big as it can realistically get on the hardware I have.

The final change is to re-generate ssh host keys and other self-signed certs in /var/local/keys, and symlink them from their default locations in /etc so that they're also on the encrypted partition.

Comment by grok-mctanys [livejournal.com]
details, details...

you have in your post:

[...]

XTERM=xterm-color LANG=C.UTF-8 chroot /target /bin/bash

The most critical parts now are to correctly save the parameters for the encrypted device, and the partitions and logical volumes. You'll need the UUID saved before:


echo 'sda2_crypt UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks' \

/target/etc/crypttab

you just chroot'ed into the installed system, there's no need for the /target/ at the beginning of the file path.

Also, the IPv6 example with 'dead:beef' becoming 'dead:beff' seems like a type (even if you said the the last byte gets replaced by 'ff').

Maybe another \n at the end of:

echo -e '0.0 0 0.0\n0\nUTC' > /etc/adjtime

Comment by StyXman
details
Styxman, thanks for the errata, I will fix the '/target' thing, and try to clarify the ipv6 issue. The adjtime this is OK, as echo adds a '\n' at the end anyway.
Comment by Martín
partitions

grok-mctanys, note that I didn't use all the available space. i like to always leave plenty of unused space on LVM and then grow the partitions as the need arises.

Also, I don't like putting in /var important stuff, for me it is a partition that can die. Also, I don't see the point of dealing with links from /etc to /var, when you can just encrypt the whole thing...

Comment by Martín
Virtual KVM Workaround fails

Thanks for the immense instructions and write-up :)

Tried following along and I've obviously done something stupid somewhere as my box never comes back.

Was going to diagnose with the virtual-KVM but the url for the kvm.tgz returns a 404 error. Don't suppose you backed up the archive before it went missing?

Comment by Daniel
KVM image

Daniel,

Sadly, I don't have the image. But try the forums, I think someone had set up a mirror somewhere..

Comment by Martín
dropbear issues

Thank you so much for this great how-to! I have reproduced on my side, and it is almost perfect. I only had issues with dropbear that made me lose hours to sort out, hopefully this comment will save people's time.

I installed this on a debian 7 too, but somehow, dropbear is very sensitive to the arguments passed on the command line of the client. I spent hours looking for issues in the initramfs while it was perfectly OK, as described here. My ssh key was always rejected by dropbear because (I'm not sure exactly) either the order of arguments like BatchMode and the key file was wrong, or a first connection attempt was required without the BatchMode enabled. My advice would be: try connecting with the minimum arguments on the ssh client you can, and once you have it working, add the BatchMode. Launch dropbear manually before rebooting for nothing and verify that you are actually able to connect. OpenSSH keys work just fine with dropbear, no need to make new ones with dropbearkey because you can't connect.

Also, I can report that this installation can be done from an Ubuntu rescue, that's what my hoster provides. You may also want to install the cryptsetup 1.6 version in it, or in the OS installed in the first place on the server before shredding it, to run 'cryptsetup benchmark' and see what kind of performance you can expect from the server.

Anyway, muchas gracias, me ayudó mucho!

Comment by vinvin
dropbear issues

Thank you so much for this great how-to! I have reproduced on my side, and it is almost perfect. I only had issues with dropbear that made me lose hours to sort out, hopefully this comment will save people's time.

I installed this on a debian 7 too, but somehow, dropbear is very sensitive to the arguments passed on the command line of the client. I spent hours looking for issues in the initramfs while it was perfectly OK, as described here. My ssh key was always rejected by dropbear because (I'm not sure exactly) either the order of arguments like BatchMode and the key file was wrong, or a first connection attempt was required without the BatchMode enabled. My advice would be: try connecting with the minimum arguments on the ssh client you can, and once you have it working, add the BatchMode. Launch dropbear manually before rebooting for nothing and verify that you are actually able to connect. OpenSSH keys work just fine with dropbear, no need to make new ones with dropbearkey because you can't connect.

Also, I can report that this installation can be done from an Ubuntu rescue, that's what my hoster provides. You may also want to install the cryptsetup 1.6 version in it, or in the OS installed in the first place on the server before shredding it, to run 'cryptsetup benchmark' and see what kind of performance you can expect from the server.

Anyway, muchas gracias, me ayudó mucho!

Comment by vinvin
Impressive

Wow!

I've been spending days to find how to do what you just described here.

Thank you very much for this great guide!

I followed everything carefully and my server is booting properly.

Only problem is: the computer i use to SSH into my remote server runs on Windows and PUTTY.

I couldn't find a way to allow dropbear to allow my windows/putty setup to connect.

Could you help me please?

TIA

Kevin

Comment by Kevin
comment 10

Kevin:

It should be the same to connect with putty, although once you've logged in, you'll have to type the command by hand, like this:

echo -n "PASSWORD" > /lib/cryptsetup/passfifo

I don't think there is any big difference in PuTTY that will prevent you from connecting. The only issue will be the different SSH keys, as PuTTY will recognise they have changed since last boot.

Comment by Martín Ferrari
Putty

Martin:

Since i'm on windows, I did not have a public RSA key to add into "/etc/initramfs-tools/root/.ssh/authorized_keys"

So I downloaded puttygen to generate a public and a private RSA key.

Then I used the command "echo $(YOUR_PUB_RSA_KEY) > /etc/initramfs-tools/root/.ssh/authorized_keys" as described in the guide.

Then I setup putty to use the private key that was also generated with puttygen.

But when I connect to my server this way, I'm getting a "server refused our key" message after login prompt :(

Comment by Kevin
PS
PS: thanks again for your help :)
Comment by Kevin
comment 13

Hi again Kevin,

Did you run update-initramfs after adding your public key to dropbear?

You can also test by adding that same key to your user in some other machine and checking if you can log in without a password from PuTTY.

Comment by Martín Ferrari
Setup

Hey Martin :)

I started your guide from start again and managed to connect to the server :)

This is how I've done it:

  • generate a public key with puttygen on Windows
  • pasted the key into /etc/initramfs-tools/root/.ssh/authorized_keys (the "rsa-key-CURRENTDATE" at the end of the key must be removed manually else it won't work)
  • saved the associated private key with puttygen using a passphrase. It's important to use a passphrase else the server won't accept it.
  • connect to the server using putty and the private key file. Once connected, the prompt asks for the private key passphrase. Once put, I'm getting a busybox shell.

I still have 1 problem tho: I cannot find out how to unlock the server from there.

I tried putting the encrypted partition passphrase in /lib/cryptsetup/passfifo with every possible command like:

echo -ne "my luks encryption passphrase" > /lib/cryptsetup/passfifo

but then I wait and nothing happens :(

Any idea?

TIA :)

Comment by Kevin
Did some test

Hey martin :)

I tested your guide on a local Virtual Machine and it's working like a charm :)

When I'm on my remote server however, after i connect to dropbear and issue the command:

echo -ne "passphrase" > /lib/cryptsetup/passfifo

The SSH prompt is then not avariable (no more prompt line, like if the system was working) but nothing happens, even if I wait some time.

When i do ctrl+c, it says "interrupted system call".

I'll try to figure out what's wrong on my remote server.

Comment by Kevin
Woot

Dear Martin,

Ok I managed to get everything working :)

I think the problem was that I had an error message when doing update-initramfs-update.

I had to install the package for the network driver of my system (bnx2) before i could update it without the error.

I did a video on youtube to help other people:

https://www.youtube.com/watch?v=Su46AjtyTqw&feature=youtu.be

Thx again for your help ;)

Comment by Kevin
comment 17

Sorry Kevin for not answering before, I was offline for a few days.

I am glad you managed to fix the issue. Yes, the network driver is an important part, regardless of encryption and all that :-)

Best of luck!

Comment by Martín Ferrari
comment 18

hi... firstly the tut are very nice...

but i have a really strange error...

zcat /boot/initrd.img-3.2.0-4-amd64 | cpio -t conf/conf.d/cryptroot \ etc/lvm/lvm.conf etc/dropbear/* root/.ssh/authorized_keys sbin/dropbear

etc/lvm/lvm.conf

etc/dropbear/dropbear_dss_host_key

etc/dropbear/dropbear_rsa_host_key

sbin/dropbear

root/.ssh/authorized_keys

28311 Blocks


i tried to edit cryptroot with ->

vim /etc/initramfs-tools/conf.d/cryptroot

target=sda2_crpyt,source=UUID=.... and so on...

but decrypt doesn´t work... any idea?

Comment by fugaru
comment 19

@fugaru:

If the cryptsetup file is not being included, it is probably because the initramfs script does not think you need it to boot. Double check the device names in /etc/fstab, I remember having some issues with this, as there are many equivalent names you can use there: UUID=xxx, LABEL=xxx, /dev/mapper/volgroup-lv, /dev/volgroup/lv, etc. I'd recommend you trying to change that and running again update-initramfs until you find the right one. Please, comment back if you find the issue.

Comment by Martín Ferrari
comment 20

hi Martin,

thanks for your support. i´m still in testing...

i had changed the entries in /etc/fstab (UUID, /dev/mapper, /dev/volgroup, ...) but they don't resolve my problem.

next step. when i include "dm-crpyt" in /etc/modules the "zcat /boot/initrd.img-3.2.0.4..." command print the missing "conf/conf.d/cryptroot" to the shell.

etc/lvm/lvm.conf

etc/dropbear/dropbear_dss_host_key

etc/dropbear/dropbear_rsa_host_key

sbin/dropbear

root/.ssh/authorized_keys

conf/conf.d/cryptroot

Blocks 68389

annoyingly the Block entry too...it doesn´t matter?

btw. i had read a hint in the stack-network. that i should escape the special character like % $ @ ? when i type my password in dropbear... ( \% \$ \@ \?)

echo -n "password\%\$" > /lib/crypsetup/passfifo

do you think this is correct?

Comment by fugaru
comment 21

additional informations...

firstly, i use the "twofish-xts-plain64" cypher, because my atom cpu had no aes-ni instruction set.

secondly, i´ve created my partitions with parted.

parted /dev/sda mklabel gpt

parted /dev/sda gb mkpart 1 0 1

parted -s /dev/sda set 1 boot on

parted /dev/sda unit gb mkpart 2 1 100%

mkfs.ext4 /dev/sda1

Comment by fugaru
comment 22

sorry for comment spam...

grub2 is able to handle gpt afaik... but i convert gpt to mbr for furhter testing...

Comment by fugaru
Rebooting

Every now and then (maybe once a year) for various reasons my host will get rebooted, as for example when the hoster had power issues.

So, the remote system will hang at dropbear until you manually help it along? Not a big deal if it is your private blog being hosted, but problematic for say, your mail server.

I like blog posts with this level of detail, keep them coming!

Also, don't you think it would be easier to simply create xen DomUs on encrypted LVM?

Comment by Jon
Rebooting

Every now and then (maybe once a year) for various reasons my host will get rebooted, as for example when the hoster had power issues.

So, the remote system will hang at dropbear until you manually help it along? Not a big deal if it is your private blog being hosted, but problematic for say, your mail server.

I like blog posts with this level of detail, keep them coming!

Also, don't you think it would be easier to simply create xen DomUs on encrypted LVM?

Comment by Jon
Re: Rebooting

Hi Jon,

Yes, it will hang on dropbear waiting for you to ssh in and enter the password. But that's on purpose, it means that if somebody takes the harddrive away, they won't be able to decrypt your data. It is a trade-off, a bit of inconvenience or a bit of security :)

You could create domUs on top of this, of course, but still the point is that I don't want the data to be so easily unencrypted, so I wouldn't store the passphrase in the same server.

Comment by Martín Ferrari
Adapting the installation to Ubuntu Server 14.04

It can be possible to adapt the procedure to Ubuntu server Trusty?

I tried it in many ways, but with no success. The error is on grub installation that refuse to install on fat or ext2 (don't know why), and even if in the past I was able to bypass this problem, once I'm on dropbear console, putting the password as described in this guide doesn't produce any effect (even if I do the same way on similar procedure on Debian, and worked smootly, so I don't think it's an error of me).

Thanks.

Comment by Mauritus
One more typo

Hi,

there is one more typo in "mkfs.ext4 -L $i /dev/mapper/vg-$i; done" there should be "mkfs.ext4 -L $i /dev/mapper/vg0-$i; done" :)

Comment by LosPollosHermanos
more typos

LosPollosHermanos, thanks for the bug report!

I fixed that and another two of the same error.

It's great to get bug reports on blog posts :-)

Comment by Martín Ferrari
vKVM Mirror
Here's a new tutorial that I found that has a vKVM mirror on it "https://trick77.com/how-to-set-up-virtual-kvm-vnc-console-ovh-server/"
Comment by Max
Jessie, systemd?

Hello again, I reinstalled my system on debian 8, and I'm not able to get the system booting, I have no idea why. Has anybody tried to use this set-up with systemd? I can connect to dropbear and enter the passphrase, the connection is then closed and an ssh server never comes back again. The server is still pinging. Is there something I can check by logging in into the dropbear environment? In rescue I have no problem mounting the encrypted root and chrooting into the system. Thanks!

Comment by vinvin
No problem with Jessie
Ok, it took me days but I finally figured it out. I made /boot ext4 while I thought it was ext2, as I documented in the /etc/fstab. I guess it was failing to mount because of the mismatch, and Jessie has strict mounting rules. Sorry, and thanks again for the great tutorial.
Comment by vinvin2