ParaVM support

[ecprice]

Make it possible to create and control ParaVM machines from the web interface and boot them through the database

[tabbott]

Since this is essentially blocking paraVMs support for the service, its priority should be higher.

[andersk]

The important blocker for paravirtualized guests is a way to load the guest kernel. Having the guest kernel locked at 2.6.18 isn’t acceptable, and even if it were, we’d have to coordinate black-mesa host kernel upgrades with all paravirt guests at once to have the right modules available.

I suspect paravirt console support is straightforward, but paravirt kernel loading requires a mechanism we don’t have, unless we decide we’re willing to trust pygrub.

[anonymous]

Using Pygrub:

• Save the kernel and ramdisk images inside of the guest domain.
• Create a grub config file, /boot/grub/menu.lst in the guest domain that contains an entry for booting off of the kernel and ramdisk images inside of the guest.
  • The root option for the kernel entry should look identical to the root option in the original guest domain's config file.
  • pygrub seems to understand only a small fraction of the options that grub understands; for example, it does not understand "boot" or "savedefault." A very basic menu.lst file is all that is needed.
• Modify the guest domain's config file to exclude the kernel and ramdisk entries and include a new entry: "bootloader = /path/to/pygrub"
  • The "root" line in the domU's config file is probably also unnecessary; it's presence does not seem to affect performance
• Start up the paravm in the same way: xm create -c /etc/xen/paravm_config_file

[price]

3 was neboat.

• Start up the paravm in the same way: xm create -c /etc/xen/paravm_config_file

There's an example of this in /etc/xen/neboat.para.test1.cfg, starting a test machine neboat_para_test1.

[neboat]

• Modify the guest domain's config file to exclude the kernel and ramdisk entries and include a new entry: "bootloader = /path/to/pygrub"

Also check that the entry for "disk" contains the device that holds the root portion of the filesystem first in the array; otherwise pygrub, when run from the paravm's config file, will fail.

[broder]

Now that we have pygrub and ParaVMs working, we need support for them from the web interface and database.

[andersk]

I guess all the new members of the sipb-xen team haven’t heard about the previous consensus on pygrub.

pygrub is sketchy. It runs a lot of code from dom0 that was never designed to be secure (both the Python code itself and libext2fs2). It’s great for managing a bunch of VMs that are all controlled by the same admins, but this security model is not appropriate for our application.

I think that we came to this decision even before ​CVE-2007-4993 was announced; in any event, it’s not a surprise that a privilege escalation attack arose from this mess. The fact that the vulnerability was rated ​less-critical by Secunia supports our hypothesis that pygrub was not designed for our security model.

The Xen people are ​aware of this problem. The solution they are working on, which I think is the right one, is to load the kernel from within the guest instead of dom0. There has been some work on extending the ​kboot bootloader to run ​within a Xen guest. I have not been able to find the current status of this work because all the relevant links seem to be broken, but I expect it would not be hard to get this working again (using more clever searches, archive.org, emailing the xen lists, or just hacking for a while).

[price]

We're aware of what was said about pygrub in the past.

Replying to andersk:

I think that we came to this decision even before ​CVE-2007-4993 was announced; in any event, it’s not a surprise that a privilege escalation attack arose from this mess.

This vulnerability is not something that came out of a subtle combination of choices that we could easily miss looking at the code. It came out of using 'exec' to take up the settings in grub.conf.

black-mesa:~# egrep -r exec\|eval /usr/lib/xen-3.1-1/{bin/pygrub,lib/python/grub} black-mesa:~#

No such blatant exposure now exists, and if you go read the code that now parses grub.conf (GrubConfigFile?.parse() in GrubConf?.py), it's quite tightly limited in what it allows itself to do based on the file's contents.

The Xen people are [​http://wiki.xensource.com/xenwiki/XenRoadMap#line-1272 aware of this problem]. The solution they are working on, which I think is the right one, is to load the kernel from within the guest instead of dom0. There has been some work on extending the ​kboot bootloader to run [​http://xen.org/files/summit_3/xen-summitHoffmankexec.pdf within a Xen guest]. I have not been able to find the current status of this work because all the relevant links seem to be broken, but I expect it would not be hard to get this working again (using more clever searches, archive.org, emailing the xen lists, or just hacking for a while).

None of this is particularly convincing that work is happening on these approaches. The relevant line in that XenRoadMap? page is in among dozens of ideas in the huge "Misc" section; it doesn't even rate a place in the "Paravirtualization" section. The kboot project hasn't released since January 2007, and the presentation you linked to is dated September 2006. I think it'd be a lot of grungy hacking to make this work ourselves, more than I expect we're going to be willing to do.

It would be really elegant and nice to do the boot-loading inside the guest. If we can find a way to do that with a level of work we're willing to put in, let's do so. Until then, we still want paravm (since scaling is one of our biggest issues) and pygrub appears to offer an acceptable level of security for that purpose.

Some sipb-xen developers are interested in reading through the pygrub code (it's ~1000 lines of Python) to check that there are no further injection attacks. This might be a reasonable thing to do to increase our confidence in the security of pygrub.

[price]

Thoroughly done. Even #65 and #66 are done.