remote-server should prevent create on two hosts at once

[price]

Repeating an observation I made on zephyr: (*)

Here's an unsettling thought. It would be bad if a VM got booted on two hosts at once.

$ remctl remote control price-test8 create \
 & remctl remote control price-test8 create
[1] 10725
Creating on host citadel-station.mit.edu...
Creating on host citadel-station.mit.edu...

If the availability were close and someone else happened to create or shut down a VM at the right time, those could go to different hosts. And then the VM would run in two places at once.

Perhaps the most worrying way this might be triggered is when we're dealing with something and booting up a user VM because we just shut it down -- even if we go through Invirt all the way, we could still end up stepping on the user booting it themselves because they noticed it go down.

Fortunately in order to deal with this we don't need to build or use a distributed system, only a concurrent one; it suffices to ensure on the remote-server that creates are serialized (or, in principle, only serializable.) So invirt-remote-create could, e.g., use a lockfile to ensure only one of itself runs at a time, and then we try to verify that when the create call returns, Xen is guaranteed to have entered a state where invirt-listvms will see the new machine.

This is probably a totally reasonable ticket for someone new to fix.

(*) And, in fact, in a TODO in the original invirt-remote-create code, back on 2008-06-28. Should have put it in Trac in the first place, perhaps.