support multiple compute servers

[price]

In order to scale up, we'll need to run VMs on more than one host machine.

Currently the code assumes that only one host exists, as it's always been running on just one. So this will require some development work; the web interface has to pick which host to create a VM on, tell the database, and when a VM is running to perform operations on the right host. The console server will also have to look up the right host to talk to. Probably something else will need fixing too.

Nelson, Tim, and I have set up the long-former office head quiche in the machine room, with a fibrechannel interface to the RAID. It remains to actually set it up as another compute server like black-mesa, and to fix up the various parts of the code to use it.

Ultimately, of course, we'll want a more powerful machine, which we'll be asking Wilson to buy for us. (Or two machines, to replace black-mesa too.) But this'll allow us to prepare the software so we're ready for when that machine arrives.

This will also be an opportunity to test the installability and completeness of our packages...

[price]

We have now six blade servers, lent via mitchb by the DARPA Grand Challenge team. We're running one of them, sx-blade-2, as a host rather like black-mesa; it takes the place of quiche in these plans. After much wrangling quentin, broder, and I were able to get it to see the RAID like black-mesa does.

I've created a proxy server remote.mit.edu to accept remctl calls and pass them on to black-mesa. (I also replaced the sipb-xen-remctl-auto system with a FUSE-based solution like on the console server.) This will become useful once we add some code so that the scripts on this server query both black-mesa and sx-blade-2 for running VMs, proxy commands like uptime and shutdown to the server a VM is running on, and choose a server to proxy commands like create to.

[price]

Also, all this proxying will be more efficient if we keep persistent connections to the servers rather than spend 400ms creating a connection for each command. I believe quentin and ecprice have studied how to do this.

[price]

Done, principally in r538, r658, and the components of #106 and #108.

It'd still be cool to have a daemon with persistent connections. Quentin wrote a proof-of-concept in r658.