Figure out CPU usage limits

[price]

Currently when a handful of VMs decide to spin, the whole system gets hosed. This doesn't seem like a sustainable situation for a production service.

I don't think any of us knows to what extent Xen supports resource limits on VMs. We should investigate, and apply whatever limits we can. Ideally the limits would be in the same units as the limits we do enforce on RAM and storage, namely by owning locker.

[broder]

We probably want to be using ​http://wiki.xensource.com/xenwiki/CreditScheduler

It's built into the version of Xen we're running, and appears to work - both the weight and capping aspects. We should figure out how we actually want to set them.

[price]

Another page on the Xen wiki has four other schedulers:

​http://wiki.xensource.com/xenwiki/Scheduling

called "Borrowed Virtual Time", "Atropos", "Round Robin", and "sEDF scheduler". We should make at least a brief examination of all five of these, and work out which one makes the most sense for what we want to do. (Also, figure out just what we do want to do.)

[price]

We've been doing this on a manual, ad-hoc basis to machines that persistently chew CPU with commands like

$ xm sched-credit -d d_<machine-name> -w 128 -c 50

which causes the machine to get half the weight in the scheduler that other machines get (128 rather than 256, in abstract units), and be capped at half a CPU.

The scheduler involved here is the new Xen default:

​http://wiki.xensource.com/xenwiki/CreditScheduler

It's possible we want to apply this more systematically.