Fix the RAID

[price]

Our RAID is still mysteriously, horrendously slow in certain circumstances. It starts spewing thousands of the same puzzling errors, and some IO hangs for minutes.

The vendor blames the errors on the cable connecting the RAID to black-mesa. Tonight we swapped out the cable and one of the two connectors on its ends; nothing improved.

Tonight we also tried booting a 2.6.24 kernel. The error messages recurred, but the delay could not be reproduced. If we fix #41, we can upgrade the kernel; or we could consider backporting the new driver to our present etch (2.6.18-something) kernel.

We should also go back to the vendor, tell them that replacing the cable didn't help, and ask them to actually help debug the problem.

[quentin]

To be clear, it seems that 2.6.24 just recovers from the errors better, rather than spewing them continuously. It is the lack of spewing errors that results in better I/O performance.

We can't actually upgrade the Xen kernel to 2.6.24; #41 refers to guest kernels, not host kernels.

I'm contacting the vendor now.

--Quentin

[tabbott]

So, what were the results of swapping out components?

[quentin]

The results were that nothing fixed it, and a few swaps made it (seem) even worse. We are awaiting a reply to our support ticket (they told us to call them if it was an emergency)

--Quentin

[price]

The current status from Quentin is that we need to try a new driver they've supplied. This apparently involves recompiling the kernel.

[price]

The new driver was in the kernel booted Saturday night, and all seems to be well so far.

I'll wait a few days before closing this, but if someone else is confident the upgrade fixed it they should feel free to close.

[price]

Closing, as it seems to be fixed.