(Apologies for the short notice. We generally want to stick to our normal advertised maintenance windows but sometimes technology works against us.)
We will have an emergency maintenance window tonight from 6pm to 11:00pm PST.
The following work will take place:
- 6:00pm PST (0200 UTC) ) EqualLogic storage array firmware upgrade. See bug 464458 for details. We’ll be locking the tree to all check-ins at 5:00pm PST tonight to start the maintenance window and will unlock it once the machines have come back up and gone green for a cycle.
Last night one of our network attached storage devices experienced an error and didn’t fail over properly. The effect was such that we had to close the tree as Tinderbox wasn’t getting proper data for updates, and we couldn’t tell what was happening with our builds. While we were able to restore that functionality, there is a high chance that any of our VMs attached to this storage device may experience further problems.
To fix this, we’ll need to pause all of our VM systems (most notably build machines & unit test machines, but also litmus and some others) for a few hours. The affected systems would be:
After discussing this with the sheriff, product leads and release engineering team, we’ve decided it would be best to do this as soon as possible. We’ll be locking the tree to all checkins at 5:00pm PST tonight to start the maintenance window, and we’ll unlock it once the machines have come back up and gone green for a cycle. Vlad will try to shepherd the remaining patches for beta 2 blockers that are in the checkin queue before 5pm, but at that time we’ll stop all work and pick it up tomorrow morning.
Let us know if there are any objections as soon as possible.