• In the first week of March 2012 we moved our nightly runs from old hardware (in Garland center) to new hardware (in Irvine center). We had a week of failures while we fixed geocentric assumptions in some of our test scripts, and a few more weeks of random network and infrastructure related failures as we got access to and within the new lab sorted out.
• In April of 2012 we were building and testing so many branches that the gitbuilders would often not have finished building the master branch by the time the nightlies ran ... resulting in failures (attributed to infrastructure) when tests tried to download and install those tar-balls.