InfoQ have a fascinating Q&A session with leading technologists about their preferred manners of ensuring their infrastructures remain performant under high traffic.
InfoQ: Scaling and Performance tuning is often seen as a fire-fighting activity; it’s all about fixing the problem right now. How would you go about tracking performance regressions over a mature codebase?
Matt Youill, Chief Technologist at Betfair: I think these are only truly detected once the application is live. Ensure that an application is partitioned effectively and deploy it onto one server in the live environment. If it looks good, deploy it onto another, and then another, and so on. Make sure that you invest in effective monitoring and measurement infrastructure to catch performance issues early in the rollout process.
Don’t try and catch performance issues before you deploy. You won’t be able to recreate the conditions that exist in live, and consequently you won’t get realistic dependable measurements.