One of the keys to any good piece of software is good testing. There are well-known testing suites for back end code – things like junit and py.test. There are also good front-end testing tools – things like Selenium. But for distributed applications there aren’t so many well-known tools – because the problem is quite different, and a bit harder. In this blog post we’ll cover the failure testing methodology used by three different successful distributed systems – its history and how and why it works.