u/waynemroseberry

>The most complex testing we ever do usually happens when we test under conditions that look like, or sometimes are, production systems. We connect to real running instances of dependencies, platforms, services, and data sources. The various network stacks, routers, identity services, request protection, configurations, service and domain lookup and registration are all real running instances. All the limitations of production apply. All the behaviors are the same.

>We often find bugs in such a configuration we do not find earlier in simpler environments. Services behave in unexpected ways, data is slightly different, errors occur we don't otherwise see. We catch a lot of unknowns this way. We are tempted to move all our testing under this semi-production, staging environments.

>The problem is we have the least amount of control over test conditions in these environments. We usually have no influence over data, some of which may be transient and changing over time. We cannot invoke behaviors of timing, order differences, failure, performance deviations. We are usually distant from diagnostic logs and signals that let us test with high precision and better information. Testing under these conditions is usually far more difficult, far more uncertain, far slower, far more expensive than in simpler environments.

>It is usually to our advantage to do the difficult work of simulating as much of that "real" environment in testing environments where we can control everything. This is almost always a deep commitment to analysis and product design, build system, and environment design that supports effective simulation. Getting this right is not easy, but it is better than trying to get by without it. The alternative is slow, unreliable, expensive testing activity, usually supported by overwrought automation suites weighed down by the attempt to control services and data sources and conditions that cannot be controlled. This doesn't mean we should never test in staging environments. We will always find problems in them and in production that we missed with earlier testing. Once we do as much testing as we can under conditions we can control efficiently, quickly, reliably, we then complement that with testing in more complex environments. While there, we expand the net as wide as we can. Take advantage of the rich collection of application activity, the rich data state. Take user scenarios that are normally meant for simple, cleaner conditions and try them out with data that looks more realistic or alter the user paths off the regular path and see what comes up. Scour system logs and application telemetry for any sign of error, failure or unexpected behavior. Try user scenarios that cross as many services and data sets as possible. Everything you do in this environment is going to take extra work to understand, it is a good idea to enrich the coverage with as much possibility of unanticipated behavior as possible.

Excerpt from Drawn to Testing Again, my second book of cartoons and articles about software testing.