USER TESTING with Wayne Pau, SAP

User Testing with Wayne Pau, SAP
By Helen Kontozopoulos, MEIC
First posted on Mobile Experience Innovation Lab, OCAD University, March 2014

Wayne makes a key point about testing: “it’s not just about finding bugs in the software, but to really making it better.”

Wayne presents testing as a series of “zero sum games.” Budgets and deadlines mean you can’t test everything, so it’s important to decide the kinds of testing that are going to be the most effective. In other words, testing itself needs to be optimised. It should get the most ‘bang for your buck’ and help to actually fix bugs, rather than just collect them.

Top 4 Recommendations for Testing
Since not everything can be tested or fixed, Wayne introduces a few scientific approaches to help pare down testing to make it more manageable. Wayne’s recommendations for testing include the following:

1. Have a Plan: When you decide to test, it’s important to have a plan. Organize your limited resources, and schedule your testing to be effective as possible. It’s a good idea the spend time identifying ‘high-yielding’ tests and prioritizing ‘high-yield’ areas of code or features before setting up testing environments and thinking about test coverage.

2. Variety is Essential: Don’t always use the same data. Use different types of tests to compensate for each test’s weaknesses.

3. Knowledge is Power: Learn from other testers’ experience and pay attention to what is happening in the development practices, then ask questions, read the bug reports, and see what is breaking for others, so you don’t repeat the same mistakes.
​
4. Be Results-Oriented: Create priorities about what needs to be tested and work on the goal of choosing quality over quantity when testing. Avoid default test data, target critical bugs in main features, reduce and eliminate redundant tests, and generate ONLY quality bug cases.

Continuous Integration
Continuous Integration is just what it sounds like: a development process in which team members’ work is integrated continuously. Integration happens multiple times per day, and is tested with an automated build to find errors whenever they crop up. Wayne believes that continuous integration is a key aspect of modern agile development practices, since constant iteration requires constant testing to make sure each version holds together.

Wayne uses the analogy of Lego blocks: When building, you want to make sure that the individual blocks and the way they are connected work, and not wait until later to see if your construction falls apart. Under older integration processes, developers might spend as much time trying to integrate the unwieldy pieces of their creations as building them. Continuous integration is about avoiding that problem by identifying and squashing bugs as soon as they occur. By reducing the risk of unpleasant surprises later in development, it also allows you to get a good idea of the quality of your code and what kind of progress you are making.
​
Wayne sounds a note of caution: Automated testing will not identify all bugs, log issues, or help to debug software. Think of it as a part of a developer’s repertoire, rather than a replacement.

The Pitfalls of testing
Wayne outlines three pitfalls of testing: Human Bias, Group Think, and Inference and Assumptions.

Human Bias: We all have biases due to our backgrounds, and these may influence our decisions. While transcending bias is easier said than done, we all benefit from becoming aware of our bias and how it might affect our testing and objectivity. If you are aware of your bias, you might be able to make choices you otherwise wouldn’t have.

Group Think: Teams have their own biases as well. If everyone is just agreeing with each other, we might be creating ‘yes teams,’ instead of creating teams that test assumptions.
​
Inference and Assumptions: There is an additional pitfall: Obvious patterns can fool us into jumping to erroneous conclusions. Indicators are not results: They just indicate those areas where we should test for results. For any testing, past experience should be only used as a guide and not as a conclusion.


UPDATE since Wayne’s presentation at MEIC!
Since Wayne’s presentation on testing at MEIC, Steve Krugh has been recently acknowledged Wayne in a new chapter on mobile in his 3rd edition of “Don’t Make Me Think,” a very popular and must have book on UX. Steve Krug is one the leading authorities in research in human-computer interaction and web usability. Wayne highly recommends Steve’s book for everyone who builds mobile apps to buy this book.
​
Steve’s website: http://www.sensible.com/
​
Video 1: Introduction to testing
Video 2: Mathematics of Testing
Video 3: Continuous Integration
Video 4: Pitfalls of Testing