1. Questions for Customer
1.1. Non-functional
1.1.1. Do you care about testability?
1.1.2. Do you care about usability?
1.1.3. Do you care about accessibility?
1.1.4. Do you care about security?
1.1.5. Do you care about performance?
1.1.5.1. Can we load or performance test against this system?
1.2. DO YOU HAVE REQUIREMENTS YOU CAN GIVE US TO USE FOR TESTING???
1.3. WHEN SHOULD WE STOP ASKING QUESTIONS?
1.4. Functional?
1.4.1. Weinberg recommends...
1.4.1.1. Attribute wish list
1.4.1.1.1. dimensions we are interested in
1.4.1.1.2. vs points along those dimensions
1.4.1.2. How would you (the client) like to use this product?
1.4.1.2.1. Decisions
1.4.1.3. What is the purpose of this product?
1.4.1.3.1. What are you most looking forward to?
1.4.1.3.2. Which part of the system will be most valuable to you?
1.4.1.3.3. Expectations list
1.4.1.3.4. Replacing existing product?
1.4.1.4. How would others like to use this product? (i.e. user list)
1.4.1.4.1. Doctrine of Reasonable Use: we'll know what is reasonable when we see it
1.4.1.4.2. Measuring satisfaction
1.4.1.4.3. What groups of people have what troubles with the present product?
1.4.1.4.4. How much time do people spend on various parts of the present product?
1.4.1.4.5. How much money do people spend on various parts of the present product?
1.4.1.5. Don't worry about how much it would cost. What would you like?
1.4.1.5.1. What's it worth?
1.4.1.6. Don't worry if it can really be done. What would you like?
1.4.1.6.1. When do you need it?
1.4.1.6.2. What is the expertise of the estimators?
1.4.1.7. Evident (visible) vs Hidden (non-obvious or taken for granted) vs Frill (only if free)
1.4.1.8. Ambiguity
1.4.1.8.1. Problem-statement
1.4.1.8.2. Design-process
1.4.1.8.3. Final-product
1.5. What risks are you concerned about?
1.5.1. What is important?
1.5.2. What do my customers care about?
1.5.3. Priority/risk
1.5.4. Consequences of failure
1.5.5. Project details
1.6. Use any of Weinberg's context-free questions?
1.6.1. Process
1.6.1.1. Where else can the solution to this design problem be obtained? Can we copy something that already exists?
1.6.1.1.1. (Doesn't seem useful)
1.6.1.2. Should we use a single design tesam or more than one?
1.6.1.2.1. (Doesn't seem useful)
1.6.1.2.2. Who should be on the teams?
1.6.1.3. Who is the client?
1.6.1.3.1. (I think we know Matt's the client rep, but may not be the ultimate end-user type of customer)
1.6.1.4. How much time do we have for this project? What is the trade-off between time and value?
1.6.1.4.1. (Doesn't seem useful)
1.6.1.5. What is a highly successful solution really worth to this client?
1.6.1.5.1. What is the real reason for wanting to solve this problem?
1.6.1.6. Are you comfortable with the process right now?
1.6.1.7. Is there any reason you don't feel you can answer freely?
1.6.2. Product
1.6.2.1. What problems does this system solve?
1.6.2.2. What problems could this system create?
1.6.2.3. What environment is this system likely to encounter?
1.6.2.4. What kind of precision is required or desired in the product?
1.6.3. Metaquestions
1.6.3.1. Am I asking you too many questions?
1.6.3.2. Do my questions seem relevant?
1.6.3.3. Are you the right person to answer these questions?
1.6.3.3.1. (Seems like Matt is all we get, maybe judge team)
1.6.3.3.2. When I asked X about that, she said Y. Do you have any idea why she might have said Y?
1.6.3.3.3. I notice that you don't seem to agree with that reply. Would you tell us about that?
1.6.3.3.4. What can you tell me about the other people on this project?
1.6.3.3.5. How do you feel about the other people working with us on this project?
1.6.3.3.6. Is there anybody we need on this project whom we don't have?
1.6.3.3.7. Is there anybody we have on this project whom we don't need?
1.6.3.3.8. Can you tell me more about that person?
1.6.3.4. Are your answers official?
1.6.3.5. In order to be sure we understand each other, I've found that it helps me to have things in writing so I can study them at leisure. May I write down your answers and give you a written copy to study and approve?
1.6.3.5.1. (Seems inherent in the YouTube commenting process)
1.6.3.6. The written material has been helpful, but I find that I understand some things better if I can discuss them face to face. Can we get together at some point so we can know each other better and can clarify some of these points?
1.6.3.6.1. (Not sure that we'll get any live "face time" with judges, so probably not relevant)
1.6.3.7. Is there anyone else who can give me useful answers?
1.6.3.7.1. (Seems like Matt is all we get, maybe judge team)
1.6.3.8. Is there someplace I can go to see the environment in which this product will be used?
1.6.3.9. Is there anything else I should be asking you?
1.6.3.9.1. I notice that you hesitated a long time before answering that question. Is there something else we should know?
1.6.3.10. Is there anything you want to ask me?
1.6.3.11. May I return or call you with more questions later, in case I don't cover everything this time?
1.6.3.11.1. (I think the process said we have a limited question-asking window and it's up-front)
1.7. from demo
1.7.1. Are we only testing U.S. site? Canada too?
2. Deliverables
2.1. Issues found
2.1.1. Lean Testing
2.1.2. Emailed instructions on bug reporting
2.2. Test report
2.2.1. Format?
2.2.1.1. Could include Mindmeister?
2.2.1.2. Describing the state of the SUT
2.2.1.3. what was most important to test
2.2.1.4. details of strategy
2.2.1.5. major issues found
2.2.2. Purpose?
2.2.2.1. Help decision-maker figure out whether to ship
2.2.2.2. Needs more fixes?
2.2.2.3. What to invest in next?
2.2.3. Content
2.2.3.1. How you decided what to test
2.2.3.2. How team spent its time
2.3. Submission
2.3.1. title
2.3.1.1. Functional Test Report for TEAM_NAME
2.3.1.2. North America in the subject line
2.3.2. email address
2.3.2.1. [email protected]
2.3.3. time
2.3.3.1. grace period of 5 minutes after end time
2.4. Score
2.4.1. On mission
2.4.1.1. <= 20 pts
2.4.2. Quality of bug reports
2.4.2.1. <= 20 pts
2.4.3. Quality of test report
2.4.3.1. <= 20 pts
2.4.4. Accuracy of test report
2.4.4.1. <= 20 pts
2.4.5. Non-functional testing
2.4.5.1. <= 20 pts
2.4.6. Interacting with judges/customer
2.4.6.1. bonus <= 10 pts
3. Focus
3.1. Major issues
3.2. Informing the customer
3.3. Suggesting fixes & their importance
3.3.1. Bug advocacy
4. Product Owner inputs!!
4.1. helping consumers make decisions
4.1.1. credit cards
4.1.1.1. helping consumer select best rewards card for them
4.1.2. bank accounts
4.1.3. cell phones
4.2. text at the top of the page
4.2.1. subject/verb mismatch
4.3. prioritized/sort list of credit cards suitable for you
4.4. filters
4.4.1. more tab
4.4.1.1. annual fee
4.4.1.2. foregin transaction fee percentage
4.5. inputs
4.5.1. credit score
4.5.2. types of cards
4.5.2.1. card issuer
4.5.3. rewards you want
4.5.3.1. redemption categories affect value of points
4.5.3.1.1. airfare makes points worth more
4.5.4. time horizon for rewards
4.5.4.1. 1 year
4.5.4.2. 10 years
4.5.5. spending patterns
4.5.5.1. monthly
4.5.5.2. categories
4.6. toggles
4.6.1. sorting preferences
4.6.1.1. plain sorting
4.6.1.1.1. avg annual reward over chosen time horizon
4.6.1.2. annual fees
4.6.1.3. prioritized
4.6.1.4. order of sorts matters
4.6.2. Pareto ranking
4.6.2.1. tradeoffs
4.6.2.2. finding best options depending on your credit score
4.7. better than diagram
4.7.1. directed graph
4.7.2. arrows point from worse option to better option
4.7.3. no errors, not much other value
4.7.4. lower priority
4.8. customize columns display
4.8.1. groupps of columns
4.9. concerns
4.9.1. accuracy
4.9.1.1. calculations
4.9.1.1.1. not focused on small details
4.9.1.1.2. respond well
4.9.1.1.3. what can people do to compromiste the calcuatlations?
4.9.1.2. avoid "gross inaccuracy"
4.9.1.2.1. legal issues
4.10. browsers
4.10.1. using websocket
4.11. accessibility
4.11.1. easy fixes
4.11.2. not complete overhaul
4.12. availability
4.12.1. concurrent users
4.12.1.1. up to 20?
4.13. privacy
4.13.1. PII????
4.13.1.1. nope
4.14. responsive design
4.15. lcoale
4.15.1. USA
4.15.1.1. larger potential user base
4.15.2. Canada
4.15.2.1. affiliate kickbck
4.15.3. wrong aspect ratio on US flag
4.16. sourece of data
4.16.1. card issuer websites
4.17. programming language
4.17.1. R
5. Heuristics we want to use?
5.1. http://karennicolejohnson.com/wp-content/uploads/2012/11/KNJohnson-2012-heuristics-mnemonics.pdf
5.2. http://karennicolejohnson.com/2012/07/testing-mnemonics-as-a-card-deck/
5.3. http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
5.4. To be clear, I'm cool with SFDPOT - Claire
6. Tools
6.1. YouTube comments
6.2. Skype/Google Hangout
6.3. Google Docs
6.4. Mindmeister
7. Specializations
7.1. Accessibility
7.1.1. Curtis
7.2. Security
7.2.1. Nawwar
7.3. Question Asking
7.3.1. Claire
7.4. Report Beautification
7.4.1. Elizabeth