TestGoal: Result-Driven Testing by Derk-Jan de Grood

아마존 링크.

아래는 목차.

1 Result-driven Testing............................................................................. 1
1.1 The Importance of IT.................................................................... 1
1.2 A Statement about Quality............................................................ 3
1.3 The Perception of Testing ............................................................. 6
1.4 A Common Goal ........................................................................... 7
1.5 Tying in with the Business............................................................ 10
1.6 Result-driven Testing.................................................................... 11
1.7 Focus on the Goal ......................................................................... 13
2 TestGoal and the Ten Test Principles................................................... 15
2.1 Test Principles............................................................................... 15
2.2 Focus on Result............................................................................. 16
2.3 Build Trust .................................................................................... 17
2.4 Take Responsibility....................................................................... 19
2.5 Master the Testing Profession....................................................... 20
2.6 Build Bridges ................................................................................ 22
2.7 Test in Phases................................................................................ 23
2.8 Facilitate the Entire Product Life Cycle........................................ 24
2.9 Provide Overview and Insight....................................................... 25
2.10 Ensure Reusability ........................................................................ 27
2.11 Keep in Mind: Testing is Fun........................................................ 28
2.12 Applying the Test Principles......................................................... 29
3 Test Expertise ......................................................................................... 31
3.1 The Test Manager ......................................................................... 32
3.2 The Test Coordinator .................................................................... 32
3.3 The Test Analyst ........................................................................... 33
3.4 The Test Engineer ......................................................................... 33
3.5 The Test Specialist ........................................................................ 33

4 The Approach......................................................................................... 37
4.1 Context of the Test Project............................................................ 38
4.2 Test Levels .................................................................................... 42
4.3 The Details of the Test Project...................................................... 44
4.3.1 The Step Plan ................................................................. 45
4.3.2 Sequence of Activities.................................................... 46
4.4 Testing a New Program................................................................. 47
4.4.1 Step 1: Goal .................................................................... 48
4.4.2 Step 2: Approach............................................................ 49
4.4.3 Step 3: Design ................................................................ 51
4.4.4 Step 4: Set up.................................................................. 54
4.4.5 Step 5: Execution............................................................ 56
4.4.6 Step 6: Assurance ........................................................... 58
4.4.7 Goal (Information and Communication) ........................ 59
4.4.8 Review and Acceptance ................................................. 61
4.5 Testing in a Maintenance Environment ........................................ 61
4.6 Testing Conformity and Interoperability....................................... 65
4.6.1 Introduction .................................................................... 65
4.6.2 Applying the Step Plan................................................... 67
4.6.3 Certification Tests .......................................................... 69
4.7 Testing Performance ..................................................................... 71
4.7.1 Introduction .................................................................... 71
4.7.2 Applying the Step Plan................................................... 72
4.7.3 Test Design Techniques ................................................. 73
4.7.4 Test Tools....................................................................... 74
4.7.5 Dependencies ................................................................. 75
4.8 Testing Security ............................................................................ 77
4.8.1 Introduction .................................................................... 77
4.8.2 Approach ........................................................................ 77
4.8.3 Applying the Step Plan................................................... 79
5 Getting Started ....................................................................................... 83
Step 1 . Goal
6 Assessing the Anticipated Goal ............................................................. 87
6.1 Introduction................................................................................... 87
6.2 Aim of the Assessment ................................................................. 88
6.3 Goal Description ........................................................................... 90
6.4 Information Gathering................................................................... 93
6.4.1 Product Development ..................................................... 93
6.4.2 People............................................................................. 94
6.4.3 Guidelines and Documentation ...................................... 95

Step 2 . Approach
7 Test Risk Analysis .................................................................................. 101
7.1 Introduction................................................................................... 101
7.2 The 1D Test Risk Analysis ........................................................... 104
7.2.1 Introduction .................................................................... 104
7.2.2 Identify Stakeholders and Kick-off ................................ 105
7.2.3 Determine the Functions and Areas of Attention ........... 106
7.2.4 Determine the Relative Importance................................ 108
7.2.5 Process the Data ............................................................. 111
7.2.6 Agree on the TRA .......................................................... 112
7.3 The 2D Test Risk Analysis ........................................................... 113
7.3.1 Introduction .................................................................... 113
7.3.2 Identify Stakeholders and Kick-off ................................ 116
7.3.3 Establish the Risks.......................................................... 116
7.3.4 Data Processing .............................................................. 117
7.3.5 Agree on the TRA .......................................................... 118
8 Generic Test Strategy............................................................................. 119
8.1 Introduction................................................................................... 119
8.2 The Generic Test Strategy............................................................. 120
8.3 Test Strategy in the DTP and MTP............................................... 123
9 Test Budget and Planning...................................................................... 125
9.1 Introduction................................................................................... 125
9.2 Create the Test Budget .................................................................. 126
9.2.1 General ........................................................................... 126
9.2.2 Work Breakdown Structure............................................ 127
9.2.3 Assessing the Requisites................................................. 135
9.2.4 Establishing the Budget.................................................. 136
9.3 Test Planning ................................................................................ 137
9.3.1 Generic Planning ............................................................ 137
9.3.2 Detailed Planning ........................................................... 138
9.4 Key Indicators............................................................................... 141
10 Test Plan.................................................................................................. 143
10.1 Introduction................................................................................... 143
10.2 Description of the Assignment...................................................... 145
10.3 Test Base....................................................................................... 146
10.4 Test Strategy ................................................................................. 148
10.4.1 Description of the Test Strategy ..................................... 149
10.4.2 Test risk analysis ............................................................ 151
10.4.3 Quality Attributes ........................................................... 152
10.4.4 Strategy Matrix............................................................... 155

10.4.5 Technique Matrix ........................................................... 157
10.4.6 Previous and Next Phases............................................... 158
10.4.7 Test Environment ........................................................... 159
10.4.8 Assuring the Quality of the Test Project ........................ 160
10.4.9 Release Advice ............................................................... 163
10.4.10 Change and Error Management ...................................... 164
10.4.11 Transfer .......................................................................... 164
10.5 Planning ........................................................................................ 166
10.6 Test Organization.......................................................................... 167
10.6.1 Organization Chart ......................................................... 167
10.6.2 Responsibilities .............................................................. 168
10.6.3 Meeting Structures ......................................................... 169
10.7 Deliverables .................................................................................. 170
10.8 Requisites for the Test Process ..................................................... 171
10.9 Changes and Deviations................................................................ 171
Step 3 . Design
11 Sanity Check........................................................................................... 175
11.1 Introduction................................................................................... 175
11.2 Filling out the Sanity Check Checklist.......................................... 176
11.3 Continuous Learning..................................................................... 178
11.4 Test Base Review.......................................................................... 178
11.5 Registration................................................................................... 180
11.6 Formal Review and Inspection Procedures................................... 181
12 Logical Test Design ................................................................................ 183
12.1 Introduction................................................................................... 183
12.2 Test Design Techniques ................................................................ 185
12.3 Use Test Design Techniques Cleverly .......................................... 191
12.4 Little Experience with Test Design Techniques?.......................... 191
12.5 No Test Design Techniques .......................................................... 192
12.6 Using Test Design Techniques...................................................... 193
12.6.1 Syntax Testing................................................................ 193
12.6.2 EP: Equivalence Partitioning.......................................... 196
12.6.3 BVA: Boundary Value Analysis .................................... 199
12.6.4 C/E: Cause-effect Graphing ........................................... 203
12.6.5 State Transition............................................................... 206
12.6.6 CRUD Testing................................................................ 208
12.6.7 PCT: Process Cycle Test / AT: Algorithm Test ............. 210
12.6.8 Load Tests ...................................................................... 215
12.6.9 Stress Testing ................................................................. 221
12.6.10 Reliability Testing .......................................................... 223

12.6.11 Concurrency Tests.......................................................... 225
12.6.12 HT: Heuristic Testing..................................................... 226
12.6.13 ET: Exploratory Testing................................................. 227
12.7 Test Design Techniques and Security Testing .............................. 234
13 The Physical Test Design ....................................................................... 237
13.1 Introduction................................................................................... 237
13.2 Relationship Between the TRA and the Logical Test Design....... 237
13.3 Physical Test Case ........................................................................ 238
13.4 Test Actions .................................................................................. 243
13.5 The Physical Test Scenario ........................................................... 245
13.6 Test Data ....................................................................................... 246
14 Test Data ................................................................................................. 247
14.1 Test Data Elements ....................................................................... 247
14.2 Test Data Repository..................................................................... 249
14.3 Live Data Versus Test Data .......................................................... 250
14.4 Test Data Management Strategy ................................................... 251
14.4.1 Input from the Application ............................................. 251
14.4.2 Input from the Database ................................................. 252
14.4.3 Closed Loop ................................................................... 253
14.5 Including Data in the Physical Test Design .................................. 254
14.6 Automated Tests ........................................................................... 255
14.7 Test Data and Exploratory Testing................................................ 256
14.8 Back-up and Restore ..................................................................... 256
15 Test Environment ................................................................................... 257
15.1 Introduction................................................................................... 257
15.2 Determine the Requirements of the Test Environment ................. 258
15.2.1 Module Tests and Module Integration Tests .................. 258
15.2.2 System Tests................................................................... 259
15.2.3 Functional Acceptation Tests ......................................... 260
15.2.4 User Acceptance Tests ................................................... 261
15.2.5 Production Acceptance Tests ......................................... 262
15.2.6 Chain Tests..................................................................... 262
15.2.7 Pilot ................................................................................ 263
15.2.8 Performance Tests .......................................................... 264
15.2.9 Security Tests ................................................................. 265
15.2.10 Training Purposes........................................................... 266
15.3 Test Environment Requirements Checklist................................... 267
15.4 Setting up the Test Environment................................................... 270
15.5 Configuration and Smoke Test...................................................... 271
15.5.1 Configuring the Test Environment ................................. 271
15.5.2 Smoke Test ..................................................................... 271

15.6 Maintaining the Test Environment................................................ 272
15.6.1 Configuration Management............................................ 272
15.6.2 Release Management...................................................... 273
15.6.3 Back-up and Restore ...................................................... 274
Step 4 . Set up
16 Test Automation..................................................................................... 277
16.1 Introduction................................................................................... 277
16.2 What is Test Automation?............................................................. 278
16.3 Dynamic Test Tools ...................................................................... 278
16.3.1 Additional Testing Possibilities...................................... 278
16.3.2 Time Saving ................................................................... 279
16.3.3 Log files.......................................................................... 279
16.3.4 Comparing Results ......................................................... 280
16.3.5 Extensive Repeatability.................................................. 280
16.4 Static Test Tools ........................................................................... 282
16.5 Supporting Tools........................................................................... 282
16.6 Test Automation: Yes/No ............................................................. 283
16.6.1 Business Case................................................................. 283
16.6.2 Making a Well-Informed Decision................................. 285
16.7 Developing Test Scripts ................................................................ 285
16.7.1 Record and Playback ...................................................... 286
16.7.2 Programming Test Scripts .............................................. 288
16.8 Automated Testing for Systems with More Than One Interface... 289
17 Smoke Test .............................................................................................. 293
17.1 Introduction................................................................................... 293
17.2 Filling out the Checklist ................................................................ 295
17.3 Maintaining the Checklist ............................................................. 296
Step 5 . Execution
18 Test Execution ........................................................................................ 299
18.1 Test Execution and its Activities................................................... 299
18.2 Activities During the Test Execution............................................ 301
18.3 Test Run and Regression Tests ..................................................... 303
18.4 Leaving the Beaten Track ............................................................. 305
18.5 When is Testing Finished? ............................................................ 306
19 Error Logging and Management .......................................................... 309
19.1 Introduction................................................................................... 309
19.2 Filling out the Error Log ............................................................... 310

19.3 Error Attributes ............................................................................. 311
19.4 Error Management ........................................................................ 316
20 Test Reporting ........................................................................................ 321
20.1 Introduction................................................................................... 321
20.2 Elements in the Test Report .......................................................... 323
20.2.1 Scope .............................................................................. 323
20.2.2 Release Advice ............................................................... 324
20.2.3 Hour Estimate................................................................. 325
20.2.4 Project Risks and Bottlenecks ........................................ 326
20.2.5 Product Status................................................................. 327
20.2.6 Completed Versus Planned Tests ................................... 327
20.2.7 Error Status..................................................................... 328
20.2.8 Defect Detection Rate .................................................... 332
20.2.9 Open Errors .................................................................... 333
20.2.10 Test Result by Risk Category or Test Cluster ................ 334
20.2.11 Test Progress . Executed Versus Planned Tests ............ 337
20.2.12 Outstanding Product Risks ............................................. 339
20.3 The Dashboard.............................................................................. 340
20.4 Clarity of the Test Report.............................................................. 342
Step 6 . Assurance
21 Assurance ................................................................................................ 349
21.1 Introduction................................................................................... 349
21.2 Evaluating the Test Project ........................................................... 349
21.2.1 Purpose of the Evaluation............................................... 349
21.2.2 Points of Attention.......................................................... 350
21.2.3 Lessons Learned Report ................................................. 352
21.3 Determining the Regression Test Set ............................................ 353
21.4 Archiving and Securing the Testware ........................................... 354
21.5 Handover....................................................................................... 354
21.6 Discharging the Test Team........................................................... 355
Appendix A . Checklist: Sanity Check on the Design................................. 357
Appendix B . Checklist: Sanity Check on the Testware.............................. 363
Appendix C . Checklist: Checklist smoke test system................................. 367
Appendix D . Checklist: Test charter exploratory testing .......................... 369
Appendix E . Glossary ................................................................................... 371
References ........................................................................................................ 389
Index................................................................................................................. 393