How to conduct regression tests
Q- How should I write a regression testing section within the test plan?
A- According to the Regression Testing course on TestingEducation.org, regression testing is a common way to manage the risks of change. We might do regression testing by repeating the exact same test as before, or we might reuse the prior test idea, using different data and different secondary conditions as varying items across different uses of the test. The course covers why we might want to regression test, covers some common methods for regression testing, and provides an extensive list of readings for more information on the topic.
When writing the regression testing section of a formal test plan, there are several things you might want to consider:
What's the goal of the testing? This will help you understand what types of risks you'll address and how much coverage you'll need. Try to define this as clearly as you possibly can. I find that confusion around the goal of what the regression tests are suppose to accomplish is largest reason why regression testing spirals out of control, becoming expensive and ineffective.
What risks will our regression testing address, and what risks won't it address? Based on the goal of your testing, what specific risks will be addressed and which won't?
What kind of coverage do we want from our regression testing? I find that coverage is one of the hardest things to keep track of in my regression testing. Often, one test may cover multiple risks or multiple areas of the application. Understanding coverage is important in communicating what your regression tests are and are not doing to other project stakeholders.
What techniques will we employ to execute and maintain the tests? Understanding execution will be important. What tests will be manual and what tests automated? If they are automated, what tools will we need? What infrastructure? How will we maintain them over time? If tests are manual, who will do the testing? Using what techniques? What skills will the manual testers need to know and what areas of the application?
What environment(s) will we need to execute the tests? Here you look at what environments you'll need and when you think you'll need them. What data will need to be available? Are there any custom configurations that will need to be deployed? Will the same tests need to be executed against different configurations? How will you manage that?
How will we report the status of the testing? Who is the audience for your status? What level of detail do they want? Which information do they want first? Are some tests more important then others? What obstacles do you have to your regression testing?
If your test plan section addresses those items in some way, I think you're ok. The specifics of the format don't really matter if you have the right content there. Just format the information using the same style and tone as the template you're using. If you don't use a template, just structure the information in a way that makes sense for your context.
By Mike Kelly
Tuesday, August 12, 2008
Wednesday, August 6, 2008
Software testing deliverables
Software testing deliverables: From test plans to status reports
There are core sets of test deliverables that are required for any software testing phase: test plan, test case, defect documentation and status report. When taken together this set of deliverables takes the testing team from planning to testing and on through defect remediation and status reporting. This does not represent a definitive set of test deliverables, but it will help any test organization begin the process of determining an appropriate set of deliverables.
One common misconception is that these must be presented as a set of documents, but there are toolsets and applications available that capture the content and intent of these deliverables without creating a document or set of documents. The goal is to capture the required content in a useful and consistent framework as concisely as possible.
Test planAt a minimum the test plan presents the test: objectives, scope, approach, assumptions, dependencies, risks and schedule for the appropriate test phase or phases. Many test organizations will use the test plan to describe the software testing phases, testing techniques, testing methods and other general aspects of any testing effort. General information around the practice of testing should be kept in a "Best Practices" repository -- testing standards. This avoids redundant and conflicting information from being presented to the reader and keeps the test plan focused on the task at hand –- planning the testing effort. (See "The role of a software test manager".)
Objectives -- mission statementThe objective of the current testing effort needs to be clearly stated and understood by the software testing team and any other organization involved in the deployment. This should not be a sweeping statement on testing the "whole application" -- unless that is actually the goal. Instead the primary testing objectives should relate to the purpose of the current release. If this were a point-of-sale system and the purpose of the current release was to provide enhanced online reporting functionality, then the objective/mission statement could be this:
"To ensure the enhanced online reporting functionality performs to specification and to verify any existing functionality deemed to be in scope."
The test objective describes the "why" of the testing effort. The details of the "what" will be described in the scope portion of the test plan. Once again, any general testing objectives should be documented in the "Best Practices" repository. General or common objectives for any testing effort could include expanding the test case regression suite, documenting new requirements, automating test cases, and updating existing test cases.
In scopeThe components of the system to be tested (hardware, software, middleware, etc.) need to be clearly defined as being "in scope." This can take the form of an itemized list of those "in scope": requirements, functional areas, systems, business functions or any aspect of the system that clearly delineates the scope to the testing organization and any other organization involved in the deployment. The "What is to be tested?" question should be answered by the in scope portion of the test plan -- the aspects of the system that will be covered by the current testing effort.
Out of scopeThe components of the system that will not be tested also need to be clearly defined as being "out of scope." This does not mean that these system components will not be executed or exercised; it just means that test cases will not be included that specifically test these system components. The "What is NOT to be tested?" question should be answered by the out of scope portion of the test plan. Often neglected, this part of the test plan begins to deal with the risk-based scheduling that all test organizations must address -- What parts of the system can I afford not to test? The testing approach section of the test plan should address that question.
ApproachThis section defines the testing activities that will be applied against the application for the current testing phase. This addresses how testing will be accomplished against the in scope aspects of the system and any mitigating factors that may reduce the risk of leaving aspects of the system out of scope.
The approach should be viewed as a to-do list that will be fully detailed in the test schedule. The approach should clearly state which aspects of the system are to be tested and how: backup and recovery testing, compatibility/conversion testing, destructive testing, environment testing, interface testing, parallel testing, procedural resting, regression testing, application security testing, storage testing, stress and performance testing, and any other testing approach that is applicable to the current testing effort. The reasoning for using any given set of approaches should be described, usually from the perspective of risk.
AssumptionsAssumptions are facts, statements and/or expectations of other teams that the test team believes to be true. Assumptions specific to each testing phase should be documented. These are the assumptions upon which the test approach was based. Listed assumptions are also risks should they be incorrect. If any of the assumptions prove not to be true, there may be a negative impact on the testing activities. In any environment there is a common set of assumptions that apply to any given release. These common assumptions should be documented in the "Best Practices" repository; only assumptions unique to the current testing effort and perhaps those common assumptions critical to the current situation should be documented.
DependenciesDependencies are events or milestones that must be completed in order to proceed within any given testing activity. These are the dependencies that will be presented in the test schedule. In this section the events or milestones that are deemed critical to the testing effort should be listed and any potential impact or risks to the testing schedule itemized.
RisksRisks are factors that could negatively impact the testing effort. An itemized list of risks should be drawn up and their potential impact on the testing effort described. Risks that have been itemized in the project plan need not be repeated here unless the impact to the testing effort has not already been clearly stated.
Schedule The test schedule defines when and by whom testing activities will be performed. The information gathered for the body of the test plan is used here in combination with the available resource pool to determine the test schedule. Experience from previous testing efforts along with a detailed understanding of the current testing goals will help make the test schedule as accurate as possible. There are several planning and scheduling tools available that make the plan easier to construct and maintain.
Test caseTest cases are the formal implementation of a test case design. The goal of any given test case or set of test cases is to detect defects in the system being tested. A test case should be documented in a manner that is useful for the current test cycle and any future test cycles. At a bare minimum, each test case should contain the author, name, description, step, expected results and status.
Test case nameThe name or title should contain the essence of the test case, including the functional area and purpose of the test. Using a common naming convention that groups test cases encourages reuse and helps prevents duplicate test cases from occurring.
Test case descriptionThe description should clearly state the sequence of business events to be exercised by the test case. The test case description can apply to one or more test cases; it will often take more than one test case to fully test an area of the application.
Test case stepEach test case step should clearly state the navigation, data and events required to accomplish the step. Using a common descriptive approach encourages conformity and reuse. Keywords offer one of the most effective approaches to test case design and can be applied to both manual and automated test cases.
Expected resultsThe expected results are the expected behavior of the system after any test case step that requires verification or validation. This could include screen pop-ups, data updates, display changes or any other discernable event or transaction on the system that is expected to occur when the test case step is executed.
StatusThis is the operational status of the test case. Is it ready to be executed?
Documenting defectsThe primary purpose of testing is to detect defects in the application before it is released into production. Furthermore, defects are arguably the only product the testing team produces that is seen by the project team. Document defects in a manner that is useful in the defect remediation process. At a bare minimum, each defect should contain the author, name, description, severity, impacted area and status.
Defect nameThe name or title should contain the essence of the defect, including the functional area and nature of the defect.
Defect descriptionThe description should clearly state what sequence of events leads to the defect. When possible include a screenshot or printout of the error.
How to replicateThe defect description should provide sufficient detail for the triage team and the developer fixing the defect to duplicate the defect.
Defect severityThe severity assigned to a defect is dependent on the phase of testing, impact of the defect on the testing effort, and the risk the defect would present to the business if the defect was rolled-out into production.
Impacted areaThe Impacted area can be referenced by functional component or functional area of the system. Often both are used.
Status reportA test organization and members of the testing team will be called upon to create status reports on a daily, weekly, monthly and project basis. The content of any status report should remain focused on the testing objective, scope and scheduled milestones currently being addressed. It is useful to state each of these at the beginning of each status report and then publish the achievements or goals accomplished during the current reporting period, as well as those that will be accomplished during the next reporting period.
Any known risks that will directly impact the testing effort need to be itemized here, especially any "showstoppers" that will prevent any further testing of one or more aspects of the system.
Reporting periodThis is the period covered in the current status report. Include references to any previous status reports that should be reviewed.
Mission statementThe objective of the current testing effort needs to be clearly stated and understood by the testing team and any other organization involved in the deployment.
Current scopeThe components of the system being tested (hardware, software, middleware, etc.) need to be clearly defined as being "in scope," and any related components that are not being tested need to be clearly itemized as "out of scope."
Schedule milestonesAny schedule milestones being worked on during the current reporting period need to be listed and their current status clearly stated. Milestones that were scheduled but not addressed during the current reporting period need to be raised as risks.
RisksRisks are factors that could negatively impact the current testing effort. An itemized list of risks that are currently impacting the testing effort should be drawn up and their impact on the testing effort described.
By: David W. Johnson
There are core sets of test deliverables that are required for any software testing phase: test plan, test case, defect documentation and status report. When taken together this set of deliverables takes the testing team from planning to testing and on through defect remediation and status reporting. This does not represent a definitive set of test deliverables, but it will help any test organization begin the process of determining an appropriate set of deliverables.
One common misconception is that these must be presented as a set of documents, but there are toolsets and applications available that capture the content and intent of these deliverables without creating a document or set of documents. The goal is to capture the required content in a useful and consistent framework as concisely as possible.
Test planAt a minimum the test plan presents the test: objectives, scope, approach, assumptions, dependencies, risks and schedule for the appropriate test phase or phases. Many test organizations will use the test plan to describe the software testing phases, testing techniques, testing methods and other general aspects of any testing effort. General information around the practice of testing should be kept in a "Best Practices" repository -- testing standards. This avoids redundant and conflicting information from being presented to the reader and keeps the test plan focused on the task at hand –- planning the testing effort. (See "The role of a software test manager".)
Objectives -- mission statementThe objective of the current testing effort needs to be clearly stated and understood by the software testing team and any other organization involved in the deployment. This should not be a sweeping statement on testing the "whole application" -- unless that is actually the goal. Instead the primary testing objectives should relate to the purpose of the current release. If this were a point-of-sale system and the purpose of the current release was to provide enhanced online reporting functionality, then the objective/mission statement could be this:
"To ensure the enhanced online reporting functionality performs to specification and to verify any existing functionality deemed to be in scope."
The test objective describes the "why" of the testing effort. The details of the "what" will be described in the scope portion of the test plan. Once again, any general testing objectives should be documented in the "Best Practices" repository. General or common objectives for any testing effort could include expanding the test case regression suite, documenting new requirements, automating test cases, and updating existing test cases.
In scopeThe components of the system to be tested (hardware, software, middleware, etc.) need to be clearly defined as being "in scope." This can take the form of an itemized list of those "in scope": requirements, functional areas, systems, business functions or any aspect of the system that clearly delineates the scope to the testing organization and any other organization involved in the deployment. The "What is to be tested?" question should be answered by the in scope portion of the test plan -- the aspects of the system that will be covered by the current testing effort.
Out of scopeThe components of the system that will not be tested also need to be clearly defined as being "out of scope." This does not mean that these system components will not be executed or exercised; it just means that test cases will not be included that specifically test these system components. The "What is NOT to be tested?" question should be answered by the out of scope portion of the test plan. Often neglected, this part of the test plan begins to deal with the risk-based scheduling that all test organizations must address -- What parts of the system can I afford not to test? The testing approach section of the test plan should address that question.
ApproachThis section defines the testing activities that will be applied against the application for the current testing phase. This addresses how testing will be accomplished against the in scope aspects of the system and any mitigating factors that may reduce the risk of leaving aspects of the system out of scope.
The approach should be viewed as a to-do list that will be fully detailed in the test schedule. The approach should clearly state which aspects of the system are to be tested and how: backup and recovery testing, compatibility/conversion testing, destructive testing, environment testing, interface testing, parallel testing, procedural resting, regression testing, application security testing, storage testing, stress and performance testing, and any other testing approach that is applicable to the current testing effort. The reasoning for using any given set of approaches should be described, usually from the perspective of risk.
AssumptionsAssumptions are facts, statements and/or expectations of other teams that the test team believes to be true. Assumptions specific to each testing phase should be documented. These are the assumptions upon which the test approach was based. Listed assumptions are also risks should they be incorrect. If any of the assumptions prove not to be true, there may be a negative impact on the testing activities. In any environment there is a common set of assumptions that apply to any given release. These common assumptions should be documented in the "Best Practices" repository; only assumptions unique to the current testing effort and perhaps those common assumptions critical to the current situation should be documented.
DependenciesDependencies are events or milestones that must be completed in order to proceed within any given testing activity. These are the dependencies that will be presented in the test schedule. In this section the events or milestones that are deemed critical to the testing effort should be listed and any potential impact or risks to the testing schedule itemized.
RisksRisks are factors that could negatively impact the testing effort. An itemized list of risks should be drawn up and their potential impact on the testing effort described. Risks that have been itemized in the project plan need not be repeated here unless the impact to the testing effort has not already been clearly stated.
Schedule The test schedule defines when and by whom testing activities will be performed. The information gathered for the body of the test plan is used here in combination with the available resource pool to determine the test schedule. Experience from previous testing efforts along with a detailed understanding of the current testing goals will help make the test schedule as accurate as possible. There are several planning and scheduling tools available that make the plan easier to construct and maintain.
Test caseTest cases are the formal implementation of a test case design. The goal of any given test case or set of test cases is to detect defects in the system being tested. A test case should be documented in a manner that is useful for the current test cycle and any future test cycles. At a bare minimum, each test case should contain the author, name, description, step, expected results and status.
Test case nameThe name or title should contain the essence of the test case, including the functional area and purpose of the test. Using a common naming convention that groups test cases encourages reuse and helps prevents duplicate test cases from occurring.
Test case descriptionThe description should clearly state the sequence of business events to be exercised by the test case. The test case description can apply to one or more test cases; it will often take more than one test case to fully test an area of the application.
Test case stepEach test case step should clearly state the navigation, data and events required to accomplish the step. Using a common descriptive approach encourages conformity and reuse. Keywords offer one of the most effective approaches to test case design and can be applied to both manual and automated test cases.
Expected resultsThe expected results are the expected behavior of the system after any test case step that requires verification or validation. This could include screen pop-ups, data updates, display changes or any other discernable event or transaction on the system that is expected to occur when the test case step is executed.
StatusThis is the operational status of the test case. Is it ready to be executed?
Documenting defectsThe primary purpose of testing is to detect defects in the application before it is released into production. Furthermore, defects are arguably the only product the testing team produces that is seen by the project team. Document defects in a manner that is useful in the defect remediation process. At a bare minimum, each defect should contain the author, name, description, severity, impacted area and status.
Defect nameThe name or title should contain the essence of the defect, including the functional area and nature of the defect.
Defect descriptionThe description should clearly state what sequence of events leads to the defect. When possible include a screenshot or printout of the error.
How to replicateThe defect description should provide sufficient detail for the triage team and the developer fixing the defect to duplicate the defect.
Defect severityThe severity assigned to a defect is dependent on the phase of testing, impact of the defect on the testing effort, and the risk the defect would present to the business if the defect was rolled-out into production.
Impacted areaThe Impacted area can be referenced by functional component or functional area of the system. Often both are used.
Status reportA test organization and members of the testing team will be called upon to create status reports on a daily, weekly, monthly and project basis. The content of any status report should remain focused on the testing objective, scope and scheduled milestones currently being addressed. It is useful to state each of these at the beginning of each status report and then publish the achievements or goals accomplished during the current reporting period, as well as those that will be accomplished during the next reporting period.
Any known risks that will directly impact the testing effort need to be itemized here, especially any "showstoppers" that will prevent any further testing of one or more aspects of the system.
Reporting periodThis is the period covered in the current status report. Include references to any previous status reports that should be reviewed.
Mission statementThe objective of the current testing effort needs to be clearly stated and understood by the testing team and any other organization involved in the deployment.
Current scopeThe components of the system being tested (hardware, software, middleware, etc.) need to be clearly defined as being "in scope," and any related components that are not being tested need to be clearly itemized as "out of scope."
Schedule milestonesAny schedule milestones being worked on during the current reporting period need to be listed and their current status clearly stated. Milestones that were scheduled but not addressed during the current reporting period need to be raised as risks.
RisksRisks are factors that could negatively impact the current testing effort. An itemized list of risks that are currently impacting the testing effort should be drawn up and their impact on the testing effort described.
By: David W. Johnson
Integartion Testing
Q- How do testers do integration testing? What are top-down and bottom-up approaches in integration testing?
Expert’s response: Ironically, integration testing means completely different things to completely different companies. At Microsoft, we typically referred to integration testing as the testing that occurs at the end of a milestone and that "stabilizes" a product. Features from the new milestone are integration-tested with features from previous milestones. At Circuit City, however, we referred to integration testing as the testing done just after a developer checks in -- it's the stabilization testing that occurs when two developers check in code. I would call this feature testing, frankly…
But to answer your question, top-down vs. bottom-up testing is simply the way you look at things. Bottom-up testing is the testing of code that could almost be considered an extension of unit testing. It's very much focused on the feature being implemented and that feature's outbound dependencies, meaning how that feature impacts other areas of the product/project.
Top-down, on the other hand, is testing from a more systemic point of view. It's testing an overall product after a new feature is introduced and verifying that the features it interacts with are stable and that it "plays well"' with other features.
The key to testing here is that you are in the process of moving beyond the component level and testing as a system. Frankly, neither approach alone is sufficient. You need to test the parts with the perspective of the whole. One part of this testing is seeing how the system as a whole responds to the data (or states) generated by the new component. You want to verify that data being pushed out by the component are not only well-formatted (what you tested during component testing) but that other components are expecting and can handle that well-formatted data. You also need to validate that the data originating within the existing system are handled properly by the new component.
Real-world examples? Well, let's assume you are developing a large retail management system, and an inventory control component is ready for integration. Bottom-up testing would imply that you set up a fair amount of equivalence-classed data in the new component and introduced that new data into the system as a whole. How does the system respond? Are the inventory amounts updated correctly? If you have inventory-level triggers (e.g., if the total count of pink iPod Nanos falls below a certain threshold, generate an electronic order for more), does the order management system respond accordingly? This is bottom-up testing.
At the same time, you want to track how well the component consumes data from the rest of the system. Is it handling inventory changes coming in from the Web site? Does it integrate properly with the returns system? When an item's status is updated by the warehouse system, is it reflected in the new component?
We see constant change in the testing profession, with new methodologies being proposed all the time. This is good -- it's all part of moving from art to craft to science. But just as with anything else, we can't turn all of our testing to one methodology because one size doesn't fit all. Bottom-up and top-down testing are both critical components of an integration testing plan and both need considerable focus if the QA organization wants to maximize software quality.
By -John Overbaugh
Expert’s response: Ironically, integration testing means completely different things to completely different companies. At Microsoft, we typically referred to integration testing as the testing that occurs at the end of a milestone and that "stabilizes" a product. Features from the new milestone are integration-tested with features from previous milestones. At Circuit City, however, we referred to integration testing as the testing done just after a developer checks in -- it's the stabilization testing that occurs when two developers check in code. I would call this feature testing, frankly…
But to answer your question, top-down vs. bottom-up testing is simply the way you look at things. Bottom-up testing is the testing of code that could almost be considered an extension of unit testing. It's very much focused on the feature being implemented and that feature's outbound dependencies, meaning how that feature impacts other areas of the product/project.
Top-down, on the other hand, is testing from a more systemic point of view. It's testing an overall product after a new feature is introduced and verifying that the features it interacts with are stable and that it "plays well"' with other features.
The key to testing here is that you are in the process of moving beyond the component level and testing as a system. Frankly, neither approach alone is sufficient. You need to test the parts with the perspective of the whole. One part of this testing is seeing how the system as a whole responds to the data (or states) generated by the new component. You want to verify that data being pushed out by the component are not only well-formatted (what you tested during component testing) but that other components are expecting and can handle that well-formatted data. You also need to validate that the data originating within the existing system are handled properly by the new component.
Real-world examples? Well, let's assume you are developing a large retail management system, and an inventory control component is ready for integration. Bottom-up testing would imply that you set up a fair amount of equivalence-classed data in the new component and introduced that new data into the system as a whole. How does the system respond? Are the inventory amounts updated correctly? If you have inventory-level triggers (e.g., if the total count of pink iPod Nanos falls below a certain threshold, generate an electronic order for more), does the order management system respond accordingly? This is bottom-up testing.
At the same time, you want to track how well the component consumes data from the rest of the system. Is it handling inventory changes coming in from the Web site? Does it integrate properly with the returns system? When an item's status is updated by the warehouse system, is it reflected in the new component?
We see constant change in the testing profession, with new methodologies being proposed all the time. This is good -- it's all part of moving from art to craft to science. But just as with anything else, we can't turn all of our testing to one methodology because one size doesn't fit all. Bottom-up and top-down testing are both critical components of an integration testing plan and both need considerable focus if the QA organization wants to maximize software quality.
By -John Overbaugh
Friday, August 1, 2008
Performance testing in a virtual environment
Software testing in a virtual environment
Q- What is the likelihood of capturing accurate load testing results in a virtual test environment? We use LoadRunner/PerformanceCenter for performance testing. Our company is in the process of making use of virtualization. It seems this may be ideal for functional test environments, but not for performance test environment. What is your opinion?
Expert’s response: There are a lot of ways to use virtual environments in your performance testing, so there's no easy answer to this question. I'm assuming that you're referring to hosting the entire application in a virtual environment and running your performance testing against that platform. My answer is that, as always, it depends.
Some research on the topic has found that virtual environments don't scale as well as non-virtual environments. In a study by BlueLock, a company that provides IT infrastructure as a service, they found that "the number of simultaneous users that could be handled by the virtualized server was 14% lower than the number of simultaneous users being handled by the traditional server configuration."
This is consistent with my experience testing financial service applications in virtual environments. He points out some other common challenges with working in a virtual environment.
If you don't have much choice, or if you have a lot of pressure to make it work, I would recommend that you perform a comparison performance test to prove out the new platform. If you can do that successfully, you'll have some confidence that the platform is comparable. But just be aware that over time, as the application changes and the server configurations change (both virtual and the physical servers in production) your comparison will become outdated. It may happen faster than you might think.
As Scott points out in his talk, the problem isn't necessarily virtualization. It's that we don't always pay attention to all the other factors that affect performance. Differences in software and hardware configurations, network devices, geographic location, firewalls and other security measures, and a host of other factors all affect performance. Virtual environments often only make it more complex to track everything since they introduce their own overhead, rely on different network devices, and can reside in different physical locations.
It's not all doom and gloom. You might be able to virtualize some parts of your application quite successfully. For example, at the April Indianapolis Workshop on Software Testing, Ken Ahrens from iTKO shared an experience where he used the iTKO LISA product to enable service-oriented virtualization -- a process where you virtualize services that your application might rely on. In this case, if you were performance testing the core application and not the services it relies on, then that virtualization wouldn't necessarily affect your performance testing at all.
In that specific case study, before virtualization Ken's customer was unable to run performance tests on a regular basis due to service availability. Testing was an "event" where tens of teams had to get together to make a load test happen, and the high cost meant that this only happened a few times a year. Since they virtualized some of those services, they can run tests daily. Virtualization at the message level also gave them a greater ability to experiment with issues such as "what if this key service slows down significantly?" Or to try different data scenarios, such as "what if the lookup returns 900 records instead of 10 records?"
By Mike Kelly
Tuesday, July 29, 2008
Increasing tester interactions with developers
What can you do if all testing at a company is done with little interaction with developers? How can you change things to be more iterative? Testing expert Scott Barber explains.
On having little interaction with the developersSome testers lament that they don't have enough interaction with the development team, while other testers complain if they are expected to interact with developers. I have my preference (I like to be integrated as a single team, working literally side-by-side), as I'm sure most people do, but the fact is that this is as much a matter of preference and policy as anything else.
The key here is to figure out why you don't have much interaction with the developers. Then, based on that answer, decide whether to champion a change or deal with the status quo.
More interaction with the developers in and of itself will not make a team more iterative, more agile or more effective. More interaction will simply make the teams more interactive -- and in some environments interactions between developers and testers is not pleasant. So, before you decide to champion a change, make sure you know what you're getting yourself/your team into.
On becoming more iterativeThis is a project- to company-level question, not a test-team level question. To be honest, of all the possible company/project roles who should be involved in deciding what approach the company/project should take for software development, testers and test managers are least relevant to that decision.
Put another way, if the testers want the Rational Unified Process (RUP), the developers want Extreme Programming (XP), and the company as a whole wants to follow STEP, which approach do you think will result in the best software come release day?
The answer is "Whatever the developers want." Why? Because if you force the developers into a development model they don't like, they're going spend their energy hating the model and longing to do it the way they like to do it and end up writing bad code that they don't care very much about.
As a tester, it's my job to adapt to whatever works best for the development team. It's my job to help the developers make better software. If I think having the developers be more interactive with the testers and being more iterative will help the developers make better software, then I'm going to demonstrate that to them by interacting with them. Once I prove that interacting with me adds tangible value for them and they trust me, then maybe I'll suggest shortening iteration cycles or developer/tester pairing for test sessions before code is ever checked in -- or whatever.
By Scott Barber
On having little interaction with the developersSome testers lament that they don't have enough interaction with the development team, while other testers complain if they are expected to interact with developers. I have my preference (I like to be integrated as a single team, working literally side-by-side), as I'm sure most people do, but the fact is that this is as much a matter of preference and policy as anything else.
The key here is to figure out why you don't have much interaction with the developers. Then, based on that answer, decide whether to champion a change or deal with the status quo.
More interaction with the developers in and of itself will not make a team more iterative, more agile or more effective. More interaction will simply make the teams more interactive -- and in some environments interactions between developers and testers is not pleasant. So, before you decide to champion a change, make sure you know what you're getting yourself/your team into.
On becoming more iterativeThis is a project- to company-level question, not a test-team level question. To be honest, of all the possible company/project roles who should be involved in deciding what approach the company/project should take for software development, testers and test managers are least relevant to that decision.
Put another way, if the testers want the Rational Unified Process (RUP), the developers want Extreme Programming (XP), and the company as a whole wants to follow STEP, which approach do you think will result in the best software come release day?
The answer is "Whatever the developers want." Why? Because if you force the developers into a development model they don't like, they're going spend their energy hating the model and longing to do it the way they like to do it and end up writing bad code that they don't care very much about.
As a tester, it's my job to adapt to whatever works best for the development team. It's my job to help the developers make better software. If I think having the developers be more interactive with the testers and being more iterative will help the developers make better software, then I'm going to demonstrate that to them by interacting with them. Once I prove that interacting with me adds tangible value for them and they trust me, then maybe I'll suggest shortening iteration cycles or developer/tester pairing for test sessions before code is ever checked in -- or whatever.
By Scott Barber
Wednesday, June 4, 2008
Top Ten Tips for Bug Tracking
- A good tester will always try to reduce the repro steps to the minimal steps to reproduce; this is extremely helpful for the programmer who has to find the bug.
- Remember that the only person who can close a bug is the person who opened it in the first place. Anyone can resolve it, but only the person who saw the bug can really be sure that what they saw is fixed.
- There are many ways to resolve a bug. FogBUGZ allows you to resolve a bug as fixed, won't fix, postponed, not repro, duplicate, or by design.
- Not Repro means that nobody could ever reproduce the bug. Programmers often use this when the bug report is missing the repro steps.
- You'll want to keep careful track of versions. Every build of the software that you give to testers should have a build ID number so that the poor tester doesn't have to retest the bug on a version of the software where it wasn't even supposed to be fixed.
- If you're a programmer, and you're having trouble getting testers to use the bug database, just don't accept bug reports by any other method. If your testers are used to sending you email with bug reports, just bounce the emails back to them with a brief message: "please put this in the bug database. I can't keep track of emails."
- If you're a tester, and you're having trouble getting programmers to use the bug database, just don't tell them about bugs - put them in the database and let the database email them.
- If you're a programmer, and only some of your colleagues use the bug database, just start assigning them bugs in the database. Eventually they'll get the hint.
- If you're a manager, and nobody seems to be using the bug database that you installed at great expense, start assigning new features to people using bugs. A bug database is also a great "unimplemented feature" database, too.
- Avoid the temptation to add new fields to the bug database. Every month or so, somebody will come up with a great idea for a new field to put in the database. You get all kinds of clever ideas, for example, keeping track of the file where the bug was found; keeping track of what % of the time the bug is reproducible; keeping track of how many times the bug occurred; keeping track of which exact versions of which DLLs were installed on the machine where the bug happened. It's very important not to give in to these ideas. If you do, your new bug entry screen will end up with a thousand fields that you need to supply, and nobody will want to input bug reports any more. For the bug database to work, everybody needs to use it, and if entering bugs "formally" is too much work, people will go around the bug database.
Sunday, June 1, 2008
What is Equivalence Partitioning
Before actually starting off the testing, it is good to have optimum test cases in place which not only cover all the desired features but also are sufficient to detect good quality bugs. There are many design techniques for writing test case. One of the most popular among them is Equivalence Partitioning. It is a Black-box (Specification Based) Test Case Design Technique with two primary goals
1. To reduce the number of test cases to necessary minimum.
To cover all possible scenarios. Key Points:
1. Although in rare cases equivalence partitioning is also applied to outputs, typically it is applied to the inputs of a component being tested.
2. This method divides the input domain of a program into classes of data from which test cases can be derived.
3. It is based on an evaluation of equivalence classes for an input condition. An equivalence class represents a set of valid or invalid states for input conditions. An input has certain ranges which are valid and other ranges which are invalid
4. Supplemented by BVA( Boundary Value Analysis).Having determined the partitions of possible inputs the method of boundary value analysis has to be applied to select the most effective test cases out of these partitions
This may be best explained by the following example of a function which accept input parameter "month" of a date. The valid range for the month is 1 ( January ) to 12 ( December ). This valid range is called a partition. In this example there are two further partitions of invalid ranges. The first invalid partition would be <= 0 and the second invalid partition would be >= 13. You can also have invalid partition for non-digit input as well.
.... -2 -1 0 1 .............. 12 13 14 15 .....
------------------------------------------------------
invalid partition 1 valid partition invalid partition 2
Why Equivalence Partitioning
The testing theory related to equivalence partitioning says that only one test case of each partition is needed to evaluate the behavior of the program for the related partition. In other words it is sufficient to select one test case out of each partition to check the behavior of the program. To use more or even all test cases of a partition will not find new faults in the program. The values within one partition are considered to be "equivalent". Thus the number of test cases can be reduced considerably.
An additional effect by applying this technique is that you also find the so called "dirty" test cases. An inexperienced tester may be tempted to use input as 1 to 12 and forget to select some invalid partitions all together. This would lead to a huge number of unnecessary test cases on the one hand, and lack of test cases on the other hand.
Equivalence Partitioning in white box testing.
The tendency is to relate equivalence partitioning to the so called black box testing which is strictly checking a software component at its interface, without consideration of internal structures of the software. But having a closer look on the subject there are cases where it applies to the white box testing as well. Imagine an interface to a component which has a valid range between 1 and 12 like in the example above. However, internally the function may have a differentiation of values between 1 and 6 and the values between 7 and 12. Depending on the input value the software internally will run through different paths to perform slightly different actions. Regarding the input and output interfaces to the component this difference will not be noticed, however in your white-box testing you would like to make sure that both paths are examined. To achieve this it is necessary to introduce additional equivalence partitions which would not be needed for black-box testing. For this example this would be:
.... -2 -1 0 1 ..... 6 7 ..... 12 13 14 15 .....
-------------------------------------------------
invalid partition 1 P1 P2 invalid partition 2
TC1 TC2 TC3 TC4
valid partitions
To check for the expected results you would need to evaluate some internal intermediate values rather than the output interface.
Types of Equivalence Classes
Continuous classes run from one point to another, with no clear separations of values. An example is a temperature range.
Discrete classes have clear separation of values. Discrete classes are sets, or enumerations.
Boolean classes are either true or false. Boolean classes only have two values, either true or false, on or off, yes or no. An example is whether a checkbox is checked or unchecked.
Thus, Equivalence classes may be defined according to the following guidelines:
1. If an input condition specifies a range, at the minimum one valid and two invalid equivalence classes can be defined.
2. If an input condition requires a specific value, then one valid and one invalid equivalence class can be defined.
3. If an input condition specifies a member of a set, then one valid and one invalid equivalence class can be defined
If an input condition is Boolean, then one valid and one invalid equivalence class can be defined.
Subscribe to:
Posts (Atom)