|
|
What are capture/replay and data-driven testing and when would each method be used? |
|
|
|
|
What is a risk-based approach to software quality? |
|
|
Is it possible for software to be perfect? |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
What are capture/replay and data-driven testing and when would each method be used?
The Capture/Replay technique has been around for sometime and has commonly been used for functional testing. Capture replay can be extremely effective in conducting quick and repeatable tests during a short implementation window. Often users and technicians work together, wherein users capture initial data and technicians continue to test with this data until the changes are implemented. The drawback to this technique is the difficulty in maintaining long-term scripts for regression testing. Applications with frequent screen changes significantly hamper re-use of captured scripts. Often significant editing of scripts is required utilizing this technique since both the data and the script are integrated. Each location where the screen has changed needs to be edited. For example, if "Phone 2" was added to the new account screen, everywhere that screen appears within the scripts will need to be edited to support entering a valid value or for tabbing past the field.
Data-driven testing separates the scripts that navigate through the code from the data that is used to populate the screens. A spreadsheet may be used to reflect field values that are called by the driver script. In this regard, if there are screen changes, there is only one script to be modified significantly reducing maintenance. In the case of "Phone 2" field being added, all that is necessary is adding a new column to the spreadsheet that contains the data and have the tool re-learn the changed screen. Where this might involve several hundred tests, minimal work effort is required to enable continued testing with the same data. Data-driven testing takes more time to orchestrate and set up, but is the better long-term solution for regression testing.
What are the most significant issues that should be addressed before implementing automated testing in my organization?
Before tools are acquired, it is most important to establish an effective testing process within which a tool can be integrated. Process definition should always precede tool acquisition. Those that have tried the reverse are unable to realize the benefits to be obtained from automation. Automation requires greater orchestration of resources than manual techniques. There are seven essential steps to properly assess, plan, and implement automated testing effectively:
Step 1:
Assess Environment
Step 2: Identify Gaps
Step 3:
Establish Framework
Step 4:
Select Tools and Prototype
Step 5:
Make Changes / Apply Lessons Learned From Prototype
Step 6:
Integrate Tools and Processes
Step 7:
Monitor and Measure Results
What is a risk-based approach to
software quality?
A risk-based approach to software quality utilizes risk as the determinant for priority of testing activities. Risk analysis involves evaluating the Impact of failure and determining the Likelihood that failure will occur. The process involves segmenting the application into logical pieces, such as function, modules, or business events. Each of these pieces is then analyzed to determine the types of errors that could occur, the impact of these errors on the business, and the likelihood that the errors will occur. The errors with the most impact and/or highest likelihood would, of course, be tested first. Depending on the nature of the application, potential failures with high impact, but low likelihood, may be candidates for thorough testing; an example would be miscalculation of drug dosage from a pharmaceutical system – it might not ever happen, but if it did the effects could be disastrous. Conversely, errors with low impact, but high likelihood, could be candidates for thorough testing depending on the volume of occurrences and resources required to correct them after implementation. Risk is viewed in terms of size, complexity, interfaces, history, and who was the programmer that wrote it. We might also include technology - is this state of the art - never been tried or based on fundamentally proven processes. These risk factors are used to determine impact and likelihood. In this regard, the most time and money is spent on testing those aspects of the system that can cause the most damage if error is not detected.
Is it possible for software to be
perfect?
It is possible, but probably not very cost effective, particularly with highly complex applications. First, we would have to spend some amount of time determining the definition of perfect. The term itself is likely to change for the business user within a short period of time. Testing is a risk mitigation process. You would not want to spend significantly more to identify risks than the consequences of them occurring. Those applications involving risks to human life, merit substantial analysis and risk aversion. Wherein the likelihood of an event occurring is small, and the consequences from the event occurring are small, less rigorous testing is often applied. One might ask: Is the system perfect that completes on time and under budget with no high priority or high severity errors, and no defects that are visible to the end user?
Software development has taken more time than anticipated and now there is not enough time to perform all of the planned tests. The delivery date cannot be slipped, so what should we do?
Test what you can, focusing on the highest risk business functions first. Create a list of priorities with the business team. Define those tests that must be completed before implementation, e.g., high priority business functions. Create a secondary list of tests, i.e., medium priority, that would be important to complete before implementation but would not prevent product implementation. Complete all high priority tests. Test all possible medium priority issues prior to implementation, but if not completed, continue testing after product implementation.
We are going to purchase software from a vendor. What should we do to ensure that they have tested their software sufficiently?
Review their risk analysis and corresponding test plans that detail the processes employed to verify the application. Review sample detail tests where possible. Another good idea is to review their defect reporting system to determine how well the vendor tests changes.
Because of the nature of our software we have not been able to create reusable test data, so every test is planned from the ground up. Is there anything that can be done to alleviate this situation?
This is often a challenge, but really not much unlike other problems that occur in this business. We find that whenever the client comes to understand the benefits associated with establishing iterative testing practices, solutions will be found. As many times as we have encountered this scenario, we have never seen a situation for which there were not solutions that built the desired equity for future testing.
I am in the process of setting up a test, but I don’t have a complete list of the requirements. What can I do to determine what to test and what to expect?
Unfortunately, as consultants, we are far too often confronted with this issue. The documentation provided does not convey a clear understanding of the expected result. The requirements should be the driving force to the testing effort, but frequently, detail requirements essential to predicting results do not provide sufficient support.
Here is one suggestion:
Create test conditions that clearly define the events and stages of validation for each event (there may be online inquiries to confirm initial input, inquiries to be performed after batch processing, or report or file outputs to be reviewed after batch processing). Conduct a walkthrough of these conditions with the business user to ensure that the proper validations are being considered for each point in time. Where possible, utilize the system to illustrate results. As you are doing this inquire as to the impact of related conditions; e.g.,
o "Would this action be the same if this were a delinquent account?"
o "Are there situations where this answer would not be correct?"
o Etc.
In this way you are building the business rules interactively with the user that can be reviewed by others.
Management says they are tired of hearing about problems, but the software still has bugs. What can I do to get their attention?
Educating management to the impacts of problems and the importance of detecting them early in the life cycle is no easy task. We have found that translating the problems into financial numbers tends to have a positive impact. It is important to understand the business and the translation into actual dollar impact. Some of these factors include:
o Production re-runs (printed media, operations time, paper costs, sorting & distribution of materials).
o Loss of Customer confidence (lose accounts/average revenue per client/ etc.) If 10, 100. 1000 customers walk away, what impact will it have on the corporation
o Labor costs - regular and overtime
Be armed with information pertaining to priority and criticality of the problems discovered. Provide meaningful actions and recommendations that can expedite discovery earlier in the life cycle. Create a cost trend line that illustrates the cost savings when problems are detected earlier in the development life cycle. By demonstrating your awareness of the business issues, you increase your credibility with management. Understand the cost issues, including the lost opportunity costs associated with late delivery.