Are Test Reports for Your Project Providing All the Information You Need?
Bill Hoberecht -
When conducting planning discussions with a test manager, Project Managers’ typically focus their attentions on the scheduling of test activities. All too often the topics of test goals and status reporting are ignored, the tacit assumption being that there is agreement in these areas. Effective Project Managers prepare for planning discussions on these often-ignored topics, ensure that these discussions are conducted, and that agreement is reached. This article outlines the Project Manager’s preparatory steps and describes a Test Reports toolkit that each project manager should have.
A Project Manager with a Problem to Solve
Drew was a project manager with a big problem: not enough relevant information about testing activities was being reported by the test team. Drew really had no idea how well (or poorly) testing activities were proceeding nor was any information available on the quality of the system under test. Imagine the problems when it came time for Drew to present project status to the PMO!
Starved for Information on Testing
Drew was leading a moderately sized project to deliver a new procurement management system. Because the existing system was not going to be made Y2K compliant, this project team had to purchase and test a new system and bring it operational well in advance of 1/1/2000. System testing was perhaps the most critical activity of the project. Unfortunately for Drew:
No information was ever provided by the test team on their pre-testing activities. Drew never knew if the test team was sufficiently prepared to start testing as scheduled.
Test execution started slightly behind schedule. Even worse, test execution status reports were not produced until nearly half-way through the test execution period.
The first few test execution status reports did not contain sufficient information to understand execution progress or defect discovery.
Although Drew was without timely or thorough test status information, there were many requests from executives and the PMO for project status information – for a few weeks, none of these requests could be fulfilled with information about test activities. Faced with this unmanaged situation, Drew’s Director eventually stepped in to establish a core set of required test status reports and was heavily involved in managing project activities during the test execution phase.
A quick post mortem on this project situation uncovered these key findings:
- Implied responsibilities. The Project Manager had an implied obligation to ensure that status reporting content and reporting schedule were defined for the project. Because Drew was an inexperienced Project Manager, these implicit responsibilities were not recognized or acted upon.
- No ‘standard’ reports. The company had no standard set of test reports that a project manager could expect an independent test team to provide. As well, the test team that was assembled for this project had no available library of standard test status reports.
Do You Have an Obvious Problem with Tests Status Reporting?
Drew’s predicament is not unique. I have repeatedly seen situations and heard from Project Managers about projects that had inadequate status reporting about a project’s test activities. Sometimes the gaps in reporting effectiveness are obvious, as in Drew’s case:
- Absence of reporting on certain test activities.
- Extraneous reporting. Irrelevant or unusable information is included in the status report.
- Delays in the start of reporting. Reports become available many days or weeks after the start of test efforts.
- Inconsistent reporting from one report to the next as the test manager or project manager discovers the need for additional information to be reported. (Those in a positive frame of mind might describe this as “test reporting content evolves to meet emerging information needs”)
I suspect that it is quite likely that even experienced project managers (including those in an organization that uses standard test reports) do not have ample visibility of testing activities on their projects; as a result, the Project Manager cannot possibly have a sufficient understanding of how testing activities are impacting the overall project status. Having inadequate status information about testing will be difficult to detect without some early groundwork, which is the topic of the next section.
Constructing your “Project Manager’s Toolkit – Test Reports”
At last we reach the key premise of this article: Project Managers are best positioned to heavily influence the definition of test status reports by preparing and giving substantial consideration to the type of status information that a project manager really needs. The insights you gain through your studies will be valuable when planning discussions are initiated with the test team.
Project managers should never follow a path of carelessly accepting the content of ‘standard’ test reports that are defined without their involvement; while a laissez-faire approach may give some useful information, the project manager does have an obligation to ensure that their complete information needs are defined and incorporated into reporting by the test organization.
These tips for Project Managers can help structure your preparations for planning discussions with the test managers on your projects. Independent of your planning for any specific project, devote some focused attention on test report contents and format. The first four activities described here involving locating and creating information that will comprise your own reusable “Project Manager’s Toolkit – Test Reports.”
Become familiar with the content and format of several different test reports.
Expand your base of knowledge by looking at test reports that others use. There are plenty of easily visited places to find example test reports: glance through some books on testing, look on the web, ask your PMO and other PMOs, and talk with other project managers throughout your company. Don’t aim low on this task, have a target of locating and understanding at least 10 different reports with more than half of these from outside your company.
Develop a framework of testing goals.
Not all test organizations have identical goals for their testing activities. For some test teams, the emphasis is on problem detection, while others may go beyond this to also include ownership for driving problem resolution. For some organizations it is inflexible that 100% of all tests will be executed and 100% of all requirements will be validated, while others may have relaxed execution and validation targets that achieve a different goal of meeting immediate business needs. Create a framework of different testing goals; the key here is to understand the different dimensions of testing goals. Your framework of testing goals becomes the basis for your discussions with the test team that will ensure alignment on each project. The goals you agree for any specific project will be the primary influencer in planning and decision-making.
Enumerate the questions you need answered by a status report.
Consider your own information needs, ensuring that these all relate to the agreed testing goals. As a project manager, you’ll need to know that testing activities are adequately planned and that execution is proceeding to achieve the agreed testing goals.
Build your toolkit of expected test reports.
Now it is time to construct your version of test reports; this will be your own thoughts on the information and format of test reports that you want to see from a test team. One easy method to create this report is simple: draw the various charts/tables freehand (using pencil and paper) and then scan that document into a pdf file; alternatively, you might construct a report using MS-Office tools. Your focus is on the content and representation, not on the underlying mechanism that the test team will use to collect the input data and producing their version of the reports you need.
Revise your toolkit as you gain experience using it on your projects.
Once you’ve created your initial toolkit of expected test reports you will have joined a relatively small population of project managers that have completed this degree of preparation. You’ll gain new insights during each project that you manage. Keep your toolkit current; re-examine your toolkit near the completion of each project and incorporate any new learnings that have emerged.
What’s Next: Using your “Project Manager’s Toolkit – Test Reports”
Because you have researched test reports and have created a toolkit of general testing goals and reports, you are now well-prepared for planning discussions on all of your projects. Your planning discussions with the test manager will now go well beyond typical discussions that only cover testing schedules. You’ll want to use your toolkit in covering these topics:
- Ensuring alignment on testing goals. Testing is much more than just a set of project activities. Assuming, without any discussion, that the testing goals are well understood and fully agreed will likely be problematic. Have the discussion
- Agreeing on the questions that must be answered by the various test reports prepared by the test team.
- Agreeing on test reports content. (Where important, agree on format as well.)
This approach of preparing a toolkit that covers your information on test reports, and then using that toolkit in planning discussions can be a very effective vehicle for ensuring that you have useful, meaningful and sufficient information about testing activities on your project.