SWT-03: Software Inspection

AlexQin

There are variuos names for the same thing. Some call it software inspection, which also could extend to the design and its documentation, some call it code inspection which relates more to the source code. A third name would be Fagan Inspection, called after the person who invented this quality assurance and testing method.

Code inspections are a highly efficient test method which can not be substituted by any other test methods. It is time consuming but according to statistics it will find up to 90% of the contained errors, if done properly. However it all depends on the methods and checks applied and on the diligence of the inspectors. It must not be confused with the so called "code review" or "walk through" which is usually done in a single meeting lasting for a couple of hours. A proper code inspection may take several days and needs the help of tools to browse the symbols in order to find the places where they are used. The code review can be used in addition to e.g. to generated acceptance of a software package by the integrators, but it must not be a substitute for a proper inspection. Proper inspections can be applied for almost all work products in the software life cycle. At the first glance they may look very time consuming. But statistical evaluations have shown that over the whole life cycle of the software development they even save resources and thus money and improve the quality of the product.





Source of the diagram: Michael Fagan


Most people are not aware that the manual static testing methods i.e. inspections, reviews and walk throughs are defined in the "IEEE Standard for Software Reviews". This is the IEEE 1028-1997 standard. I want to give a short overview on the main definitions in this standard, however I will not discuss the "Management Review" which is in the widest sense a check of a project's performance and the related documents. I will also omit a discussion of the Audits described in the standard, which a more relalted to having external checks on work products and processes. I will focus on the review techniques for technical work products as they are typically used within a company. I also want to point out the problems involved with these methods and make an attempt to present some solutions for these problems.

AlexQin

Walk-throughs

A walk-through can have a twofold purpose. First of all is can be performed to evaluate a software product with the purpose of:

• Find anomalies
• Improve the software product
• Consider alternative implementations
• Evaluate the conformance to standards and specifications

In summary you could say that this kind of walk-trough is a method which should be used throughout the design phase of a software product to collect ideas and inputs from other team members which lead to an overall improvement of the product. The second objective of a walk-through is to exchange techniques and perform training of the participants. It is a method to raise all team members to the same level of knowledge regarding programming styles and details of the product. In a sense it also generates agreement within the team about the object of the walk-through.

The formal aspects of a walk-through have a low profile. There are only a few roles defined in the standard. There is a walk-through leader, a recorder, the author of the work product and team members. The standard says that at least two members have to be assembled for the walk-through and the roles can be shared among them. The walk-through has to be planned which means that the participants have to be defined and the meeting has to be scheduled. Further the findings and outcomes of the meeting have to be recorded.

In total this is a nice and easy to use method for everyday technical work from which you can expect good benefit.

AlexQin

Technical Reviews

About the purpose of techincal reviews the IEEE standard says: "The purpose of a technical review is to evaluate a software product by a team of qualified personnel to determine its suitability for its intended use and identify discrepancies from specifications and standards." In other words the technical review is a meeting in which a team analyzes a work product to see it its quality is as expected or if it needs some improvement. The standard further states that not necessarily all aspects of the review object have to be examined and that it is a possible purpose of the meeting to come up with alternatives for a better design. The list of work products for which the review can be applied is quite big: Software requirements specification , Software design description, Software test documentation, Software user documentation, Maintenance manuals, System build procedures, Installation procedures and Release notes are possible candidates for the review. The review meetings should be planned in the project plan or they can be held on request e.g. by the quality group. The roles involved in a technical review are as follows:

• Decision maker
• Review leader
• Recorder
• Technical staff
• Management staff (optional)
• Other team members (optional)
• Customer or user representative (optional)

According to the IEEE standard the input to the technical review shall include the following:

• A statement of objectives for the technical review (mandatory)
• The software product being examined (mandatory)
• Software project management plan (mandatory)
• Current anomalies or issues list for the software product (mandatory)
• Documented review procedures (mandatory)
• Relevant review reports (should)
• Any regulations, standards, guidelines, plans, and procedures against which the software product is to be examined (should)
• Anomaly categories (See IEEE Std 1044-1993 [B7]) (should)

Instead of describing the technical review procedure in words I put this into a diagram. In the following flow chart you find the steps of the review, the expected inputs i.e. work product and other documents, and a rough description of each process step in the comment boxes:

AlexQin

Inspections

The inspection as decribed in the IEEE standard is basically the same as the Fagan Inspection, as invented and described by Michael Fagan in 1976. Compared to the technical review there is more formalism and more roles, etc. involved. Further down you will find a comparison and discussion of these review techniques so I will not go into details here. The only thing which needs to be pointed out at this place, is that the IEEE standard states that the inspection should be done according to the project plan. It is usually not just done on demand. Further there is an explicit reference to the software verifcation and validation plan where the inspections should be reflected. An inspection is therefore regarded as a proper testing activity rather than an activity to evaluate a work product for suitability. The IEEE standard says that the following roles shall be established for the inspection:

• Inspection leader
• Recorder
• Reader
• Author
• Inspectors

According to the IEEE standard the input to the inspection shall include the following:

• A statement of objectives for the inspection (mandatory)
• The software product to be inspected (mandatory)
• Documented inspection procedure (mandatory)
• Inspection reporting forms (mandatory)
• Current anomalies or issues list (mandatory)
• Inspection checklists (should)
• Any regulations, standards, guidelines, plans, and procedures against which the software product is to be inspected (should)
• Hardware product specifications (should)
• Hardware performance data (should)
• Anomaly categories (see IEEE Std 1044-1993 [B7]) (should)

Again here I have a description of the inspection procedure in a diagram. In the following flow chart you find the steps of the inspection, the expected inputs i.e. work product and other documents, and a rough description of each process step in the comment boxes:

AlexQin

Comparison of Inspections and Reviews

As you can see from the diagrams a technical review in not the same as an inspection. There are big differences and I want to summarize them in the following table:



There are some additional things to be observed. According to the standard the inspection leader has to have a formal training about the inspection process. Further from practical experience it can be determined that an inspection or review meeting should not be longer than 2 hours. After that time the quality of the inspection or review decreases. Another rule is that in an inspection you should cover a maximum of 100 lines of code per hour and a maximum of 400 lines of code per day.

AlexQin

Practical Problems with Inspections

An inspection requires a big effort and lots of resources. The number of participants in an inspection is usually around 5 to 6 persons. However in modern industry it is very difficult to get 6 people who fulfill the "requirements" to attend in numerous inspection meetings. We do not live in 1976 any more. Time pressure is high and development teams are global. If you think of a small piece of software, let's say 2000 lines of code and if you try to keep the inspection rules this would mean that you have to schedule approximately 10 inspection meetings to do a complete inspection of the code. In case the leader finds that preparation was not well done he has to re-schedule the meetings. In a modern company with high time pressure in development and a global setup of the teams this seems to be almost impossible to accomplish, especially if you consider that a medium microcontroller application consists of 20000 and more code lines!

The formalism in the inspection process is heavy. However, the formalism does not guarantee that bugs are detected! Formalism in a way guarantees that the inspection meetings are performed in the intended way and by this it may be implied that the quality of the inspection has a certain standard, but that's all. You guarantee that a number of meetings were held according to the rules. But there is no real benefit in it and no guarantee that the real task i.e. TO FIND ANOMALIES is really accomplished to an extend that justifies the heavy effort.

The preparation period as described in the standard and as performed in real life is very vague! The checklists may not cover everything that is necessary. Usually they are too general. A typical question in such a checklist is: "Are overflows/underflows in calculations considered?". Then it is left up to the inspector to find a way to check this. Some experienced inspectors may have a standard method for themselves and they produce good results, but there is no clear method about the exact procedure that has to be followed in the preparation period or in the inspection meeting. This is left completely open. However, this activity IS THE BUG FINDING ACTIVITY! Especially during the preparation time the bugs are found. In the meeting you may have the effect that things become more clear when they are presented in the words of the reader and additional problems are found, but the main bug finding activity is the preparation time.

AlexQin

The "Specified Inspection" - a better Inspection Process

The following suggestion for a different inspection process was not developed on a theoretical basis. It is based on real life observations and after an adaptation of the inspection process also a very successful usage for real projects. Some years ago I was part of a team performing inspections on source code. As the years went on I had the increasing feeling during these meetings that what was done was not good enough. It left the impression on me to be superficial. On top of that we increasingly had problems to run the inspections according to the process.

The inspections we did have not been 100% according to the process anyway. Some of the roles were assigned to one person and the follow up process was never part of the inspection but we always used our established standard problem resolution process. However, the inspections still worked o.k. But as time went on we did not manage to set up the meetings any more. They were scheduled, but since the team size decreased we ended up in having only half of the expected participants attending, because the other team members were caught up in meetings with the customer, or other important activities. On the non-working level we realized that the times we formerly had for a coffee and a chat had also decreased and it was not only in our department. It was a company wide and even industry wide phenomenon. After suffering from a slow change of worklife we suddely realized that we were not living in 1976 any more. Therefore a change of the inspection process had to be considered as well.

The old process just did not match modern work life any more! After one of the inspections it was more out of curiosity that I asked a colleague from a different department to have another look on the source code to search for bugs. This colleague had never done any reviews or inspections. So I had to give him more details on each question of the checklist, explaining how to examine the code. The bad thing was, that this colleague found some bugs, really critical bugs which slipped our previous inspection. This was a clear sign that we had to do something about our inspection process. It was impossible to turn the clock back and get more manpower and a more relaxed planning, so we had to go a different way.
Basically this was the birth of the new Specified Inspection method. It is based on the inspection we knew so far, but reduces formalism to a level that can be lived. Further from the experience of success I had with a different kind of "checklist" I emphasized the "preparation period" to make it a specified activity with clearly expected work steps, thus turning it into a real bug finding phase. I want to go into the details of this process now and things can be best explained with this picture followed by an explanation of the details and differences:

AlexQin

Reduced Roles

Manpower became very expensive and teams are much smaller than they have been 30 years ago. Therefore one of the first optimizations was to focus on the reduction of manpower in the inspection process. On the one hand a reduction is possible due to a different procedure which does not need an inspection meeting, on the other hand a reduction is achieved by using already existing roles as they are usually present in a software development process. Thus embedding the inspection process into testing rather than keeping it as a separate process. The following roles are needed:

• The autor. The author is responsible to collect the work product and all necessary supporting documents for the inspection. This is e.g. the source code and the related design document. Further the author shall initiate the inspection by a "hand over" document in which the work product and supporting documents are clearly identified and other information is given to the inspectors as e.g. known anomalies, the scope of the inspection or things to observe. Further the author has the responsibility to support the inspectors during their inspection and answer questions regarding the work product or the intended functionality. The author also has to participate in the problem resolution meeting and participate in the definition of action items as a result of the inspection.
• The inspector(s). An inspector is a specially educated person who's job is to perform inspections and testing. The inspectors of the normal inspection as described in the IEEE 1028 are also required to be educated. However this is more an education about the inspection process itself. The inspector of the Specified Inspection however needs a special education concerning the error finding techniques, and if the inspection is related to source code, he also needs an education about the programming language and it's traps and pitfalls. Further it is recommendable to have two inspectors examining the same work product at the same time.
• The test manager. The test manager in a software development process is responsible for the test planning. He also shall perform the planning for inspections. As it was pointed out in the presentation of the IEEE 1028 inspection process an inspection is regarded to be a testing activity. Therefore there is no reason to have a separate planning for inspections. It just can be planned together with the other testing. Further the test manager has the responsibility to perform the moderation and if necessary arbitration in the inspection process. Whenever inspector and autor enter into unresolvable discussions the test manager has to intervene.
• Other team members. The involvement of other team members is optional. They are not regarded to be a proper role in the Specified Inspection process. Other team members may be involved in the problem resolution meeting when it comes to the definition of action items. The author may not be able to decide about schedules and activities so that other colleagues or the project manager needs to be involved.

AlexQin

Planning

The planning of inspections has to be part of the overall test planning. The strengths and weaknesses of inspections have to be considered and the planning has to reflect the use of the right testing methods at the right time. Testing is typically done on source code, also the original Fagan Inspection was focusing on source code. However inspections may extend also to design documents, requirements or other work products. In this case the test planning is not the right place for the inspection planning. It should be rather part of the overall project planning in case inspection is performed on other work products. However, the overall principles and process of the Specified Inspection can be also applied on other work products, if appropriate inspection specifications are used.

AlexQin

Inspection

The inspection process itself is different to the so far know inspection process. It can be summarized in the following features:

• Interaction with Static Code Analysis by Tools. During the past few years powerful static code analysis tools were developed. These tools have to be used, since they cover a large amount of checks which 20 or 30 years ago had to be done by inspections! An example of this may be the order of evaluation problem in C. A code line like: x = i++;is dangerous. It is not clear if the indexing is done first or the increment. Some years ago it was subject of an inspection to check the code for problems like this. However now there are tools which find these kind of problems easily. Inspection nowadays should only supplement these tools in areas where an automatic check is not possible. These areas are mainly the remaining statically detectable coding problems like underflow and overflow at calculations and a check of the conformance of the work product with it's specification. Below is an example of possible items that remain to be checked by an inspection after the check by a static code analysis tool is done.
• Specification. The inspection steps have to be clearly specified. The usual checklists are not enough. They only say "what to do". A proper specification goes into details about "how to do" an inspection step. An example of an inspection specification is given below.
• Inspection Conduct. The inspector uses the inspection specification and follows it step by step and applies it to the work product. This will usually take the inspector several times through the code examining it each time for a different purpose. The inspection is performed at the inspector's desk as part of his usual woking hours. The inspector shall observe some rules regarding the inspection. He should not be faster than a maximum of 150 lines of code per hour and shall perform a maximum of 4 hours inspection a day. The lines of code have to be netto lines of code, i.e. without comments and blank lines. This is to maintain the quality of the inspection. These rules have to be considered in the planning of the inspection. The inspector will record any anomalies during his inspection. Although the inspection has to follow a specification the inspector is obliged to apply his expert knowledge and also report anomalies which may not be covered by the specification. In case a found defect is not covered by the specification it shall lead to an improvement of the specification.
• Multiple Inspectors. Four eyes seem more than two. This is an old principle which also applies to software inspection. It is recommended that you have always two inspectors working on the same work product. During the inspection they can compare their results and come to a combined output. But at least at the end of the inspection they have to put their results together into one report. This will inevitably lead to a discussion about found or not found anomalies and this way the inspection will be of better quality. The preferences of a single person will be reflected against the opinion of the second person, thus avoiding extremes and false alarms on the one hand and confirming real problems on the other hand.
• Reporting. The inspection results shall be recorded in a proper test report. In case there were multiple inspectors their results have to be arbitrated first and a common conclusion of the two inspectors has to be documented in the test report. The contents of the report have to be the same as for a standard test report according to IEEE 829. These are e.g. The name of the tester(s), date and duration of the inspection, an identification of the inspection object, the objective of the inspection, the used specification, a list of the found defects including a classification of the defects, the overall result of the test i.e. passed or failed, etc. The report has to be build up in a way that a defect data collection is supported.