Evaluation of the Classification Performance of Surface Inspection Systems

Surface inspection systems (SIS) are used in flat steel production for the inspection of strip surfaces. For the evaluation and specification of these systems, it is important to be able to quantify the performance of an SIS reproducibly with reasonable effort. With the draft guideline VDI/VDE/VDMA 2632 Part 4.2 published in April 2023, this is now possible. The method can be transferred to many classification tasks where the determination of a "ground truth" is not possible. With a view on data usage in the context of Industry 4.0, the guideline offers valuable assistance, as an increasing importance of valid data is to be expected.

"In the surface inspection of flat steel, it is a matter of detecting defects that are often only a few square millimetres in size and assigning them to a correct defect class. In the case of tin plates, for example, we are talking about steel strips about 15 km long and 1 m wide, which are inspected at a strip speed of up to 54 km/h. In the process, there may well be 200,000 events on one strip," said Dr. Jens Brandenburger, chairman of the VDI working group "Surface Inspection Systems." 

He added: "That's like trying to detect and classify the small stones on the road with the width of a bicycle lane, on a speedy drive from Düsseldorf to Duisburg." Since such an inspection result cannot be verified with reasonable effort, the so-called "ground truth," the basic truth about how many and which defects are actually on the strip, is missing.

At most, it is possible to compare the results of the SIS with those of the inspection staff for partial sections of individual strips. But what conclusions can be drawn from such random samples?  An example: A person looks at 1000 individual images that have been assigned to a certain type of defect, such as "scratches". If this person then finds that three scratches are actually cracks, what does this say about the performance of the classifier? In the next collection of 1000 images, there could be five misclassified images or there could be two.

Challenges in SIS assessment

This example shows some challenges in evaluating OIS classification performance in flat steel production. The following points summarise framework conditions in flat steel production that make it impossible to evaluate the classification result according to common methods as presented in VDI/VDE/VDMA 2632 Part 3.1:

• A very large number of events to be classified exist. Practically, it is therefore impossible to assign a "true" class to each event.  
• There is a lack of separating characteristics for an unambiguous class assignment.
• There is a high proportion of events that are not relevant to quality.
• The assessment of the product as "good" may be permissible even if detected individual events were classified as defects.
• Products and/or product areas can be divided into different quality classes or released for different purposes depending on the classification results. Not only binary good/bad decisions exist.
• Deferring samples for later comparison measurements is not possible.

The VDI/VDE/VDMA 2632 Part 4.2

The draft guideline VDI/VDE/VDMA 2632 Part 4.2 published in April 2023 presents a procedure for the performance evaluation of inspection systems that is practicable under the framework conditions of flat steel production. The draft was published in a bilingual (German/English) edition and can be commented on until 30.06.2023.

The guideline was written for the needs of surface inspection systems in steel production, but the methodology described in the guideline can also be transferred to many other areas of application for classifying image processing systems where no "ground truth" can be determined.

Production data as part of the company assets

An important aspect of Industry 4.0 is to use data from sensors, measuring and testing systems not only to automate the manufacturing process or for quality control, but also to use this data for continuous process optimization and thus also for strategic corporate decisions. In this context, quality data often form the target variable for higher-level analysis, for example in accordance with the VDI/VDE 3714 "Big Data" series of guidelines. The value of valid data is once again increased by the new methods for data analysis. As a result, the new draft guideline VDI/VDE/VDMA 2632 Part 4.2 on the performance evaluation of classification acquires particular relevance.

The demand on the reliability of the SIS results increases depending on the selected field of application and ranges from simple defect trend reports for the evaluation of the process quality and the early detection of quality deviations (quality monitoring) to automated release decisions for each individual strip. The new draft guideline VDI/VDE/VDMA 2632 part 4.2 therefore provides additional assistance for the evaluation of SIS results and the performance requirements depending on the selected field of application.