Defect Classification of Electronic Circuit Board Using Multi-Input Convolutional Neural Network

Automatic Optical Inspection (AOI) is introduced in the manufacturing process. Detected defect is classified by the human eys check and human eye check may cause problem of unbalanced accuracy and that of cost. Based on these reasons, automatic defect classification is desired to the manufacuturing process. This paper proposes a convolutional neural network (CNN) of multiple input images with two different connection layers using two test images taken under two different conditions of illumination. Comparison is demonstrated in the experiments and the result suggests that better accuracy is obtained from the multi-input CNN which connects the two different connection layers near input layer. The performance of the proposed approach was validated with the obtained result of experiments.


Introduction
Printed Circuit Board (PCB) is crucial part of electronic device it needs to be properly investigated before get launched.Automatic inspection systems are used for this purpose but due to more complexity in circuits, PCB inspections are now more problematic.This problem leads to new challenges in developing advanced automatic visual inspection systems for PCB.
Automatic Optical Inspection (AOI) has been commonly used to inspect defects in printed circuit board during the manufacturing process.An AOI system generally uses meth-ods which detects the defects by scanning the PCB board and analyzing it.AOI uses methods like Local Feature matching, image Skeletonization and morphological image comparison to detect defects and has been very successful in detecting defects in most of the cases but production problems like oxidation, dust, contamination and poor reflecting materials leads to most inevitable false alarms.To reduce the false alarms is the concern of this paper.
There are previous approaches for detecting defects of elec-tronic circuit board such as papers [1,2].Maeda et al. [1] uses infrared light image to detect the defect by testing electrically, while Numada et al. [2] extracts the global features using the interest point and its surrounding points and detects the defect using Mahalanobis distance.
Papers for defect classification have been also proposed in papers [3] and [4].Paper [3] improves the accuracy of classification using two images taken under the different Lighting conditions, and detects the defect region from the difference between the test image and the reference image which is prepared in advance.Support Vector Machine (SVM) [5] is used to classify into two classes of true defect and pseudo defect.Multiple classifiers are used with multiple subsets using random sampling of data set.The approach has an advantage to take voting processing for the defect classification when number of data set is large but it is still necessary to generate and prepare the reference image.
Paper [4] has been proposed as an approach without using reference image for the test of target defect.Key point extraction is used to crop the defect candidate region and SVM is used to classify the cropped defect region using the extracted features when cropped defectregion is input to the learned Convolutional Neural Network (CNN).
However, this model is not applied to the multiple input images and it is not possible to make classification using two images taken under the different Lighting conditions.
In the computer vision fields such as image matching, channelwise concatenations of inputs or feature maps have been widely studied [6][7][8].Research for classification of the flower image is also represented in the image classification [9].This paper improves the accuracy by learning CNN using two images taken under the different lighting conditions.The proposed method tries to apply the multiple dimensional features originally under two different lighting conditions to the research field of defect classification of electronic board.No similar researches have been proposed in this defect classification field since the most previous researches of this defect classification of electronic board just use the feature vectors obtained from the original image and this paper tris to use two different lighting conditions simultaneously and develops the new contribution of this defect classification field using deep learning architecture.It is shown that the proposed approach improves the classification accuracy for the electronic board image with the defect through the comparison with the previous approaches.The effectiveness is also confirmed in the experimental results.

Lighting Conditions
Test image used for this research is the detected defect by AOI, which was taken again by the device with the human eye check called "verification device".Verification device takes two kinds of images under two different lighting conditions.Image taken with large angle lighting is named "coaxial main" and image taken with both side lighting and large angle lighting is named "side main" in this paper.
Figure 1 shows the outline of each lighting condition.Coaxial main and side main are ring lights structure and diameter of the side main is larger than that of coaxial main.Taking image with this structure enables not to make difference of position between two kinds of image.Coaxial main illuminates the board from the vertical direction as shown in Figure 1(a) while side main illuminates board from side directions in addition to the vertical direction as shown in Figure 1(b).

Kinds of defect
Defects of the electronic board consists of true defect and pseudo defect.True defect cannot be used as a product and it is necessary toremove the true defect at the checking test, while pseudo defect is no problem to be used as a product by removing the dust and so on.Images of true defects in the electronic board are shown in Figure 2, Similarly images of pseudo defects in the electronic board are shown in Figure 3. Upper images are taken by thecoaxial main lighting conditions and lower images are taken by the side main lighting conditions.Coaxial main lighting is useful to the attached foreign matter, color change, and lack and so on, while side main lighting is useful to the edge, 3D defect, projection, and foreign matter.

Multi-Input CNN model
Deep learning and CNN technologies have been introduced as the effective pattern recognition approach in computer vision.CNN has been high contributions since AlexNet [10] (2012) in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [11].CNN mainly consists of convolutional layers, pooling layers and full connected layers.Convolutional layers and pooling layers are important since convolutional layers calculate convolution of image and multiple filters, extract the features where abstraction features are

Dataset and data augmentation
Data set consists of 805 samples respectively for coaxial main lighting and side main lighting image with 256 × 256 pixels.Numbers of true defect and pseudo defect are 470 and 335 respectively.50 samples of true defect and 50 samples of pseudo defect are randomly selected for the testing evaluation and remained 420 true defects and 285 pseudo defects are used for the learning.
From the situation that number of learning data is not enough for the sufficient learning, data augmentation is applied and number of learning images were increased to 5 times of the original data with flip horizontal, rotation and scaling transformations.

Experiments
Two Multi-inputs CNN were used for the learning by inputting actual electronic circuit board images.Keras [12] was introduced as a high level neural network library for the implementation.Binary cross entropy was used for the loss function and SGD and Adam [13] were used for the optimization.Initial value for the learning rate was 5e-4 and momentum parameter was 0.9 and parameter to decreasing of learning rate at each epoch was 1e-4 as parameters of SGD.Learning rate was lr=5e-4 and parameter to decreasing of learning rate at each epoch was 1e-4 as parameters of Adam.Other parameters were given in the similar way as β 1 = 0.9, β 2 = 0.999.
Let the learning rate of Adam be lr, initial learning rate be lr 0 , parameter to decreasing at each epoch be k, number of epoch be t, then learning rate is updated using Equation (1).
Learning was done under the condition of batch size 32 and maximum epoch 100.Table 1 shows the confusion matrix when positive sample means true defect while negative sample means pseudo defect.Table 2 shows the classification results.Here let the CNN connected at around output layer be CNN1 and the CNN connected at around input layer be CNN2.
Table 2 suggests that CNN2 gives the higher accuracy than CNN1 using both optimization algorithm.It is suggested that CNN connected at around input layer can learn the better features from the property that position does not change between both lighting conditions.It is also confirmed that CNN2 gives the same level accuracy in both optimization algorithms but CNN1 suggests Adam gives better accuracy than SGD.

Comparative experiments
Proposed approach was compared with two previous classification approaches.One is to use SVM for the classification after using a learned CNN as feature extractor as shown in paper [4].The other  is Fine-tuning of learned CNN using small number of data set.ImageNet [14] was learned to VGG16 [15] and this CNN model used for the feature extraction and finetuning.CNN features were extracted as 4096 dimensional features from fc7 layer of learned VGG 16 of ImageNet.Then extracted features were learned by linear SVM.Three cases of learning CNN features by SVM are evaluated for the cases of using only the coaxial main, only the side main and using both of them.When two lighting images are used for the classification, 4096 dimensional features are extracted, connected and 8192 dimensional features were used.Classification results are shown in Table 3 using the extracted CNN features and SVM.Table 3 suggest that using both lighting condition gives best classification accuracy.
In the fine-tuning of VGG16 which learned ImageNet, size of input layer was changed to (256,256,3), number of output layer was changed to 2 from the original 1024.Binary cross entropy was used as loss function and SDM and Adam [13] were used for the optimization.
Initial value for the learning rate was 1e-4 and momentum parameter was 0.9 and parameter to decreasing of learning rate at each epoch was 1e-4 as parameters of SGD.Learning rate was lr=1e-4 and parameter to decreasing of learning rate at each epoch was 1e-4 as parameters of Adam.Other parameters were given in the similar way as β 1 = 0.9, β 2 = 0.999.Learning rate is updated using Equation (1).Learning was done under the condition of batch size 32 and maximum epoch 100.Table 2 shows the classification results.Classification result of fine-tuning of VGG16 which learned ImageNet is shown in Table 4.
Accuracy Graph of comparison with CNN1, CNN2, CNN feature and SVM, Fine-tuning of Learned CNN is shown in Figure 10.Fine-tuning1 represents the learned result of coaxial main images, while Fine-tuning 2 represents the learned result of side main images.CNN Feature + SVM represents the result using both coaxial main and side main images.CNN1, CNN2, Fine-tuning represent the accuracy for he learned CNN using Adam optimization algorithm.
Figure 10 suggests that CNN2 connected at around input layer gives the best accuracy and Fine-tuning 2 (of VGG16 learned for ImageNet) gives the second best accuracy.It is also confirmed that side main lighting images gives better accuracy than coaxial main images from Table 3 and Figure 10.These results suggest that CNN2 gives the higher classification accuracy by learning the features between two different lighting images at around input layer.Some misclassified images by Fine-tuning 2 but correctly classified images by CNN2 are shown in Figure 11.Upper images are side main images while lower images are coaxial main images.
Fine-tuning 2 misclassified the image which were difficult to see the defect position by the coaxial main lighting but CNN2 using both images could correctly classify the defects in Figure 11.

Conclusion
This paper proposed a new approach using multi-input CNN architecture to use two different lighting images and evaluated the classification performance using defect images of electronic circuit board.Previous approaches need the reference images under two lighting conditions, while the proposed approach does not need reference images and could correspond to the defect classification using CNN from the feature extraction and classification process.
The evaluation in the experiments suggest that CNN connected at around input layer gives the highest accuracy in defect classification via comparison with two different CNN architectures.Further the proposed approach was compared with the CNN feature + SVM, and Fine-tuning from the learned CNN.It is shown that the proposed Multi-input CNN2 gave the best classification accuracy among them.
True defects consist of connection, projection and lack while pseudo defects consist of large dust, dust and oxidation.Classification was performed to the true and pseudo defect but classifications into multiple cases or corresponding to the multiple defects in an image are remained as future works.

1
Department of Computer Science, Chubu University, Japan 2 Department of Computer Engineering, Chulalongkorn University, Bangkok, Thailand True and pseudo defects are classified into several kinds based on Citation: Shiina T, Iwahori Y, Kijsirikul B (2018) Defect Classification of Electronic Circuit Board Using Multi-Input Convolutional Neural Network.Int J Comput Softw Eng 3: 137.doi: https://doi.org/10.15344/2456-4451/2018/137Page 2 of 7 the color and shape.Same kind of defects have different features in an image and this may cause difficult classification problem.

Figure 4 :
Figure 4: CNN Connected at around Output Layer.

Figure 6 -
Figure 6-9 show the transitions of Accuracy and Loss for each condition.Blue line represents "Training Accuracy", red line represents "Training Loss", orange line represents "Validation Accuracy" and green line represents "Validation Loss", respectively.Figure 6-9 suggests Adam gives faster optimization in the learning than SGD in any Multi-input CNN.
Figure 6-9 show the transitions of Accuracy and Loss for each condition.Blue line represents "Training Accuracy", red line represents "Training Loss", orange line represents "Validation Accuracy" and green line represents "Validation Loss", respectively.Figure 6-9 suggests Adam gives faster optimization in the learning than SGD in any Multi-input CNN.

Figure 5 :
Figure 5: CNN connected at around input layer.

Figure 11 :
Figure 11: Misclassified Images by Fine-tuning2 but Correctly Classified Image by CNN2

Table 3 :
Result of multi-input CNN.

Table 3 :
Result of CNN features and SVM.

Table 4 :
Result of fine-tuning.