face recognition systems provide the benefit 
 of collecting a large amount of biometric information in a relatively easy and 
 cost - effective manner ,   because they do not require subjects to bring any part 
 of their body in contact with the recognition 
 device intentionally ,   which results in fewer repercussions and less inconvenience 
 when collecting the biometric information .
an additional advantage exists , that is , the widely 
 deployed image acquisition equipment can be used without modification . in 
 particular ,
various face recognition algorithms and commercial systems have 
 been developed and proposed , and the marketability of face recognition systems 
 has increased , with many immigration - related facilities such as air and sea 
 ports in many countries anxious to introduce face recognition systems after the 
 9.11 terror attacks in the us .
these benefits , and the perceived necessity of 
 increased security , have led to a rising social demand for face recognition 
 systems ; and certified performance evaluation has become important as a means 
 of evaluating these face recognition systems . this paper proposes a performance evaluation model ( pem ) to evaluate the performance of 
 biometric recognition systems ; and designs and implements a performance 
 evaluation tool that enables comparison and evaluation of face recognition 
 systems , based on the proposed pem . the pem is designed to be compatible with 
 related international standards , and contributes to the consistency and enhanced 
 reliability of the performance evaluation tool that is developed with reference 
 to the model .
section 2 
 outlines existing studies related to performance evaluation of face recognition 
 systems .
section 4 describes the 
 design and implementation of the performance evaluation tool , based on the 
 proposed pem .
section 5 compares the performance evaluation method , utilizing 
 the performance evaluation tool proposed in this paper , with existing 
 performance evaluation programs .
the last section provides a conclusion , and 
 suggests some remarks for future research .
the representative certified performance evaluation programs 
 for facial recognition systems are face recognition technology ( feret ) and face 
 recognition vendor testing ( frvt ) .
as has been used since 1993 , feret includes not only performance 
 evaluation of facial recognition systems but also the development of algorithms 
 and the collection of a face recognition database .
department of defense ( usdod ) , feret 
 is an evaluation tool that has been systematically executed from 1993 through 
 1997 by testing changes in the environment ( e.g. , size , posture , background , 
 etc . ) , differences in the time when pictures are taken , and the performance of 
 algorithms in processing a mass - volume database . in particular ,
the feret performance evaluations have been 
 general evaluations designed to measure algorithms at the level of research 
 centers .
the major purpose of the feret performance evaluation tool has been to 
 implement adaptations to the latest facial recognition technologies and their 
 flexibility .
therefore , the feret test is neither used to clearly measure the influence 
 of algorithms on the performance of individual components nor to assess 
 performance in fully organized scientific manners under all operating 
 conditions of a system [ 13 ] . face 

recognition vendor testing ( frvt ) was a performance test for the face 
 recognition system that was implemented using three face recognition technology 
 ( feret ) performance evaluations ( 1994 , 1995 , and 1996 ) .
the feret program 
 introduced the evaluation technique in the face recognition area , and developed 
 the face recognition area at the earliest level ( system prototype development ) . 

however , as face recognition technology matured from the prototype level to the 
 commercial system level , frvt 2000 measured the performance of these commercial 
 systems , and evaluated how far the technology had evolved through comparison 
 with the last feret evaluation .
as a consequence , frvt 2002 measured the degree 
 of technical development since 2000 , evaluated the large - size databases that 
 were in use , and introduced new experiments to better understand the 
 performance of face recognition .
size , difficulty , and complexity of this 
 performance evaluation were on the rise as the evaluation theory as well as the 
 face recognition technology grew .
for example , feret sep96 performed just 14.5 
 million comparisons over a period of 72 hours , while frvt 2000 carried out 192 
 million comparisons in 72 hours .
in contrast , frvt 2002 introduced an 
 evaluation that made 15 billion comparisons in 264 hours [ 46 ] . certified 
 performance evaluation programs like feret and frvt
were designed to measure 
 the algorithm accuracy of face recognition systems . for these projects ,
a 
 common face image database was provided for the test , face recognition was 
 performed for a certain period of time according to the respective method , and 
 the results were evaluated . however , this method provides evaluation only for 
 face recognition technology vendors that participated in the program during the 
 evaluation period . in particular ,
database items were limited to image size , 
 target posture , image acquisition environment , and time , which left the problem 
 that various conditions of algorithm evaluation were not satisfied dynamically . 

moreover , the algorithm evaluation environment was commissioned to each of the 
 face recognition system developers , creating the problem of inconsistency in 
 establishing a performance evaluation system environment .
furthermore , 
 additional tasks were required in order to determine the accuracy of each 
 algorithm , and to analyze the algorithm implementation result again .
therefore , 
 it is necessary to design an algorithm evaluation method that can resolve these 
 problems , to build a standardized evaluation environment , and to automatically 
 figure out the evaluation result of the algorithm whose performance is measured 
 in this environment .
results from the performance evaluation of facial 
 recognition systems change in accordance with varying factors , such as 
 lighting , posture , facial expression , and elapsed time .
the jtc 1/sc 
 37/wg 5 international standard   classifies those factors affecting the performance of biometric systems .
as outlined in table 1 , there are a number of factors 
 affecting the performance of facial recognition systems , and such factors must 
 be prudently taken into consideration when the probe and gallery are selected 
 during the performance evaluation of these systems .
algorithm performance evaluation of biometric recognition technology is 
 conducted in such a way that the standard gallery is trained or registered , and 
 is then compared with the test biometric information ( probe ) to be recognized , after 
 which the similarities between the two sets of information are measured .
generally , basic factors such as posture , angle , facial 
 expression , lighting brightness , and gender are considered in the construction 
 of a face image database .
however , records of the face image database need to 
 be further subdivided in order to process the test under conditions similar to those 
 prevalent in the real world .
the research facial database that was developed 
 by korea information security agency ( kisa ) from 2002 to 2004 was used for 
 performance evaluation .
many factors 
 must be considered when building a fair performance evaluation system for 
 biometric recognition systems .
for example , to evaluate the performance of face 
 recognition systems , a database of facial information for use in face 
 recognition should be collected , and performance evaluation items ( changes in facial 
 expression , lighting , etc . ) as well as performance evaluation 
 measurement criteria such as false acceptance rate ( far ) , false rejection rate ( frr ) , 
 and equal error rate ( eer ) should be selected . the face recognition system to 
 be evaluated and a standardized interface for the performance evaluation system 
 should be designed and international standards need to be applied at each stage 
 of performance evaluation , in order to enhance fairness and reliability .
thus , the performance 
 evaluation model ( pem ) is created to analyze and arrange the criteria to be considered 
 in building up the performance evaluation system , and to support the 
 development of the performance evaluation system .
the pem presents the basic 
 system structure , guidelines , and development process used to build a system 
 for performance evaluation .
the pem 
 proposed in this paper is designed to ( 1 ) evaluate the performance of the 
 biometric recognition algorithm , and ( 2 ) to build a system that automatically evaluates 
 performance and outputs the results in tandem with the biometric recognition 
 system .
the pem structure for the system that evaluates the biometric recognition 
 algorithm is composed of a data preparation module , an execution model , and a 
 result analysis module , as shown in figure 1 .
the data preparation module prepares the 
 biometric information used for performance evaluation , for which the development 
 of a biometric information database and the design of the test criteria are the 
 major elements to consider .
as the biometric information database used affects 
 the evaluation reliability of the performance evaluation system to a large 
 extent , it should be considered a priority at the initial stage of system development . 

in addition , the biometric information used for performance evaluation should 
 never be exposed , so that evaluation reliability can be improved .
generally , algorithm performance evaluation of 
 biometric recognition technology is conducted in such a way that the standard 
 gallery is trained or registered , and is then compared with the test biometric 
 information ( probe ) to be recognized , after which the similarities between the 
 two sets of information are measured . at this time ,
algorithm performance 
 varies according to the information generation environment or conditions . for 
 example
, if an expressionless front - view facial image is registered in the 
 gallery , and a smiling facial image photographed at a 15-degree angle from the 
 left is used as the probe , we can compare the strength of the different 
 algorithm technologies in terms of facial expression and angle . in this paper , 
 the  test criteria 
refers to an item that could affect the performance 
 evaluation result , and the performance evaluation system developer should 
 design test criteria that are suitable for the objective of the evaluation .
the 
 test criteria selected by the pem are limited to the classification criteria of 
 the biometric information database .
the execution module activates the biometric recognition system to be evaluated , 
 and executes a performance evaluation .
there are two methods for establishing an 
 interface between the performance evaluation and the biometric recognition 
 system .
the first one consists in developing two systems as independently 
 applied programs , while the second one consists in creating the biometric 
 recognition algorithm as a component or library and then inserting it into the 
 performance evaluation system .
the former requires advance agreement between 
 two systems with regards to the input / output file format , since the input data 
 used for performance evaluation and the performance evaluation execution result 
 data are generally transferred in a predefined form ( generally , xml ) .
even 
 though frvt 2006 did not use the performance evaluation tool , participating 
 companies submitted the biometric recognition system as an execution file , and 
 the name of the input file used for evaluation and the output file that records 
 the evaluation result were transferred as the program argument . for the 
 component ( or library )
the agreed interface should be as simple as possible , and 
 compatibility with international standards is desirable .
the related 
 international standards include biometric application programming interface ( bioapi ) 1.1   and bioapi 2.0 .
the result analysis module performs a final 
 analysis of recognition algorithm performance , using the result value obtained 
 from the execution module .
the performance of the specific algorithm can be 
 expressed using several measurement criteria , and the appropriate measurement 
 factor is decided upon depending on the objectives of the performance 
 evaluation .
error rates are basically derived from matching errors and 
 sample acquisition errors , and the focus is on whether the algorithm is working 
 properly and accurately .
the throughput rate shows the number of users that the 
 face recognition system can process in a given unit time .
this throughput rate 
 has significant meaning when performing the verification in a large image 
 database .
as biometric products 
 are being used in establishing national infrastructure , a need for more 
 effective and objective biometric performance test is on the increase . 

as examples are being 
 announced such as frvt , however , no objective and proved methodology is 
 reported yet . in this paper , by presenting and formalizing biometric system 
 performance test models , firstly , securing efficiency by eliminating unnecessary 
 processes or factors that can occur when evaluating the performance of a biometric 
 recognition system , and secondly , guaranteeing objectivity may be accomplished 
 by generating credible test factors and processes for the performance test , 
 which is completely dependent on 
 heuristic methodology .
the performance test 
 was executed by elevating the usability of presented models in this paper , and by 
 formulating model - based tools for validation .
pem is defined as a 
 structure =(db , attr , int , met ) with the following meanings . 


db is a set of all images of the test database.attr is a set of pair pattr , gattr representing 
 classification factors for probe and gallery set . 

pattr , gattrfact where fact is a set of factors influencing performance.fact={age , sex , background , 
 expression , pose , illumination,  }. 
 int is an interface of 
 biometric recognition system being tested.met is a set of performance metrics . 

db represents all 
 facial images in the image database to be used for the purpose of face 
 recognition performance evaluation .
the factors include age , sex , photographing time , expression ,   background , 
 posture , lighting , and costume ; 
 and these factors are used when selecting probe and gallery image set .
probe 
 image set is pset=pattr(db ) , and gallery image set is gset=gattr(db ) .
for example , in case of attr={{man , exprsmile},{front , normal} } to perform the performance test for laughing men s faces , the 
 probe image set used in the performance test is man , exprsmile(db ) . 

attr is a set of pair pattr , gattr representing 
 classification factors for probe and gallery set . 

pattr , gattrfact where fact is a set of factors influencing performance.fact={age , sex , background , 
 expression , pose , illumination, 
fact={age , sex , background , 
 expression , pose , illumination,  }. int is an interface of 
 biometric recognition system being tested .
that is , all image matching set mset is a cartesian product of pset and gset . 

( a ) mset={(x , y)xpset  and  ygset}int is the interface of face recognition module , 
 an object of the performance test . through this interface ,
. the interface should be as 
 simple as possible , and it is desirable to be compatible with the international 
 standards .
int should include the matching function fmatching , and 
 this function outputs the matching results ( accept or reject ) by accepting one factor of image matching set . 
 
int is the interface of face recognition module , 
 an object of the performance test . through this interface ,
. the interface should be as 
 simple as possible , and it is desirable to be compatible with the international 
 standards .
int should include the matching function fmatching , and 
 this function outputs the matching results ( accept or reject ) by accepting one factor of image matching set . 

reject ,  where   amsetthe execution result of 
 face recognition function for the performance evaluation is expressed as the 
 results from calling fmatching function using 
 all element of mset as factors , and this can be expressed as a two dimensional 
 matrix , rmatrix .
when the result value is 
 accept , the outcome is 1 , when 
 the resulting value is reject , it is 0 .
for example , pset={p1,p2,p3 } , and gset={g1,g2,g3 } ,   mset   becomes { p1,g1,p1,g2,p1,g3,p2,g1,p2 ,  g2,p2,g3 , p3,g1,p3,g2,p3,g3}. if the resulting values of applying each element of mset to fmatching is 1 , 0 , 0 , 0 , 1 , 1 , 0 , 0 , 1 , consecutively , 
 this can be expressed 
 as the following 2-dimensional 
 matrix : 
 ( 1)rmatrix = p1p2p3g1g2g3 . 

met is a set of performance test measures such as fail - to - enroll 
 rate ( fter ) , fail - to - acquire rate ( ftar ) , and false nonmatch rate ( fnmr ) , false 
 match rate ( fmt ) . such matching error - related metrics as fnmr , fmt can be 
 calculated using rmatrix : 

 ( 2)fnmr=i=1n  j=1 m 
( same(pseti , gsetj)(rmatrixi , j=0))i=1n  j=1 m  same(pseti , gsetj),fmr=i=1n  j=1m(diff(pseti , gsetj)rmatrixi , j)i=1n  j=1mdiff(pseti , gsetj ) , 
 where the following hold : 
 n is the size of pset , m is the size of gset , same(a , b ) is 1 , if a and b are images of same person , otherwise same(a , b ) is 0,diff(a , b ) is 0 if a and b 
 are images of same person , otherwise , diff(a , b ) is 1,ab is 1 , if a and b are 1 , otherwise it is 0,a = b is 1 , if values of a and b are same , otherwise it is 0 . 

  the execution result of 
 face recognition function for the performance evaluation is expressed as the 
 results from calling fmatching function using 
 all element of mset as factors , and this can be expressed as a two dimensional 
 matrix , rmatrix .
when the result value is 
 accept , the outcome is 1 , when 
 the resulting value is reject , it is 0 .
for example , pset={p1,p2,p3 } , and gset={g1,g2,g3 } ,   mset   becomes { p1,g1,p1,g2,p1,g3,p2,g1,p2 ,  g2,p2,g3 , p3,g1,p3,g2,p3,g3}. if the resulting values of applying each element of mset to fmatching is 1 , 0 , 0 , 0 , 1 , 1 , 0 , 0 , 1 , consecutively , 
 this can be expressed 
 as the following 2-dimensional 
 matrix : 
 ( 1)rmatrix = p1p2p3g1g2g3 . 

met is a set of performance test measures such as fail - to - enroll 
 rate ( fter ) , fail - to - acquire rate ( ftar ) , and false nonmatch rate ( fnmr ) , false 
 match rate ( fmt ) . such matching error - related metrics as fnmr , fmt can be 
 calculated using rmatrix : 

 ( 2)fnmr=i=1n  j=1 m 
( same(pseti , gsetj)(rmatrixi , j=0))i=1n  j=1 m  same(pseti , gsetj),fmr=i=1n  j=1m(diff(pseti , gsetj)rmatrixi , j)i=1n  j=1mdiff(pseti , gsetj ) , 
 where the following hold : 
 n is the size of pset , m is the size of gset , same(a , b ) is 1 , if a and b are images of same person , otherwise same(a , b ) is 0,diff(a , b ) is 0 if a and b 
 are images of same person , otherwise , diff(a , b ) is 1,ab is 1 , if a and b are 1 , otherwise it is 0,a = b is 1 , if values of a and b are same , otherwise it is 0 . 
 
n is the size of pset ,   m is the size of gset , same(a , b ) is 1 , if a and b are images of same person , otherwise same(a , b ) is 0 , diff(a , b ) is 0 if a and b 
 are images of same person , otherwise , diff(a , b ) is 1 , ab is 1 , if a and b are 1 , otherwise it is 0 , a = b is 1 , if values of a and b are same , otherwise it is 0 .
the following section describes how to build a performance evaluation system according to the pem . 

describe the objectives of developing and evaluating a performance evaluation system.develop or select the biometric information database that will be used for performance evaluation.design test criteria that fit into the evaluation objectives.determine the type of interface to be used between the performance evaluation system and the biometric recognition system .
if it runs as a standalone program , select the input / output file format ; or , if it is linked by the component method , design the component interface.select the measurement criteria that fit into the evaluation objectives.implement the  data preparation module  which reads the biometric information according to the test criteria through the interface with the biometric information database.implement the  execution 
 module  which executes the recognition algorithm with the gallery / probe 
 biometric information provided by the data preparation module .
the execution 
 result ( degree of similarity ) should be saved in the database containing the 
 results of performance evaluation.calculate the value of the measurement 
 criteria by analyzing the similarity saved in the performance evaluation result 
 database , and implement the  result analysis module  to generate a report on 
 the results of the performance evaluation . 
 
determine the type of interface to be used between the performance evaluation system and the biometric recognition system .
if it runs as a standalone program , select the input / output file format ; or , if it is linked by the component method , design the component interface .
select the measurement criteria that fit into the evaluation objectives . implement the  data preparation module  which reads the biometric information according to the test criteria through the interface with the biometric information database .
execution 
 module  which executes the recognition algorithm with the gallery / probe 
 biometric information provided by the data preparation module .
the execution 
 result ( degree of similarity ) should be saved in the database containing the 
 results of performance evaluation .
calculate the value of the measurement 
 criteria by analyzing the similarity saved in the performance evaluation result 
 database , and implement the  result analysis module  to generate a report on 
 the results of the performance evaluation .
the test items and measurement criteria can be 
 decided when performing the actual performance evaluation instead of building 
 the performance evaluation system . in this case , select test items and 
 measurement criteria that can be selected at steps 3 and 5 as described above , 
 the test should be able to select the necessary items from among these when 
 running the performance evaluation tool .
the performance evaluation tool was designed 
 and implemented using the pem proposed by this paper .
the following section 
 describes the contents and results by step , according to the evaluation system 
 development process .  
the purpose of performance evaluation is to identify the 
 technology level of the face recognition system through objective performance 
 evaluation and certification , so as to encourage public trust in face 
 recognition products and enhance their competitiveness .
the test criteria are designed in such way that 
 those do not have to be selected when developing the evaluation system , enabling 
 the tester to select it in the course of performing the actual performance 
 evaluation .
basically , the test item lists all the classification criteria so 
 that the tester can select from them separately , based on the condition of the 
 gallery image set and the probe image set .
the gallery image set and the probe 
 image set , each of which is composed of several items , are referred to as the 
  test set .
 one performance evaluation project can generate several test sets , 
 and each test set can generate a different result report .
for instance , even 
 though the collection of facial images to be registered in the face recognition 
 system is white - front ( normal ) or purple - front ( illum .
yellow ) , the actual 
 image acquired by the image acquisition device to identify the user can be 
 normal , eye - closed , or tilted slightly to the left or right .
the test set can 
 be configured as in figure 2 . by assuming this kind of face recognition system .
if we assume that 10 face images exist per 
 category in the above example , there are 20 facial images in the gallery set , 
 and 40 facial images in the probe set .
therefore , the template generation 
 function will be invoked 20 times for the gallery image , and image processing 
 will be performed 40 times for the probe image , which means that 800 ( 2040 ) 
 comparison operations will be executed if performance evaluation is performed 
 on the above test set . among these comparison operations
, image comparison will 
 be performed 80 times for all the persons involved , because the image for a 
 specific person appears twice in the gallery set and 4 times in the probe set , 
 which results in an image comparison being performed 8 times for the same 
 person , with 10 persons in total being compared .
therefore , 720 different image 
 comparisons are made for the different persons , based on this system . using 
 this method ,
image comparison times can be estimated in advance , and the tester 
 can estimate the time required for performance evaluation in advance , because 
 the comparison times are calculated beforehand .
the performance evaluation tool provides a 
 standard interface for the face recognition system , and the examinee provides 
 the face recognition module that satisfies this interface as the dynamic 
 library .
the performance evaluation tool is designed to enable the tester to 
 change the face recognition module during the run time so that the tester can 
 perform evaluation by changing the face recognition module without modifying 
 the performance evaluation tool .
bioapi 1.1 was applied for the compatibility with 
 international standards , and only the minimum number of functions required for 
 algorithm performance evaluation was selected , in order to reduce the 
 examinee s development burden .
the performance evaluation metrics used by 
 feret and frvt , as well as the metrics proposed by jtc 1/sc 37/wg 5 standard , 
 were analyzed . among these metrics ,
the criteria related with the technology 
 evaluation of the face recognition system were chosen , as shown in table 4 . within the performance evaluation 
 tool implemented in this paper
, individual projects created for performance 
 evaluation internally generate metadata in a specific structure in order to 
 save the settings related to performance evaluation and performance evaluation 
 results .
whenever a new project is created , this class 
 creates an instance in connection with the project .
it maintains the path used to 
 save the project name and the project itself , and the path to access a database 
 in character strings .
this information is related to the project 
 configurations , and contains values established by the tester upon the project s 
 creation .
in addition , the list of face image data designed by the tester for performance 
 evaluation receives the lists of cmetatestset class .
a single project may have at least one group 
 of face image data for its performance evaluation , and independently create a 
 report based on these individual evaluation results .
moreover , this class allows users to 
 ascertain the number of total images to test ( compare ) , and that of probes and 
 gallery images with regards to such information .
face images are influenced by the direction of the picture 
 taken , the location of lighting , and posture , and so forth . according to these 
 conditions ,
they are classified , and the tester may choose some of the 
 classified images by using the performance test tool as the test subject group . 
 the chosen information is individually saved in the cmetacategory class .
these metadata 
 contain variables , such as the category name of each criterion , the total number 
 of images in the category , and the number of images failed to enroll ( for gallery 
 items ) or acquire ( for probe items ) .
each face image has a unique i d for the photograph target , its 
 location , and the face image items .
additionally , it contains boolean variables 
 in order to save whether each item failed to enroll ( for gallery items ) and 
 acquire ( for probe items ) or not .
this 
 class stores the verification results of the comparison of one probe item with one 
 gallery item in order to determine whether or not they come from an identical 
 person .
it contains each item s criteria , location , and i d information , along 
 with variables , such as the similarity 
 value and the comparison time created as a result of a comparison between two 
 images .
this 
 class saves the template data created so as to recognize a face from the image 
 data for the system to execute the enrollment of a gallery item .
it holds not 
 only the item s criteria and location information but also a binary space to 
 store template data , as well as data used to save the results of the template creation 
 and the time required to create a template .
this class includes information about the face 
 image probe group and gallery group in order to conduct the test .
one project may 
 have several test groups , and each test group individually saves the 
 performance evaluation results .
firstly , the class contains cmetacategory which is the 
 information holding the face image criteria as the list information for each of 
 the probe and gallery groups .
namely , each of the probe and gallery groups may 
 include a face image group with several different criteria .
in addition , the 
 class maintains the list of cmetatestitem for each of the probe and gallery groups . 

this is not the criteria information but the metadata with individual image 
 item information used for the test .
it also contains the list of cmetatestresult , 
 containing the test results between a single probe item and a single gallery 
 item .
furthermore , where a system enrolls a gallery item , this 
 class will contain the list of cmetaenrolled classes in order to store template 
 data that are created when each face recognition module generates its own 
 template data .
this is accomplished by using the face image data . as general data of such list data
, the 
 class contains variables to contain the test start time , end time , number of 
 total probe items , number of total gallery items , number of total gallery items 
 that failed to enroll , and number of total probe items that the system failed 
 to acquire .
we developed the six 
 major modules and metadata classes that we examined above with the program for 
 windows , under microsoft s visual studio development configuration .
the face 
 recognition modules , which are the subject of the performance evaluation , work in 
 connection with the performance evaluation tool in the format of a dynamic link 
 library .
the performance evaluation 
 tool was developed as an application program running on windows os , and the 
 face recognition module to be evaluated was implemented as a dynamic link library 
 ( dll ) .
the data preparation module that has the function of connecting with the 
 biometric database and of setting the gallery and probe image set was 
 implemented for use in the performance 
 evaluation , as shown in figure 4 . the face recognition module provided by the vendor
was checked to verify 
 that it provides the functions presented in table 3 , and the execution module 
 that performs evaluation was implemented , using the functions of the selected 
 face recognition module .
a function that visually displays whether performance 
 evaluation is progressing properly or not was included in the execution module , 
 as shown in figure 5 .
finally , the value of evaluation criteria is 
 calculated by analyzing the similarity saved in the performance evaluation 
 result database , and the result analysis module that generates the performance 
 evaluation result report is implemented .
this performance evaluation tool is 
 equipped with a function that generates the evaluation result , as well as a 
 function that issues the certificate for the face recognition module , depending 
 on the evaluation result .
table 5 shows a comparison made between 
 feret and frvt , which are the representative face recognition evaluation cases , 
 and the evaluation method that uses the performance evaluation tool proposed by 
 this paper .
compared with performance evaluation programs 
 such as feret and frvt , the performance evaluation tool proposed by this paper 
 provides the following benefits . 


disclosure of the face image database can be fundamentally prevented.development of a face recognition module that complies with international standards will be encouraged.the performance evaluation target can be separated from the performance tester.the evaluation cost can be reduced significantly , and individual evaluations can be performed for each vendor . 
 
the evaluation cost can be reduced significantly , and individual evaluations can be performed for each vendor .
the proposed pem is designed for compatibility 
 with the related international standards , thereby contributing to the enhanced 
 consistency and reliability of the performance evaluation tool that is 
 developed according to this design .
it represents a model and development method for 
 the performance evaluation system.it applies the related 
 international standards to the performance evaluation system.it enhances the consistency and reliability of the performance evaluation system.it provides guidelines for the 
 design and implementation of the performance evaluation system by formalizing 
 the performance test process .  
it provides guidelines for the 
 design and implementation of the performance evaluation system by formalizing 
 the performance test process . 

in addition , a performance evaluation tool capable 
 of comparing and evaluating the performance of the commercialized facial 
 recognition systems was designed and implemented , and an evaluation that executed 
 800 billion comparisons in 596 hours using the kfdb   was conducted .
the 
 certificate issuance criteria regarding the performance of the face recognition 
 systems should be presented systematically , and a method should be prepared 
 that can promote certification .
 due to usability features , practical applications , and its lack of intrusiveness , face recognition technology , based on information , derived from individuals ' facial features , has been attracting considerable attention recently . reported recognition rates of commercialized face recognition systems can not be admitted as official recognition rates , as they are based on assumptions that are beneficial to the specific system and face database . 
 therefore , performance evaluation methods and tools are necessary to objectively measure the accuracy and performance of any face recognition system . in this paper 
 , we propose and formalize a performance evaluation model for the biometric recognition system , implementing an evaluation tool for face recognition systems based on the proposed model . 
 furthermore , we performed evaluations objectively by providing guidelines for the design and implementation of a performance evaluation system , formalizing the performance test process . 
1. INTRODUCTION
2. RELATED STUDIES
3. PERFORMANCE EVALUATION MODEL (PEM)
4. DESIGNING AND IMPLEMENTING   THE  PERFORMANCE EVALUATION TOOL
5. COMPARISON OF PERFORMANCE EVALUATION METHODS
6. CONCLUSION