Do these images contain only cropped portions of face? or it contains the entire person with body and other background?
Ways to improve performance:

More data
If you see two faces are not much distinct except for few fine grained facial aspects, you can try using a deep network(especially densenets), attention networks or handcrafted features from both faces.