This study has proposed a new automatic system that interfaces image analysis methods with the V-CNN model for the segmentation and classification of phase contrast microscopy images, using the morphological and textural features of iPSC colonies. To our best knowledge, this is the first study to have designed a vector-based deep CNN for classification of iPSC colony quality. The motivation for implementing the V-CNN in this study was to investigate the suitability of this classifier model in accomplishing the classification of feature vectors of healthy and unhealthy colonies, which is the main contribution of this study. The V-CNN model classifier had a higher discriminant ability with colony morphologies than with colony textures. The model revealed highly acceptable classification accuracy (95.5%) compared with the SVM classifier (86.7%) that was implemented for similar feature vectors in this study. The overall accuracy using five-fold cross-validation with V-CNN and SVM was <90% and 75–77%, respectively. Furthermore, the precision, recall, and F-measure values of the V-CNN model were much higher than those of the SVM classifier. The differences in performance could be due to the transfer of the prior knowledge of feature maps of the V-CNN model across the stacked layers of the network capable of minimizing the classification error. Furthermore, the local shared weight capacity of the neurons drastically reduced the network complexity and number of parameters, thus contributing to the robustness of the performance in classifying the stem cell colonies.