colorectal cancer ( crc ) , also known as colon cancer , is due to the abnormal growth of cells proliferating throughout the colon .
the american cancer society estimates that almost 136,830 people will be diagnosed and 50,310 will die due to crc in 2016 , and that the average lifetime risk of developing this type of cancer is one in 20 ( 5% ) . as with most types of cancer ,
the early detection of crc is key to improving the chances of a successful treatment .
crc is typically diagnosed through the microscopic analysis of colon biopsy images . however , this process can be time consuming and subjective , often leading to significant inter / intraobserver variability . as a result
, many efforts have been made toward the development of reliable techniques for the automated detection of crc .
a number of studies have investigated the development of automated methods for the assessment and classification of crc tissue . in fu
, a computer - aided diagnostic system was developed to classify colorectal polyp types using sequential image feature selection and support vector machine ( svm ) classification . a processing pipeline , including microscopic image segmentation , feature extraction , and classification ,
was also proposed in kumar et al . for the automated detection of cancer from biopsy images . in jass ,
a study based on clinical , morphological , and molecular features showed the usefulness of using such features for the diagnosis and treatment of crc .
a combination of geometric , morphological , texture , and scale invariant features was also investigated in rathore et al .
et al . , a similar set of hybrid features was used with an ensemble classifier to enhance the classification accuracy . in rathore
, structural features based on the white run length and percentage cluster area were also shown to be useful for the classification of biopsy images .
a few studies have also focused on the detection of crc using multispectral microscopy images .
one such work , presented in chaddad et al . , uses three - dimensional gray - level co - occurrence matrix features to classify crc tissue types in multispectral biopsy images .
recently , convolution neural network ( cnn ) models have resulted in state of the art performance on a broad range of computer vision tasks such as face recognition , large - scale object classification , and document analysis .
unlike methods based on handcrafted features , such models have the ability to build high - level features from low - level ones in a data - driven fashion . in medical image analysis ,
cnns have shown great potential for various applications such as medical image pattern recognition , abnormal tissue detection , and tissue classification . in this paper
, we propose a new approach for assessing crc progression that applies cnns to multispectral biopsy images . in this approach
, the progression of crc is modeled using three types of pathological tissues : ( 1 ) benign hyperplasia ( bh ) representing an abnormal increase in the number of noncancerous cells , ( 2 ) intraepithelial neoplasia ( in ) corresponding to an abnormal growth of tissue that can form a mass ( tumor ) , and ( 3 ) carcinoma ( ca ) , in which the abnormal tissue develops into cancer .
a cnn is used to determine the tissue type of biopsy images acquired with an optical microscope at different wavelengths . by identifying this tissue type ,
our approach can determine and track the progression of crc , thereby facilitating the selection of an optimal treatment plan . to the best of our knowledge
, this work is the first to use cnns to model the progression of crc tissues based on multispectral microscopy images .
the rest of this paper is organized as follows : section 2 describes the data used in this study , the preprocessing steps , and the proposed cnn - based model .
section 3 then presents the experimental methodology and results , highlighting the performance of our model . in section 4
finally , we conclude by summarizing the contributions of this work and proposing some potential extensions .
multispectral crc biopsy images are first acquired using an optical microscope system with a charge - coupled device ( ccd ) camera and a liquid crystal tunable filter ( lctf ) .
these images are then used as input to the cnn classifier , both for training the model and classifying the tissue type of a new image .
alternatively , a segmentation technique based on active contours can be used to extract regions of interest corresponding to pathological tissues , before the classification step . in section 3 , we show that presegmenting images can improve the classification accuracy of the proposed approach .
convolutional neural network classification of colorectal cancer tissues based on multispectral biopsy images histological crc data are obtained from anatomical pathology at the chu nancy brabois hospital .
part of these data was used in a previous study for classifying the abnormal pt from texture features .
sections of 5 m thickness were extracted and stained using hematoxylin and eosin to reduce image processing requirements .
multispectral images of 512  512 pixels were then acquired using a ccd camera integrated with a lctf in an optical microscopy system . for each tissue sample ,
the lctf was used to provide 16 multispectral images sampled uniformly across the wavelength range of 500650 nm .
since multispectral imaging considers a broader range of wavelengths , it can capture physiological characteristics of tissues beyond those provided by standard grayscale or trichromatic photography . as mentioned before ,
the progression of crc was modeled by considering three types of pathological tissues : bh , in , and ca .
each of the thirty biopsy samples used in this study was labeled by a senior histopathologist , for a total of ten bh samples , ten in samples , and ten ca samples .
although unsegmented images can also be used directly , a segmentation step is added in the proposed pipeline to isolate the crc tissues from nonrelevant tissues and structures such as lumen . for this step ,
we used a semi - automatic segmentation technique based on the active contour algorithm , which can accurately delineate the boundaries of irregularly shapes and has been shown to perform well for tissue segmentation .
briefly , the active contour model can be described as a self - adaptive search for a minimal energy state eac , defined as the sum of an internal energyand an external energy eext . the segmentation contour in an image i ( x , y ) is represented as a parametric function p ( s ) = ( x ( s ) , y ( s ) ) , s [ 0 , 1 ] and is updated iteratively to minimize eac .
the internal energy is defined as where p ( s ) and p ( s ) represent the first and second derivatives of p ( s ) , respectively , and  and  are constants weighting the relative importance of the derivatives . intuitively , the internal energy favors short and smooth contour configurations .
likewise , the external energy is defined as : where g ( x , y ) is the 2d gaussian function with standard deviation  and  is the gradient operator .
the external energy ensures agreement between the contour and image gradient information arising from tissue boundaries .
segmentation contours were initialized as centered rectangles of 508  508 pixels . as shown in figure 2 , the segmentation algorithm partitions the image in two nonoverlapping regions , containing pixels that are on one side or the other of the segmentation contour . in a postprocessing step , a trained user ( e.g. , a pathologist )
is asked to select the region of interest as one of the two segmented regions [ figure 2c ] .
although outside the scope of this paper , a supervised learning model such as svm could also be trained to select the region of interest automatically .
( a ) original image , ( b ) segmentation obtained by the active contour model , ( c ) selected region of interest to evaluate the performance of our segmentation model , we used manually annotated ground truth provided by the chu nancy - brabois hospital . the jaccard similarity coefficient ( jsc ) ,
dice similarity coefficient ( dsc ) , false positive rate ( fpr ) , and false negative rate ( fnr ) were considered as performance metrics .
the jsc and dsc metrics evaluate the degree of the correspondence between two segmentations ( i.e. , segmentation output and ground truth ) and are defined as : where a and b represent the compared segmentations , tp / tnisthe number of correctly classified foreground / background pixels ( i.e. , true positives / negatives ) and fp / fnisthe number of incorrectly labels foreground / background pixels ( i.e. , false positives / negatives ) .
moreover , fpr / fnr is the ratio between the number of pixels incorrectly labeled as foreground / background and the total number of background / foreground pixels : we compared the performance of our active contour model with two standard segmentation approaches : otsu 's thresholding method and edge detection .
as in most cnn - based classification approaches , we adopted an architecture consisting of three types of layers : convolution layers , subsampling ( max - pooling ) layers , and a fully - connected output layer .
these types of layer can be described as follows : this layer type receives as input either the image to classify or the output or the previous layer and applies a set of nl convolution filters to this input .
the output of the layer corresponds to nl feature maps , each one the result of a convolution filter and some additive bias .
note that the convolution process trims output maps by a border of ml  1 pixels , where ml  ml is the size of convolution filters .
this parameter - less type of layer reduces the size of input feature maps through subsampling , thereby supporting local spatial invariance .
it divides the input maps into nonoverlapping subregions and applies a specific pooling function to each one of them . in our architecture , we considered the max - pooling strategy , which outputs the maximum value of each subregion .
note that the pooling process reduces the feature maps by a factor of ml , where ml  ml is the size of pooling subregions .
this type of layer captures the relationship between the final layer feature maps and class labels .
the output of the layer is a vector of k elements , each one representing the score of a class ( e.g. , k = 3 in our network ) .
fully - connected layers can be seen as convolution operations , in which filters have the same size as their input maps .
this architecture is composed of two convolution layers ( c1 and c3 ) , each one followed by max - pooling layers ( s2 and s4 ) , and an output layer ( f6 ) . although not represented in the figure , a layer of rectified linear units ( relus ) is added after each max - pooling layer to improve the convergence of the learning process , which is based on the stochastic gradient descent algorithm .
proposed convolutional neural network architecture with two convolution layers ( c1 and c3 ) , two max - pooling layers ( s2 and s4 ) , and one fully - connected layer ( f5 ) . for each layer ,
the filter size and number of output features are given for training and evaluating the cnn , the data were split based on the patients .
we randomly selected the data of 21 patients ( 7 bh , 7 in and 7 ca ) for training and used the data of the remaining 9 patients ( 3 bh , 3 in , and 3 ca ) for testing .
furthermore , the biopsy images of three training patients were held out in a validation set and used to determine the optimal network architecture and number of training epochs . for each patient in the training , validation , and testing
sets , we obtained multiple examples by running a 60  60  16 sliding window across the original 512  512  16 multispectral images .
these examples were given the same label as the original image and used as input to the cnn .
for segmented images , only examples whose center is within the region of interest were kept .
thus , training examples obtained from segmented images contain more relevant information for the target classification problem .
accuracy , which corresponds to the percentage of correctly classified examples , was used as a measure of classification performance .
part of these data was used in a previous study for classifying the abnormal pt from texture features .
sections of 5 m thickness were extracted and stained using hematoxylin and eosin to reduce image processing requirements .
multispectral images of 512  512 pixels were then acquired using a ccd camera integrated with a lctf in an optical microscopy system . for each tissue sample ,
the lctf was used to provide 16 multispectral images sampled uniformly across the wavelength range of 500650 nm .
since multispectral imaging considers a broader range of wavelengths , it can capture physiological characteristics of tissues beyond those provided by standard grayscale or trichromatic photography . as mentioned before ,
the progression of crc was modeled by considering three types of pathological tissues : bh , in , and ca .
each of the thirty biopsy samples used in this study was labeled by a senior histopathologist , for a total of ten bh samples , ten in samples , and ten ca samples .
although unsegmented images can also be used directly , a segmentation step is added in the proposed pipeline to isolate the crc tissues from nonrelevant tissues and structures such as lumen . for this step ,
we used a semi - automatic segmentation technique based on the active contour algorithm , which can accurately delineate the boundaries of irregularly shapes and has been shown to perform well for tissue segmentation .
briefly , the active contour model can be described as a self - adaptive search for a minimal energy state eac , defined as the sum of an internal energyand an external energy eext . the segmentation contour in an image i ( x , y ) is represented as a parametric function p ( s ) = ( x ( s ) , y ( s ) ) , s [ 0 , 1 ] and is updated iteratively to minimize eac .
the internal energy is defined as where p ( s ) and p ( s ) represent the first and second derivatives of p ( s ) , respectively , and  and  are constants weighting the relative importance of the derivatives . intuitively , the internal energy favors short and smooth contour configurations .
likewise , the external energy is defined as : where g ( x , y ) is the 2d gaussian function with standard deviation  and  is the gradient operator .
the external energy ensures agreement between the contour and image gradient information arising from tissue boundaries .
segmentation contours were initialized as centered rectangles of 508  508 pixels . as shown in figure 2 , the segmentation algorithm partitions the image in two nonoverlapping regions , containing pixels that are on one side or the other of the segmentation contour . in a postprocessing step , a trained user ( e.g. , a pathologist )
is asked to select the region of interest as one of the two segmented regions [ figure 2c ] .
although outside the scope of this paper , a supervised learning model such as svm could also be trained to select the region of interest automatically .
( a ) original image , ( b ) segmentation obtained by the active contour model , ( c ) selected region of interest to evaluate the performance of our segmentation model , we used manually annotated ground truth provided by the chu nancy - brabois hospital . the jaccard similarity coefficient ( jsc ) , dice similarity coefficient ( dsc ) , false positive rate ( fpr ) , and false negative rate ( fnr ) were considered as performance metrics .
the jsc and dsc metrics evaluate the degree of the correspondence between two segmentations ( i.e. , segmentation output and ground truth ) and are defined as : where a and b represent the compared segmentations , tp / tnisthe number of correctly classified foreground / background pixels ( i.e. , true positives / negatives ) and fp / fnisthe number of incorrectly labels foreground / background pixels ( i.e. , false positives / negatives ) .
moreover , fpr / fnr is the ratio between the number of pixels incorrectly labeled as foreground / background and the total number of background / foreground pixels : we compared the performance of our active contour model with two standard segmentation approaches : otsu 's thresholding method and edge detection .
as in most cnn - based classification approaches , we adopted an architecture consisting of three types of layers : convolution layers , subsampling ( max - pooling ) layers , and a fully - connected output layer .
these types of layer can be described as follows : this layer type receives as input either the image to classify or the output or the previous layer and applies a set of nl convolution filters to this input .
the output of the layer corresponds to nl feature maps , each one the result of a convolution filter and some additive bias .
note that the convolution process trims output maps by a border of ml  1 pixels , where ml  ml is the size of convolution filters .
this parameter - less type of layer reduces the size of input feature maps through subsampling , thereby supporting local spatial invariance .
it divides the input maps into nonoverlapping subregions and applies a specific pooling function to each one of them . in our architecture , we considered the max - pooling strategy , which outputs the maximum value of each subregion .
note that the pooling process reduces the feature maps by a factor of ml , where ml  ml is the size of pooling subregions .
this type of layer captures the relationship between the final layer feature maps and class labels .
the output of the layer is a vector of k elements , each one representing the score of a class ( e.g. , k = 3 in our network ) .
fully - connected layers can be seen as convolution operations , in which filters have the same size as their input maps .
is composed of two convolution layers ( c1 and c3 ) , each one followed by max - pooling layers ( s2 and s4 ) , and an output layer ( f6 ) .
although not represented in the figure , a layer of rectified linear units ( relus ) is added after each max - pooling layer to improve the convergence of the learning process , which is based on the stochastic gradient descent algorithm .
proposed convolutional neural network architecture with two convolution layers ( c1 and c3 ) , two max - pooling layers ( s2 and s4 ) , and one fully - connected layer ( f5 ) . for each layer ,
the filter size and number of output features are given for training and evaluating the cnn , the data were split based on the patients .
we randomly selected the data of 21 patients ( 7 bh , 7 in and 7 ca ) for training and used the data of the remaining 9 patients ( 3 bh , 3 in , and 3 ca ) for testing .
furthermore , the biopsy images of three training patients were held out in a validation set and used to determine the optimal network architecture and number of training epochs . for each patient in the training , validation , and testing sets
, we obtained multiple examples by running a 60  60  16 sliding window across the original 512  512  16 multispectral images .
these examples were given the same label as the original image and used as input to the cnn .
for segmented images , only examples whose center is within the region of interest were kept .
thus , training examples obtained from segmented images contain more relevant information for the target classification problem .
accuracy , which corresponds to the percentage of correctly classified examples , was used as a measure of classification performance .
this layer type receives as input either the image to classify or the output or the previous layer and applies a set of nl convolution filters to this input .
the output of the layer corresponds to nl feature maps , each one the result of a convolution filter and some additive bias .
note that the convolution process trims output maps by a border of ml  1 pixels , where ml  ml is the size of convolution filters .
this parameter - less type of layer reduces the size of input feature maps through subsampling , thereby supporting local spatial invariance .
it divides the input maps into nonoverlapping subregions and applies a specific pooling function to each one of them . in our architecture , we considered the max - pooling strategy , which outputs the maximum value of each subregion .
note that the pooling process reduces the feature maps by a factor of ml , where ml  ml is the size of pooling subregions .
this type of layer captures the relationship between the final layer feature maps and class labels .
the output of the layer is a vector of k elements , each one representing the score of a class ( e.g. , k = 3 in our network ) .
fully - connected layers can be seen as convolution operations , in which filters have the same size as their input maps .
this architecture is composed of two convolution layers ( c1 and c3 ) , each one followed by max - pooling layers ( s2 and s4 ) , and an output layer ( f6 ) .
although not represented in the figure , a layer of rectified linear units ( relus ) is added after each max - pooling layer to improve the convergence of the learning process , which is based on the stochastic gradient descent algorithm .
proposed convolutional neural network architecture with two convolution layers ( c1 and c3 ) , two max - pooling layers ( s2 and s4 ) , and one fully - connected layer ( f5 ) . for each layer ,
the filter size and number of output features are given for training and evaluating the cnn , the data were split based on the patients .
we randomly selected the data of 21 patients ( 7 bh , 7 in and 7 ca ) for training and used the data of the remaining 9 patients ( 3 bh , 3 in , and 3 ca ) for testing .
furthermore , the biopsy images of three training patients were held out in a validation set and used to determine the optimal network architecture and number of training epochs . for each patient in the training , validation , and testing sets , we obtained multiple examples by running a 60  60  16 sliding window across the original 512  512  16 multispectral images .
these examples were given the same label as the original image and used as input to the cnn .
for segmented images , only examples whose center is within the region of interest were kept .
thus , training examples obtained from segmented images contain more relevant information for the target classification problem .
accuracy , which corresponds to the percentage of correctly classified examples , was used as a measure of classification performance .
since this method uses segmentation as preprocessing step , we first assess the ability of the proposed segmentation algorithm to extract regions of interest corresponding to crc tissues .
table 1 gives the average performance of the three tested segmentation methods ( otsu 's thresholding , edge detection , and active contour ) on tissue samples corresponding to bh , in , and ca .
we observe that our active contour method outperforms the other two approaches , for all performance metrics and tissue types ( with p < 0.01 in a paired t - test ) .
with respect to tissue types , the best performance of our method is obtained for ca , with a jsc of 0.86 , compared to 0.80 and 0.82 for bh and in , respectively .
examples of segmentation results , for each tissue type , are shown in figure 4 .
we see that the active contour method finds more consistent regions that better delineate the crc tissues in the image .
average performance obtained by three tissue segmentation methods on bh , in and ca tissue samples examples of results obtained by the segmentation methods for the benign hyperplasia , intraepithelial neoplasia , and carcinoma tissue types .
( a ) original image , ( b ) otsu 's thresholding , ( c ) edge detection , ( d ) active contour classification results are summarized in table 2 , showing the accuracy obtained on examples of the test set by our proposed cnn model , with and without the image segmentation step . for comparison
, we also report accuracy values obtained by various tissue classification approaches on the same data .
these approaches are categorized according to the type of texture or shape features used ( e.g. , gray - level co - occurrence matrix [ glcm ] , and statistical moments ) , classifier model ( e.g. , svm and nearest - neighbors ) , and whether image segmentation is required or not .
note that these approaches work by quantifying a region ( segmented or the whole image ) with generic features and using these features as input to a classifier .
in contrast , our proposed cnn method learns the features from training data , providing a better representation of the different crc tissue types .
comparison of tissue classification methods on the same data from these results , we observe that extracting regions of interest through segmentation enhances the accuracy of our method .
thus , while an accuracy of 79.23% is obtained without segmentation , the accuracy of our method reaches 99.17% on presegmented images . while this could be due to various other factors , extracting regions corresponding to crc tissues provide more discriminative examples to train the cnn . in comparison to other tissue segmentation approaches , our cnn method with segmentation provides the highest accuracy ( i.e. , 99.17% vs. 98.92% for chaddad et al . ) .
although relatively small , such improvement in accuracy can have a significant impact considering that the problem is cancer detection . to illustrate the convergence of the parameter optimization phase ( i.e. , stochastic gradient optimization ) , figure 5 shows the mean squared error ( mse ) measured after each training epoch , corresponding to the mean squared difference between the network output yiand target label vector yi : variation of mean squared error across training epochs we see that the optimization converges after 500 epochs and that the mse upon convergence is nearly 0
this shows that the proposed architecture is complex enough to learn a discriminative representation of tissue types and that the learning rate is adequate . in practice ,
however , eight segmented images were tested , and the performance metrics were increased with mse and accuracy value of 0.0001 and 99.168% , respectively .
the crc tissue classification approaches presented in table 2 are based on the assumption that such tissues can be effectively described using a generic texture or morphological features .
for example , chaddad et al . used texture features derived from glcms , discrete wavelet transforms , and laplacian of gaussian filters , computed on presegmented regions , to classify the same images with an accuracy of 98.92% . likewise , peyret et al .
computed texture features such as local binary and local intensity order patterns , on unsegmented to images , obtaining an accuracy of 91.3% .
in contrast , we proposed a data - driven method , based on cnns , to learn an optimal representation of tissues from training data .
our experiments showed this method outperforms existing approaches , even with a small number of tissue samples , with an accuracy of 99.17% .
results have also shown the usefulness of using presegmented images , which significantly improves the accuracy by focusing computation on relevant tissue regions within the image .
first , it is based on a single small cohort of thirty patients . having a larger set of biopsy images from different patients would help capture the full variability of tissues in the progression of crc .
moreover , to obtain an optimal accuracy , our method currently requires the pathologist to select the region of interest from a segmented image . to have a fully automated pipeline , this step
should be replaced by a supervised learning model which would determine the region of interest from training data .
we have presented a method for the classification of crc tissues from multispectral biopsy images , based on active contour segmentation and cnns . unlike traditional approaches , which extract generic texture or shape features from the image
experiments on multispectral images of thirty patients show our method to outperform traditional approaches when using presegmented images . in future work
, we will extend this study by including a larger number of patients and using a fully automated segmentation step .
 background : colorectal cancer ( crc ) is the third most common cancer among men and women . 
 its diagnosis in early stages , typically done through the analysis of colon biopsy images , can greatly improve the chances of a successful treatment . 
 this paper proposes to use convolution neural networks ( cnns ) to predict three tissue types related to the progression of crc : benign hyperplasia ( bh ) , intraepithelial neoplasia ( in ) , and carcinoma ( ca).methods : multispectral biopsy images of thirty crc patients were retrospectively analyzed . 
 images of tissue samples were divided into three groups , based on their type ( 10 bh , 10 in , and 10 ca ) . 
 an active contour model was used to segment image regions containing pathological tissues . 
 tissue samples were classified using a cnn containing convolution , max - pooling , and fully - connected layers . 
 available tissue samples were split into a training set , for learning the cnn parameters , and test set , for evaluating its performance.results:an accuracy of 99.17% was obtained from segmented image regions , outperforming existing approaches based on traditional feature extraction , and classification techniques.conclusions:experimental results demonstrate the effectiveness of cnn for the classification of crc tissue types , in particular when using presegmented regions of interest . 
I
M
Data acquisition
Pathological tissue segmentation
Convolutional neural network-based classification
Convolution layer
Subsampling (pooling) layer
Output (fully-connected) layer
R
D
C
Financial support and sponsorship
Conflicts of interest