<p dir="auto">Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on <a href="http://stackoverflow.com/questions/tagged/keras" rel="nofollow">StackOverflow</a> or <a href="https://keras-slack-autojoin.herokuapp.com/" rel="nofollow">join the Keras Slack channel</a> and ask there instead of filing a GitHub issue.</p>
<p dir="auto">Thank you!</p>
<ul class="contains-task-list">
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Check that you are up-to-date with the master branch of Keras. You can update with:<br>
pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps</p>
</li>
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found <a href="https://www.tensorflow.org/get_started/os_setup" rel="nofollow">here</a>.</p>
</li>
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:<br>
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps</p>
</li>
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).</p>
</li>
</ul>
<p dir="auto">Hi all,</p>
<p dir="auto">I was wondering if you would be able to help or advice me on an issue I am having. I am not sure that this is a bug but I would like some input on this problem to be sure of what is happening!</p>
<p dir="auto">I have built a multi-label classifier  network in order to classify categories in some data i have that includes classes such as age etc... My question is related to the predict function and just wanted some clarification as to the confidence scores assigned by the network. I have some unlabelled data in my data set, and I would like also the network to predict what class it thinks the data belongs to. The reason it is unknown is because it has not been measured, however, given that it hasn't been measured, I would like to know what the network thinks it is.</p>
<p dir="auto">For each sample, multiple labels exist (age, sex) etc...</p>
<p dir="auto">My code is as follows:</p>
<p dir="auto">For label binarizing, the unknown class are given a 0 and when one-hot encoding, they are left out as follows:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
Metadataencoded= Normcountstransposemetadata.apply(le.fit_transform)

cat_Organ=to_categorical(Metadataencoded[&quot;Organ&quot;],12)
cat_Age=to_categorical(Metadataencoded[&quot;Age&quot;],6)
cat_Sex=to_categorical(Metadataencoded[&quot;Sex&quot;],3)
cat_LLGC=to_categorical(Metadataencoded[&quot;Life_long_Generative_capacity&quot;],5)
cat_LLGC=cat_LLGC[:,1:5]
cat_Sex=cat_Sex[:,1:3]"><pre class="notranslate"><code class="notranslate">from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
Metadataencoded= Normcountstransposemetadata.apply(le.fit_transform)

cat_Organ=to_categorical(Metadataencoded["Organ"],12)
cat_Age=to_categorical(Metadataencoded["Age"],6)
cat_Sex=to_categorical(Metadataencoded["Sex"],3)
cat_LLGC=to_categorical(Metadataencoded["Life_long_Generative_capacity"],5)
cat_LLGC=cat_LLGC[:,1:5]
cat_Sex=cat_Sex[:,1:3]
</code></pre></div>
<p dir="auto">Sex and LLGC are the categories for which there are unknown or unlabelled data. The first column is removed as label binarizer assigns these samples a class.. so after this, the unknown samples contain all 0's for one-hot encoded data.</p>
<p dir="auto">Merge one-hot encoded classes:</p>
<p dir="auto"><code class="notranslate">trainingtarget1 = np.concatenate((cat_Age, cat_Organ,cat_Sex,cat_LLGC), axis=1) </code></p>
<p dir="auto">Test and split and compile model</p>
<p dir="auto">`X_train, X_test, y_train, y_test = train_test_split(Normcountstrain,trainingtarget1, train_size = 0.8)</p>
<p dir="auto">numpy.random.seed(7)</p>
<p dir="auto">model= Sequential()<br>
model.add(Dense(units=64, input_dim=5078, activation="relu"))<br>
model.add(Dense(units=32, activation="relu"))<br>
model.add(Dense(units=100, activation="relu"))<br>
model.add(Dense(units=24, activation="sigmoid"))</p>
<p dir="auto">model.compile(loss="binary_crossentropy", optimizer='adam', metrics=['accuracy'])</p>
<p dir="auto">model.fit(X_train, y_train,<br>
epochs=200,<br>
batch_size=32,<br>
verbose=1,<br>
validation_split=0.15,<br>
callbacks=[EarlyStopping(monitor='val_loss', patience=5,)])</p>
<p dir="auto">`<br>
Validation loss and accuracy etc (the model performs relatively well)... and now skipping to the part I have a question about: I run the following code to get prediction scores.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="trainpred =  model.predict(X_train)
trainpred=pd.DataFrame(trainpred, index=X_train.index)"><pre class="notranslate"><code class="notranslate">trainpred =  model.predict(X_train)
trainpred=pd.DataFrame(trainpred, index=X_train.index)
</code></pre></div>
<p dir="auto">I have attached a screenshot of of a portion of the unknown samples under the 4 classes I would like to know about (the 0.9 values is a sample with a known class). As you can see it can assign some confidence value to the samples with unknown labels I would like the network to try and classify, however, they are all very very low values. Should I go with the highest value in the categories and then assume the network thinks it is most likely that class?  I will say that for the most part, even though the values are low, the highest value is normally assigned to what I would assume is the correct class... so another question would then be, why is the value so low?</p>
<p dir="auto">Is this how it is for classification of unknown labels or is there another way of evaluating this/tweaking the model to get a higher confidence value etc?</p>
<p dir="auto">I will add, I dont want to force these in to the test unseen set as this class is not known at all, and so would like to try and train on these to learn unique features within this class. They are samples at different ages where the class is not known, but it is known for two ages.. I would like to see where they fit in between...</p>
<p dir="auto">Thank you for your time and any help and advice is much appreciated!!</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/33659783/58969099-10a84300-87af-11e9-96c8-7c354499137e.png"><img src="https://user-images.githubusercontent.com/33659783/58969099-10a84300-87af-11e9-96c8-7c354499137e.png" alt="Screenshot 2019-06-05 at 16 20 00" style="max-width: 100%;"></a></p>