title: Tensorflow - Dst tensor is not initialized
body: <p>I’m trying to train a convolutional neural net and am running into some issues with tensorflow. My process is being killed and the following error is thrown:</p>
  
  <pre><code>2018-03-14 15:56:57.850610: W tensorflow/core/framework/op_kernel.cc:1142] Internal: Dst tensor is not initialized.
      /var/spool/slurmd/job06322/slurm_script: line 23: 416703 Killed
  </code></pre>
  
  <p>After looking up the issue, I see that it’s memory related. The node in the cluster has a gpu partition with 93GB of RAM. So, it seems kinda odd that there is a memory issue. 
  I think there’s an error with the way that I am splitting my dataset, but I’m not really sure. I included a code snippet about what I’m doing below. If I should provide any other information about the data that I am working with that would help with figuring out the issue, I can provide it.</p>
  
  <pre><code>comments = pd.read_csv('set1.tsv', sep = '\t', index_col = 0)
  annotations = pd.read_csv('set2.tsv',  sep = '\t')
  labels = annotations.groupby('rev_id')['score'].mean()
  labels = pd.DataFrame(data=labels.values, columns=['score'])
  scaler = MinMaxScaler(feature_range=(0, 1))
  labels = 1-(scaler.fit_transform(labels))
  comments['score'] = labels
  comments['comment'] = comments['comment'].apply(lambda x: x.replace("NEWLINE_TOKEN", " "))
  comments['comment'] = comments['comment'].apply(lambda x: x.replace("TAB_TOKEN", " "))
  comments = comments.reset_index()
  vocab_size = 20000
  max_length = 300  # max length of the comments in the model
  tok = Tokenizer(num_words=vocab_size)  # make the Keras tokenizer
  tok.fit_on_texts(comments['comment'])  # fit it to comments
  train = comments.query("split=='train'")
  test = comments.query("split=='test'")
  x_train = pad_sequences(tok.texts_to_matrix(train[['comment']], mode='count'), maxlen=max_length, padding='post')
  y_train = np.array([[float(d)] for d in train[['score']].pop('score')]).T
  x_test = pad_sequences(tok.texts_to_matrix(test[['comment']], mode='count'), maxlen=max_length, padding='post')
  y_test = np.array([[float(d)] for d in test[['score']].pop('score')]).T
  embedding_vecor_length = 128
  model = Sequential()
  model.add(Embedding(vocab_size, embedding_vecor_length, input_length=max_length))
  model.add(Convolution1D(64, 3, padding='same'))
  model.add(Convolution1D(32, 3, padding='same'))
  model.add(Convolution1D(16, 3, padding='same'))
  model.add(Flatten())
  model.add(Dropout(0.2))
  model.add(Dense(95692,activation='sigmoid'))
  model.add(Dropout(0.2))
  model.add(Dense(95692,activation='sigmoid'))
  tensorBoardCallback = TensorBoard(log_dir='./logs', write_graph=True)
  model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  model.fit(x_train, y_train, epochs=3, callbacks=[tensorBoardCallback])
  </code></pre>