<p dir="auto">I'm trying to train a model in a multiprocessing <code class="notranslate">Process</code>, but when a model already exist in the parent scope, the process will freeze at the initialization of the <code class="notranslate">Embedding</code> layer. I'm using Keras 2.0.5, Tensorflow 1.1.0, and only running on the CPU. Here's a simple duplicatable example.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from multiprocessing import Process, Pipe
import numpy as np
from keras.models import Model
from keras.layers import Input, Dense, Embedding
from keras.optimizers import Adam
import tensorflow as tf


def make_model(vecs, weights=None):
    inp = Input((5,))
    embd = Embedding(len(vecs), 50, weights=[vecs], trainable=False)(inp)
    out = Dense(5, activation='softmax')(embd)
    model = Model(inp, out)
    model.compile(Adam(0.001), 'categorical_crossentropy', metrics=['accuracy'])
    return model


def f(vecs, conn):
    model = make_model(vecs)
    conn.send('done')
    conn.close()


if __name__ == '__main__':
    vecs = np.random.random((100000, 50))
    model1 = make_model(vecs)

    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(vecs, child_conn), daemon=True)
    p.start()

    print('starting model two')
    print(parent_conn.recv())
    print('completed')"><pre class="notranslate"><code class="notranslate">from multiprocessing import Process, Pipe
import numpy as np
from keras.models import Model
from keras.layers import Input, Dense, Embedding
from keras.optimizers import Adam
import tensorflow as tf


def make_model(vecs, weights=None):
    inp = Input((5,))
    embd = Embedding(len(vecs), 50, weights=[vecs], trainable=False)(inp)
    out = Dense(5, activation='softmax')(embd)
    model = Model(inp, out)
    model.compile(Adam(0.001), 'categorical_crossentropy', metrics=['accuracy'])
    return model


def f(vecs, conn):
    model = make_model(vecs)
    conn.send('done')
    conn.close()


if __name__ == '__main__':
    vecs = np.random.random((100000, 50))
    model1 = make_model(vecs)

    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(vecs, child_conn), daemon=True)
    p.start()

    print('starting model two')
    print(parent_conn.recv())
    print('completed')
</code></pre></div>
<p dir="auto">When this script is run as it's currently written, it will never print the 'completed' message. If, however, I comment out the line <code class="notranslate">model1 = make_model(vecs)</code> then it will work just fine.</p>
<p dir="auto">I feel like this is a graph/session issue, but none of my attempts to fix it have been successful. Any thoughts?</p>
<p dir="auto">The graph visualization is confusing. There are too many nodes on the graph compared to the actual model, seems the nodes are repeatedly logged many times.</p>
<p dir="auto">For example, see parts of the graph of a simple 0-hidden layer network:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/4527536/28496230-4a663a7a-6fa1-11e7-939d-6dc14c83879a.png"><img src="https://user-images.githubusercontent.com/4527536/28496230-4a663a7a-6fa1-11e7-939d-6dc14c83879a.png" alt="screen shot 2017-07-23 at 12 18 48" style="max-width: 100%;"></a></p>