In the following snippet, I construct a very simple neural network and evaluate it on some synthetic data. I don’t train the network, just evaluate it with the initial weights. I’m using binary cross entropy, and I compute the loss in two ways:

By calling model.evaluate().
By calling model.predict() and manually computing the loss.

I expect to get the same result in each case, but I consistently get very different values.
import keras
import numpy as np

input_dimension = 100

rng = np.random.default_rng(0)
train_xs = rng.uniform(0, 255, (1, input_dimension))
train_ys = rng.uniform(0, 1, (1, 1))

keras_model = keras.Sequential([
    keras.layers.Dense(1000, activation="relu"),
    keras.layers.Dense(1, activation="sigmoid")
])

keras_model.compile(
    optimizer=keras.optimizers.SGD(learning_rate=0.01, momentum=0.0, nesterov=False),
    loss=keras.losses.binary_crossentropy,
    metrics=["accuracy"]
)

keras_model.build(input_shape=(1, input_dimension))

y = keras_model.predict(train_xs)

loss, _ = keras_model.evaluate(train_xs, train_ys, verbose=0)

print(loss)
print(keras.losses.binary_crossentropy(train_ys, y))

If the activation in the hidden layer is changed from ReLU to sigmoid, the behavior goes away. What’s going on?