If I can offer a suggestion … This probably isn’t what you’re looking for (I’m actually having a similar problem getting Inception nets and things to recognise pyramids vs cubes – generated on the fly … Which I thought would be 1,000x easier … Same models that get 98% on MNIST in seconds can barely get above 50:50 … Fully batch normalised conv nets and everything).
However, if the task is principally what you’re trying to achieve, a route I believe human expression detection uses would be to first identify facial landmarks, then train another network to identify expressions on that data.
And for that you can use this to extract features:
from PIL import Image, ImageDraw
import face_recognition

image = face_recognition.load_image_file("obama.jpg")

face_landmarks_list = face_recognition.face_landmarks(image)

print("I found {} face(s) in this photograph.".format(len(face_landmarks_list)))

for face_landmarks in face_landmarks_list:

    for facial_feature in face_landmarks.keys():
        print("The {} in this face has the following points: {}".format(facial_feature, face_landmarks[facial_feature]))

    pil_image = Image.fromarray(image)
    d = ImageDraw.Draw(pil_image)

    for facial_feature in face_landmarks.keys():
        d.line(face_landmarks[facial_feature], width=5)

    pil_image.show()

And that will give you this relatively quickly:

And you could use a very basic neural net on the vector data that generates – only a few dozen numbers; neatly categorised by facial landmark … Certainly how I’d do it – simplify the problem as much as possible … Partly because I’m on a Macbook – first thought is always: how can I avoid having to throw 10 billion numbers at a problem?