how did you save the model?, does it have custom ops?, what is your tf version?
if you want to use the SaveModel format and H5, do it in this way:
Create the model and save it:
import os 

model = FancyCNN()
mode.save(os.path.join(your_model_path, 'model.kerasmodel'))
model.save(os.path.join(your_model_path, 'model.h5')))


that will create an h5 file containing the model and other objects and a folder named model.kerasmodel with 3 things (a folder called variables, other called assets and the saved_model.pb)
when you load the SavedModel you have to point to the folder model.kerasmodel like this:
model = tf.keras.models.load_model(os.path.join(your_model_path, 'model.kerasmodel'))
model = tf.keras.models.load_model(os.path.join(your_model_path, 'model.h5'))

Remember, if you have custom layers you have to pass the argument custom_objects in this call tf.keras.models.load_model(), custom objects should be a dictionary where the keys are the names of the custom layers and values the objects themselves.