events.action: opened
events.author: nttgithubntt
events.masked_author: username_0
events.text: Hj, Thanks for creating tfrs. I examinate a recommendation model based on tfrs, after that, I fit, save model and load model again. the summary after loading model:
  Model: "ranking_model"
  
  Layer (type)                 Output Shape              Param #   
  =================================================================
  user_model (UserModel)       multiple                  5248      
  
  card_model (CardModel)       multiple                  957640    
  
  sequential_5 (Sequential)    (None, 4)                 4644      
  
  ranking (Ranking)            multiple                  2         
  =================================================================
  Total params: 967,534
  Trainable params: 967,532
  Non-trainable params: 2
  
  None
  
  But when predicting: the loaded model show error:
  
  ValueError: Could not find matching function to call loaded from the SavedModel. Got:
        Positional arguments (3 total):
          * {'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None, 1) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None, 1) dtype=string>, 'user_id': <tf.Tensor 'inputs_5:0' shape=(None, 1) dtype=string>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None, 1) dtype=string>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None, 1) dtype=float32>}
          * False
          * None
        Keyword arguments: {}
      
      Expected these arguments to match one of the following 4 option(s).....
  
  
  How to solve this issues? Tks again.
events.title: can't predict after saving then loading model
events.type: issue
events.action: created
events.author: maciejkula
events.masked_author: username_1
events.text: This most commonly happens where the features you pass to `call` in training are different than the features you pass to `predict`. For example, you might accidentally pass labels or example weights to `call` during training, but these are not available in prediction.
  
  To avoid this, make sure you only pass the features you have in prediction into `call` when defining your `compute_loss` method.
  
  Can you provide all the options that the saved model error gives you?
events.type: comment
events.action: created
events.author: nttgithubntt
events.masked_author: username_0
events.text: Hj @username_1 , thanks you for responding. Here are the ranking model:
  
  class RankingModel(tfrs.models.Model):
  
      def __init__(self, user_ids, topic_ids, card_ids, card_titles):
          # We take the loss weights in the constructor: this allows us to instantiate
          # several model objects with different loss weights.
          super().__init__()
  
          self.query_model = UserModel(user_ids, topic_ids)
          self.candidate_model = CardModel(card_ids, card_titles, topic_ids)
  
          # A small model to take in user and card embedding and predict interactions.
          # We can make this as complicated as we want as long as we output a vector
          # as our prediction. A vector prediction is [Favorites, Shares, Discussions_Started, Press]
          self.interacting_model = tf.keras.Sequential([
              tf.keras.layers.Dense(64, activation="relu"),
              tf.keras.layers.Dense(32, activation="relu"),
              tf.keras.layers.Dense(4),
          ])
  
          # The ranking tasks.
          self.ranking_task: tf.keras.layers.Layer = tfrs.tasks.Ranking(
              loss=tf.keras.losses.MeanSquaredError(),
              metrics=[tf.keras.metrics.RootMeanSquaredError()]
          )
  
      def call(self, inputs):
          print("call-ranking model inputs info:")
          print(inputs)
          user_embeddings = self.query_model(inputs)
          card_embeddings = self.candidate_model(inputs)
          return self.interacting_model(tf.concat([user_embeddings, card_embeddings], axis=1))
  
      def compute_loss(self, features: Dict[Text, tf.Tensor], training=False) -> tf.Tensor:
          inputs = features.copy()
          del inputs["favorites"], inputs["share"], inputs["discussions_started"], inputs["press"]
          interacting_predictions = self(inputs)
          interacting = tf.concat(
              [tf.expand_dims(features["favorites"], -1),
               tf.expand_dims(features["share"], -1),
               tf.expand_dims(features["discussions_started"], -1),
               tf.expand_dims(features["press"], -1)
               ], axis=1)
  
          return self.ranking_task(
              labels=interacting,
              predictions=interacting_predictions
          )
  
  model = RankingModel(user_ids=vocabularies[0], topic_ids=vocabularies[1], card_ids=vocabularies[2],
                                         card_titles=vocabularies[3])
  model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))
  
  train, test = self.data_manager.get_train_test_batch_set()
  model.fit(train.batch(8192), epochs=3)
  print("ranking model summary:")
  print(self._model.summary())
  
  print("save ranking model:")
  model.save(DEEP_MODEL_PATH)
  
  
  model = tf.keras.models.load_model(DEEP_MODEL_PATH)
  # Check its architecture
  print("loaded ranking model ok")
  print("loaded ranking model summary:")
  print(model.summary())
  
  predict_dataset = DeepDataManager.get_df_score(user_info, df_card).batch(5)
  results = model.predict(predict_dataset)
  
  
  Here are the logs:
  
  /home/ai/Documents/AI/gh-ai/eurekai/bin/python /home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py
  2020-10-06 11:58:08.502124: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  /home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py:23: DtypeWarning: Columns (7,16,17,18,19,20,21,23) have mixed types.Specify dtype option on import or set low_memory=False.
    self.data_manager = DeepDataManager(data_path=data_path)
  2020-10-06 11:58:10.596744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
  2020-10-06 11:58:10.662514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
  pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
  coreClock: 1.77GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
  2020-10-06 11:58:10.663019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
  pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
  coreClock: 1.77GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
  2020-10-06 11:58:10.663046: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  2020-10-06 11:58:10.663244: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
  2020-10-06 11:58:10.663287: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
  2020-10-06 11:58:10.663326: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
  2020-10-06 11:58:10.663365: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
  2020-10-06 11:58:10.663403: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
  2020-10-06 11:58:10.666135: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
  2020-10-06 11:58:10.666155: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
  Skipping registering GPU devices...
  2020-10-06 11:58:10.666427: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2020-10-06 11:58:10.678109: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3299990000 Hz
  2020-10-06 11:58:10.679010: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5cb4090 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
  2020-10-06 11:58:10.679039: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
  2020-10-06 11:58:10.680719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
  2020-10-06 11:58:10.680732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      
  fitting ranking model:
  Epoch 1/3
  WARNING:tensorflow:Layer ranking_model is casting an input tensor from dtype float32 to the layer's dtype of float64, which is new behavior in TensorFlow 2.  The layer has dtype float64 because its dtype defaults to floatx.
  
  If you intended to run this layer in float64, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.
  
  To change all layers to have dtype float32 by default, call `tf.keras.backend.set_floatx('float32')`. To change just this layer, pass dtype='float32' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.
  
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'IteratorGetNext:9' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'IteratorGetNext:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'IteratorGetNext:10' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'IteratorGetNext:6' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'IteratorGetNext:2' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'IteratorGetNext:3' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'Cast:0' shape=(None,) dtype=float64>}
  WARNING:tensorflow:Layer ranking is casting an input tensor from dtype float32 to the layer's dtype of float64, which is new behavior in TensorFlow 2.  The layer has dtype float64 because its dtype defaults to floatx.
  
  If you intended to run this layer in float64, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.
  
  To change all layers to have dtype float32 by default, call `tf.keras.backend.set_floatx('float32')`. To change just this layer, pass dtype='float32' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.
  
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'IteratorGetNext:9' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'IteratorGetNext:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'IteratorGetNext:10' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'IteratorGetNext:6' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'IteratorGetNext:2' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'IteratorGetNext:3' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'Cast:0' shape=(None,) dtype=float64>}
  1/1 [==============================] - 0s 389us/step - root_mean_squared_error: 12.4154 - loss: 154.1427 - regularization_loss: 0.0000e+00 - total_loss: 154.1427
  Epoch 2/3
  1/1 [==============================] - 0s 435us/step - root_mean_squared_error: 12.1651 - loss: 147.9898 - regularization_loss: 0.0000e+00 - total_loss: 147.9898
  Epoch 3/3
  1/1 [==============================] - 0s 442us/step - root_mean_squared_error: 11.2242 - loss: 125.9836 - regularization_loss: 0.0000e+00 - total_loss: 125.9836
  ranking model summary:
  Model: "ranking_model"
  
  Layer (type)                 Output Shape              Param #   
  =================================================================
  user_model (UserModel)       multiple                  5248      
  
  card_model (CardModel)       multiple                  957640    
  
  sequential_5 (Sequential)    (None, 4)                 4644      
  
  ranking (Ranking)            multiple                  2         
  =================================================================
  Total params: 967,534
  Trainable params: 967,532
  Non-trainable params: 2
  
  None
  save ranking model:
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>}
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>}
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>}
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'inputs_5:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs_6:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None,) dtype=float64>}
  WARNING:tensorflow:From /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/training/tracking/tracking.py:111: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
  Instructions for updating:
  This property should not be used in TensorFlow 2.0, as updates are applied automatically.
  WARNING:tensorflow:From /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:2309: Layer.updates (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
  Instructions for updating:
  This property should not be used in TensorFlow 2.0, as updates are applied automatically.
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'inputs/user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs/card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs/view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs/interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs/card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs/card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs/card_recency:0' shape=(None,) dtype=float64>}
  call-ranking model inputs info:
  {'user_id': <tf.Tensor 'inputs/user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs/card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs/view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs/interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs/card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs/card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs/card_recency:0' shape=(None,) dtype=float64>}
  2020-10-06 11:58:13.487382: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
  WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7f4ab8ff99d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
  loaded ranking model ok
  loaded ranking model summary:
  Model: "ranking_model"
  
  Layer (type)                 Output Shape              Param #   
  =================================================================
  user_model (UserModel)       multiple                  5248      
  
  card_model (CardModel)       multiple                  957640    
  
  sequential_5 (Sequential)    (None, 4)                 4644      
  
  ranking (Ranking)            multiple                  2         
  =================================================================
  Total params: 967,534
  Trainable params: 967,532
  Non-trainable params: 2
  
  None
  sys:1: DtypeWarning: Columns (7,16,17,18,19,20,21,23) have mixed types.Specify dtype option on import or set low_memory=False.
  /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:69: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    df_interactions["user_id"] = user_info["user_id"]
  /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:70: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    df_interactions["interested_topic_id"] = user_info["interested_topic_id"]
  /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:71: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    df_interactions['view_time'] = 0
  /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:18: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    df_interactions['updated_at'] = pd.to_datetime(df_interactions['updated_at'])
  /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:21: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    lambda x: (pd.Timestamp.now() - x['updated_at']).days + 1, axis=1)
  /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:23: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    df_interactions['card_recency'] = 1 / df_interactions['card_recency']
  /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/pandas/core/frame.py:4167: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    errors=errors,
  /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/pandas/core/frame.py:3065: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead
  
  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    self[k1] = value[k2]
  Traceback (most recent call last):
    File "/home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py", line 82, in <module>
      score_phase()
    File "/home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py", line 76, in score_phase
      results = recommender_.predict(predict_dataset)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 130, in _method_wrapper
      return method(self, *args, **kwargs)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1599, in predict
      tmp_batch_outputs = predict_function(iterator)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
      result = self._call(*args, **kwds)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call
      self._initialize(args, kwds, add_initializers_to=initializers)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
      *args, **kwds))
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
      graph_function, _, _ = self._maybe_define_function(args, kwargs)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
      graph_function = self._create_graph_function(args, kwargs)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
      capture_by_value=self._capture_by_value),
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
      func_outputs = python_func(*func_args, **func_kwargs)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
      return weak_wrapped_fn().__wrapped__(*args, **kwds)
    File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
      raise e.ag_error_metadata.to_exception(e)
  ValueError: in user code:
  
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1462 predict_function  *
          return step_function(self, iterator)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1452 step_function  **
          outputs = model.distribute_strategy.run(run_step, args=(data,))
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
          return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
          return self._call_for_each_replica(fn, args, kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
          return fn(*args, **kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1445 run_step  **
          outputs = model.predict_step(data)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1418 predict_step
          return self(x, training=False)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__
          outputs = call_fn(inputs, *args, **kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:71 return_outputs_and_add_losses
          outputs, losses = fn(inputs, *args, **kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:170 wrap_with_training_arg
          lambda: replace_training_and_call(False))
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py:65 smart_cond
          pred, true_fn=true_fn, false_fn=false_fn, name=name)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py:56 smart_cond
          return false_fn()
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:170 <lambda>
          lambda: replace_training_and_call(False))
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:165 replace_training_and_call
          return wrapped_call(*args, **kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:780 __call__
          result = self._call(*args, **kwds)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:823 _call
          self._initialize(args, kwds, add_initializers_to=initializers)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:697 _initialize
          *args, **kwds))
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:2855 _get_concrete_function_internal_garbage_collected
          graph_function, _, _ = self._maybe_define_function(args, kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:3213 _maybe_define_function
          graph_function = self._create_graph_function(args, kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:3075 _create_graph_function
          capture_by_value=self._capture_by_value),
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py:986 func_graph_from_py_func
          func_outputs = python_func(*func_args, **func_kwargs)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:600 wrapped_fn
          return weak_wrapped_fn().__wrapped__(*args, **kwds)
      /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/saved_model/function_deserialization.py:257 restored_function_body
          "\n\n".join(signature_descriptions)))
  
      ValueError: Could not find matching function to call loaded from the SavedModel. Got:
        Positional arguments (2 total):
          * {'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None, 1) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None, 1) dtype=string>, 'user_id': <tf.Tensor 'inputs_5:0' shape=(None, 1) dtype=string>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None, 1) dtype=string>, 'view_time': <tf.Tensor 'inputs_6:0' shape=(None, 1) dtype=int32>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None, 1) dtype=float64>}
          * False
        Keyword arguments: {}
      
      Expected these arguments to match one of the following 4 option(s):
      
      Option 1:
        Positional arguments (2 total):
          * {'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/user_id'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_title'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='inputs/view_time'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/interested_topic_id'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='inputs/card_recency')}
          * True
        Keyword arguments: {}
      
      Option 2:
        Positional arguments (2 total):
          * {'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_topic_id'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='view_time'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='user_id'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='interested_topic_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='card_recency'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='card_title'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_id')}
          * False
        Keyword arguments: {}
      
      Option 3:
        Positional arguments (2 total):
          * {'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/interested_topic_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='inputs/card_recency'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='inputs/view_time'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_title'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/user_id')}
          * False
        Keyword arguments: {}
      
      Option 4:
        Positional arguments (2 total):
          * {'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='card_recency'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='interested_topic_id'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_id'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='card_title'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='view_time'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='user_id')}
          * True
        Keyword arguments: {}
  
  
  Process finished with exit code 1
events.type: comment
events.action: created
events.author: nttgithubntt
events.masked_author: username_0
events.text: Hj @username_1 , I had find the cause of the problem, due to the mismatch shape between 'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, and requirement input 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'). I have fixed it, tks you again :)
events.type: comment
events.action: closed
events.author: nttgithubntt
events.masked_author: username_0
events.type: issue
events.action: created
events.author: yoke2c
events.masked_author: username_2
events.text: Hi @username_0 I encountered the same error as you while trying to call `model.predict` or `model.evaluate` after saving the model using `model.save` and reloading the model again using `tf.keras.models.load_model` and understand that you have managed to fix it. Perhaps could you share or advise where the shape mismatch can be fixed? Thank you.
events.type: comment
events.action: created
events.author: nttgithubntt
events.masked_author: username_0
events.text: Hj @username_2 , you push the log to check more detail. the common cause of this error is the tensor variable shape in training phase does not match the evaluating or predicting phase.
  I my case error:
  In training phase, the tensor variable 'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 
  has shape = shape=(None, 1)
  But in predicting phase, we input it with  'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id')
  has shape =(None,)
  Clearly, shape=(None, 1) not match shape =(None,).
events.type: comment
events.action: created
events.author: yoke2c
events.masked_author: username_2
events.text: Hi @username_0 thank you for sharing! While I'm still trying to troubleshoot the shape issue, a work-around now is using `model.save_weights()` and then using `model.load_weights()` after re-creating the model.
events.type: comment
events.action: created
events.author: maciejkula
events.masked_author: username_1
events.text: I suspect it might be related to different batching that you're doing between training/evaluation and prediction.
events.type: comment
events.action: created
events.author: yoke2c
events.masked_author: username_2
events.text: Thanks for the advice @username_1, shall explore this as well.
events.type: comment
events.action: created
events.author: cory1219
events.masked_author: username_3
events.text: Hello @username_0, can you explain how you changed tensor variable shape from (None, 1) to (None,) thank!
events.type: comment