SnakeIsTheName:

Why do the attention mechanisms require the memory argument? According to the docs, it’s optional.


There’s an option to use this layer by passing the memory to call instead of init:

  

      github.com
  

  
    tensorflow/addons/blob/v0.15.0/tensorflow_addons/seq2seq/attention_wrapper.py#L212-L231



      
      
          if setup_memory:
              if isinstance(inputs, list):
                  if len(inputs) not in (1, 2):
                      raise ValueError(
                          "Expect inputs to have 1 or 2 tensors, got %d" % len(inputs)
                      )
                  memory = inputs[0]
                  memory_sequence_length = inputs[1] if len(inputs) == 2 else None
                  memory_mask = mask
              else:
                  memory, memory_sequence_length = inputs, None
                  memory_mask = mask
              self.setup_memory(memory, memory_sequence_length, memory_mask)
              # We force the self.built to false here since only memory is,
              # initialized but the real query/state has not been call() yet. The
              # layer should be build and call again.
              self.built = False
              # Return the processed memory in order to create the Keras
              # connectivity data for it.
              return self.values
      
    
  


  

  
    
    
  

  



The way I’ve set this up, it receives the normalized inputs whereas it should be receiving the LSTM hidden layer inputs, no?

IIUC The AttentionWrapper is implementing something similar to this decoder but wrapped up in the keras-RNN api.
“LSTM hidden layer inputs” is ambiguous to me. So I’m not 100% sure what you’re trying to do.
Remember that the AttentionWrapper architecture was designed for sequence to sequence type problems, and here it looks like you’re just using a single input sequence.
Looking at the code for AttentionWrapper, I see it’s doing what I expect: At each time step the output from the LSTM is returned (return_attention=False), and then used as the query for the attention. The output of the attention is stacked with the input of the next timestep.

  

      github.com
  

  
    tensorflow/addons/blob/v0.15.0/tensorflow_addons/seq2seq/attention_wrapper.py#L1986



      
      
          Returns:
            A tuple `(attention_or_cell_output, next_state)`, where:
          
          
  - `attention_or_cell_output` depending on `output_attention`.
            - `next_state` is an instance of `tfa.seq2seq.AttentionWrapperState`
               containing the state calculated at this time step.
          
          
Raises:
            TypeError: If `state` is not an instance of `tfa.seq2seq.AttentionWrapperState`.
          """
          if not isinstance(state, AttentionWrapperState):
              try:
                  state = AttentionWrapperState(*state)
              except TypeError:
                  raise TypeError(
                      "Expected state to be instance of AttentionWrapperState or "
                      "values that can construct AttentionWrapperState. "
                      "Received type %s instead." % type(state)
                  )
          
          
# Step 1: Calculate the true inputs to the cell based on the
      
    
  


  

  
    
    
  

  


So the LSTM is seeing the input sequence both from the current time’s direct input and its attention over the whole input sequence.

TypeError: To be compatible with tf.eager.defun… <function while_loop… found return value of type <class ‘keras.engine.keras_tensor.KerasTensor’>, which is not a Tensor.

I think what this is trying to tell you is that you can’t use one of those RNN components with the keras-functional interface. Which line throws that error? Is it the RNN line?
Try building that as a Layer subclass instead of a keras-functional model.


Is this even the optimal flow? Input > Normalize > LSTM > Attention > Bidirection > Dense
I’m using BahdanauAttention/LuongAttention with AttentionWrapper solely because of a Tensorflow example; but if there’s a way of using Keras’ Attention class in simpler fashion I’d be happy to learn how that works.


Transformers work by applying attention layers without the RNNs. You can definitely stack Transformer and LSTM layers. Maybe that’s more inline with what you’re trying to do.
This tutorial implements the transformer layer from scratch:

  
      

      TensorFlow
  

  
    

Transformer model for language understanding  |  Text  | ...



  

  
    
    
  

  


But I should update it to use the new MultiHeadAttentionLayers from keras:

  
      

      TensorFlow
  

  
    

tf.keras.layers.MultiHeadAttention  |  TensorFlow Core v2.7.0

  MultiHeadAttention layer.