I was able to fix it by casting reward_batch to float().

  
      github.com
  
  
    edowson/alphapilot_openai_ros/blob/master/ardrone_race_track/src/ardrone_v1_ddqn.py#L437-L438 5

# Compute the expected Q values
expected_state_action_values = (next_state_values * GAMMA) + reward_batch.float()