Ok, I figured out a solution. I changed my loss function to recalculate my transition logits and recreate the hmm on each training step. My new loss function looks like this:
def compute_loss():
    hmm = tfd.HiddenMarkovModel(
        initial_distribution = initial_distribution,
        transition_distribution = tfd.Categorical(logits=get_transition_logits()),
        observation_distribution = joint_dists,
        num_steps = num_observations,
        time_varying_transition_distribution = True
    )
    return -tf.reduce_logsumexp(hmm.log_prob(observations))