text: Title: topologySpreadConstraint doesn't work fine when only one worker node is eligible for workload
  Body: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!
  
  If the matter is security related, please disclose it privately via https://kubernetes.io/security/
  -->
  
  
  #### What happened:
  Created a deployment with topologySpreadConstraint with topology key kubernetes.io/hostname, with the following config
  topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              foo: bar
  
  Created deployment with 2 replicas and both the pods landed on the same worker node. 
  #### What you expected to happen:
  I expected that second pod would have gone in Pending state.
  
  #### How to reproduce it (as minimally and precisely as possible):
  
  Create following deployment:
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: foo-deploy
    labels:
      foo: bar
  spec:
    replicas: 2
    selector:
      matchLabels:
        foo: bar
    template:
      metadata:
        labels:
          foo: bar
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: node-name
                  operator: In
                  values:
                  - node1
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              foo: bar
        containers:
        - name: nginx
          image: nginx
          ports:
          - containerPort: 80
  Make sure that only one worker node is eligible for workload using nodeSelectors or nodeAffinity.
  
  foo-deploy-7d7bd695b8-xcdw2   1/1     Running   0          12s    192.168.107.140   node1 <none>           <none>
  foo-deploy-7d7bd695b8-xsk2k   1/1     Running   0          12s    192.168.107.141   node 1<none>           <none>
  
  
  #### Anything else we need to know?:
  
  #### Environment:
  - Kubernetes version (use `kubectl version`): 
  **kubectl version**
  Client Version: version.Info{Major:"1", Minor:"20", GitVersion:**"v1.20.4"**, GitCommit:"ec2760d6d916781de466541a6babb4309766c995", GitTreeState:"clean", BuildDate:"2021-02-27T17:24:15Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"ec2760d6d916781de466541a6babb4309766c995", GitTreeState:"clean", BuildDate:"2021-02-27T17:18:03Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
  
  - Cloud provider or hardware configuration:
  - OS (e.g: `cat /etc/os-release`):
  NAME="SLES"
  VERSION="15-SP1"
  VERSION_ID="15.1"
  PRETTY_NAME="SUSE Linux Enterprise Server 15 SP1"
  ID="sles"
  ID_LIKE="suse"
  ANSI_COLOR="0;32"
  CPE_NAME="cpe:/o:suse:sles:15:sp1"
  
  - Kernel (e.g. `uname -a`):
  Linux XXXXX 4.12.14-195-default #1 SMP Tue May 7 10:55:11 UTC 2019 (8fba516) x86_64 x86_64 x86_64 GNU/Linux
  
  - Install tools:
  - Network plugin and version (if this is a network-related bug):
  - Others:
  