Question: I am running a pipeline using the Apache Beam model in Google Cloud Dataflow but I am unable to scale it up from 8 workers, even though the maximum number of workers is 32. 

When I try to run the same pipeline setting the number of workers to 32, it gives me the following warnings:

`Autoscaling: Startup of the worker pool in zone us-central1-f reached 30 workers, but the goal was 32 workers. The service will retry. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded. Limit: 4096.0`

`Autoscaling: Unable to reach resize target in zone us-central1-f. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded. Limit: 4096.0`

But still doesn't pass 8 workers. Is there any particular reasons why a pipeline won't use more than 8 workers?

Answer: 
The problem was quota limits. Google Dataflow uses behind the scenes VMs of Google Compute Engine and their quotas apply. The specific limitation of 8 was being caused by the In use external IP adresses quota limitation. Others quotas were also violated when I tried to scale to 32, like the Disk space. So if anyone is having the same problem I suggest going to `IAM Admin > Quotas` on the console while the pipeline is running to check which quotas your pipeline may violate. 

Also, the logs are different if you run using a deployed template or use the Eclipse plugin to run in debug mode. The later will give much more details than the first.
Dataflow will use whatever number of workers you can get. In your case, it will reach 30 workers and will use them. It will however retry constantly to reach 32, as quota could be given back by other workflows.