Question: I am running a pipeline using the Apache Beam model in Google Cloud Dataflow but I am unable to scale it up from 8 workers, even though the maximum number of workers is 32. 

When I try to run the same pipeline setting the number of workers to 32, it gives me the following warnings:

`Autoscaling: Startup of the worker pool in zone us-central1-f reached 30 workers, but the goal was 32 workers. The service will retry. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded. Limit: 4096.0`

`Autoscaling: Unable to reach resize target in zone us-central1-f. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded. Limit: 4096.0`

But still doesn't pass 8 workers. Is there any particular reasons why a pipeline won't use more than 8 workers?

Answer: 
The problem was quota limits. Google Dataflow uses behind the scenes VMs of Google Compute Engine and their quotas apply. The specific limitation of 8 was being caused by the In use external IP adresses quota limitation. Others quotas were also violated when I tried to scale to 32, like the Disk space. So if anyone is having the same problem I suggest going to `IAM Admin > Quotas` on the console while the pipeline is running to check which quotas your pipeline may violate. 

Also, the logs are different if you run using a deployed template or use the Eclipse plugin to run in debug mode. The later will give much more details than the first.
1. Visit <https://console.cloud.google.com/iam-admin/quotas>
2. Use the filter menu that says Quota type 

[![enter image description here](https://i.stack.imgur.com/LmBxT.png)](https://i.stack.imgur.com/LmBxT.png)
3. You can tell by the color of the Current Usage column that API limit has reached limit.
4. Click **Edit Quotas** for the API that has exceeded usage and request for new limit. This will take from few hours to day.