Ch7 - can't increase quota of Dataproc cluster; workaround not working

[so-james-gammerman]

Unfortunately I can't run create_large_cluster.sh :

 - Insufficient 'CPUS' quota. Requested 404.0, available 12.0.
 - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 404.0, available 20.0.
 - Insufficient 'DISKS_TOTAL_GB' quota. Requested 25500.0, available 2596.0.
 - Insufficient 'INSTANCES' quota. Requested 51.0, available 21.0.
 - Insufficient 'IN_USE_ADDRESSES' quota. Requested 51.0, available 5.0.

The chapter 7 README says: If it fails with quota issues, get increased quota. If you can't have more quota, reduce the number of workers appropriately

Unfortunately if you're using the GCP Free trial then you can't get the increased quota, and I've tried reducing the number of workers several times but the requested number of CPUs is still too large- this was the output with the following parameters:
--num-workers 3 --num-secondary-workers 3:

 - Insufficient 'CPUS' quota. Requested 52.0, available 12.0.
 - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 52.0, available 20.0.
 - Insufficient 'DISKS_TOTAL_GB' quota. Requested 3500.0, available 2596.0.
 - Insufficient 'IN_USE_ADDRESSES' quota. Requested 7.0, available 5.0.

Any advice would be appreciated otherwise I'll have to give up on the rest of this chapter...

[lakshmanok]

Yes, you may not be able to do the large cluster on the free trial. You could do the rest of the chapter, but sticking to the small dataset that the small cluster is sufficient for. thanks, Lak

…

On Wed, Feb 1, 2023, 10:27 AM so-james-gammerman ***@***.***> wrote: Unfortunately I can't run create_large_cluster.sh : - Insufficient 'CPUS' quota. Requested 404.0, available 12.0. - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 404.0, available 20.0. - Insufficient 'DISKS_TOTAL_GB' quota. Requested 25500.0, available 2596.0. - Insufficient 'INSTANCES' quota. Requested 51.0, available 21.0. - Insufficient 'IN_USE_ADDRESSES' quota. Requested 51.0, available 5.0. The chapter 7 README says: If it fails with quota issues, get increased quota. If you can't have more quota, reduce the number of workers appropriately Unfortunately if you're using the GCP Free trial then you can't get the increased quota, and I've tried reducing the number of workers several times but the requested number of CPUs is still too large: - Insufficient 'CPUS' quota. Requested 52.0, available 12.0. - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 52.0, available 20.0. - Insufficient 'DISKS_TOTAL_GB' quota. Requested 3500.0, available 2596.0. - Insufficient 'IN_USE_ADDRESSES' quota. Requested 7.0, available 5.0. Any advice would be appreciated otherwise I'll have to give up on the rest of this chapter... — Reply to this email directly, view it on GitHub <#163>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANJPZYRMRTJFLBRAJYUACDWVJ6HLANCNFSM6AAAAAAUN3XS3U> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>