Manage Azure Virtual Machine Scale Sets with Terraform

Azure Virtual Machine Scale Sets let you create and manage a group of load balanced virtual machines. You can then use autoscale rules to automatically increase or decrease the number of virtual machines that run your application.

In this tutorial, you will create a Virtual Machine Scale Set using Terraform, then configure two autoscale rules that adjust your application’s capacity based on CPU load.

Prerequisites

This tutorial assumes that you are familiar with the standard Terraform workflow. If you are new to Terraform, complete the Get Started tutorials first.

For this tutorial, you will need:

• Terraform v1.4+ CLI installed locally.
• An Azure account and subscription.
• Azure CLI installed and authenticated locally.

Clone example repository

In your terminal, clone the example repository.

$ git clone https://github.com/hashicorp-education/learn-terraform-azure-scale-sets

Navigate to the cloned repository.

$ cd learn-terraform-azure-scale-sets

Review configuration

This repository contains the configuration to deploy a Virtual Machine Scale Set as well as the required network infrastructure.

Here, you will find the following files:

• The main.tf file defines the Azure provider version and configures it to deploy to westus3 Azure region.
• The network.tf file defines the required network configuration, such as the virtual network, subnet, security group, and load balancer.
• The outputs.tf file declares the output from the Terraform configuration that will display the application endpoint.
• The compute.tf file creates the Virtual Machine Scale Set.
• The autoscale.tf file creates an autoscale profile for the scale set with two autoscale rules.

Scale set configuration

Open the compute.tf file to review the scale set configuration.

resource "azurerm_orchestrated_virtual_machine_scale_set" "vmss_terraform_tutorial" {
  name                        = "vmss-terraform"
  resource_group_name         = azurerm_resource_group.rg.name
  location                    = azurerm_resource_group.rg.location
  sku_name                    = "Standard_D2s_v4"
  instances                   = 3
  platform_fault_domain_count = 1     # For zonal deployments, this must be set to 1
  zones                       = ["1"] # Zones required to lookup zone in the startup script
## ...

The azurerm_orchestrated_virtual_machine_scale_set resource creates the scale set and configures the VM template, including the OS, source image, and networking. This resource then defines an os_profile.

## ...
user_data_base64 = base64encode(file("user-data.sh"))
os_profile {
    linux_configuration {
        disable_password_authentication = true
        admin_username                  = "azureuser"
        admin_ssh_key {
        username   = "azureuser"
        public_key = file("~/.ssh/id_rsa.pub")
        }
    }
}
## ...

The user_data_base64 parameter uploads the user-data.sh script to configure the software running on each machine. The os_profile parameter sets the administrator username and uploads your SSH public key, allowing you to connect to each instance with SSH.

Finally, the scale set resource defines the source_image_reference.

## ...
source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-LTS-gen2"
    version   = "latest"
}
## ...

This parameter defines the OS image. In this case, the scale set creates virtual machines running Ubuntu 22.04.

Autoscale configuration

Open autoscale.tf to review the autoscaling configuration.

resource "azurerm_monitor_autoscale_setting" "autoscale" {
  name                = "autoscale"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  target_resource_id  = azurerm_orchestrated_virtual_machine_scale_set.vmss_terraform_tutorial.id
  enabled             = true
  profile {
    name = "autoscale"
    capacity {
      default = 3
      minimum = 1
      maximum = 10
    }
## ...

The azurerm_monitor_autoscale_setting resource configures the autoscale rules of the scale set created in compute.tf and references it with the target_resource_id parameter. It creates an autoscale profile that can deploy between 1 and 10 VMs. The default parameter configures the scale set to deploy 3 VMs if there are no scaling rules or metrics available.

The example configuration has two autoscale rules commented out by default, which you will enable later.

Apply Configuration

In your terminal, initialize the configuration.

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/azurerm from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/azurerm v3.51.0...
- Installed hashicorp/azurerm v3.51.0 (signed by HashiCorp)
- Installing hashicorp/random v3.5.1...
- Installed hashicorp/random v3.5.1 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Now, apply the configuration to create the scale set, autoscale configuration, and network resources. Respond yes to the prompt to confirm the operation.

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:
##...
Plan: 18 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + application_endpoint  = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes
##...

Apply complete! Resources: 18 added, 0 changed, 0 destroyed.

Outputs:

application_endpoint = "http://terraform-tutorial-enabling-krill.westus3.cloudapp.azure.com/index.php"

Visit the application_endpoint address output value in your browser to test out your application.

Add autoscale rules

Use the az CLI to confirm that the scale set created 3 virtual machines.

$  az vm list --resource-group terraform-tutorial --output table

Name                    ResourceGroup       Location    Zones
-----------------------  ------------------  ----------  -------
vmss-terraform_29eed767  terraform-tutorial  westus3    1
vmss-terraform_71bc4dcc  terraform-tutorial  westus3    1
vmss-terraform_b713a96c  terraform-tutorial  westus3    1

Open autoscale.tf and uncomment the two autoscale rules.

/*
rule {
    metric_trigger {
        metric_name     = "Percentage CPU"
            metric_resource_id = azurerm_orchestrated_virtual_machine_scale_set.vmss_terraform_tutorial.id
        operator        = "LessThan"
        statistic       = "Average"
        time_aggregation   = "Average"
        time_window     = "PT2M"
        time_grain      = "PT1M"
        threshold       = 10
    }
    scale_action {
        direction = "Decrease"
        type    = "ChangeCount"
        value   = "1"
        cooldown  = "PT1M"
    }
}
rule {
    metric_trigger {
        metric_name     = "Percentage CPU"
        metric_resource_id = azurerm_orchestrated_virtual_machine_scale_set.vmss_terraform_tutorial.id
        operator        = "GreaterThan"
        statistic       = "Average"
        time_aggregation   = "Average"
        time_window     = "PT2M"
        time_grain      = "PT1M"
        threshold       = 90
}
    scale_action {
        direction = "Increase"
        type    = "ChangeCount"
        value   = "1"
        cooldown  = "PT1M"
    }
}
*/

These two rules tell the scale set to add a VM when the average CPU load is greater than 90% and to remove a VM when it is less than 10%. The scale_action parameter in each autoscale rule configures the scale set to only add or remove one VM at a time, and the cooldown parameter configures the scale to wait one minute between scaling actions. Remember that the autoscale profile that you created in the previous step ensures a minimum of 1 and a maximum of 10 virtual machines will be deployed.

Apply the updated configuration to add the two rules to your autoscale settings.

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # azurerm_monitor_autoscale_setting.autoscale will be updated in-place
##..

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes
##..

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Since each VM is using less than 10% of their CPU, the scale set scales down the number of VMs, waiting one minute between each action. Run the following command after a few minutes to verify that only a single VM is running.

$ az vm list --resource-group terraform-tutorial --output table 

Name                    ResourceGroup       Location    Zones
-----------------------  ------------------  ----------  -------
vmss-terraform_71bc4dcc  terraform-tutorial  westus3    1

Destroy configuration

Now that you have completed this tutorial, destroy the Azure resources you provisioned to avoid incurring unnecessary costs. Respond yes to the prompt to confirm the operation.

$ terraform destroy

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:
##...
Plan: 0 to add, 0 to change, 18 to destroy.

Changes to Outputs:
  - application_endpoint = "http://terraform-tutorial-enabling-krill.westus3.cloudapp.azure.com/index.php" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes
##...
Destroy complete! Resources: 18 destroyed.

Next steps

In this tutorial, you used Terraform to provision a Virtual Machine Scale Set that deployed three virtual machines. You then added two rules to the scale set autoscale profile to scale up and down based on VM CPU load.

Learn more about Azure Virtual Machine Scale Sets and Terraform:

• Read about Azure Virtual Machine Scale Sets.
• Learn how to Get Started with the Azure Terraform provider.