---
layout: docs
page_title: Automated upgrades
description: >-
  Automated upgrades simplify the process for updating Consul. Learn how Consul can gracefully transition from existing server agents to a new set of server agents without Consul downtime.
---

# Automated upgrades

<EnterpriseAlert>

This feature requires
HashiCorp Cloud Platform (HCP) or self-managed Consul Enterprise.
Refer to the [enterprise feature matrix](/consul/docs/enterprise#consul-enterprise-feature-availability) for additional information.

</EnterpriseAlert>

Consul Enterprise enables the capability of automatically upgrading a cluster of Consul servers to a new
version as updated server nodes join the cluster. This automated upgrade will spawn a process which monitors the amount of voting members
currently in a cluster. When an equal amount of new server nodes are joined running the desired version, the lower versioned servers
will be demoted to non voting members. Demotion of legacy server nodes will not occur until the voting members on the new version match.
Once this demotion occurs, the previous versioned servers can be removed from the cluster safely.

## Introduction

Upgrades are always a delicate moment for any production environment and it is
important to have to have best practices in place that simplify the process where
possible.

Consul Enterprise provides automated upgrades with the autopilot feature that
enable you to test the upgrade process before making changes
to production. The feature allows you to start new Consul servers alongside
the old ones and automatically switch to the new servers, as soon as they are able to
reach quorum.

This process automates the leader election process and ensures that the new leader is
elected among the new nodes so that removing the old nodes from the datacenter
does not trigger a leader election.

<Warning>

This tutorial describes a cluster upgrade process that deploys new servers and then removes the old ones. To upgrade the version of the `consul` binary running on servers in your cluster without deploying new servers, follow the [general upgrade process](/consul/docs/upgrade/instructions/general) and disable autopilot's upgrade migration feature.

</Warning>

Using this tutorial you will test the functionality and get examples on how to:

1. Upgrade Consul servers to a newer version
1. Patch configuration for server nodes without changing Consul version

This will help you create a standardized and repeatable internal process for server upgrades.

## Prerequisites

To test the automated upgrades feature explained in this tutorial you will need:

- A Consul Enterprise datacenter with three servers running Consul Enterprise
  1.7.0.
- Three extra nodes with Consul Enterprise 1.7.2 binary installed to be used as
  the new servers once the upgrade is concluded.

You will also need a text editor, the `curl` executable to test the API
endpoints, and optionally the `jq` command to format the output for `curl`.

## Upgrade to a new Consul version

The automated upgrades feature is enabled by default in Consul Enterprise. You
can verify the configuration for your datacenter by using the `consul operator autopilot` command.

```shell-session
$ consul operator autopilot get-config
```

```plaintext
##...
DisableUpgradeMigration = false
UpgradeVersionTag = ""
```

~> In case the feature is disabled in your datacenter you can enable it using
the `consul operator autopilot set-config -disable-upgrade-migration=false`
command.

### Add new servers

Once you have verified that `DisableUpgradeMigration` is set to `false`, you can
start the new servers and make sure they join the existing datacenter with the Consul 1.7.0 servers.
In this example the old servers with Consul version 1.7.0 are identified as `server-1`,
`server-2`, and `server-3`. While the new ones with Consul version 1.7.2 are identified as `server-1a`,
`server-2a`, and `server-3a`.

When autopilot detects
that a server with a newer version of Consul has joined the datacenter,
it will wait to promote the new server as voter until enough
newer-versioned servers have been added to the datacenter to reach quorum. When the count of new
servers equals or exceeds that of the old servers, autopilot will begin
promoting the new servers to voters and demoting the old servers. After this is
finished, the old servers can be safely removed from the datacenter.

To check the Consul version on the old servers, you can either use the [autopilot
health](/consul/api-docs/operator) endpoint or the `consul members`
command.

<Tabs>
<Tab heading="CLI command">


```shell-session
$ consul members
```

```plaintext hideClipboard
Node      Address           Status  Type    Build      Protocol  DC   Segment
server-1  10.20.10.11:8301  alive   server  1.7.0+ent  2         dc1  <all>
server-2  10.20.10.12:8301  alive   server  1.7.0+ent  2         dc1  <all>
server-3  10.20.10.13:8301  alive   server  1.7.0+ent  2         dc1  <all>
```

</Tab>
<Tab heading="API call using cURL">


```shell-session
$ curl http://127.0.0.1:8500/v1/operator/autopilot/health | jq .
```

```json
{
  "Healthy": true,
  "FailureTolerance": 1,
  "Servers": [
    {
      # ...
      "Name": "server-1",
      "Address": "10.20.10.11:8300",
      "SerfStatus": "alive",
      "Version": "1.7.0",
      "Leader": true,
      # ...
      "Voter": true,
      # ...
    },
    {
      # ...
      "Name": "server-2",
      "Address": "10.20.10.12:8300",
      "SerfStatus": "alive",
      "Version": "1.7.0",
      "Leader": false,
      # ...
      "Voter": true,
      # ...
    },
    {
      # ...
      "Name": "server-3",
      "Address": "10.20.10.13:8300",
      "SerfStatus": "alive",
      "Version": "1.7.0",
      "Leader": false,
      # ...
      "Voter": false,
      # ...
    }
  ]
}
```

</Tab>
</Tabs>


Now that you have checked the autopilot configuration and Consul version on your existing three servers, start two new servers with Consul version 1.7.2. Be sure to join them to the existing datacenter.

Verify the new servers are not yet part
of the voters pool.

```shell-session
$ consul operator raft list-peers
```

```plaintext hideClipboard
Node       ID                                    Address           State     Voter  RaftProtocol
server-1   64858627-0565-0c81-234f-4d84b3455303  10.20.10.11:8300  leader    true   3
server-2   5b10789f-3fe6-2407-e6f2-9e85cf9cfeb3  10.20.10.12:8300  follower  true   3
server-3   b0ecba67-a422-e46d-ac53-8f48aa403944  10.20.10.13:8300  follower  true   3
server-1a  15a7b243-95ad-8c7a-4335-4c1690120bbd  10.20.10.21:8300  follower  false  3
server-2a  fd1e7110-cc90-cf0c-d1e1-a22190208b0a  10.20.10.22:8300  follower  false  3
```

Finally, start the third new server on Consul version 1.7.2. Once the third new server is started and autopilot detects the possibility to
reach a quorum among the new servers only it promotes them as voters, triggers a
new leader election, and demotes the old nodes as non-voters.

Verify the three new servers on Consul version 1.7.2 are now voters.

```shell-session
$ consul operator raft list-peers
```

```plaintext hideClipboard
Node       ID                                    Address           State     Voter  RaftProtocol
server-1   64858627-0565-0c81-234f-4d84b3455303  10.20.10.11:8300  follower  false  3
server-2   5b10789f-3fe6-2407-e6f2-9e85cf9cfeb3  10.20.10.12:8300  follower  false  3
server-3   b0ecba67-a422-e46d-ac53-8f48aa403944  10.20.10.13:8300  follower  false  3
server-1a  15a7b243-95ad-8c7a-4335-4c1690120bbd  10.20.10.21:8300  leader    true   3
server-2a  fd1e7110-cc90-cf0c-d1e1-a22190208b0a  10.20.10.22:8300  follower  true   3
server-3a  36d57c89-dc39-056d-f8eb-b1a63faa85b8  10.20.10.23:8300  follower  true   3
```

### Stop old servers

Once the upgrade is rolled out you can proceed by removing the old servers from
the datacenter using the `consul leave` command. This will leave only the new
servers and will prevent a leader election during the process that you would
have experienced without autopilot.

## Migrations without a Consul version change

In some environments, Consul is not the only component that needs to be upgraded
to ensure security or stability for the node. For example, you might
need to roll out node updates to apply OS patches
or new configuration settings.

To cover these cases, Consul autopilot offers the possibility to set the
`UpgradeVersionTag` parameter to override the version information used during
automated upgrades. The same logic can be used for updating the
datacenter nodes even without the need to change Consul version.

If the `UpgradeVersionTag` is configured, Consul will use its value to look for
a version in each server's specified
`-node-meta` tag.
For example, if `UpgradeVersionTag` is set to `build`, and `-node-meta build:0.0.2` is used when starting a server, that server's version will be
`0.0.2` when considered in a migration. The upgrade logic will follow semantic
versioning and the version string must be in the form of either `X`, `X.Y`, or
`X.Y.Z`.

### Configure autopilot for existing servers

-> For this example, you can use the three servers started in the previous example (`server-1a`, `server-2a` and `server-3a`). To perform the migration you will start three extra servers called `server-1b`, `server-2b` and `server-3b`. Using three new servers (`server-1b`, `server-2b` and `server-3b`) will help you differentiate the command output from the previous example. If you do not want to start three new nodes you can reuse the ones used previously as long as the Consul binary on the servers is changed to match the one running on `server-1a`, `server-2a` and `server-3a`.

Before proceeding with the upgrade it is necessary to prepare the existing
servers to be compared with the new ones. For all the existing servers, add the `node_meta` option to the
configuration.

<CodeTabs heading="Consul server agent configuration">

```hcl
## ...
node_meta {
  build = "0.0.1"
}
## ...
```

```json
## ...
"node_meta": {
    "build": "0.0.1"
  },
## ...
```

</CodeTabs>


This change requires a reload.

```shell-session
$ consul reload
```

```plaintext hideClipboard
Configuration reload triggered
```

Once the configuration is updated, you should
modify the Consul autopilot configuration to reflect the `node_meta` configuration.

```shell-session
$ consul operator autopilot set-config -upgrade-version-tag=build
```

```plaintext hideClipboard
Configuration updated!
```

Ensure the autopilot configuration has been successfully updated.

```shell-session
$ consul operator autopilot get-config
```

```plaintext hideClipboard
## ...
DisableUpgradeMigration = false
UpgradeVersionTag = "build"
```

The `UpgradeVersionTag` should be set to `build`.

### Start new servers

For the new servers, you will define the `UpgradeVersionTag` directly in the
configuration.

<CodeTabs>

<CodeBlockConfig highlight="6">

```hcl
## ...
node_meta {
  build = "0.0.2"
}
autopilot {
  upgrade_version_tag = "build"
}
## ...
```

</CodeBlockConfig>

<CodeBlockConfig highlight="6">

```json
# ...
	"node_meta": {
		"build": "0.0.2"
	},
	"autopilot":{
		"upgrade_version_tag":"build"
	},
# ...
```

</CodeBlockConfig>

</CodeTabs>


Once the third new server is started and autopilot detects the possibility to
reach a quorum among the new servers only it promotes them as voters, triggers a
new leader election, and demotes the old nodes as non-voters. This time the
version comparison happened not using the Consul internal version but the value
expressed in the `build` value.

```shell-session
$ consul operator raft list-peers
```

```plaintext hideClipboard
Node       ID                                    Address           State     Voter  RaftProtocol
server-1a   dcc542fc-ff83-2cab-5bbb-7615584be019  10.20.10.11:8300  follower  false  3
server-2a   f38a430b-dda0-ad3d-a394-08c59052723c  10.20.10.12:8300  follower  false  3
server-3a   647a98a1-d2ab-f5d3-e911-91c1c10a92f4  10.20.10.13:8300  follower  false  3
server-1b   7770d10c-fcca-f6f9-222a-e1777a7fd29e  10.20.10.21:8300  leader    true   3
server-2b   bace6b4d-5e2e-8942-d553-ca527cbc2386  10.20.10.22:8300  follower  true   3
server-3b   221de624-0f5c-29da-ffaf-39481adcd234  10.20.10.23:8300  follower  true   3
```

### Stop old servers

Once the migration is rolled out you can proceed by removing the old servers from
the datacenter using the `consul leave` command. This will leave only the new
servers but will prevent a leader election during the process that you would
have experienced without autopilot.

## Next steps

In this tutorial you upgraded your Consul datacenter by using autopilot's automated
upgrades functionality. The automated upgrade feature allows you to both upgrade the Consul version of your servers and to deploy updated version for your server instances without the need to change the Consul version to distribute patches. You now have an extra tool to use when planning upgrades
of critical system and to reduce manual steps to a minimum.

Before planning any Consul upgrade make sure to review version specific
instructions provided by:

- [Upgrading Consul](/consul/docs/upgrade)
- [Upgrading specific Consul versions](/consul/docs/upgrade/version-specific)

You can learn more on other autopilot functionalities by checking our [autopilot
tutorial](/consul/tutorials/datacenter-operations/autopilot-datacenter-operations).
