---
layout: docs
page_title: Observe service mesh on Docker containers
description: >-
   Deploy a Consul service mesh observability stack with Docker Compose to gain complete insight into metrics, logs, and traces.
---

# Observe service mesh on Docker containers

This page describes the configuration process for service mesh observability features when Consul runs on Docker containers.

## Overview

In this tutorial, you will create a local Consul service mesh and observability
stack using Docker Compose. You will then explore your Consul service mesh with 
the observability suite.

The complexity of a service mesh environment requires advanced mechanisms to 
determine where services are running, the traffic patterns of the data flowing, 
and the health of the connections between them. Having this information makes 
it easier to not only understand failures occurring in your
deployments, but also gives you the granularity needed to identify more subtle 
failures within your network such as performance degradation, packet loss, 
memory pressure, etc.

True service mesh observability consist of three core elements: metrics, logs
and traces. To explore each of these elements, you will create an observability
suite comprised of Prometheus (metrics), Loki (logs), Tempo (traces), and Grafana 
(visualization). Your Consul service mesh will host an example microservice 
application that will exhibit the ingress, API, and web layers. As you access
the application; metrics, logs, and traces will be sent to your observability
stack for you to explore.

This tutorial uses elements that are not suitable for production
environments including an unsecured Consul datacenter, relaxed security policies,
and container configurations only suited for development. The tutorial will teach you
the core concepts for understanding and exploring observability in a Consul service mesh.
Refer to the [Consul Reference Architecture](/consul/tutorials/production-deploy/reference-architecture)
for Consul best practices and the [Docker Documentation](https://docs.docker.com/)
for Docker best practices.

## Prerequisites

### Linux or MacOS

Elements of this tutorial require a Linux or MacOS environment for deployment. See the [Docker plugin page](https://docs.docker.com/engine/extend/) for more information about OS requirements.

### Docker

You will need a local install of Docker running on your machine for this tutorial. You can find the instructions for installing Docker on your specific operating system [here](https://docs.docker.com/install/). Docker Engine 19.03.0 was tested in this tutorial.

### Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Visit the [Docker Compose install guide](https://docs.docker.com/compose/install/) for operating system specific installation instructions. Docker Compose format 3.7 was used in this tutorial.

### Loki logging drivers

Install the Loki logging drivers for Docker. See the [Loki Docker driver plugin page](https://grafana.com/docs/loki/latest/clients/docker-driver/) for more information.

```shell-session
$ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
```

### GitHub repository

Clone the GitHub repository containing the configuration files and resources.

<CodeTabs tabs={[ "HTTPS", "SSH" ]}>

```shell-session
$ git clone https://github.com/hashicorp/learn-consul-docker.git
```

```shell-session
$ git clone git@github.com:hashicorp/learn-consul-docker.git
```

</CodeTabs>

Change into the directory with the newly cloned repository. This directory contains the complete configuration files.

```shell-session
$ cd learn-consul-docker/datacenter-deploy-observability
```

## Environment overview

The observability suite provides insight into the metrics, logs, and traces that occur 
throughout the Consul service mesh. This diagram illustrates the components of the
observability suite and Consul service mesh that you will create in this tutorial.

![Image of architecture diagram for this environment.](/img/docker/observability_architecture_diagram.png 'There are two main logical environments in this architecture diagram: a Consul service mesh with each of its components in the left diagram, and the observability platform with each of its components in the right diagram. Both of these logical environments are encapsulated in a Docker Compose environment.')

## Create environment

From within your working directory, run the following Docker Compose command to
create your local service mesh environment and observability suite.

```shell-session
$ docker-compose up --detach

Creating network "datacenter-deploy-observability_vpcbr" with driver "bridge"
Creating node-exporter ... done
Creating tempo         ... done
Creating grafana       ... done
Creating consul-server ... done
Creating prometheus    ... done
Creating loki          ... done
Creating ingress       ... done
Creating web           ... done
Creating api           ... done
Creating api_proxy     ... done
Creating ingress_proxy ... done
Creating web_proxy     ... done
```

<Note>

 Your first run will take the longest as Docker will pull the respective images from the Docker Hub repository.
Extra runs will not need you to download the image again and should only take a few seconds to complete.

</Note>

## Perform testing procedures 

Perform the following steps of each section to explore the elements involved in the service mesh and observability suite.

### Explore the Consul UI

Navigate to [http://localhost:8500/](http://localhost:8500/) on your browser to access the Consul UI. Consul will navigate to the services tab by default. 

![Image of Nodes page in Consul UI.](/img/docker/observability_consul_services.png 'There is a navigation bar at the top of the page with multiple options. The Nodes tab is selected. In the main area of the screen, three Consul servers and one Consul client are shown as healthy nodes. One of the Consul servers is marked as a leader node.')

Notice the services monitored by Consul. Each of the applications (ingress, web, and api) includes a sidecar proxy that makes up the service mesh.

<Note>

 If your Consul UI details do not match the image above, restart your container stack environment.

</Note>

### Explore the sample application

Navigate to [http://localhost:9090/ui/](http://localhost:9090/ui/) and refresh the page to generate traffic.

![Image of sample application UI.](/img/docker/observability_example_service.png 'There are three squares, each representing an application, connected by lines. The Ingress service is at the top, with web below it, and API at the bottom.')

The example application shows the architecture of your complete application stack. Each refresh of this page will generate extra logs, metrics, and traces that your observability suite will ingest.

### Explore Prometheus targets

Navigate to [http://localhost:9092/targets](http://localhost:9092/targets).

![Image of targets page in Prometheus UI.](/img/docker/observability_prometheus_targets.png 'There are four target endpoints: consul-server, node, services, and tempo. The status shows "up" for each of these targets.')

Notice that Prometheus is pre-configured to scrape metric-related data from four endpoints: Consul, Node-Exporter, Tempo, and itself.

### Explore Grafana data sources

Navigate to [http://localhost:3000/datasources](http://localhost:3000/datasources).

![Image of data sources page in Grafana UI.](/img/docker/observability_grafana_datasources.png 'Three datasources are present in the list: Loki, Prometheus, and Tempo.')

Grafana is pre-configured with Prometheus, Loki, and Tempo as data sources. These sources enable Grafana to visualize metrics, logs, and traces.

### Explore Logs and Traces

Navigate to [http://localhost:3000/explore](http://localhost:3000/explore).

Execute a search with these parameters: `{container_name="web"} |= "trace_id"`
     
Open one of the log lines, locate the `TraceID` field, then click the nearby `Tempo` link to jump directly from logs to traces. 

![Image of linked logs and traces in Grafana UI.](/img/docker/observability_grafana_logs_traces.png 'There are two panes within the window: a left and right pane. In the left pane, the Loki logs are shown, with one of its entries selected. On the right pane, a Tempo traces graph is shown. The Tempo traces graph shows communication details between the three services, ingress, web, and api.')

Using this functionality, notice how logs and traces provide insight into service mesh performance.

To learn more about how application communication traverses through the service mesh, see the [distributed tracing article](https://www.hashicorp.com/blog/enabling-distributed-tracing-with-hashicorp-consul).

### Explore Metrics

Navigate to [http://localhost:3000/dashboards](http://localhost:3000/dashboards).

Explore the metrics-related dashboards: Consul Server Monitoring and Node Exporter.

![Image of Consul Server Monitoring metrics dashboard in Grafana UI.](/img/docker/observability_grafana_consul_metrics.png 'A dashboard is shown with four panes present. Each of the panes shows details around various metrics generated by Consul including Raft commit time, catalog operation time, and autopilot health.')

The Consul Server Monitoring dashboard presents important application-level metrics for Consul while the Node Exporter dashboard presents system-level metrics. Each of these dimensions provides important insight into the health and performance of your service mesh.

To learn more about the various runtime metrics reported by Consul, see the [Consul telemetry docs](/consul/docs/reference/agent/telemetry).

## Clean up your environment

To clean up your environment, execute the following command.

```shell-session
$ docker-compose down --rmi all

Stopping ingress_proxy ... done
Stopping api_proxy     ... done
Stopping web_proxy     ... done
Stopping web           ... done
Stopping api           ... done
Stopping ingress       ... done
Stopping grafana       ... done
Stopping node-exporter ... done
Stopping prometheus    ... done
Stopping consul-server ... done
Stopping tempo         ... done
Stopping loki          ... done
Removing ingress_proxy ... done
Removing api_proxy     ... done
Removing web_proxy     ... done
Removing web           ... done
Removing api           ... done
Removing ingress       ... done
Removing grafana       ... done
Removing node-exporter ... done
Removing prometheus    ... done
Removing consul-server ... done
Removing tempo         ... done
Removing loki          ... done
Removing network datacenter-deploy-observability_vpcbr
Removing image hashicorp/consul:1.8.10
Removing image nicholasjackson/fake-service:v0.21.0
Removing image nicholasjackson/consul-envoy:v1.6.1-v0.10.0
Removing image prom/node-exporter:v1.1.2
Removing image prom/prometheus:v2.26.0
Removing image grafana/tempo:1f1c40b3
Removing image grafana/grafana:7.5.3
Removing image grafana/loki:2.1.0
```

## Next steps

In this tutorial, you learned how to deploy and configure a local Consul service mesh and observability
stack using Docker Compose. You also learned how to explore your Consul service mesh with the 
open source observability suite. Finally, you learned how to clean up your Docker environment.

You can continue learning the concepts of a secure Consul datacenter by completing the [deploy a secure Consul datacenter with Docker Compose tutorial](/consul/tutorials/docker/docker-compose-datacenter). This will teach you the concepts of Access Control Lists (ACLs), gossip encryption, and transport encryption. 

You can also extend your Consul skills by exploring the following tutorials:

- [Running Consul on Docker](/consul/tutorials/day-0/docker-container-agents)
- [Running Consul on Kubernetes](/consul/tutorials/get-started-kubernetes)
- [Service Discovery](/consul/tutorials/developer-discovery)
- [Service Mesh](/consul/tutorials/developer-mesh)

For additional reference documentation on the technologies and Docker images used in this guide, refer to the following websites:

- [hashicorp/docker-consul GitHub Repository](https://github.com/hashicorp/docker-consul)
- [grafana/grafana GitHub Repository](https://github.com/grafana/grafana)
- [grafana/loki GitHub Repository](https://github.com/grafana/loki)
- [grafana/tempo GitHub Repository](https://github.com/grafana/tempo)
- [prometheus/prometheus GitHub Repository](https://github.com/prometheus/prometheus)
- [prometheus/node-exporter GitHub Repository](https://github.com/prometheus/node_exporter)
