Load Balancing Options available

Once jobs have been migrated to running on Nomad, you’ll need to put something in front of it in order to route traffic to the correct destination, if you are running multiple web sites.

There are a multitude of options here, such as HAProxy, Nginx or Envoy. HAProxy and Nginx use consul-template to generate configuration on the fly. There could probably be a similar setup created for Envoy. I was hoping to be able to use Envoy for my front-end load balancer as Consul makes use of it for Consul Connect (service mesh). However, at this time there aren’t the correct hooks in place to easily make use of the Consul service catalog. That leaves two main options left - Fabio and Traefik. Fabio is super easy to use, but isn’t very fancy. Traefik is a little more complex, but very powerful. Both of these have native support for getting service information from Consul. Having used Fabio in the past, I decided to see if the extra effort was worth it for Traefik. Traefik has a commercial offering as well as the open source version, and seems to be in active development. Development rate for Fabio is a little slower, but again, it does work for what it does. Traefik also has support for ACME, while Fabio does not. For my home lab, this was important.

The Hashicorp Learn documentation for Load Balancing with Traefik (available here) does a great job of getting you going. On top of that, the upstream documentation provides a very large amount of information.

Unfortunately, both fail to address a scenario which I think one may see quite a bit - backend instances which are running HTTPS. Security concerns drive the need to have all traffic encrypted, even internally. For some scenarios, Consul Connect (service mesh) may be the answer but I don’t think this appropriate in all scenarios.

Configuration Overview

While the tutorial has the configuration in a TOML file, most of that can be specified on the command line.

For instance, this is the stanza for my Traefik setup:

task "traefik" {
  driver = "docker"

  config {
    image        = "traefik:v2.4"
    network_mode = "host"
    args = [
      "--api.dashboard=true",
      "--api.insecure=true",
      "--entrypoints.traefik.address=<Internal IP>:8888",
      "--entrypoints.web.address=:80",
      "--entrypoints.web.http.redirections.entryPoint.to=websecure",
      "--entrypoints.web.http.redirections.entryPoint.scheme=https",
      "--entrypoints.websecure.address=:443",
      "--entrypoints.websecure.http.tls=true",
      "--providers.consulCatalog=true",
      "--providers.consulCatalog.prefix=traefik",
      "--providers.consulCatalog.exposedByDefault=false",
      "--providers.file.directory=/configuration/",
      "--providers.file.watch=true",
    ]
    volumes = [
      "/home/docker_data/traefik/static:/configuration",
    ]
  }
  resources {
    cpu    = 500
    memory = 128
  }
}

Some things to note here: providers.consulCatalog.prefix=traefik – this value must match the tags of your jobs. If you say foobar instead of traefik, then it must be foobar.http…

I have auto-redirection to https set up for everything. For my home lab, I don’t have this as the default, and instead put it in each service config.

In addition to using the Consul Catalog, we do have file-based configuration, which you’ll see why shortly.

Problem

So the assumption is that the TLS (HTTPS) session will end at Traefik and a cleartext connection will be opened to the HTTP server in the background at whatever port it is registered at in Nomad / Consul. The documentation implies that you can have HTTPS but to me it was very unclear.

Solution

The answer is that whenever you register a service like this:

  service {
    name = "drobnak-com-ssl"
    port = "https"
    tags = [
      "traefik.enable=true",
      "traefik.http.routers.dc.rule=Host(`drobnak.com`) || Host(`www.drobnak.com`)",
    ]
  }

You are implicitly creating a loadbalancer for the service. Therefore we can set additional options which allow us to update parameters for the connection to the backend servers.

By updating the block to look like this:

  service {
    name = "drobnak-com-ssl"
    port = "https"
    tags = [
      "traefik.enable=true",
      "traefik.http.routers.dc.rule=Host(`drobnak.com`) || Host(`www.drobnak.com`)",
      "traefik.http.services.dc.loadbalancer.server.scheme=https",
    ]
  }

You can see we have told Traefik that the connection to the backend server will be https. However, this will probably give you errors, unless you’ve added the IP address of the backend into the SAN (Subject Alternative Name) value for the SSL certificate. If we write it like this:

  service {
    name = "drobnak-com-ssl"
    port = "https"
    tags = [
      "traefik.enable=true",
      "traefik.http.routers.dc.rule=Host(`drobnak.com`) || Host(`www.drobnak.com`)",
      "traefik.http.services.dc.loadbalancer.server.scheme=https",
      "traefik.http.services.dc.loadbalancer.serversTransport=drobnakCom@file",
    ]
  }

We can Tell Traefik that there are specific options to use when talking to the backend servers. Unfortunately, serversTransport configuration settings at this time cannot be read in via the tags method that the other items are. So your options are kubernetes, or in a file. Obviously we’re going to use a file here.

So, inside of /home/docker_data/traefik/static/transports.toml we have:

[http.serversTransports.drobnakCom]
  serverName = "drobnak.com"
  rootCAs = [ "/configuration/cloudflare-root.crt" ]

This tells Traefik to use the server name drobnak.com in the HTTPS request to the backend. Problem solved.

Manual configuration

What if we have some services which aren’t in Nomad quite yet? I had two such things running on the server itself, and that was easily solved by putting another file in place.

Inside of /home/docker_data/traefik/static/static_urls.tom we have:

  [http.routers.drobnak-commento]
    rule = "Host(`commento.drobnak.com`)"
    service = "drobnak-commento"

[http.services]
    [http.services.drobnak-commento]
        [[http.services.drobnak-commento.loadBalancer.servers]]
          url = "http://localhost:8080"

Which basically mirrors the setup of the tags.

Conclusion

After figuring out the HTTPS backend problem, I’ve been very happy with Traefik so far. While I would have liked to deploy only one technology (Envoy), without extra glue (like Gloo – get it), it wasn’t possible. So I’ll let consul handle the Envoy details, and try and minimize the amount of manual configuration with Traefik. I hope this information helps you.