Using Dependencies

In the getting started guide we talked about dependencies, and briefly mentioned that they’re super important for Converge to work properly. But we didn’t really go into them there… so here we are!

Graph Walking

Briefly, Converge operates by thinking about your deployment as a graph of tasks that need to be done. It walks around the graph from the leaves (AKA tasks with no dependencies) all the way to to the root (AKA Rome, where all roads lead.) Let’s explore what that means with one of our graphs from before:

What does Converge do when you ask it to apply this graph? When Converge loads this file, it will load then file and then start walking at the node that doesn’t have any dependencies. In this case, that’s param.name = "World". When param.name has been successfully walked, it will move on to File: hello.txt. If we’re successful, the root (/) will be marked as successful, and our graph will be successful. Neat!

The Graph Command

All the graphs we’ve been seeing so far have just been the output of Converge’s graph command. When asked, Converge will load up any modules you specify and then render them as Graphviz dot output. You can render that like so:

$ converge graph --local yourModule.hcl | dot -Tpng > yourModule.png

When you’re developing modules, make a habit of rendering them as graphs. It makes it easier to think about how the graph will be executed.

Cross-Node References

Resources may references one-another as long as the references do not introduce circular dependencies. When creating a reference from one node to another we can use the lookup command to reference fields of an entry that are provided by that entries module. The available fields will vary depending on the module and should be documented along with each module. The example below illustrates using lookup to access fields from a docker.image node from within docker.container

docker.image "nginx" {
  name    = "nginx"
  tag     = "1.10-alpine"
  timeout = "60s"
}

docker.container "nginx" {
  name  = "nginx-server"
  image = "{{lookup `docker.image.nginx.name`}}:{{lookup `docker.image.nginx.tag`}}"
  force = "true"
  expose = [
    "80",
    "443/tcp",
    "8080",
  ]
  publish_all_ports = "false"
  ports = [
    "80",
  ]
  env {
    "FOO" = "BAR"
  }
  dns = ["8.8.8.8", "8.8.4.4"]
}

As we can see, lookup syntax resembles that of parameters and add implicit dependencies between nodes.

Explicit Dependencies

When we’re walking our graph, there are a lot of operations that can be done in parallel. For this to work, you will need to specify dependencies between resources in the same file. Let’s take the following example:

task "names" {
  check = "test -d names"
  apply = "mkdir names"
}

file.content "hello" {
  destination = "names/hello.txt"
  content     = "Hello, World!"
}

As a human reading this, we can clearly see that file.content.hello is dependent on task.names, because the file needs the directory to be created before it can write files into it. But Converge doesn’t know that yet, so here’s how it looks:

To fix this, we’ll need to specify depends on our file.content. depends is a list of resources in the current module that must be successfully walked before walking ours. They’re specified as the resource type, a dot, then the resource name. So task "names" above becomes task.names.

task "names" {
  check = "test -d names"
  apply = "mkdir names"
}

file.content "hello" {
  destination = "names/hello.txt"
  content     = "Hello, World!"

  depends = ["task.names"] # added in the resource that needs the dependency
}

Now Converge correctly sees that it needs to walk task.names before file.content.hello:

Grouping

There can be some scenarios where a group of tasks are not explicitly dependent on each other but also cannot be run in parallel. A good example of this is package management tools like apk or apt. As an example, let’s look at this file which installs three packages:

task "install-tree" {
  check = "dpkg -s tree >/dev/null 2>&1"
  apply = "apt-get install -y tree"
}

task "install-jq" {
  check = "dpkg -s jq >/dev/null 2>&1"
  apply = "apt-get install -y jq"
}

task "install-build-essential" {
  check = "dpkg -s build-essential >/dev/null 2>&1"
  apply = "apt-get install -y build-essential"
}

Here is what the corresponding graph looks like:

If you were to execute apply against this graph, you would end up with errors that look something like this:

E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/

This is because multiple apt-get commands cannot be run at the same time.

You could certainly use depends to chain these tasks together but this is tedious and error prone. Luckily, Converge supports groups which makes this much easier. We can add a named group to each task and Converge will modify the graph so that these tasks are not run in parallel.

task "install-tree" {
  check = "dpkg -s tree >/dev/null 2>&1"
  apply = "apt-get install -y tree"
  group = "apt"
}

task "install-jq" {
  check = "dpkg -s jq >/dev/null 2>&1"
  apply = "apt-get install -y jq"
  group = "apt"
}

task "install-build-essential" {
  check = "dpkg -s build-essential >/dev/null 2>&1"
  apply = "apt-get install -y build-essential"
  group = "apt"
}

And the corresponding graph: