Exploring Nornir, the Python Automation Framework

This blog contributed by Patrick Ogenstad. Patrick works at Conscia Netsafe, a Cisco Gold partner in Sweden. He also writes about automation and development on his blog Networklore 

A while back a client asked me, “I’m guessing you are pretty good at Python multiprocessing?” I assumed that his goal wasn’t to evaluate my skills. Instead of answering, I asked a question of my own. “What problem is it you want to solve?”

It turned out that he had a couple of thousand devices and needed to collect data from each one. As connecting and gathering the information from each device was taking one or two seconds it would add up to a couple of hours to run the entire job, so he needed to add parallelization. I told him about the Norns, or Nornir, from the Norse mythology. They spun the threads of fate that connected all beings.

I asked him if his goal was to have the power of the Nornir, or if he wanted a lecture about multiprocessing in Python. As he remained silent, I told him about the Nornir automation framework.

A Python Automation Framework

Enough with the theatrics. What is Nornir? In short, Nornir is plugable multi-threaded framework with inventory management to help operate collections of devices. The way it differs from other automation frameworks is that it is used directly from Python. Compare this to Ansible where you write playbooks in a DSL (Domain specific language) based on YAML.

For our friend from the intro, this wasn’t a problem. As he asked for a Python solution Nornir fit perfectly.

There is a myth that writing code is hard, that it something meant for others, but not you. While it can be hard and take years to master, the goal should be to learn enough to help you with your job and be happy with that.

What Does Nornir Do?

Nornir works with collections of data. It runs tasks against this data and keeps track of all the threads. In a network environment, this typically means that you have a device inventory with data associated with each node. You can define tasks, and those tasks can contain plugins shared by the community or your custom plugins or python code. Nornir then ties everything together and lets you run those tasks against all, or a subset of, devices handling the data, parallelization and keeping track of the errors for you.

Your First Interaction with Nornir

Like most Python packages you install Nornir using pip.

[cc lang=”Python”]pip install nornir[/cc]

The first thing you need before you can use Nornir is an inventory. In its most basic form, a Nornir inventory is a Python dictionary containing one or more hosts. Optionally you can have a second dictionary which defines groups, as you might imagine this lets you use any source as the input to your inventory. To make things simpler, Nornir includes a few inventory plugins the default one being SimpleInventory which uses YAML files. Let’s create a few example hosts and groups.
hosts.yaml:

---
al-rtr-1:
    site: alderaan
    groups:
      - routers
    loopback0: 10.230.12.1

en-rtr-1:
    site: endor
    groups:
      - routers
    nornir_username: wicket
    nornir_password: G0lden_god
    loopback0: 10.230.12.2
    nornir_host: 127.0.0.1

ta-rtr-1:
    site: tatooine
    groups:
      - routers
    loopback0: 10.230.12.3

ta-sw-1:
    site: tatooine
    groups:
      - switches

ta-sw-2:
    site: tatooine
    groups:
      - switches

The keys under each host could, for the most part, be anything. There is, however, one restriction you have to follow; the groups key must be a list, the use of groups is optional though. You might also have noticed that one of the hosts has a username and password defined. The plugins that ship with Nornir uses some of these variables. If the idea of entering a password in clear text like above horrifies you, don’t worry this is just an example. You can load passwords from a secret store of your choice.
groups.yaml

---
defaults:
    nornir_username: luke
    nornir_password: blue_saber
    domain: empire.space
    contact: Unknown

routers:
    nornir_username: han
    nornir_password: I_shot_first
    nornir_nos: iosxr
    contact: Router team

switches:
    nornir_nos: ios
    contact: Switch group

Once you have these files, you can explore them with Nornir using the InitNornir function, which is a quick way to get started using sane defaults.

from nornir.core import InitNornir

nr = InitNornir()

You can see that the nornir_username gets assigned to the username property of each host. Anything you set in the “defaults” group gets applied to all hosts. You can override the value of each property at the group or host level. Most data you define in your inventory is accessible through the Host objects in the same way as you would access a key in a Python dictionary. An example of this is the “contact” key above. As you can see from the initial test, the username value can be accessed either as an attribute or as a key in a dictionary.

>>> nr.inventory.hosts

{'al-rtr-1': Host: al-rtr-1, 'en-rtr-1': Host: en-rtr-1, 'ta-rtr-1': Host: ta-rtr-1,
 'ta-sw-1': Host: ta-sw-1, 'ta-sw-2': Host: ta-sw-2}

>>> nr.inventory.hosts['al-rtr-1'].username
'han'
>>> nr.inventory.hosts['en-rtr-1'].username
'wicket'
>>> nr.inventory.hosts['ta-sw-1'].username
'luke'
>>>

>>> nr.inventory.hosts['ta-sw-1']['nornir_username']
'luke'
>>>

>>> nr.inventory.hosts['ta-sw-1'].nos
'ios'
>>> nr.inventory.hosts['ta-sw-1']['contact']
'Switch group'
>>>

In our other thread back in the real world, my friend didn’t have an inventory defined in the SimpleInventory format. All he had was a text file containing 10 000 devices, or mac addresses to be more specific. One option would be to convert the text file into the YAML format.
Another option is to read the content and send it directly to the base Inventory class. A consequence of this approach is that we can no longer use the InitNornir function and have to initialize the code a bit different. Not to worry though, this could be considered an advanced topic and I only show it here to illustrate how easy it is to extend things in Nornir.

from nornir.core import Nornir
from nornir.core.configuration import Config
from nornir.core.inventory import Inventory


with open("source_mac.txt", 'r') as fs:
    hosts = {
        h: {
            'mac': h
        } for h in fs.read().splitlines()
    }


inv = Inventory(hosts=hosts)
conf = Config(num_workers=100)
nr = Nornir(inventory=inv, dry_run=False, config=conf)

The source_mac.txt is just a file containing a mac address on each line. A subset of the source_mac.txt looks like this:

[cc lang=”Python”]B0:98:2B:76:F3:E9
B0:98:2B:76:F9:C3
B0:98:2B:76:F5:94
B0:98:2B:76:ED:98
B0:98:2B:76:F1:5E
B0:98:2B:76:FA:56
B0:98:2B:76:FB:0C
B0:98:2B:76:F3:79
B0:98:2B:76:EF:12[/cc]

Writing a Nornir Application

Not to say that it isn’t educational to explore the inner details of the inventory, but it isn’t that helpful unless you can run tasks. Returning to a galaxy more than fifteen hops away all of our hosts belonged to a specific site. All of the hosts within each site share common data, so let’s define a data source for the sites.
endor.yaml

---
asn: 66401
networks:
  - net: 10.12.0.0
    mask: 255.255.0.0
  - net: 10.16.0.0
    mask: 255.255.0.0
  - net: 10.192.25.0
    mask: 255.255.255.0

tatooine.yaml

---
asn: 66801
networks:
  - net: 192.168.23.0
    mask: 255.255.255.0
  - net: 192.168.24.0
    mask: 255.255.255.0
  - net: 192.168.38.0
    mask: 255.255.255.0

I know what you are thinking, why aren’t they using IPv6? Remember, all this played out a long time ago. Our first goal is to load the data from these files so that it’s associated with each of the hosts from that site. There is a plugin to read YAML files in Nornir which we are going to use. We do however need to write a bit of code to tie it together. Let’s take a look!

from nornir.core import InitNornir
from nornir.plugins.tasks.data import load_yaml
from nornir.plugins.functions.text import print_result


def load_data(task):
    data = task.run(
           task=load_yaml,
           file=f'{task.host["site"]}.yaml'
    )

    task.host["asn"] = data.result["asn"]
    task.host["networks"] = data.result["networks"]


nr = InitNornir()
r = nr.run(task=load_data)
print_result(r)

A few new things get introduced here. The load_yaml task plugin, which read data from a YAML file. We create a load_data task that in turn uses load_yaml as a subtask. The goal is to load data from a file that corresponds to the site associated with each host. Note the f-strings introduced in Python 3.6. We save the content of the site file and store the values on each host. As a final step, we use the print_result function to see what happens.

It looks like the site file for Alderaan is missing, that’s terrible. By default, Nornir doesn’t particularly care about the fate of Alderaan. If you feel differently, it is possible to cause Nornir to raise an exception if something breaks instead.

Rendering a Configuration

Now that we have some data for our hosts, most of them anyway, we can start to do something more interesting. Let’s use this data together with the Jinja2 templating engine. Nornir currently includes two plugins to render templates with Jinja2, one for strings and another one for files. If you would rather work with the Mako or Velocity engine, it would be simple enough to create a plugin for that.

router bgp {{ asn }}
 bgp router-id {{ loopback0 }}
 address-family ipv4 unicast
{% for n in networks %}
  network {{ n.net }} {{ n.mask }}
{% endfor %}

An observant reader might note that we are referencing the loopback0 variable in the template. We only defined this variable for the hosts in the router group so it won’t work if we try to render this template for the switches. To make sure we only generate the configuration for the router group we can use the filter function.

from nornir.core import InitNornir
from nornir.plugins.tasks.data import load_yaml
from nornir.plugins.tasks.text import template_file
from nornir.plugins.functions.text import print_result


def load_data(task):
    data = task.run(
           task=load_yaml,
           file=f'{task.host["site"]}.yaml'
    )

    task.host["asn"] = data.result["asn"]
    task.host["networks"] = data.result["networks"]
    task.host["template_config"] = task.run(task=template_file,
                                            template="router.j2", path="")


nr = InitNornir()
routers = nr.filter(nornir_nos="iosxr"
r = routers.run(load_data)
print_result(r)

Configuring Network Devices

Once we have the rendered configuration, we can configure the network devices. Remembering back to what we learned earlier we know that Nornir works on collections of data and let you run tasks against that data. Nothing in this explanation specifically mention network devices or how to connect to them. Nornir is agnostic regarding how it accesses devices. So while it currently ships with plugins for Paramiko, Napalm, and Netmiko, anything else would also be possible. If you would like to create a plugin for Restconf or something else that would be perfectly fine. There are a lot of explosions and fire in Star Wars, so Napalm seems appropriate in this context. In this final example of the code, we import the napalm_configure plugin and choose to extend our filter further to only target Endor.

from nornir.core import InitNornir
from nornir.plugins.tasks.data import load_yaml
from nornir.plugins.tasks.text import template_file
from nornir.plugins.tasks.networking import napalm_configure
from nornir.plugins.functions.text import print_result


def load_data(task):
    data = task.run(
           task=load_yaml,
           file=f'{task.host["site"]}.yaml'
    )

    task.host["asn"] = data.result["asn"]
    task.host["networks"] = data.result["networks"]
    r = task.run(task=template_file, template="router.j2", path="")
    task.host["template_config"] = r.result
    task.run(task=napalm_configure, configuration=task.host["template_config"])


nr = InitNornir()
routers = nr.filter(nornir_nos="iosxr")

endor = routers.filter(site="endor")
r = endor.run(load_data)
print_result(r)

While learning about network automation chances are that you’ve crossed paths with Vagrant which is a great tool when testing and learning. It lets you create a virtual router on the fly to run tests. If you haven’t tried Vagrant, I strongly recommend that you do so. Above I relied upon the DNS entries for each host and connected to the default ports. To use the napalm_configure task against a device running in Vagrant you would have to make a small modification to the inventory. An example would look like this:

---
en-rtr-1:
    nornir_username: vagrant
    nornir_password: vagrant
    loopback0: 10.230.12.2
    nornir_host: 127.0.0.1
    nornir_network_api_port: 12202

Keeping it real

Returning to our reality, what my friend wanted to do was to lookup the IP address for each device and then query the device for information. Nornir didn’t do any of the needed tasks out of the box, so he had to build them himself. An advantage of using a python framework is that your own extensions will be more powerful, feel more natural and will be easier and more efficient to implement. The number of plugins that ships with Nornir will grow over time, but it will also be a platform for you to write creations of your own.

Your Next Adventure

While we did create a complete working Nornir application in the example above, there is still a lot more to explore. There are lots of stand-alone programs that you could create, or you could integrate Nornir into something bigger. The above example works well in a learning environment, but often what happens in the real world is more interesting. I look forward to hearing about what you create with Nornir.

More Information About the Project

David Barroso, who also created Napalm, is the initial developer behind Nornir. Currently, the team also consists of Kirk Byers author of Netmiko as well as myself Patrick Ogenstad.
The source code for Nornir is available on GitHub.  Documentation is here: https://nornir.readthedocs.io/

Share