{"id":7380,"date":"2023-01-10T16:16:31","date_gmt":"2023-01-10T22:16:31","guid":{"rendered":"https:\/\/gregsowell.com\/?p=7380"},"modified":"2023-01-11T11:26:26","modified_gmt":"2023-01-11T17:26:26","slug":"ansible-looping-with-lists-and-dictionaries-with-advanced-filtering","status":"publish","type":"post","link":"https:\/\/gregsowell.com\/?p=7380","title":{"rendered":"Ansible Looping With Lists And Dictionaries With Advanced Filtering"},"content":{"rendered":"

\"\"<\/a>
\nOne of the real strengths of Ansible is it’s ability to loop through information. This can be done to generate reports or configuring network or server kit.
\nThe aim of this post is teach you how to loop over various data types be them lists(which are easy) or dictionaries(a little more challenging). Another important piece here is the fact that often I need to do some advanced filtering. This could be done with complex nested loops, OR I can do it with some special plugins(which will be demonstrated).<\/p>\n

Sample Files<\/h2>\n

My public git repository with the files can be found here<\/a>.
\nIf you are in a workshop you can issue the following command to clone the repository to your workstation in your current working directory:<\/p>\n

git clone https:\/\/github.com\/gregsowell\/ansible-looping-lesson.git<\/code><\/pre>\n
Looping Over Lists<\/h2>\n
The file I’m pulling from is loop-test.yml.  I’ll break down the various parts below.
\nThe play section and variables I’m working with are as follows:<\/p>\n
---\r\n- name: loop examples\r\n  hosts: localhost\r\n  gather_facts: false\r\n  vars:\r\n    list1:\r\n      - thing1\r\n      - thing2\r\n      - thing3\r\n\r\n    list2:\r\n      - fname: Greg\r\n        lname: Sowell\r\n      - fname: Andy\r\n        lname: Garrett\r\n\r\n    list3:\r\n      - 1:\r\n        - n: 1\r\n        - m: 1\r\n      - 2:\r\n        - n: 2\r\n        - m: 2<\/code><\/pre>\n
Looking at the above, “list1” variable is a standard list.  It has just values as the items inside the list.
\n“list2” is actually a list of dictionaries.  List items begin with a “-” character.  Each list item has two variables, both “fname” and “lname”.
\n“list3 is a list of lists.
\nYou will often see variables returned in ansible in these forms.  I would say I see combinations of list2 and list3 most often…probably list2 the most often.<\/p>\n
Here are two tasks that will display list1 in different ways:<\/p>\n
  - name: debug list1\r\n    ansible.builtin.debug:\r\n      var: list1\r\n\r\n  - name: loop through standard list\r\n    ansible.builtin.debug:\r\n      msg: "{{ item }}"\r\n    loop: "{{ list1 }}"<\/code><\/pre>\n
Here’s the returned format once they are run:<\/p>\n
TASK [debug list1] *************************************************************\r\nok: [localhost] => {\r\n    "list1": [\r\n        "thing1",\r\n        "thing2",\r\n        "thing3"\r\n    ]\r\n}\r\n\r\nTASK [loop through standard list] **********************************************\r\nok: [localhost] => (item=thing1) => {\r\n    "msg": "thing1"\r\n}\r\nok: [localhost] => (item=thing2) => {\r\n    "msg": "thing2"\r\n}\r\nok: [localhost] => (item=thing3) => {\r\n    "msg": "thing3"\r\n}<\/code><\/pre>\n
The initial debug isn’t that complex; all I did was request that variable 1 be dumped out to screen.
\nThe second debug where I “loop through standard list” is slightly more interesting.  I added a “loop” option with the “list1 variable.  Notice the format:<\/p>\n
loop: "{{ list1 }}"<\/code><\/pre>\n
Have a look at how the word loop is on the same column level as the module itself.  This says that the loop option is applied to the task itself and is not a sub parameter of the module.
\nAlso notice, in the line below, that the result of the loop is contained in a temporary variable named “item”.  This is the default behavior, though the returned iteration name “item” can be changed to something else if you want to.<\/p>\n
msg: "{{ item }}"<\/code><\/pre>\n
Picking specific items inside the array<\/strong>
\nA list is like an array.  You can pick specific values like so:<\/p>\n
  - name: pulling items from list 1\r\n    ansible.builtin.debug:\r\n      msg: "{{ list1[1] }}"<\/code><\/pre>\n
This returns the following results:<\/p>\n
TASK [pulling items from list 1] ***********************************************\r\nok: [localhost] => {\r\n    "msg": "thing2"\r\n}<\/code><\/pre>\n
This list contained 3 items and I specified 1, so why did it return thing2 and not thing1?  That’s because ansible, like most programming languages, numbers its arrays starting at 0.  So for this list it goes 0, then 1, and last 2.<\/p>\n
Loop through lists of dictionaries<\/strong>
\nLooping through lists of dictionaries is virtually the same as before, but with more options.  See the following example:<\/p>\n
  - name: loop through list of dictionaries\r\n    ansible.builtin.debug:\r\n      msg: "First name is:{{ item.fname }} last name is:{{ item.lname }}"\r\n    loop: "{{ list2 }}"<\/code><\/pre>\n
Notice the loop section is the same, and it returns iterations as “item” just the same, but look at the variables in the “msg” option…not quite the same.
\nThe contents of item for each iteration will be the dictionary key\/value pairs.  This allows me to reference the contents of the variable, so in each list item I have a fname and a lname.  So to reference each I say item.fname or item.lname.<\/p>\n
Here are the results of the run:<\/p>\n
TASK [loop through list of dictionaries] ***************************************\r\nok: [localhost] => (item={'fname': 'Greg', 'lname': 'Sowell'}) => {\r\n    "msg": "First name is:Greg last name is:Sowell"\r\n}\r\nok: [localhost] => (item={'fname': 'Andy', 'lname': 'Garrett'}) => {\r\n    "msg": "First name is:Andy last name is:Garrett"\r\n}<\/code><\/pre>\n
Looping Over Dictionaries<\/h2>\n
This gets a little harrier.  I say that because ansible won’t natively loop over a dictionary.  The trick is, I need to convert the dictionary into a list.  I’m specifically going to be focusing on the “dict2items” plugin.  This will take that dictionary and create key\/value pairs with it.  Often the easiest way to understand it is an example.<\/p>\n
Task:<\/p>\n
  - name: debug dict3 with dict2items\r\n    ansible.builtin.debug:\r\n      var: dict3 | dict2items<\/code><\/pre>\n
Dictionary variable I am manipulating:<\/p>\n
    dict3:\r\n      1:\r\n        name: New Greg\r\n        color: Bright\r\n      2:\r\n        name: Andy\r\n        color: Green\r\n      3:\r\n        name: Andrew\r\n        color: Baby blue<\/code><\/pre>\n
Now the results:<\/p>\n
TASK [debug dict3 with dict2items] *********************************************\r\nok: [localhost] => {\r\n    "dict3 | dict2items": [\r\n        {\r\n            "key": 1,\r\n            "value": {\r\n                "color": "Bright",\r\n                "name": "New Greg"\r\n            }\r\n        },\r\n        {\r\n            "key": 2,\r\n            "value": {\r\n                "color": "Green",\r\n                "name": "Andy"\r\n            }\r\n        },\r\n        {\r\n            "key": 3,\r\n            "value": {\r\n                "color": "Baby blue",\r\n                "name": "Andrew"\r\n            }\r\n        }\r\n    ]\r\n}<\/code><\/pre>\n
First, notice in the original dictionary variable, there are no “-“s.  That shows you that it is, in fact, not a list.  Remember, a list has dashes in standard variable form.
\nSecond, take a look at the output…there are no “-” in there, so how is it a list?  In json output a list is surrounded by square brackets like so “[]”.  If you notice at the very beginning and very end of the output you can see the square brackets!  So, dashes in defined variables and square brackets in output means list.
\nNow for the dict2items output.  I see that it breaks the original dictionary into two sections, first a key and then a value.
\nI can see that the left most part of the dictionary column becomes the “key”.  So in this example it’s 1, 2, and 3.
\nNext, the indented information to the right of the key gets put into the “value” section.  <\/p>\n
I can actually access the individual pieces under the value section…let me show you an example:<\/p>\n
  - name: loop through dictionary with many entries\r\n    ansible.builtin.debug:\r\n      msg: "What is the key:{{ item.key }}. What is the name:{{ item.value.name }}. => {{ item.value.color }}"\r\n    loop: "{{ dict3 | dict2items }}"<\/code><\/pre>\n
Notice in the above that inside the loop section I piped the dictionary over to the dict2items plugin right there; I don’t have to do this ahead of time, rather I can do it right when I need it.
\nAlso notice that in the returned info I referenced the key directly with “item.key” and then referenced the value information directly by following the variable chain with “item.value.name”.
\nImagine that any time I’m moving a column to the right to access another variable name, just add a “.” between the variable names.  So looking above where I debugged the dict2items I can pretty easily see myself moving to the right and just add periods.<\/p>\n
Results:<\/p>\n
TASK [loop through dictionary with many entries] *******************************\r\nok: [localhost] => (item={'key': 1, 'value': {'name': 'New Greg', 'color': 'Bright'}}) => {\r\n    "msg": "What is the key:1. What is the name:New Greg. => Bright"\r\n}\r\nok: [localhost] => (item={'key': 2, 'value': {'name': 'Andy', 'color': 'Green'}}) => {\r\n    "msg": "What is the key:2. What is the name:Andy. => Green"\r\n}\r\nok: [localhost] => (item={'key': 3, 'value': {'name': 'Andrew', 'color': 'Baby blue'}}) => {\r\n    "msg": "What is the key:3. What is the name:Andrew. => Baby blue"\r\n}<\/code><\/pre>\n
Advanced Filtering with selectattr<\/h2>\n
Additional resources here<\/a> and here<\/a>.
\nThere are MANY ways to do additional filtering, but what I’m going to focus on is using the “selectattr” plugin.  This plugin takes a list of dictionaries and applies a “test” to each iteration in the list.  <\/p>\n
It’s utilization is something like the following:<\/p>\n
selectattr('key','search','^Ethernet.*')<\/code><\/pre>\n
So selectattr(field to be evaluated, jinja2 test, optional value to add to the test)<\/strong>.
\nThere are several options for the test(some examples follow, but there are more): search, match(search and match are pretty much the same), eq, ==(eq and == are the same), >, <, >=, <=, !=, defined.\n\nLet's see a few examples based on some of this variable:\n\n\n
    dict3:\r\n      1:\r\n        name: New Greg\r\n        color: Bright\r\n      2:\r\n        name: Andy\r\n        color: Green\r\n      3:\r\n        name: Andrew\r\n        color: Baby blue<\/code><\/pre>\n
Equals or not equals<\/strong>
\nTasks:<\/p>\n
  - name: loop through dict3 selectattr == 1\r\n    ansible.builtin.debug:\r\n      msg: "{{ item }}"\r\n    loop: "{{ dict3 | dict2items | selectattr('key','==',1) }}"\r\n\r\n  - name: loop through dict3 selectattr != 1\r\n    ansible.builtin.debug:\r\n      msg: "{{ item }}"\r\n    loop: "{{ dict3 | dict2items | selectattr('key','!=',1) }}"<\/code><\/pre>\n
Notice here I’m specifying that they key field either matches 1 in the first task or doesn’t match 1 in the second task.<\/p>\n
Results:<\/p>\n
TASK [loop through dict3 selectattr == 1] **************************************\r\nok: [localhost] => (item={'key': 1, 'value': {'name': 'New Greg', 'color': 'Bright'}}) => {\r\n    "msg": {\r\n        "key": 1,\r\n        "value": {\r\n            "color": "Bright",\r\n            "name": "New Greg"\r\n        }\r\n    }\r\n}\r\n\r\nTASK [loop through dict3 selectattr != 1] **************************************\r\nok: [localhost] => (item={'key': 2, 'value': {'name': 'Andy', 'color': 'Green'}}) => {\r\n    "msg": {\r\n        "key": 2,\r\n        "value": {\r\n            "color": "Green",\r\n            "name": "Andy"\r\n        }\r\n    }\r\n}\r\nok: [localhost] => (item={'key': 3, 'value': {'name': 'Andrew', 'color': 'Baby blue'}}) => {\r\n    "msg": {\r\n        "key": 3,\r\n        "value": {\r\n            "color": "Baby blue",\r\n            "name": "Andrew"\r\n        }\r\n    }\r\n}<\/code><\/pre>\n
Regex searching<\/strong>
\nTask:<\/p>\n
  - name: loop through dict3 selectattr search for A in value.name\r\n    ansible.builtin.debug:\r\n      msg: "{{ item }}"\r\n    loop: "{{ dict3 | dict2items | selectattr('value.name','search','^A.*') }}"<\/code><\/pre>\n
This one is pretty darn handy.  It allows you to do a regex search on any field, so it will narrow down what results are returned based on what you match.<\/p>\n
Results:<\/p>\n
TASK [loop through dict3 selectattr search for A in value.name] ****************\r\nok: [localhost] => (item={'key': 2, 'value': {'name': 'Andy', 'color': 'Green'}}) => {\r\n    "msg": {\r\n        "key": 2,\r\n        "value": {\r\n            "color": "Green",\r\n            "name": "Andy"\r\n        }\r\n    }\r\n}\r\nok: [localhost] => (item={'key': 3, 'value': {'name': 'Andrew', 'color': 'Baby blue'}}) => {\r\n    "msg": {\r\n        "key": 3,\r\n        "value": {\r\n            "color": "Baby blue",\r\n            "name": "Andrew"\r\n        }\r\n    }\r\n}<\/code><\/pre>\n
Compound tests<\/strong>
\nYes, as with any filter plugin you can stack them to check for multiple conditions.  I’ll take the example from before and build on it.<\/p>\n
Task:<\/p>\n
  - name: loop through dict3 selectattr search for A in value.name and like green\r\n    ansible.builtin.debug:\r\n      msg: "{{ item }}"\r\n    loop: "{{ dict3 | dict2items | selectattr('value.name','search','^A.*') | selectattr('value.color','eq','Green') }}"<\/code><\/pre>\n
It is as simple as adding another pipe and then the next test.  Notice too that I checked different variables in each test.<\/p>\n
Results:<\/p>\n
TASK [loop through dict3 selectattr search for A in value.name and like green] ***\r\nok: [localhost] => (item={'key': 2, 'value': {'name': 'Andy', 'color': 'Green'}}) => {\r\n    "msg": {\r\n        "key": 2,\r\n        "value": {\r\n            "color": "Green",\r\n            "name": "Andy"\r\n        }\r\n    }\r\n}<\/code><\/pre>\n
Make sure the variable is defined<\/strong>
\nThe selectattr plugin will fail if one of the entries happens to not have the field to be checked not exist, so I can first check defined, then build on that as follows.
\nTask:<\/p>\n
  - name: loop through dict3 selectattr search for A in value.name and check that it is defined\r\n    ansible.builtin.debug:\r\n      msg: "{{ item }}"\r\n    loop: "{{ dict3 | dict2items | selectattr('value.name','defined') | selectattr('value.name','search','^A.*') }}"<\/code><\/pre>\n
So in this example, it will first check that value.name exists for this host, and if it does, then it will do a regex search on it.  You don’t necessarily need to do this for every variable you plan to check, but if there’s a chance it may not be instantiated, then add it; it won’t hurt to have it and not need it.<\/p>\n
Use jinja2 loop to build a variable<\/strong>
\nHere I’m breaking into a jinja2 loop in conjunction with selectattr to build a custom variable:<\/p>\n
  - name: create variable in a jinja2 loop using selectattr\r\n    ansible.builtin.set_fact:\r\n      new_var: "{% for result in dict3 | dict2items | selectattr('value.name','defined') | selectattr('value.name','search','^A.*') %}\\\r\n                {{ result.key }}: {{ result.value.name }}_{{ result.value.color }},\r\n                {% endfor %}"\r\n\r\n  - name: print out new_var\r\n    ansible.builtin.debug:\r\n      var: new_var.split(',')<\/code><\/pre>\n
Notice that in the debug statement I’m using the split option to break the string into a list separated via a comma.<\/p>\n
Results:<\/p>\n
TASK [create variable in a jinja2 loop using selectattr] ***********************\r\nok: [localhost]\r\n\r\nTASK [print out new_var] *******************************************************\r\nok: [localhost] => {\r\n    "new_var.split(',')": [\r\n        "2: Andy_Green",\r\n        " 3: Andrew_Baby blue",\r\n        " "\r\n    ]\r\n}<\/code><\/pre>\n
Advanced Filtering with map<\/h2>\n
Additional resources on map here<\/a> and here<\/a>.
\nThe map plugin allows me to take a list of dictionaries and make a simpler list from it.  Let’s jump into some examples.
\nTasks:<\/p>\n
  - name: loop through dict3 using map to create a list of names\r\n    ansible.builtin.debug:\r\n      msg: "{{ item }}"\r\n    loop: "{{ dict3 | dict2items | map(attribute='value.name') }}"<\/code><\/pre>\n
Results:<\/p>\n
TASK [loop through dict3 using map to create a list of names] ******************\r\nok: [localhost] => (item=New Greg) => {\r\n    "msg": "New Greg"\r\n}\r\nok: [localhost] => (item=Andy) => {\r\n    "msg": "Andy"\r\n}\r\nok: [localhost] => (item=Andrew) => {\r\n    "msg": "Andrew"\r\n}<\/code><\/pre>\n
So as you can see it simply returns a list of values, not key and value.<\/p>\n
Say, for example, I wanted to return it all on a single line instead of a list I can do something like the following:<\/p>\n
  - name: loop through dict3 using map to create a list of names all on one line\r\n    ansible.builtin.debug:\r\n      msg: "{{ dict3 | dict2items | map(attribute='value.name') | join(',') }}"<\/code><\/pre>\n
I’m simply using the join plugin and instructing it to use a comma as the joining character.<\/p>\n
Results:<\/p>\n
TASK [loop through dict3 using map to create a list of names all on one line] ***\r\nok: [localhost] => {\r\n    "msg": "New Greg,Andy,Andrew"\r\n}<\/code><\/pre>\n
Conclusion<\/h2>\n
This is a living document, so I plan to make updates periodically.  I hope this gives you a starting point to build some awesome automations.  If you have any questions or comments, please let me know.<\/p>\n
Thanks and happy automated looping!<\/p>\n","protected":false},"excerpt":{"rendered":"
One of the real strengths of Ansible is it’s ability to loop through information. This can be done to generate reports or configuring network\u2026<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[49],"tags":[],"class_list":["post-7380","post","type-post","status-publish","format-standard","hentry","category-ansible"],"_links":{"self":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts\/7380","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7380"}],"version-history":[{"count":32,"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts\/7380\/revisions"}],"predecessor-version":[{"id":7413,"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts\/7380\/revisions\/7413"}],"wp:attachment":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7380"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7380"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7380"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}