Skip to content
Aug 2 / thebrotherswisp

The Brothers WISP 116 – Irish Politics, Finding A Job Now, Automation Story



This week we have Greg, Mike, Wilson, and Tom Smyth; our rebel brother has returned for the holidays and he’s back to his old shenanigans.

**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
Irish politics
Covid for the Irish – the pubs are still closed!
Are people equipped to work from home
Fresh out of college, how do I get an IT job right now; what will I be missing out on being remote
Learning how to troubleshoot and experience
Automated network troubleshooting with zabbix/ansible
Ansible Omnibus – Things I learned along the way
Zach Biles blog – tips on ansible as he learns
What is a compelling automation story?

Here’s the video:(if you don’t see it, hit refresh)

Jul 30 / Greg

Mikrotik Dude Display System Identity

When the dude discovers devices it will by default name them as just the IP address of the device; not always overly useful. If you have SNMP enabled on the devices you can use the OID for “system identity” to show up as the device name.

First select all of your devices, then right click on one and choose appearance:

Next paste in the following to the label:

1
2
[oid("iso.org.dod.internet.mgmt.mib-2.system.sysName.0")]
[Device.Name]

Good luck and happy monitoring.

Jul 29 / Greg

Ansible Omnibus Examples – Filtering, Loops, Conditionals, Tips


The great sales leader Tony Owens once said “Steal the wheel, don’t reinvent it.” With that in mind, here’s my github repo; feel free to grab and modify when/where you can.

I plan to revisit and update this article on a semi regular basis to keep adding new content, so keep checking back.
Also I welcome any and all feedback.

Loops

Loops give you the ability to iterate through a list which makes task completion much more efficient(or can at any rate).

In this example I use the yum module with a list, to install/update packages on a RHEL based device. Notice the variable “{{ item }}” here. By default when you define a loop, the product of that loop is referred to as item in variable form. This means at first iteration item will equal httpd, then at the next iteration it will equal dovecot.

1
2
3
4
5
6
- yum:
    name: "{{ item }}"
    state: latest
  loop:
    - httpd
    - dovecot

Have a look here at version two of the above loop. This is what you are more likely to encounter when working with other modules to gather information. The variable mail_services is a list(which equates to an array). Here I specify the loop should use mail_services and iterate through each member.

1
2
3
4
5
6
7
8
9
10
vars:
  mail_services:
    - httpd
    - dovecot
 
tasks:
  - yum:
      name: "{{ item }}"
      state: latest
    loop: "{{ mail_services }}"

Efficiency Note
Looping in modules isn’t always the most efficient way of processing information, though. The yum module allows you to specify multiple services in the form of a list like this:

1
2
3
4
5
  - yum:
      name: 
        - httpd
        - dovecot
      state: latest

The difference seems subtle, does it not? When you perform a loop it has to recall the module for each item in the array, which can take quite a while. When you just specify all of the modules in the name section(which can just be an list variable), it processes that block of code at once, making it much more efficient. Having said all of this, not every module allows you to specify multiple items like this, so sometimes a loop is the only option; be sure to check the documentation to see what options are available to you.

Looping through lists of dictionaries:
This adds a little more complication, and you may see it defined in multiple forms. Here’s two version of the same thing:

1
2
3
4
5
6
7
- user:
    name: "{{ item.name }}"
    state: present
    groups: "{{ item.groups }}"
  loop:
    - { name: 'greg', groups: 'tacos' }
    - { name: 'sowell', groups: 'robot' }
1
2
3
4
5
6
7
8
9
10
11
vars:
  sys_users:
    - name: greg
      groups: tacos
    - name: sowell
      groups: robot
- user:
    name: "{{ item.name }}"
    state: present
    groups: "{{ item.groups }}"
  loop: "{{ sys_users }}"

Looking at the above two options I actually think version two makes more sense(this is how you are most likely to encounter this information). Most often you won’t statically define this stuff, rather it will be returned from some other task.
Make note how the name and groups are referenced in the loop as “item.keyvalue”.
That’s one of the tricky bits about Ansible, there are a lot of ways to accomplish the same task 🙂

Nested Loops
This is a bit of a trick question 🙂 Loops can’t be nested, BUT what you can do is have a loop that calls a role or another task yaml file. Take for example the following task

1
2
3
4
5
6
7
8
9
10
11
12
13
---
- name: nested loop test
  hosts: localhost
  gather_facts: faluse
 
  tasks:
  - name: loop through items and include task file for further looping
    include_tasks: tasks/2nd_loop.yml
    loop:
      - Greg
      - Jimmy
    loop_control:
      loop_var: outter_names

Contents of tasks/2nd_loop.yml

1
2
3
4
5
6
7
8
9
10
---
- name: the inner loop
  debug:
    msg: "{{ outter_names }} loves {{ inner_food }}"
  loop:
    - burgers
    - pizza
    - waffle fries
  loop_control:
    loop_var: inner_food

You’ll notice some new stuff here; namely “loop_control”. This gives you multiple new abilities on loops, but here you see that I use “loop_var”. This allows me to rename how the loop is referenced in the resulting task(or called tasks). In the original playbook I name the outer loop variables outer_names and on the called task I rename them inner_food. The reason this is so important is that by default loop results are named “item”. If you are nesting loops, then both loops would attempt to use the same name of “item” and that would create a conflict. If I rename my loop variables, then everything keeps working correctly and is also much easier to read.

This will result in:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[[email protected] ansible]# ansible-playbook outerloop.yml
 
PLAY [nested loop test] ***************************************************************************************************************************************************************
 
TASK [loop through items and include task file for further looping] *******************************************************************************************************************
included: /etc/ansible/tasks/2nd_loop.yml for localhost
included: /etc/ansible/tasks/2nd_loop.yml for localhost
 
TASK [the inner loop] *****************************************************************************************************************************************************************
ok: [localhost] => (item=burgers) => {
    "msg": "Greg loves burgers"
}
ok: [localhost] => (item=pizza) => {
    "msg": "Greg loves pizza"
}
ok: [localhost] => (item=waffle fries) => {
    "msg": "Greg loves waffle fries"
}
 
TASK [the inner loop] *****************************************************************************************************************************************************************
ok: [localhost] => (item=burgers) => {
    "msg": "Jimmy loves burgers"
}
ok: [localhost] => (item=pizza) => {
    "msg": "Jimmy loves pizza"
}
ok: [localhost] => (item=waffle fries) => {
    "msg": "Jimmy loves waffle fries"
}
 
PLAY RECAP ****************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Code Block Loops
Here’s another gotcha; code blocks like this can’t be looped. The following fails:

1
2
3
4
5
6
7
8
- name: why can't we loop this?
  block:
    - name: print stuff
      debug:
        msg: "{{ item }}"
  loop:
    - too bad
    - so sad

Lookups

Lookup plugins allow access to outside data sources. This can be pulling info from files on the local system to say pulling the time from your ansible control node. Keep in mind that they run on the control node only(not on the remote host). There are SO MANY lookup plugins. I suppose I’ll add my favorites from time to time here.

Lookup plugins are usually called something like this:

1
2
vars:
  file_contents: "{{lookup('file', 'path/to/file.txt')}}"

In the above notice it is called within the mustaches “{{}}”. This tells ansible to break out into jinja2 for processing. Here the lookup plugin calls for a file, then supplies the file to pull it from.
Lookup plugins also have error handling in the form of the errors option:

1
2
- name: file doesnt exist, but i dont care .. file plugin itself warns anyways ...
  debug: msg="{{ lookup('file', '/idontexist', errors='ignore') }}"

Options are ignore, warn, or strict(strict is the default and will cause the task to fail).

By default a lookup plugin returns a comma separated list of values, which if you want to use it in a loop isn’t great. You can add the option wantlist=True or you can substitute the lookup plugin for the query plugin if you want a list returned. The query plugin returns a list by default. Here are the options side by side:

1
2
3
4
5
lookup('dict', dict_variable, wantlist=True)
 
query('dict', dict_variable)
 
q('dict', dict_variable)

q = query in the above.

Filters

Stuff here soon.

1
 

Conditionals

Stuff here soon.

1
 

Random Tips

Spanning Multiple Lines with Values
You can quickly test what it does with this website(don’t even have to do it within ansible).
The pipe keeps everything just as seen on multiple lines(it puts line breaks after each line).

1
2
3
4
5
6
7
multiple_lines: |
  this line
  will read just
  how you see
  it here.  Where
  these will all be 
  multiple lines.

The greater than will concatenate everything into a single line

1
2
3
4
5
6
7
single_line: >
  all of this
  shows up
  as a single
  line.  This
  allows you to just
  break up long lines.

Both of the above | and > have “\n” new lines at the end of them. You can add the – to the end of them to remove new lines as in >- or |-.

1
2
3
4
5
single_line_no_newline: >-
  this is all 
  one line
  with no
  new line at the end of it.

1
 
1
 
Jul 28 / Greg

Ansible Zabbix Modules – Controlling Maintenance Mode

Ansible has a lot of good Zabbix control modules found here:.
zabbix_action – Create/Delete/Update Zabbix actions
zabbix_group – Create/delete Zabbix host groups
zabbix_group_info – Gather information about Zabbix hostgroup
zabbix_host – Create/update/delete Zabbix hosts
zabbix_host_info – Gather information about Zabbix host
zabbix_hostmacro – Create/update/delete Zabbix host macros
zabbix_maintenance – Create Zabbix maintenance windows
zabbix_map – Create/update/delete Zabbix maps
zabbix_mediatype – Create/Update/Delete Zabbix media types
zabbix_proxy – Create/delete/get/update Zabbix proxies
zabbix_screen – Create/update/delete Zabbix screens
zabbix_template – Create/update/delete/dump Zabbix template

I decided that in my Cisco IOS upgrade workflow I wanted the devices being updated to be in maintenance mode to cut down on notification spam. The playbook I used is linked here, so the version you see below may be old.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
---
- name: Put Zabbix groups into maint
  hosts: all
  gather_facts: false
  vars:
    maint_desc: IOS upgrades
    maint_group: cat_update_lab
    maint_mins: 90
    zab_host: 192.168.51.5
# gen1_user/pass are custom credentials I use in tower
#    gen1_user: username
#    gen1_pword: password
 
  tasks:
  - name: put the devices undergoing work into maint
    run_once: True
    zabbix_maintenance:
      name: "{{ maint_desc }}"
      host_groups: 
        - "{{ maint_group }}"
      state: present
      minutes: "{{ maint_mins }}"
      server_url: "http://{{ zab_host }}/zabbix"
      login_user: "{{ gen1_user }}"
      login_password: "{{ gen1_pword }}"

I’ve gotten where I variablize as much of the work as I can so that I can just pass parameters in via Tower.
In this case I created a custom credential for username and password, so that is passed in at run time.
zab_host is the IP or DNS name of the user.
Also notice the server_url; the default install path is /zabbix, so if you changed that adjust it here.
I’m using host_groups, which means I’m disabling a group of devices at a time. You could also just do the individual hosts or hosts and groups if you want.

Something important to note is that the zabbix API must be installed on the zabbix server:

1
pip3 install zabbix-api

Here’s a shot of my workflow.

First I put the kit into maintenance mode.
Second I perform a backup of the devices to be upgraded.
Last I check if the devices are at the correct version and if not, perform the upgrade.

With so many options, I’m sure I’ll keep building; let me know what you envision having Ansible do?

Jul 27 / Greg

Automated Network Troubleshooting With Ansible Tower And Zabbix


In the role of a network engineer one of the most common tickets/requests we receive is “I can’t reach this thing.” Well that or “the internet is down.” It’s at this point that I ask for the destination IP, then I start the standard pings/traceroutes…but what if I could do all of this before the user ever contacts me? That’s the plan; a device goes down and the monitoring system initiates troubleshooting through Ansible.

Demo Video Of System In Action

I’ve recently been experimenting with Zabbix, which is an opensource monitoring tool(I think it’s made over in Latvia too 🙂 I’ve been looking at it more since there is a plethora of Ansible modules that can make administration of the system much easier. It’s also possible to use the Zabbix servers as your Ansible dynamic inventory(see the .py script and ini file here), which I need to do a little more testing with to get just the way I want.

Configure Zabbix

First things first; configure a Host in Zabbix to be monitored however you like. I did just a simple ICMP ping test to 8.8.8.8 following these instructions.

I then create a trigger on this same host using the ping item just created. This trigger is activated when the ping fails to 8.8.8.8:

I’ll now create an action that will be fired once the trigger is activated:

The trigger is set as the ping down trigger I specified above:

Last the operation is set to run a remote command via the zabbix server:

The script being run is this:

1
curl -X POST --user zabbix-tower:zabbix -H 'Content-Type: application/json' -d '{"extra_vars": "{\"destIP\": \"{HOST.IP}\"}"}' https://192.168.51.6/api/v2/job_templates/27/launch/ -k -s

The curl command is using user zabbix-tower with password zabbix. It is launching job 27 and passing via extra_vars the variable destIP as the zabbix monitored host IP(in this case 8.8.8.8).

The tower template is this one on my git hub. This playbook connects to a couple of my mikrotik probes and does a ping and traceroute to the supplied destIP from each probe. After it gathers that information it will then email or slack the results to you.

Configure Ansible Tower

One of the brilliant things about Tower is its ability to create custom credentials and pass that info over to playbooks. Here I’ll show how to create custom creds for gmail used in the playbook(I also have custom creds created for slack, but I’ll let you try this one on your own based on my example).
First under admin choose credential type then hit the green plus to add new:

Next fill out the credential type to meet your needs:

The “input configuration” portion is what you collect from the user. Here’s my configuration in YAML:

1
2
3
4
5
6
7
8
9
10
11
fields:
  - id: supp_email_user
    type: string
    label: Email Username
  - id: supp_email_pword
    type: string
    label: Email Password
    secret: true
required:
  - supp_email_user
  - supp_email_pword

In the above notice the following fields:
– id: This is what the variable will be temporarily stored as during collection.
– type: This defines what the input type will be. For username/pass/tokens string is perfect.
– label: This is what is the prompt the user sees when entering information.
– secret: When set to true, this tells Tower to encrypt this value and also hide it in log output. Good for passwords or tokens.
The “required” section indicates these aren’t optional fields.

The “injector configuration” section is how you send this information over to your playbooks:

1
2
3
extra_vars:
  email_pword: '{{ supp_email_pword }}'
  email_user: '{{ supp_email_user }}'

In my case I’ll be sending them as extra_vars. In my playbook the variables “email_pword” and “email_user” are used to set email creds, which is why that’s what is being set via the supplied information.

Now that I’ve got the custom credential type created, I’ll add the custom cred:

I’m next going to tell Tower to use my git repo:

I’ll then create a job template for my troubleshooting probe playbook:

Notice that in extra_vars I supplied it with method of slack. This tells it to use slack rather than email. Also keep in mind that I told extra_vars to “prompt on launch”. This is required for the API call to pass in additional extra_vars.
Also note that I added my credentials for both my slack token as well as my email addresses. These are securely passed to the playbook at runtime!

What the sequence is up to this point:
1. Zabbix is setup to ping 8.8.8.8.
2. When the ping item returns 0(failed state), the trigger is activated.
3. An action is fired when the trigger is activated.
4. The action calls the Ansible API to run a job passing the IP being monitored.
5. Tower takes the IP supplied and pings/traceroutes to that destination from multiple probes.
6. Tower will then take this aggregated information and send it to you either via slack or email.

Slack Output

Once the tshooting completes, this is what message is supplied to slack:

I’ve now updated the playbook so that it creates an incident in ServiceNow and adds the tshooting as comments:

The idea behind all this is that your monitoring system isn’t just telling you there’s an issue, it’s actively performing troubleshooting for you. It can do more than just that; imagine it also takes this information and opens an ticket. Imagine that it also notifies the customer that there’s an issue identified and that you are actively working on it.

By the time the issue notification arrives, there will also be some advanced troubleshooting already performed for you and waiting.

Let me know how you envision using something like this!

Jul 26 / thebrotherswisp

Greg Talks 15 – Technical Sales, Vulnerability In Selling, Selling Techniques

Greg talks to Tony Owens a lifelong technical sales leader.

This week we talk about:
Technical sales
Consultitave selling
Learning to be a manager
Learning to be a leader
Age = respect
Pattern interrupt
Showing vulnerability in your job
People will rescue you, they want to save you
Upfront contracts
3 seconds of guts
Sandler training – seemed like you really liked it.
Sports and your relationship with them.
Sports is the only real thing on TV
Why do you drink so much water?

Join the patron only slack at http://patreon.com/thebrotherswisp

Here’s the video:(if you don’t see it, hit refresh)

Jul 19 / thebrotherswisp

The Brothers WISP 115 – WISP After Death, Unifi Video EOL, RPKI Basics



This week we have Greg, Mike Dave, and Wilson…all the familiar faces coming back!
**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
WISP Virtual Summit July 28th
Save Dave’s brain
RIP Ubiquiti Unifi Video – EOL 1/1/2021
zwift.com
Cloudflare DNS outage
David – Arduino, PHP programming, cycling and weight loss, new kid,
Wilson’s RPKI
Mike got some new hardware; a stent!
I’m done with my sales training – I’m a real boy.
WLED esp8266 library

Here’s the video:(if you don’t see it, hit refresh)