Skip to content
Sep 14 / Greg

Self Service Troubleshooting Using ServiceNow and Ansible Tower


Holy cow did I learn a lot on this one. It took me the better part of two weeks to put it all together/learn everything required. This was actually my first go at automating Windows and I learned a LOT.

The idea behind this setup is that if a user can’t access a TCP based service or can’t browse to a web page, they can pop into ServiceNow (which I’ll call SNOW moving forward) and create a service order for it, which will then call Tower to perform some automated troubleshooting.

Demo Video

Windows Machines

First things first, all windows devices that we will be connecting to need WinRM enabled. This is the Windows Remote Management system that allows me to connect in and run commands. The easiest way to go about it is to create a group policy and have the windows machines automatically enable it.

In my example I’m performing something known as double-hopping.

What I’m doing is having Tower reach out to Server A only. I will do testing from Server A, but I’ll also tell Server A to connect to Client 1 and perform some tests. This hop from Server A to Client 1 is considered a “double hop”. Windows prevents this behavior by default for security reasions, but there are some work arounds as seen here. The work around I used is CredSSP. This is easy to use by setting this host variable for these machines:

1
ansible_winrm_transport=credssp

This is what the host variables look like for each of these first hop devices I’ll be controlling:

1
2
3
4
5
6
---
ansible_winrm_transport: credssp
ansible_host: 10.1.1.10
ansible_connection: winrm 
ansible_winrm_scheme: http
ansible_port: 5985

Notice also that in the above I’m specifying the scheme as http and the port as 5985. HTTP indicates I’m not performing encryption, which you would want to do in production(this is just a lab setup). When performing the connection unencrypted I also need to specify the port as 5985(when encrypted the port would be 5986 and is the default).

If you run into any issues when using WinRM with Ansible to connect to your clients check this quick guide.

I wanted it to be easy for someone to find their computer name, and a simple solution is to add a toolbar that has their machine name(either bginfo or what I did here:

1
2
3
4
Right-click on Taskbar -> 
Go to Toolbars -> 
Choose New Toolbar, 
type in \\%computername%,  and Click Select Folder.

ServiceNow Configuration

I won’t cover the full setup since Michael Ford has already done that for us. That link is a walk through that will get you to the point where SNOW makes API calls to Tower to fire off job templates(while passing over variables).

Since I’m standing on the shoulders of giants, I’ll skip right to my service catalog item:

As you can see I kept it simple. I am by no means a SNOW developer, which you will quickly see once I jump into the processing section LOL.
I’m gathering the destination, so https://gregsowell.com or if it’s some random TCP service it could just be gregsowell.com or 1.2.3.4.
I’m also getting their machine name from their taskbar.
Last if it is a standard TCP service they put the port number there, otherwise it stays as 0.

In Tower I did have to pull some entries from the SNOW request table, but not knowing anything about the structure of the tables…or even what the tables were I was a little lost. I ended up finding the SNOW “REST API Explorer”. This gives you the ability to find the tables, and explore their structure along with creating cURL API calls based on what you build.
Once in the rest explorer you can choose table name, add additional query parameters(like ordering by newest entry), and picking individual fields you are interested.

I like the output to come back as json, that way I can convert it to yaml with some ansible filters.

Once you click send it gives you your desired output right in the browser;

You can also click the output method of choice in “Code Samples” for some copy/paste content:

Tower Configuration

I’ll start with the playbook that will ultimately call my processing role:

I setup my basic variables that are passed from SNOW:

1
2
3
    tcp_port: "{{ variable_3 }}"
    test_url: "{{ variable_2 }}"
    test_client: "{{ variable_1 }}"

tcp_port is always zero for web testing.
test_url is essentially the destination we are trying to reach(for either webpage or TCP based service.
test_client is the hostname of the user’s PC.

I include a single task, which is just calling the role detailed below.

Here is the role I’ve created to perform all of the processing.
I’ll look at the main files required:
Remember when I mentioned I’m not a SNOW dev…keep that in mind.

In the above at the top you see I make a restful call to SNOW and request the last created request from the request table(I pull the req ID and the username). I’m banking on the fact that no other requests came in in the seconds it took for this to fire(not what you would do in production). Obviously I’d build this into my workflow in an actual production environment.
I then call either the web tshooting or TCP tshooting task files depending on what is required.
Once that processing is complete I open a SNOW incident as the requesting user and add my tshooting info and assign it to the correct group.

I use several jinja2 templates that have powershell code to extract IP info as well as to perform the TCP and web testing. Feel free to browse away.
For example, here’s the curltest powershell template.

As you can see it’s simply issuing the curl test and returning status code as well as raw content length(size of content returned by query).

The fun starts to happen in the web tshooting task file.

I’m using the jinja templates to first resolve the client name.
Then I use one of the template files to double hop to the client machine and perform a powershell version of cURL to test pulling the web page. After the client is tested, I test from the local domain controller, then I connect to an external canary server and perform the same test.

If any of my tests fail on the inside I then connect to the local Cisco ASA firewall and perform a packet tracer test. Packet tracer in an ASA will take the source/destination info and generate a virtual packet. It will then pass this virtual packet through the firewall noting each step what happens to the packet and whether it is allowed through or blocked at various steps. In the end it will supply me with an allowed or dropped.
As a side note, here’s what my ASAv hosts file entry looked like while testing(prior to entry into Tower):

1
asav ansible_host=10.1.12.15 ansible_connection=network_cli  ansible_network_os=asa ansible_user=admin ansible_password=lab ansible_become=yes ansible_become_method=enable ansible_become_pass="lab"

At the bottom of this task file I then essentially go through various condition checks and when one is met I set what the incident message will be along with which support group it will be assigned to.

Decision Tree Example

1. If all tests pass(internals and canary both get a status 200 and the website retrieved were within 10 bytes of each other), then the user is instructed to please test again as it looks like service may have been restored. This was likely just a quick blip.
2. If the client fails, but everything else passes(including the firewall rules saying the client should have access), then it is assigned to the Service Desk team as it’s likely just a local PC issue.
3. If the client and firewall fails, then sounds like we have a network issue(needs firewall mod), so assign it to the Network team.
And so on.

The TCP test file is much the same, but it uses a slightly different method to test service(all still using powershell, though).

Wrap Up

Conceptually nothing is too crazy here, but all of the individual pieces can take some time, especially when they are new to you, as much of it was for me here. I hope this post has you thinking about various time sinks within your current workflow that could possible be automated with just a little creativity. What do most of your trouble tickets deal with, and how can you automate their troubleshooting? Let me know how you would implement something like this in your environment.

Thanks and happy automating!

Sep 13 / thebrotherswisp

The Brothers WISP 119 – $1800 PTZ, 100% SLAs, Centurylink Split



This week we have Greg, Mike, and Tommy C. keeping you in the know.

**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
Nick A found a $1,800 PTZ camera in the beta store – 22x optical zoom full PTZ
Mikrotik Net Power Lite 7R – now with 100% more video
Mikrotik HA router scripts(heartbeat link with config sync and peer failure detection) – thanks for the link Landon
100% uptime to customers
Moving DCs…time to register for my ORG/ASN/IP
More Zhone complaining
In automation config pushing is so boring…what’s interesting?
Self Service troubleshooting.
Toilet potatos
Tommy’s failure story
Is Level 3 Down?

Here’s the video:(if you don’t see it, hit refresh)

Aug 30 / thebrotherswisp

The Brothers WISP 118 – Mikrotik Updates, Your Looking Glass, Our Biggest Failures



This week we have Greg, Mike, and Sean Scarfo talk about our greatest failures(PS, Greg has many more LOL).

**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
Mikrotik CRS3xx initial Bridge Port Extender support added – 802.1BR
Mikrotik CCR2004s – a few reports of lockups
Wireguard release in beta
More L3 HW offload
NLNOG Ring – shared shells to test from
Pretty looking glass server – hyper glass
FB WISP Talk Poll: Unexpected results – pole1pole2
Our failures: sharing stories about our biggest oopsies.

Here’s the video:(if you don’t see it, hit refresh)

Aug 22 / Greg

Authenticating To Ansible Tower Via Windows Active Directory


The Ansible automation platform has many awesome features, and one in particular is its ability to authenticate off of something other than itself. A common one I’m running into with customers is having Tower authenticate off of their Active Directory, so that’s what I’m going to tackle here. This assumes you have a functioning AD environment(though turning up a domain controller is still a very simple process(I haven’t stood up a DC since windows 2000 LOL)). Here’s Ansible’s documentation on it.

Demo Video

Active Directory Configuration

My AD server is windows server 2016 essentials and all of my screen shots are done from my laptop using Active Directory Explorer.

My AD domain is DEMO.local:

I stuck everything in the Users CN:

Notice in the above that I have two users:
– test1
– test2
I also have 4 groups defined:
– tower_admins
– tower_auditors
– tower_users
– tower_team_network

The groups will be mapped to various things in tower, which I’ll cover in just a bit.

Tower Configuration

So navigating to system -> authentication will present me with my external authentication options. From here I’m going to choose LDAP since AD is really just LDAP under the hood.

I’ve highlighted…highlit…multiple things here.
For LDAP server I’m doing an unencrypted connection so I just use ldap:x.x.x.x:389; if it was to be secure I’d specify ldaps:x.x.x.x:636.
For LDAP BIND DN I’ve specified a user that has access to access the various users and groups(this is the account Tower uses to authenticate with the LDAP server).
I use MemberDNGroupType for my LDAP group type, though there is an AD option.
The LDAP Require Group is an optional field. If this option is set, then only AD users that are members of this group are allowed to connect. This prevents random AD users from connecting.
If you don’t specify anything here, then any AD user can connect to the system, but they won’t have access to do anything(other than create a credential) by default. In my case I use the tower_users group.

Here’s my next set of options:

LDAP User Search is where in LDAP the system will look for users. I have them all in the users group as specified here, but I also use scope_subtree, which tells Tower to look here, but also look in other groups under this. I also specify the sAMAccountName object to be the username as shown in the following screenshot:

1
2
3
4
5
[
 "CN=Users,DC=DEMO,DC=local",
 "SCOPE_SUBTREE",
 "(sAMAccountName=%(user)s)"
]

Next will be the LDAP Group Search option, which tells tower where to look in the LDAP tree for the configured groups used for users.

1
2
3
4
5
[
 "CN=Users,DC=DEMO,DC=local",
 "SCOPE_SUBTREE",
 "(objectClass=group)"
]

Then we hit the LDAP User Attribute Map, which maps LDAP options to tower options.

1
2
3
4
5
{
 "first_name": "givenName",
 "last_name": "sn",
 "email": "mail"
}

The LDAP Group Type Parameters do a little more mapping.

1
2
3
4
{
 "member_attr": "member",
 "name_attr": "cn"
}


LDAP User Flags By Group is where I map my singleton roles roles for admins and auditors. The AD tower_admins group and tower_auditors group get mapped to system admins/auditors in tower.

1
2
3
4
5
6
7
8
{
 "is_superuser": [
  "CN=tower_admins,CN=Users,DC=DEMO,DC=local"
 ],
 "is_system_auditor": [
  "CN=tower_auditors,CN=Users,DC=DEMO,DC=local"
 ]
}

LDAP Organization Map maps groups to various levels as shown below. tower_admins become admins and every other group are users.

1
2
3
4
5
6
7
8
9
10
11
{
 "Default": {
  "remove_admins": true,
  "admins": "CN=tower_admins,CN=Users,DC=DEMO,DC=local",
  "users": [
   "CN=_tower_auditors,CN=Users,DC=DEMO,DC=local",
   "CN=tower_users,CN=Users,DC=DEMO,DC=local",
   "CN=tower_team_network,CN=Users,DC=DEMO,DC=local"
  ]
 }
}

Last is the LDAP Team Map. This is where various AD groups are mapped to teams within Ansible Tower. If the team doesn’t exist, then the first time a user authenticates, the teams will be created. After the groups exist it’s an admin’s job to go and add permissions for the various groups.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
 "LDAP Admins": {
  "remove": true,
  "organization": "Default",
  "users": "CN=tower_admins,CN=Users,DC=DEMO,DC=local"
 },
 "LDAP Auditors": {
  "remove": true,
  "organization": "Default",
  "users": "CN=tower_auditors,CN=Users,DC=DEMO,DC=local"
 },
 "LDAP Users": {
  "remove": true,
  "organization": "Default",
  "users": "CN=tower_users,CN=Users,DC=DEMO,DC=local"
 },
 "LDAP Network": {
  "remove": true,
  "organization": "Default",
  "users": "CN=tower_team_network,CN=Users,DC=DEMO,DC=local"
 }
}

So any AD user that is a member of the tower_team_network group is put into Tower’s “LDAP Network” team.

LDAP Troubleshooting On Tower

When things inevitably go wrong, here’s some troubleshooting options.
In tower, create this file(/etc/tower/conf.d/ldap.py) with the following option:

1
LOGGING['handlers']['tower_warnings']['level'] =  'DEBUG'

After that, restart the Tower services

1
ansible-tower-service restart

You can now do a tail on the /var/log/tower/tower.log log file to see LDAP specific messages.

You also want to use the LDAP search command(you will have to download the tool) to do some testing with the following command:

1
ldapsearch -x  -H ldap://192.168.51.6 -D "CN=Greg Sowell,CN=Users,DC=DEMO,DC=local" -b "dc=DEMO,dc=local" -w MyPassword

Big thanks to Jimmy for helping me suss the last bug out of my config. Turns out if any group that Tower is searching LDAP for isn’t found, then it will just fail with a very nondescript error(so double check all of your spellings).

Hope this helps and happy authenticating.

Aug 17 / Greg

Formatting Slack Messages Sent By Ansible

I’ve been sending some messages via ansible playbooks to slack and I ran into issues when I tried sending clickable links. Luckily the ansible slack module has the options you need.

Here’s an example of sending a standard message:

1
2
3
4
5
6
7
8
9
    - name: send message through slack
      slack:
        token: "{{ slack_token }}"
        msg: |
          "http://GregSowell.com"
          "This is a great site"
        channel: "{{ slack_channel }}"
        parse: 'none'
      delegate_to: localhost

Unfortunately the URL above just shows up as standard text and isn’t clickable.

I can instead use the attachments module to make it clickable like so:

1
2
3
4
5
6
7
8
9
10
    - name: send message through slack
      slack:
        token: "{{ slack_token }}"
        attachments:
          - text: This is a great site
            title: "Go to GregSowell.com"
            title_link: "http://GregSowell.com"
        channel: "{{ slack_channel }}"
        parse: 'none'
      delegate_to: localhost

Now slack creates an attachment post that has the text along with the title being a clickable link.
Slack has an API message reference here that helps build the above.

Aug 16 / thebrotherswisp

The Brothers WISP 117 – Mike vs Zhone, VoIP APIs, Netonix and 4011 Bugs



This week we have Greg and Mike; just like old times 🙂

**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
Mike’s Zhone Issues
College hacking contest – how fun would that be.
Using public resovlers 8.8.8.8, 4.2.2.2, 1.1.1.1. Should you have some language in your contracts?
VoIP system API – 3cx doesn’t have a native, but there are 3rd parties.
Yealink phones have a rest API
Yealink CAs
Cisco Devnet reserved labs have direct access only via anyconnect client *but why*
Do any of you use Cisco ACI?
Netonix 1.5.5 bug where you have to reapply over the top with the same firmware
The random 4011 lockups continue…*sigh*
Mikrotik V7 routing documentation…a bit more anyway

Here’s the video:(if you don’t see it, hit refresh)

Aug 13 / Greg

Backdoor Access To Cisco Devnet Reserved Sandboxes – Windows Portproxy

I recently needed to do a demo of my Cisco ACI Ansible stuff, and unfortunately the always on service was down for maint…NBD I’ll reserve one. “Uhhh you can only get direct access via the anyconnect client…oh no…”

So to get the lab up quick fast I ran the anyconnect client on my workstation, then ran a TCP port map on my machine with the following:

1
netsh interface portproxy add v4tov4 listenport=443 connectaddress=10.10.20.14 connectport=443 listenaddress=192.168.51.254 protocol=tcp

The above command will listen on my local 443, then forward it to the remote host through the anyconnect client on 443 also. You can listen on one port and proxy to another, so keep that in mind.
Basic command settings:

1
netsh interface portproxy add v4tov4 listenport=80 connectaddress=ip-of-server-on-internet connectport=23 listenaddress=ip-of-windows-machine protocol=tcp

Also, don’t forget to open up your windows firewall for whatever the “listen” port is, otherwise the remote machines won’t be able to connect.

At this point I opened up my ansible tower and adjusted the creds for the ACI server(using the IP address of my workstation as that of the ACI) and awwwwaaaayyyy it went 🙂