Skip to content
Oct 19 / Greg

ServiceNow Orders VMWare VMs Via Ansible Tower


This is a quick demo that orders a server in ServiceNow, then once approved it calls Ansible Tower which then provisions VMs in VMWare.

This is built using this ansible blog post that shows how to tie together ServiceNow and Ansible Tower.

Video Demo

ServiceNow

First thing to do is create the OAuth connection between tower and SNOW. I’ve spoken about this in a previous post and it is outlined in the blog post linked above, so I won’t rehash that. Once OAuth is solid I create an outbound rest message:

Two things of note are the endpoint I’m calling which is the launch URL for my tower job and the HTTP query parameters(HQP). The HQP is where I map variables learned in SNOW to variables I will pass to tower. So variable1 is passed to tower as vm_name.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "extra_vars":
  {
    "vm_name": "${variable1}",
    "vm_template": "${variable2}",
    "vm_disksize": "${variable10}",
    "vm_memory": "${variable9}",
    "vm_cpus": "${variable8}",
    "vm_ip": "${variable7}",
    "vm_netmask": "${variable6}",
    "vm_gateway": "${variable5}",
    "vm_folder": "${variable4}",
    "vm_datacenter": "${variable3}",
    "vm_datastore": "SSD"
  }
}

I’ll now create a workflow that ties an approval and my custom script together:

Here’s a peek at my approval workflow item:

As you can see I selected a group to approve these requests through.

My custom script looks like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 try { 
 //var r = new sn_ws.RESTMessageV2('Tower Default Job', 'Tower Default Job Run');
var r = new sn_ws.RESTMessageV2('Tower VMWare Provision REST', 'Tower VMWare Provision REST HTTP Method');
 r.setStringParameterNoEscape('variable4', current.variables.variable4);
 r.setStringParameterNoEscape('variable7', current.variables.variable7);
 r.setStringParameterNoEscape('variable5', current.variables.variable5);
 r.setStringParameterNoEscape('variable6', current.variables.variable6);
 r.setStringParameterNoEscape('variable1', current.variables.variable1);
 r.setStringParameterNoEscape('variable10', current.variables.variable10);
 r.setStringParameterNoEscape('variable2', current.variables.variable2);
 r.setStringParameterNoEscape('variable8', current.variables.variable8);
 r.setStringParameterNoEscape('variable3', current.variables.variable3);
 r.setStringParameterNoEscape('variable9', current.variables.variable9);
 
//override authentication profile 
//authentication type ='basic'/ 'oauth2'
//r.setAuthenticationProfile(authentication type, profile name);
 
//set a MID server name if one wants to run the message on MID
//r.setMIDServer('MY_MID_SERVER');
 
//if the message is configured to communicate through ECC queue, either
//by setting a MID server or calling executeAsync, one needs to set skip_sensor
//to true. Otherwise, one may get an intermittent error that the response body is null
//r.setEccParameter('skip_sensor', true);
 
 var response = r.execute();
 var responseBody = response.getBody();
 var httpStatus = response.getStatusCode();
}
catch(ex) {
 var message = ex.message;
}

I just went with generic variable names and mapped them from the self service item over.

My service item points towards the workflow:

I then created simple input and multiple choice variables with all of my options:

Once I choose the item from the service catalog I’m presented with a simple form to complete:

Ansible Tower

My VMWare github repo is here.

The playbook in use here is this one:

Taking a look at the playbook, you can see that I’m using the vmware_guest module.
This module is surprisingly easy to use. This module requires the connection info for the VCenter server(IP, username, password). I use a custom credential to supply this at run time(this way I can keep the info secure.
After that I pass the template I want the VM created from, then standard info like HD size, how much RAM, how many CPUs, etc.

When you add the playbook to your tower, be sure to add “prompt on launch” for extra variables to that SNOW is allowed to pass variables in:

Conclusion

Once SNOW launches the job it only really takes about 1.5 minutes for Linux and about 2 minutes for Windows. The VMWare module is actually really easy to work with and is quite functional. SNOW can be a bit more finicky, but once it is working it’s not usually so bad.

Something I learned is that while you can have a tower job template use a survey and be called by SNOW, it’s likely better(at least while testing) to not use surveys. If there is a survey, then tower will only launch the job if the survey variables are all passed in at launch(which can cause you issues when trying to troubleshoot why your SNOW service orders aren’t lunching your tower jobs).

If you have any questions or comments, please let me know.

Thanks and happy automating.

Oct 15 / thebrotherswisp

The Brothers WISP – NEW Kwikbit 60Ghz Radios



This **sponsored** podcast has us talking to Vladimir Kelman(CTO Kwikbit), Travis Carter(CEO US Internet), and Justin Miller about Kwikbit’s new 60Ghz radios!

This cast we talk about:
Outdoor – 90 horizontal/40 vertical beam
128 antenna elements
1500 foot range
Adaptive resource allocation?
RSTP
Why RESTful API
Kwikbit tools?
k60x $499 2x 2.5Gb and 1Gb
k60 $299 2 Gb
LACP?

Indoor
k60i $209 3x 1Gb
500 foot range at full throughput
64 antenna array

Kwikbit Edge GUI management/link planning
non-technical install
zero touch install
GUI design tool?
Certificate based management protection

Qualcom chipsets
Up to 8 clients
Single antenna tile
Mesh-like capabilities
Manufactured in USA
Full monitoring access via SNMP and API

Kwikbit.com for more info!

Here’s the video:(if you don’t see it, hit refresh)

Oct 11 / thebrotherswisp

The Brothers WISP 121 – Mikrotik Corrections, NAS HDs, Datacenter Newbie Guide



This week we have Greg, Mike, Tommy C., and Thrift getting down with the get down.

**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
Ansiblefest coming this week
About RB3011 port speed mismatch on same switch and flapping with v 6.46+.
Thrift will do news corrections
CCR2004 tentatively stable on 6.48beta40 – Chad and Joe both report good stuff so far.
fq_codel & cake rumors
Netpower Lite 7R & CSS610 – A new switch chipset appears
Synology DS920+ SAN – about to plug it in LOL
Backblaze
Datacenter Tips and tricks – a newbie guide

Here’s the video:(if you don’t see it, hit refresh)

Oct 7 / Greg

Using Ansible To Create DNS Certificates And Install Them On F5 Big IP Loadbalancers


This demo will first use the DNS method to create a letsencrypt cert, then it will build a VIP in the F5, and last it will install the letsencrypt certificate for that VIP. I then have a workflow for updating the cert and pushing it to the F5. I suppose this allows you to live in a world where you don’t have to worry about your certificates expiring anymore…it just magically works LOL.

Demo Video

Github Repo

You can find the playbooks here in my github repo.

Let’s Encrypt

First is creating certificates. LE allows for a few different methods to create certs. The most common is http, which creates a file on your webserver they verify. For this demo I wanted to create the certs on another device, then send them over to the F5, and for this I chose to use LE’s DNS option. The DNS process has you create a custom txt record on your DNS server that they can verify. I’m using cloudflare’s DNS and their API makes this stupid simple.
Let’s take a look at my cert creation playbook:

This playbook has a lot going on. I’m going to point out the important bits here.
First is how I create two private keys, one for the letsencrypt account and one for the certificate itself. Keep in mind that these MUST be different keys. I tried using the same key initially and got this error:

1
2
3
4
TASK [Let the challenge be validated and retrieve the cert and intermediate certificate] ******************************************************************
fatal: [localhost]: FAILED! => changed=false
  msg: 'Error new cert: CODE: 400 RESULT: {''type'': ''urn:ietf:params:acme:error:malformed'', ''detail'': ''Error finalizing order :: certificate public key must be different than account key'', ''status'': 400}'
  other: {}

Also note that I use acme version 2 in all of my certificate calls. This is because v1 has been deprecated, but is still shown on all of the ansible examples. If you use v1 you will have a bad time.

1
2
      acme_directory: https://acme-v02.api.letsencrypt.org/directory
      acme_version: 2

Once the initial challenge is issued, it will pass back info that’s used to create the DNS TXT record. I’m using cloudlfare and the API call does it’s thing, then it moves on to the final step where the challenge is validated. It, unfortunately, takes a little time for the DNS entry to go completely live, so the final play has some additional configurations that account for that:

1
2
    retries: 10
    delay: 12

Retries says that it will try this command 10 times, and delay says wait 12 seconds between each retry. This means the task will attempt for about 2 minutes, which so far has been enough time to complete everything successfully.

At this point you should now successfully have your certificates!

F5 Certificate Application

I’m now going to connect to my F5 big IP and apply the certs:

I’m using this to create my environment and install the cert, but I could continue to run this to perform cert upgrades as all of it is idempotent. This means if the script needs to make a change it will, if the F5 already exists in the desired state, then no change is made(one of my favorite features of Ansible!).

Take note when we reach the “#SSL Upload and Modification of VIP to use New Certificate” section.
I first upload the newly created cert and key pair. Next I create a client ssl profile with the newly uploaded files. Last I create a virtual server using the client ssl profile.

At this point, I’m done!

I can now schedule the cert create playbook to run on a 30 day schedule to check if there’s less than 60 days left on the cert; if there is, it will refresh the cert. Now the install script can be run again, OR this simple cert update script can be run:

This script really just runs the certificate upload task again. If the certs are the same, nothing is done, if however, they are different, then the new certificate will be applied to the F5. Pretty slick, eh?

As always, let me know how you would utilize this in your environment, how you would change it, your comments, and your questions.
Thanks and happy certing 😉

Sep 27 / thebrotherswisp

The Brothers WISP 120 – Mikrotik Updates, Time Off In IT, One Tip For Students Heading Into IT



This week we have Greg, Mike, Tommy C., and Miller using some new video technology!

**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
Simon from Sonar will be a guest; what questions do you have?
Nvidia purchases ARM for $40B
Cradlepoint sells to Ericcson
H.266 video compression promisses to use 50% less bandwidth – thanks for the tip Mike
Major windows security hole plugged for AD; update ASAP if you haven’t already.
ROS 6.47.4 gives more u-nii-2 support, fixed CRS3xx IGMP and STP issues.
Thrift spotted in the newsletter the line “the network will be upgraded to a 100 Gbps data transfer rate, using MikroTik devices that will be released soon.”
Mikrotik hAP AC3 non LTE $99 – ARM based CPU, should we chance it? Bonus pirate video on the product page.
Mikrotik CSS610-8G-2S+IN Marvel Chipset $100 for 2x SFP+ ports!!!
Mikrotik 3011 has known switch chip issue where a mix of port speeds isn’t allowed. 1Gb devices with 100Mb devices result in port flaps.
Small indoor UPS that if drained requires no interaction to get it back online?
Ubiquiti asks for your input
Chad reminds us why fire code exists for our cabling; ” If your cabling, equipment or holes are listed in the fire investigation report, the ensuing insurance suits will do everything possible to have someone else pay.”
Time off in IT; different perspectives from around the world.
Looks like I’ll be mentoring some College Students.

Here’s the video:(if you don’t see it, hit refresh)

Sep 14 / Greg

Self Service Troubleshooting Using ServiceNow and Ansible Tower


Holy cow did I learn a lot on this one. It took me the better part of two weeks to put it all together/learn everything required. This was actually my first go at automating Windows and I learned a LOT.

The idea behind this setup is that if a user can’t access a TCP based service or can’t browse to a web page, they can pop into ServiceNow (which I’ll call SNOW moving forward) and create a service order for it, which will then call Tower to perform some automated troubleshooting.

Demo Video

Windows Machines

First things first, all windows devices that we will be connecting to need WinRM enabled. This is the Windows Remote Management system that allows me to connect in and run commands. The easiest way to go about it is to create a group policy and have the windows machines automatically enable it.

In my example I’m performing something known as double-hopping.

What I’m doing is having Tower reach out to Server A only. I will do testing from Server A, but I’ll also tell Server A to connect to Client 1 and perform some tests. This hop from Server A to Client 1 is considered a “double hop”. Windows prevents this behavior by default for security reasions, but there are some work arounds as seen here. The work around I used is CredSSP. This is easy to use by setting this host variable for these machines:

1
ansible_winrm_transport=credssp

This is what the host variables look like for each of these first hop devices I’ll be controlling:

1
2
3
4
5
6
---
ansible_winrm_transport: credssp
ansible_host: 10.1.1.10
ansible_connection: winrm 
ansible_winrm_scheme: http
ansible_port: 5985

Notice also that in the above I’m specifying the scheme as http and the port as 5985. HTTP indicates I’m not performing encryption, which you would want to do in production(this is just a lab setup). When performing the connection unencrypted I also need to specify the port as 5985(when encrypted the port would be 5986 and is the default).

If you run into any issues when using WinRM with Ansible to connect to your clients check this quick guide.

I wanted it to be easy for someone to find their computer name, and a simple solution is to add a toolbar that has their machine name(either bginfo or what I did here:

1
2
3
4
Right-click on Taskbar -> 
Go to Toolbars -> 
Choose New Toolbar, 
type in \\%computername%,  and Click Select Folder.

ServiceNow Configuration

I won’t cover the full setup since Michael Ford has already done that for us. That link is a walk through that will get you to the point where SNOW makes API calls to Tower to fire off job templates(while passing over variables).

Since I’m standing on the shoulders of giants, I’ll skip right to my service catalog item:

As you can see I kept it simple. I am by no means a SNOW developer, which you will quickly see once I jump into the processing section LOL.
I’m gathering the destination, so https://gregsowell.com or if it’s some random TCP service it could just be gregsowell.com or 1.2.3.4.
I’m also getting their machine name from their taskbar.
Last if it is a standard TCP service they put the port number there, otherwise it stays as 0.

In Tower I did have to pull some entries from the SNOW request table, but not knowing anything about the structure of the tables…or even what the tables were I was a little lost. I ended up finding the SNOW “REST API Explorer”. This gives you the ability to find the tables, and explore their structure along with creating cURL API calls based on what you build.
Once in the rest explorer you can choose table name, add additional query parameters(like ordering by newest entry), and picking individual fields you are interested.

I like the output to come back as json, that way I can convert it to yaml with some ansible filters.

Once you click send it gives you your desired output right in the browser;

You can also click the output method of choice in “Code Samples” for some copy/paste content:

Tower Configuration

I’ll start with the playbook that will ultimately call my processing role:

I setup my basic variables that are passed from SNOW:

1
2
3
    tcp_port: "{{ variable_3 }}"
    test_url: "{{ variable_2 }}"
    test_client: "{{ variable_1 }}"

tcp_port is always zero for web testing.
test_url is essentially the destination we are trying to reach(for either webpage or TCP based service.
test_client is the hostname of the user’s PC.

I include a single task, which is just calling the role detailed below.

Here is the role I’ve created to perform all of the processing.
I’ll look at the main files required:
Remember when I mentioned I’m not a SNOW dev…keep that in mind.

In the above at the top you see I make a restful call to SNOW and request the last created request from the request table(I pull the req ID and the username). I’m banking on the fact that no other requests came in in the seconds it took for this to fire(not what you would do in production). Obviously I’d build this into my workflow in an actual production environment.
I then call either the web tshooting or TCP tshooting task files depending on what is required.
Once that processing is complete I open a SNOW incident as the requesting user and add my tshooting info and assign it to the correct group.

I use several jinja2 templates that have powershell code to extract IP info as well as to perform the TCP and web testing. Feel free to browse away.
For example, here’s the curltest powershell template.

As you can see it’s simply issuing the curl test and returning status code as well as raw content length(size of content returned by query).

The fun starts to happen in the web tshooting task file.

I’m using the jinja templates to first resolve the client name.
Then I use one of the template files to double hop to the client machine and perform a powershell version of cURL to test pulling the web page. After the client is tested, I test from the local domain controller, then I connect to an external canary server and perform the same test.

If any of my tests fail on the inside I then connect to the local Cisco ASA firewall and perform a packet tracer test. Packet tracer in an ASA will take the source/destination info and generate a virtual packet. It will then pass this virtual packet through the firewall noting each step what happens to the packet and whether it is allowed through or blocked at various steps. In the end it will supply me with an allowed or dropped.
As a side note, here’s what my ASAv hosts file entry looked like while testing(prior to entry into Tower):

1
asav ansible_host=10.1.12.15 ansible_connection=network_cli  ansible_network_os=asa ansible_user=admin ansible_password=lab ansible_become=yes ansible_become_method=enable ansible_become_pass="lab"

At the bottom of this task file I then essentially go through various condition checks and when one is met I set what the incident message will be along with which support group it will be assigned to.

Decision Tree Example

1. If all tests pass(internals and canary both get a status 200 and the website retrieved were within 10 bytes of each other), then the user is instructed to please test again as it looks like service may have been restored. This was likely just a quick blip.
2. If the client fails, but everything else passes(including the firewall rules saying the client should have access), then it is assigned to the Service Desk team as it’s likely just a local PC issue.
3. If the client and firewall fails, then sounds like we have a network issue(needs firewall mod), so assign it to the Network team.
And so on.

The TCP test file is much the same, but it uses a slightly different method to test service(all still using powershell, though).

Wrap Up

Conceptually nothing is too crazy here, but all of the individual pieces can take some time, especially when they are new to you, as much of it was for me here. I hope this post has you thinking about various time sinks within your current workflow that could possible be automated with just a little creativity. What do most of your trouble tickets deal with, and how can you automate their troubleshooting? Let me know how you would implement something like this in your environment.

Thanks and happy automating!

Sep 13 / thebrotherswisp

The Brothers WISP 119 – $1800 PTZ, 100% SLAs, Centurylink Split



This week we have Greg, Mike, and Tommy C. keeping you in the know.

**Sponsors**
Sonar.software
Cambium ePMP Bundle
**/Sponsors**

This week we talk about:
Nick A found a $1,800 PTZ camera in the beta store – 22x optical zoom full PTZ
Mikrotik Net Power Lite 7R – now with 100% more video
Mikrotik HA router scripts(heartbeat link with config sync and peer failure detection) – thanks for the link Landon
100% uptime to customers
Moving DCs…time to register for my ORG/ASN/IP
More Zhone complaining
In automation config pushing is so boring…what’s interesting?
Self Service troubleshooting.
Toilet potatos
Tommy’s failure story
Is Level 3 Down?

Here’s the video:(if you don’t see it, hit refresh)