Skip to content
Sep 13 / thebrotherswisp

The Brothers WISP 143 – ROS Docker, Zerotier, Cloud DNS

This week we have Greg, Nick A., and Alex Hart giving us the Hart to Heart.


This week we talk about:
ROS V7.rc3 adds docker containers(additional package)
Colin’s blog shout out
ROS V7.rc3 adds docker containers(additional package)
Thrift says 7rc1 mlag seems stable, but Ole says wifi is not on some of the older hardware
Zerotier has some folks excited…or perhaps spicy is a better word for it LOLOL
Mikrotik IP cloud bricked for a day or so

Here’s the video:(if you don’t see it, hit refresh)

Aug 29 / thebrotherswisp

The Brothers WISP 142 – ROS V7 RC, Unifi Talk, dPSK In MDUs

This week we have Greg and Nick A. giving our annual fire side chat.


This week we talk about:
Mikrotik ROS V7.1rc1 released
Unifi Talk
Satellite Azure deploy VM with AAP final config
dPSK vs SSID per resident in MDU
Aliexpress blackholes

Here’s the video:(if you don’t see it, hit refresh)

Aug 23 / Greg

Using Ansible Automation Platform For Post Configuration With Red Hat Satellite Provisioning Via Callbacks

That title is a mouthful.
In essence, what I’m doing is:
– Connecting my Satellite install to Azure so I can deploy RHEL images
– Once I use Satellite to deploy the image and it is provisioned, the VM will contact my AAP server via a “Callback”
– The AAP(call it Tower or Control, whichever you prefer) will then sync its inventory from Satellite, and connect to the VM to perform whatever configuration you like

Demo Video

Products Used

Azure is the cloud provider I’ll be deploying my VMs into. I’m using the free tier to do all my lab work in. Key here is to kill or pause anything you deploy but don’t use.

Red Hat Satellite is primarily a life-cycle management system for RHEL environments…think of it as a kind of WSUS for RHEL.

The Ansible Automation Platform is kind of all I talk about these days :). It’s the automation platform that will be doing the post provision configurations.

Red Hat Enterprise Linux(RHEL). Enterprise ready, fully supported Linux OS.

Satellite Install/Configuration

I started with a standard RHEL7 server in my lab with it pretty much a flat configuration. The role and following playbook gets Satellite up and running(found in my repo here). The role is from here originally.
The install playbook can be found here:

I won’t cover everything in the playbook, but in short it setsup a fully qualified domain name and hosts entry on the satellite server, issues a yum update, then calls the satellite install role. The FQDN configuration is required for satellite, otherwise it barfs.

After the install script completes I should be able to browse over to it(I’ve also created a DNS entry for the host):

AAP Configuration

This section ultimately sets up a callback method to active a job template. What it does is allows any host to issue a curl command to AAP and it will then fire off a job template directed towards that host. You can find the blog post about it here that I used as reference. Note that the method for calling this via Satellite as shown in that post did not work out of the box for me, so I pivoted to something else, which I’ll outline later. So the section where they are setting up the custom parameters can be skipped.

I need to setup credentials for AAP to connect to my satellite server so that it can pull all of the hosts into an inventory.
I’ll go to credentials and create a new entry entry with credential type of Red Hat Satellite 6:

I’ll now be able to use that to connect to a custom inventory source.

You’ll also need to now create a user account that matches what satellite will be setting up on the VMs when it provisions them. In my case it was user “ansible”, so I’ll create the cred too.

I’ll now create the inventory for my satellite server:

^^ This is one of the VERY important pieces here. In the inventory source I choose Red Hat Satellite 6 and it gives me some additional options. I can now choose the satellite credential I just created, thus it knows where to go to grab the info. I then enable all of the update options, this will cause the satellite server to be requeried each time this inventory is accessed. It will also clean out any old entries and only install active hosts from Satellite.
Now for an extremely important piece! Notice that I have something in the source variables section. The compose option will allow me to build any new variables I want from the information returned from the quireied source. In this case, everything returned has “foreman_” before the variable. The callback method being used here has a process by which it has to ensure the host calling has an entry in the inventory. If you are registering hosts in a DNS server as they are being provisioned, then all will work just fine as the inventory saves the FQDN in to connect to hosts. If, however, like me you aren’t doing auto DNS updates it will fail at this step. If I have the ansible_host variable present it will work just fine, sooooo that’s why I have the compose command create the ansible_host entry, it makes this all work sans DNS auto updates.
Source Variable settings:
ansible_host: foreman_ip

I should be able to sync the inventory and check one of the hosts to see that it created the ansible_host variable:

I’ll now add a job template in my AAP based on this playbook(obviously you will want to pull it in with the rest of the files via an inventory).

I’ll take a look at this playbook first, then look at the job template.
The playbook starts by making sure there isn’t already a host key saved in the system for this IP. For some reason when using the callback method it will add a key to the key file, so when you are labbing things it will cause you issues. This is why the first task clears out any entries for the host IP in question.

Next I setup the FQDN, then download and install the katello RPMs from the satellite server(you will need to update the satellite URL here).

After that I do a quick hack that fixes a wacky DNS issue. When you create an Azure resource group I know it assigns it a random domain to reference those objects fully qualified. My entries in Satellite, however, are using my domain, so when I issue the subscription-manager register command on the VM it creates a duplicate entry sourced from the hostname + the crazy domain. So the hack around this is to create a katello.facts file and populate the network.fqdn entry with my inventory_hostname.

Now I register the host to the Satellite server using the subscription-manager register --force --org="NASA_JPL" --activationkey="library-activation" command. You’ll notice that there’s a second commented out entry, and I left this so that I would add in here how I attempted to use the “–name” option. I figured this would adjust the FQDN, but it does not LOL, so don’t waste your time with it.

I next pull down my satellite tools and install the katello packages.

This is technically the end of the playbook here, but it doesn’t have to be. This is the point where I could do further checking…say for example I can check which host group it’s a member of and call another playbook based on it for say a www group or perhaps a db group.
For example, I added this host using a Satellite host group named “default”, so it shows up in my inventory as being a member of the “foreman_default” group, which I could match on:

For that matter I could check the server name and see if it contains www or db and then process further. Keep in mind that this is just the kicking off point.

Now that I’ve looked at the playbook, I’ll examine the job template used:

The important bits here are the satellite2 inventory that I created above, I’m using the same credential set that I configured in Satellite, and I enable provisioning callbacks. Once PCs are enabled it will give you the provisioning callback URL as well as the host config key to utilize. The provisioning callback and host key are used in the compute resource Custom Script section below.

Azure Configuration

To provision VMs one needs to create several things. One is a resource group under your subscription. A RG holds all of your assets(networks, interfaces, VMs, etc.). One can manually create all of this or you can use AAP again to build it all for me. My playbook for prepping the environment is here.

The script first creates a storage account to keep all of your items in.
It then creates a virtual network and a subnet inside of that network.
Next it creates a security group and allows SSH in. Your mileage may vary on this, I’m just in a lab and there won’t be any public IPs on my resources so plan accordingly.
After that I create a virtual NIC so that I have something I can do test pings to. This is completely optional as I only have it in place for testing, but it’s not a bad idea.
The last section is prep work for my LAN to LAN VPN connection, which again, is optional depending on what you are doing in your environment. I currently have my AAP and satellite servers installed in my lab, so to reach the private addressing back and forth I have a VPN tunnel setup.

I ran into a lot of random issues like I hit a microsoft.compute resource missing error. What you need to do is enable it in Azure so that you can create virtual machines. What you want to do is go into your subscription > resource providers > search for microsoft.compute and add it:

Azure Resource Manager To Satellite Configuration

The official Red Hat guide I followed is here.

You first need to create an application that can be used by Satellite to login and manipulate resources on Azure. Follow the link here to get the info for the next section.

Infrastructure > Compute Resources > create Compute Resource

All of the info here is derived from adding your application above as well as the subscription section itself.

Once you fill in the blanks you want to load in the different regions:

Then once you choose a region, any resource provisioned from this Azure resource will forever be in this region. Something to keep in mind is that not all resources are available in all regions. For example I live in Texas, so I was originally using the South Central region. I kept having failures trying to deploy standard RHEL7 images…and it turns out they aren’t available in this region. Soooo, my solution(since it is just a lab) was to deploy to the EastUS region that happily deploys the image. Once you create this compute resource in a region, you can’t go back and edit it. You can add an additional resource and just put it in a different region if you want, but these aren’t modifiable.

I’ll now want to add some images for this resource to be able to deploy.
from inside the compute resource click on images and create image:

I’ll now fill out the details for the image. Here I’m just going with the latest LVM RHEL7 image. Notice that you have to preface the name of the image with “marketplace://”. Also note that you can’t login directly as root, so create a username that works for you, since I’m using Ansible for all of the things I’m making the user Ansible.

Next, click on Compute Profile from inside of the Azure resource manager details. By default it has three t-shirt sizes: small, medium, and large. I’m going to click on small and do some additional configuration just for that(just for labbing I’m keeping it simple).

The first options are setting up some of the VM options:

The compute resource was created by my earlier Azure provisioning playbook(see above).
These images have a recommended disk size of Standard_D2s_v3, but by all means customize it as you see fit.
Username/password is my ansible creds I’ll use to connect in; this is also your opportunity to enter in an SSH key to be installed. I’m just going to be using username/pass.
I’ll now add a network interface(again from the previously provisioned playbook above):

I now have my Azure configured subnet available:

I’ll now add a storage device and give it about 50GB.

I’m going to scroll back up and add a custom script. I’ll show it first, then explain what it’s doing as this is VERY important:

Custom Script:nohup sleep 45 && curl --insecure --data "host_config_key=3216011a-750d-4e38-b387-f0d533ee" &
So first I used nohub. This will throw a command into the background, so that even if a terminal is disconnected it will still complete. This is important as Satellite will connect to the VM, throw the command, and leave.
Next it will sleep for 45 seconds. This is in place so that Satellite will have enough time to put the VM into it’s database, so that when AAP is called this host will show up in the inventory file.
After that I use the standard curl callback command that instructs my tower server to kickoff my specific callback job template.
Last I have a lone “&”, which will kick all of this command into the background.
This custom script command is what kicks off all of the magic.

Satellite Prereqs

Most likely you have all of this in place, I’m just putting this here as a reminder to myself since I built this from scratch.
I’m first going to add my subscription. Well, I suppose first you want to generate the manifest file and download it(follow this link for that).I’ve connected to and downloaded my manifest, then imported the zip file into Satellite:

I’ll then add my repos:

Next I’ll search for “rhel-7” and hit the button for only available repos:

Next I’ll expand, then hit the + on any repos I want. In this case I want satellite tools for sure as I’m going to be pushing those in my demo:

Now I synchronize my repos:

You’ll likely want to create a sync plan that will update your repos on a regular schedule.

I’m now creating an activation key: content > activation key:

You can see that I was lazy and didn’t create a new life-cycle, instead I just used the library, which is the catch all.
I’ll now hit the repos section and enable all of my repos:

Notice the activation key from the details menu, make note of it as I’ll be using it later in the process. It’s what the VMs phone back home with to register to the Satellite server:

Create Satellite Host Group

Configure > host group

Now I create a new host group:

You will undoubtedly know more about all of the various lifecycles and content views you want to configure. The deploy on and compute profile are the sections of particular interest. I choose the azure resource and small compute profile just configured. It can also save a lot of time if you fill out all other sections that are of interest to you.
In this host group I’ll click on activation key and add the key I saved from the satellite prereq section:

Deploy VMs

In Satellite I go to hosts > create hosts:

I now choose the host group that I’ve created, and it will populate virtually everything I need:

I’ll now pop over to the operating system section and choose my rhel7 image and setup the root password:

Once I click submit it will take a minute or two, but I should see an AAP job kick off, first to update the inventory, then the job template will fire and complete.

If I go to hosts > content hosts, I can see that the newly provisioned VM is in the list, and it shows up as registered, which means the AAP job template completed and had the VM phone home to the Satellite server!


I know this has a lot of moving pieces, but if you already have your Satellite setup and you have your Azure environment up and working, then this is a really quick project. Most of the above is me building everything from scratch, which the vast majority of you won’t have to do LOL.

As with anything, there are likely faster and more efficient ways of doing these things, and if that’s the case, please let me know what you come up with/how you would do it. Questions/comments always welcome.

Thanks, and happy Satelliting!

Aug 16 / thebrotherswisp

The Brothers WISP 141 – Mikrotik RB5009, CCR2004, Marvell Prestera

This week we have Greg, Mike, Tommy, and Andrew Thrift doing some hardware catchup.


This week we talk about:
RB5009 video with Janis Megis!
RB5009 – 1GB RAM/Flash, 4 cour 1.4Ghz ARMv8, 7 gig ports, 1 2.5Gb, 1 SFP+ – $219 USD
RB5009 rack mount – 4 per 1U
New CCR2004 video with Pauls Jukonis
CCR2004-1G-12S+2XS – 16 copper gig ports, 2 SPF+, dual AC PSU, 2 switch chips each with 10Gb lane – $465 USD
V7.1 route filter updates
Thrift V7 testing CRS3XX
MCLAG/MLAG on CRS3XX testing
Marvell Acquiring Innovium for High-End Switch ASIC
Marvell Prestera 7k – 3 Chipsets. DX7312 300G (12x 25G SERDES), DX7325 1.2T (24x 50G SERDES), DX7335 1.6T (32x 50G SERDES) – MACSec, SRv6, 400G Optics

Here’s the video:(if you don’t see it, hit refresh)

Jul 31 / thebrotherswisp

The Brothers WISP – GPON vs Active Ethernet

This special cast has Greg, Tommy C, Ole, Brett, and Brad chat about the basics of GPON and how it compares to Active Ethernet. This is all about delivering fiber to the customer and there is a LOT of information in here. It is a longer episode, but we don’t finish everything, so let us know what we missed for the follow up episode.


This week we talk about:
Define GPON
Define Active Ethernet
How does GPON work-splitters/oversubscription/ONT/OLT
How strong is vendor lock-in on GPON
Vendor cost vs support game
Who can you call for help
What is XGSPON
What is FEC
Cabinet types
Cabinet locations
GPON vs XGSPON speeds
GPON in the MDU vs copper cabling
ONT with wifi vs ONT with standard router
Distance from OLT to customers
Distance between customers
How TDMA works in GPON
Locate kits
Pigtails vs mechanical connections
Live fiber monitoring
APC vs UPC connections

Here’s the video:(if you don’t see it, hit refresh)

Jul 18 / thebrotherswisp

The Brothers WISP 140 – Plooza Meetup, Travel Kit, MTK cAP XL AC

This week we have Greg, Mike, and Colin comin in hot and shootin that shot.


This week we talk about:
Anyone going to ‘Plooza this year…Greg is thinking about it?
New Mikrotik cAP XL AC is a pretty beefy antenna…not sure what the gain is yet, but it’s chonky
Do you use a travel AP/router on the road?
What’s in your work travel bag(you are taking a flight)?
What kind of bag do you use for travel?
Digital business cards
Deleting Posts on Vendor Forums
Facebook trolling
Customer touches
Many more things that I can’t remember

Here’s the video:(if you don’t see it, hit refresh)

Jun 30 / Greg

Windows Updates With Snapshots Using Ansible Automation Platform

I was recently asking my teammate Jimmy(who’s brilliant BTW) what he shows as a compelling demo with Ansible and Windows, and he he had a more complicated version of what I’m about to show(most of his stuff is over the top(with Stallone)).

Video Demo


Playbooks for Windows found here.

In this simple demo I’ll connect to my VMWare Vcenter and perform a snapshot with memory.
I have a next step of putting the system in maintenance mode in monitoring to stop notifications.
I’ll then perform a simple Windows update in a Windows server using WinRM. I do this from inside of a block. I do this because a block has some special functionality as in “rescue” and “always”.
I use a rescue block which is called if anything inside of the block fails. So, if my Windows host fails to update for any reason, it will call this rescue block which will rollback the snapshot to undue any potential damage caused.
My following step is to delete the snapshot(at this point I’m done with it). Take note that since I did a memory snapshot also, the machine will cleanly keep running. If you fail to add the memory option and restore the VM will restore in a stopped state.
The last step would be to remove the host from maintenance mode.


This isn’t something that’s relegated to just updates. Any potentially failure inducing operation could be protected against using this. This simple method could be used to protect anything, often even appliances could take advantage.

Let me know if you have any questions or comments.
Thanks and happy automating.