I’ve been using dynamic IP address services from no-ip.com for a long time to give a DNS name to my dynamic home IP. I remember having a no-ip.com address in college around the early 2000s for my dorm, then apartment, computer. I went through a hiatus after college where I managed my domain name to IP mappings manually for a few years (and my own Bind setup), until it seemed like the ISPs started giving me a new address monthly and the burden was too much. I also sold the hardware the DNS server ran on at some point during my moves.
I was happy to see that after over 10 years away, no-ip was still there, and got signed up right away. I have an Ubuntu Linux system as my home desktop (and server really) and was glad to find a client that seemed to run on it. It was a little finicky to get going and was a black box sort of client to install. That wasn’t a good sign to start.
Then came the emails. Oh my god, the emails. Every 30 days they want to make sure you’re still there, and have you verify your domain name. They give you a 1 week notice too, so this means every 21 days you’re getting an email to verify that you’re still there. If you don’t verify – they say they’ll delete your account. When you try to click the email and go to the site it’s VERY clear they’d rather have you sign up for a paid account. Every 21 days for a few years I got nagged until I couldn’t take it any more! They clearly told me that I should go somewhere else for my business if I didn’t want to pay.
This sort of things just bugs me.
At the Open Source conference All Things Open, someone mentioned DuckDNS and I decided to give it a try. It’s been a night and day difference from no-ip so far, for the better.
Here’s what I like:
There is no black box app or script to install on my Linux box. They just use cron and wget.
The instructions are clear and easy to copy and paste.
No one has emailed me anything yet, and since they have ZERO marketing team, I’m assuming no one will.
If you want to get started I highly recommend them. If you’re running a Linux server it’s dead simple.
The whole idea is that you sign up on duckdns.org and get a unique key for your desired domain name. You take this unique key and their copy and paste script and run that every 5 minutes as a cron job.
Every 5 minutes, your server uses wget (called via cron) with the duckdns URL and key and their server figures out your external IP and updates it accordingly. It was ridiculously easy to get started and used systems I already know and love (bash and cron).
Let’s Encrypt was really easy to setup, but Cron was less so. I kept getting emails that the Let’s Encrypt renewal was failing:
2017-03-09 02:51:02,285:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/bbbburns.com.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: NoInstallationError(). Skipping.
1 renew failure(s), 0 parse failure(s)
I had a cron job setup with the absolute bare minimum:
crontab -e
56 02 * * * /usr/bin/letsencrypt renew >> /var/log/le-renew.log
When I ran /usr/bin/letsencrypt renew
at the command line, everything worked just fine. I was like, “Oh – this must be some stupid cron thing that I used to know, but never remember.”
Turns out the problem was the cron environment PATH variable. Cron didn’t have access to /usr/sbin and apparently certbot was using that for access to the apache2 binary. The fix was to change the cron to the following:
56 02 * * * /root/le-renew.sh
Then create a script that runs the renewal after the PATH variable is set correctly:
cat /root/le-renew.sh
#!/bin/bash
#Automate the LE renewal process
#Need /usr/sbin for apache2
# https://github.com/certbot/certbot/issues/1833
export PATH=$PATH:/usr/sbin
#Renew the certs and log the results
/usr/bin/letsencrypt renew >> /var/log/le-renew.log
It was a good thing I put the link to the problem right in the script, or I never would have been able to find it again to write this blog.
NOW my renewal works absolutely fine. Problem solved. Thanks Cron.
I have seen articles flying around about Let’s Encrypt for a while now. The idea seemed awesome, but the website seemed so light on technical instructions that I didn’t know if it would actually work. I wanted to know EXACTLY what lines it would propose to hack into my carefully manicured Apache configuration. And by carefully manicured, I mean “strung together with stuff I copied and pasted from stack overflow“.
I couldn’t find the information I really wanted – so I just JUMPED in and started installing things and running commands. 30 seconds later, I had a fully functioning cert on my site. I was blown away. It copied my existing non-ssl vhost config and created a new vhost with SSL enabled. All I had to do was enter my email address, select the vhost to enabled SSL for, and hit GO.
I had to put in a crontab entry myself to get the auto-renewal to work but that wasn’t so bad. I would hope they improve that in the future – but cron is no big deal.
I’m interested to see if everything works when my web certs expire 90 days from now! Crazy times. I used to do this and dread it once per year because the process was so manual. Now that it’s automated – I’ll get new certs while I’m sleeping. Woohoo.
Nutanix recently released the AHV hypervisor, which means I get a new piece of technology to learn! Before I started this blog post I had no idea how Open vSwitch worked or what KVM and QEMU were all about.
Since I come from a networking background originally, I drilled down into the Open vSwitch and KVM portion of the Nutanix solution. Here’s what I learned! Remember my disclaimer – I didn’t know anything about this before I started the blog. If I’ve got something a bit wrong feel free to comment and I’m happy to update or correct.
KVM Host Configuration
AHV is built on the Linux KVM hypervisor so I figured that’s a great place to start. I read the Nutanix Bible by Steve Poitras and saw this diagram on networking.
AHV OvS Networking
The CVM has two interfaces connecting to the hypervisor. One interface plugs into the Open vSwitch and the other goes to “internal”. I wasn’t sure what that meant. Looking through the hypervisor host config though I saw the following interfaces:
[root@DRM-3060-G4-1-1 ~]# ifconfig
br0 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:50
inet addr:10.59.31.77 Bcast:10.59.31.255 Mask:255.255.254.0
eth0 Link encap:Ethernet HWaddr 0C:C4:7A:3B:1C:8C
eth1 Link encap:Ethernet HWaddr 0C:C4:7A:3B:1C:8D
eth2 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:50
eth2.32 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:50
eth3 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:51
eth3.32 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:51
lo Link encap:Local Loopback
virbr0 Link encap:Ethernet HWaddr 52:54:00:74:F9:B0
inet addr:192.168.5.1 Bcast:192.168.5.255 Mask:255.255.255.0
vnet0 Link encap:Ethernet HWaddr FE:54:00:9C:D8:CD
vnet1 Link encap:Ethernet HWaddr FE:54:00:BE:99:B3
The next place I went was routing with netstat -r to see which interfaces were used for each next hop destination.
[root@DRM-3060-G4-1-1 ~]# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.5.0 * 255.255.255.0 U 0 0 0 virbr0
10.59.30.0 * 255.255.254.0 U 0 0 0 br0
link-local * 255.255.0.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 eth1
link-local * 255.255.0.0 U 0 0 0 eth2
link-local * 255.255.0.0 U 0 0 0 eth3
link-local * 255.255.0.0 U 0 0 0 br0
default 10.59.30.1 0.0.0.0 UG 0 0 0 br0
I omitted a lot of text just to be concise here. We can see there are two interfaces with IPs, br0 and virbr0. Let’s start with virbr0, which is that internal interface. You can tell because it’s the 192.168 private IP used for CVM to hypervisor communication. I found that it was a local linux bridge, not an Open vSwitch controlled device:
[root@DRM-3060-G4-1-1 ~]# brctl show virbr0
bridge name bridge id STP enabled interfaces
virbr0 8000.52540074f9b0 no virbr0-nic
vnet1
This bridge virbr0 has the vnet1 interface headed up to the internal adapter of the CVM – so THIS is where the CVM internal interface terminates.
That’s one side of the story – the next part is Open vSwitch
[root@DRM-3060-G4-1-1 ~]# ovs-vsctl show
be65c814-5d7c-46ab-bfb1-7b2bea19d954
Bridge "br0"
Port "tap345"
tag: 32
Interface "tap345"
Port "vnet0"
Interface "vnet0"
Port "br0"
Interface "br0"
type: internal
Port "bond0"
Interface "eth2"
Interface "eth3"
Port "br0-dhcp"
Interface "br0-dhcp"
type: vxlan
options: {key="1", remote_ip="10.59.30.82"}
Port "br0-arp"
Interface "br0-arp"
type: vxlan
options: {key="1", remote_ip="192.168.5.2"}
ovs_version: "2.1.3"
OvS has a vSwitch called br0. The CVM vnet0 is a port on this bridge, and so is bond0 (the combination of the 10GbE interfaces). We also see this special “type:internal” interface – this one with the IP address assigned to it. This is the external facing IP of the AHV / KVM hypervisor host.
In addition to the CVM, external, and internal interfaces we see a tap345 interface tagged in VLAN 32. This matches the tagged interfaces from our “ifconfig -a” command above: eth2.32 and eth3.32. It’ll be used for a VM that has a network interface in VLAN 32.
Finally – we come to the IP Address Management (IPAM) interfaces, br0-arp, and br-dhcp. Steve mentions VXLAN and here’s where we see those concepts. The OvS can either intercept and respond to DHCP traffic, or just let it through. If we allow OvS to intercept the traffic this means Acropolis and Prism now become the point of control for giving out IP addresses to VMs that boot up. Very cool!
Now let’s take a look at the config parameters passed to the running CVM. Right now this box has ONLY the CVM running on it so only one instance of qemu-kvm running.
We saw a new reference in that last command, NTNX-Local-Network. If we look at virsh for information about defined networks we see the following:
[root@DRM-3060-G4-1-1 ~]# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
NTNX-Local-Network active yes yes
VM-Network active yes yes
If we look in the /root/ partition there are definitions for these:
These two pieces of information tie everything together neatly for us. The internal network given to the CVM is the linux virbr0 device. The external network given to the CVM is OvS br0.
Now I think I finally understand that image presented at the beginning!
CVM Guest Configuration
Since we understand the KVM/AHV host configuration lets take a look in the CVM guest. This should be a little easier.
nutanix@NTNX-15SM60140129-A-CVM:10.59.30.77:~$ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.5.0 * 255.255.255.128 U 0 0 0 eth1
192.168.5.0 * 255.255.255.0 U 0 0 0 eth1
10.59.30.0 * 255.255.254.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 eth1
default 10.59.30.1 0.0.0.0 UG 0 0 0 eth0
The routing table shows the internal and external networks, and just two network adapters. The eth1 adapter has a subinterface. This one is eth1:1 as opposed to .1. Not entirely sure what that one means – but I’ll keep it in mind in case I come across something later on.
nutanix@NTNX-15SM60140129-A-CVM:10.59.30.77:~$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 52:54:00:9C:D8:CD
inet addr:10.59.30.77 Bcast:10.59.31.255 Mask:255.255.254.0
eth1 Link encap:Ethernet HWaddr 52:54:00:BE:99:B3
inet addr:192.168.5.2 Bcast:192.168.5.127 Mask:255.255.255.128
eth1:1 Link encap:Ethernet HWaddr 52:54:00:BE:99:B3
inet addr:192.168.5.254 Bcast:192.168.5.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
That’s it – just two simple interfaces in the CVM. One for internal traffic to the hypervisor directly, another for receiving any external requests from remote CVMs, the management APIs, and all of the other magic that the CVM performs!
This concludes our walkthrough of networking inside a Nutanix AHV machine. I hope you learned as much as I did going through these items! Please comment or reach out to me directly if you have any questions.
Call me crazy, but I really love reading legal analysis at CYB3RCRIM3. I discovered the site via an OPML file passed to me by a friend in the security industry. Thanks Devon!
I can see how the walls of text might not be appealing, but the material is great. Legal writing is just so different; almost formulaic and with zero emotion. If you can get over the hump of reading court case opinions I think the subject matter is important to everyone. True human computer interaction… Here is where your technology comes into play to exonerate or incriminate you. In this particular instance we’ll say it’s where the rubber meets the road. (Sorry)
Here’s a great example using a car data recorder where the defendant is appealing on a technicality that I won’t pretend I’m fully behind. I LOVE reading between the lines though:
This driver was a huge dick and went out of their way to brake check someone. The data recorder in the car backs this up. This data is being used against the driver for a conviction (along with other testimony) and there is no way to contest these basic facts.
Are you aware that your car is spying on you? That your brake and throttle application along with your speed will be analyzed in full detail if you’re ever in an incident that goes to court? They don’t advertise that on the TV commercials for new cars. Do you know which car manufacturers record what data and for how long? Do you know how reliable that recorder is? Can it be hacked? Can I make it say I was going 1 mph with the gas and brake fully applied at all times (even when stopped)?
I’m not saying cars shouldn’t do this tracking after a crash. I’m also not saying the driver shouldn’t have been convicted. It seems like the justice system worked like it should have. I just like to know ahead of time how it is I’m being tracked and monitored. Personally, I think I’d opt out of this recording technology if it was possible to do so.
Thanks CYB3RCRIM3 for calling out the interesting cases!
I thought I was done writing – but just look at the next post involving a gang member and the forfeit of passwords. As a condition of probation this guy had to surrender all passwords to all social media sites. I didn’t know it was even possible for the state to make that demand. Now I know! How would you feel about that if it happened to you?
