Let’s Encrypt – How do I Cron?

Let’s Encrypt was really easy to setup, but Cron was less so. I kept getting emails that the Let’s Encrypt renewal was failing:

2017-03-09 02:51:02,285:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/bbbburns.com.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: NoInstallationError(). Skipping.
1 renew failure(s), 0 parse failure(s)

I had a cron job setup with the absolute bare minimum:

crontab -e
56 02 * * * /usr/bin/letsencrypt renew >> /var/log/le-renew.log

When I ran
/usr/bin/letsencrypt renew
at the command line, everything worked just fine. I was like, “Oh – this must be some stupid cron thing that I used to know, but never remember.”

Turns out the problem was the cron environment PATH variable. Cron didn’t have access to /usr/sbin and apparently certbot was using that for access to the apache2 binary. The fix was to change the cron to the following:

56 02 * * * /root/le-renew.sh

Then create a script that runs the renewal after the PATH variable is set correctly:

cat /root/le-renew.sh
#Automate the LE renewal process

#Need /usr/sbin for apache2
# https://github.com/certbot/certbot/issues/1833
export PATH=$PATH:/usr/sbin

#Renew the certs and log the results
/usr/bin/letsencrypt renew >> /var/log/le-renew.log

It was a good thing I put the link to the problem right in the script, or I never would have been able to find it again to write this blog.

NOW my renewal works absolutely fine. Problem solved. Thanks Cron.

Let’s Encrypt – Easy – Free – Awesome

I recently saw a news article about StartCom being on Mozilla and Google’s naughty list. Things looked bad, and my StartCom certs were up for renewal on the blog.

I have seen articles flying around about Let’s Encrypt for a while now. The idea seemed awesome, but the website seemed so light on technical instructions that I didn’t know if it would actually work. I wanted to know EXACTLY what lines it would propose to hack into my carefully manicured Apache configuration. And by carefully manicured, I mean “strung together with stuff I copied and pasted from stack overflow“.

I couldn’t find the information I really wanted – so I just JUMPED in and started installing things and running commands. 30 seconds later, I had a fully functioning cert on my site. I was blown away. It copied my existing non-ssl vhost config and created a new vhost with SSL enabled. All I had to do was enter my email address, select the vhost to enabled SSL for, and hit GO.

I had to put in a crontab entry myself to get the auto-renewal to work but that wasn’t so bad. I would hope they improve that in the future – but cron is no big deal.

I’m interested to see if everything works when my web certs expire 90 days from now! Crazy times. I used to do this and dread it once per year because the process was so manual. Now that it’s automated – I’ll get new certs while I’m sleeping. Woohoo.

Networking Exploration in Nutanix AHV

Nutanix recently released the AHV hypervisor, which means I get a new piece of technology to learn! Before I started this blog post I had no idea how Open vSwitch worked or what KVM and QEMU were all about.

Since I come from a networking background originally, I drilled down into the Open vSwitch and KVM portion of the Nutanix solution. Here’s what I learned! Remember my disclaimer – I didn’t know anything about this before I started the blog. If I’ve got something a bit wrong feel free to comment and I’m happy to update or correct.

KVM Host Configuration

AHV is built on the Linux KVM hypervisor so I figured that’s a great place to start. I read the Nutanix Bible by Steve Poitras and saw this diagram on networking.

Networking diagram inside the Acropolis KVM Host
AHV OvS Networking

The CVM has two interfaces connecting to the hypervisor. One interface plugs into the Open vSwitch and the other goes to “internal”. I wasn’t sure what that meant. Looking through the hypervisor host config though I saw the following interfaces:

[root@DRM-3060-G4-1-1 ~]# ifconfig 
br0 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:50 
  inet addr: Bcast: Mask:
eth0 Link encap:Ethernet HWaddr 0C:C4:7A:3B:1C:8C 
eth1 Link encap:Ethernet HWaddr 0C:C4:7A:3B:1C:8D 
eth2 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:50 
eth2.32 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:50 
eth3 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:51 
eth3.32 Link encap:Ethernet HWaddr 0C:C4:7A:58:91:51 
lo Link encap:Local Loopback 
virbr0 Link encap:Ethernet HWaddr 52:54:00:74:F9:B0 
  inet addr: Bcast: Mask:
vnet0 Link encap:Ethernet HWaddr FE:54:00:9C:D8:CD 
vnet1 Link encap:Ethernet HWaddr FE:54:00:BE:99:B3

The next place I went was routing with netstat -r to see which interfaces were used for each next hop destination.

[root@DRM-3060-G4-1-1 ~]# netstat -r 
Kernel IP routing table
Destination  Gateway Genmask       Flags MSS Window irtt Iface  * U     0 0 0           virbr0   * U     0 0 0           br0
link-local   *   U     0 0 0           eth0
link-local   *   U     0 0 0           eth1
link-local   *   U     0 0 0           eth2
link-local   *   U     0 0 0           eth3
link-local   *   U     0 0 0           br0
default    UG    0 0 0           br0

I omitted a lot of text just to be concise here. We can see there are two interfaces with IPs, br0 and virbr0. Let’s start with virbr0, which is that internal interface. You can tell because it’s the 192.168 private IP used for CVM to hypervisor communication. I found that it was a local linux bridge, not an Open vSwitch controlled device:

[root@DRM-3060-G4-1-1 ~]# brctl show virbr0
bridge name bridge id         STP enabled interfaces
virbr0      8000.52540074f9b0 no          virbr0-nic

This bridge virbr0 has the vnet1 interface headed up to the internal adapter of the CVM – so THIS is where the CVM internal interface terminates.

That’s one side of the story – the next part is Open vSwitch

[root@DRM-3060-G4-1-1 ~]# ovs-vsctl show
 Bridge "br0"
  Port "tap345"
    tag: 32
    Interface "tap345"
  Port "vnet0"
    Interface "vnet0"
  Port "br0"
    Interface "br0"
      type: internal
  Port "bond0"
    Interface "eth2"
    Interface "eth3"
  Port "br0-dhcp"
    Interface "br0-dhcp"
      type: vxlan
      options: {key="1", remote_ip=""}
  Port "br0-arp"
    Interface "br0-arp"
      type: vxlan
      options: {key="1", remote_ip=""}
ovs_version: "2.1.3"

OvS has a vSwitch called br0. The CVM vnet0 is a port on this bridge, and so is bond0 (the combination of the 10GbE interfaces). We also see this special “type:internal” interface – this one with the IP address assigned to it. This is the external facing IP of the AHV / KVM hypervisor host.

[root@DRM-3060-G4-1-1 network-scripts]# cat ifcfg-br0 
OVSREQUIRES="eth3 eth2 eth1 eth0"

In addition to the CVM, external, and internal interfaces we see a tap345 interface tagged in VLAN 32. This matches the tagged interfaces from our “ifconfig -a” command above: eth2.32 and eth3.32. It’ll be used for a VM that has a network interface in VLAN 32.

Finally – we come to the IP Address Management (IPAM) interfaces, br0-arp, and br-dhcp. Steve mentions VXLAN and here’s where we see those concepts. The OvS can either intercept and respond to DHCP traffic, or just let it through. If we allow OvS to intercept the traffic this means Acropolis and Prism now become the point of control for giving out IP addresses to VMs that boot up. Very cool!

Now let’s take a look at the config parameters passed to the running CVM. Right now this box has ONLY the CVM running on it so only one instance of qemu-kvm running.

[root@DRM-3060-G4-1-1 ~]# ps -ef | grep qemu
qemu 9250 1 61 Jun26 ? 1-21:13:21 /usr/libexec/qemu-kvm -name NTNX-DRM-3060-G4-1-1-CVM 
-S -enable-fips -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,mem-merge=off -cpu host,+kvm_pv_eoi -m 24576 -realtime mlock=on -smp 8,sockets=8,cores=1,threads=1 
-uuid 1323cbbc-a20d-d66a-563e-ca7a8609cb73 -no-user-config -nodefaults 
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/NTNX-DRM-3060-G4-1-1-CVM.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control 
-rtc base=utc -no-shutdown -boot menu=off,strict=on 
-kernel /var/lib/libvirt/NTNX-CVM/bzImage -initrd /var/lib/libvirt/NTNX-CVM/initrd 
-append init=/svmboot quiet console=ttyS0,115200n8 
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 
-netdev tap,fd=20,id=hostnet0,vhost=on,vhostfd=21 
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9c:d8:cd,bus=pci.0,addr=0x3 
-netdev tap,fd=22,id=hostnet1,vhost=on,vhostfd=23 
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:be:99:b3,bus=pci.0,addr=0x4 
-chardev file,id=charserial0,path=/tmp/NTNX.serial.out.0 
-device isa-serial,chardev=charserial0,id=serial0 
-vnc -vga cirrus 
-device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x5,rombar=0 
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

Maybe a better way to look at the CVM details is via the XML configuration:

[root@DRM-3060-G4-1-1 ~]# cat /etc/libvirt/qemu/NTNX-DRM-3060-G4-1-1-CVM.xml
<domain type='kvm'>
 <memory unit='KiB'>25165824</memory>
 <currentMemory unit='KiB'>25165824</currentMemory>
 <vcpu placement='static'>8</vcpu>
 <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
 <cmdline>init=/svmboot quiet console=ttyS0,115200n8</cmdline>
 <boot dev='hd'/>
 <bootmenu enable='no'/>
 <apic eoi='on'/>
 <cpu mode='host-passthrough'>
 <clock offset='utc'/>
 <controller type='usb' index='0'>
 <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
 <controller type='ide' index='0'>
 <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
 <controller type='pci' index='0' model='pci-root'/>

##Here is the CVM interface being linked to the br0 OvS interface
 <interface type='bridge'>
 <mac address='52:54:00:9c:d8:cd'/>
 <source bridge='br0'/>
 <virtualport type='openvswitch'>
 <parameters interfaceid='7a0a4887-b8cd-4f02-960d-cca5c1ca73cc'/>
 <model type='virtio'/>
 <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

##Here is the CVM interface going to the local link bridge.
 <interface type='network'>
 <mac address='52:54:00:be:99:b3'/>
 <source network='NTNX-Local-Network'/>
 <model type='virtio'/>
 <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

 <serial type='file'>
 <source path='/tmp/NTNX.serial.out.0'/>
 <target port='0'/>
 <console type='file'>
 <source path='/tmp/NTNX.serial.out.0'/>
 <target type='serial' port='0'/>
 <input type='mouse' bus='ps2'/>
 <input type='keyboard' bus='ps2'/>
 <graphics type='vnc' port='-1' autoport='yes' listen=''>
 <listen type='address' address=''/>
 <model type='cirrus' vram='16384' heads='1'/>
 <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
 <hostdev mode='subsystem' type='pci' managed='yes'>
 <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
 <rom bar='off'/>
 <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
 <memballoon model='virtio'>
 <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

We saw a new reference in that last command, NTNX-Local-Network. If we look at virsh for information about defined networks we see the following:

[root@DRM-3060-G4-1-1 ~]# virsh net-list --all 
 Name               State  Autostart Persistent
 NTNX-Local-Network active yes       yes
 VM-Network         active yes       yes

If we look in the /root/ partition there are definitions for these:

[root@DRM-3060-G4-1-1 ~]# cat net-NTNX-Local-Network.xml 
<network connections='1'>
 <bridge name='virbr0' stp='off' delay='0' />
 <ip address='' netmask=''>

[root@DRM-3060-G4-1-1 ~]# cat net-VM-Network.xml 
<network connections='1'>
 <forward mode='bridge'/>
 <bridge name='br0' />
 <virtualport type='openvswitch'/>
 <portgroup name='VM-Network' default='yes'>

These two pieces of information tie everything together neatly for us. The internal network given to the CVM is the linux virbr0 device. The external network given to the CVM is OvS br0.

Now I think I finally understand that image presented at the beginning!

CVM Guest Configuration

Since we understand the KVM/AHV host configuration lets take a look in the CVM guest. This should be a little easier.

nutanix@NTNX-15SM60140129-A-CVM:$ netstat -r
Kernel IP routing table
Destination Gateway Genmask         Flags MSS Window irtt Iface * U     0 0 0           eth1 *   U     0 0 0           eth1  *   U     0 0 0           eth0
link-local  *     U     0 0 0           eth0
link-local  *     U     0 0 0           eth1
default      UG    0 0 0           eth0

The routing table shows the internal and external networks, and just two network adapters. The eth1 adapter has a subinterface. This one is eth1:1 as opposed to .1. Not entirely sure what that one means – but I’ll keep it in mind in case I come across something later on.

nutanix@NTNX-15SM60140129-A-CVM:$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 52:54:00:9C:D8:CD 
  inet addr: Bcast: Mask:

eth1 Link encap:Ethernet HWaddr 52:54:00:BE:99:B3 
  inet addr: Bcast: Mask:

eth1:1 Link encap:Ethernet HWaddr 52:54:00:BE:99:B3 
  inet addr: Bcast: Mask:

lo Link encap:Local Loopback 
  inet addr: Mask:

That’s it – just two simple interfaces in the CVM. One for internal traffic to the hypervisor directly, another for receiving any external requests from remote CVMs, the management APIs, and all of the other magic that the CVM performs!

This concludes our walkthrough of networking inside a Nutanix AHV machine. I hope you learned as much as I did going through these items! Please comment or reach out to me directly if you have any questions.

CYB3RCRIM3 Reading

Call me crazy, but I really love reading legal analysis at CYB3RCRIM3. I discovered the site via an OPML file passed to me by a friend in the security industry. Thanks Devon!

I can see how the walls of text might not be appealing, but the material is great. Legal writing is just so different; almost formulaic and with zero emotion. If you can get over the hump of reading court case opinions I think the subject matter is important to everyone. True human computer interaction… Here is where your technology comes into play to exonerate or incriminate you. In this particular instance we’ll say it’s where the rubber meets the road. (Sorry)


Here’s a great example using a car data recorder where the defendant is appealing on a technicality that I won’t pretend I’m fully behind. I LOVE reading between the lines though:

This driver was a huge dick and went out of their way to brake check someone. The data recorder in the car backs this up. This data is being used against the driver for a conviction (along with other testimony) and there is no way to contest these basic facts.

Are you aware that your car is spying on you? That your brake and throttle application along with your speed will be analyzed in full detail if you’re ever in an incident that goes to court? They don’t advertise that on the TV commercials for new cars. Do you know which car manufacturers record what data and for how long? Do you know how reliable that recorder is? Can it be hacked? Can I make it say I was going 1 mph with the gas and brake fully applied at all times (even when stopped)?

I’m not saying cars shouldn’t do this tracking after a crash. I’m also not saying the driver shouldn’t have been convicted. It seems like the justice system worked like it should have. I just like to know ahead of time how it is I’m being tracked and monitored. Personally, I think I’d opt out of this recording technology if it was possible to do so.

Thanks CYB3RCRIM3 for calling out the interesting cases!

I thought I was done writing – but just look at the next post involving a gang member and the forfeit of passwords. As a condition of probation this guy had to surrender all passwords to all social media sites. I didn’t know it was even possible for the state to make that demand. Now I know! How would you feel about that if it happened to you?