What is DevOps?

I realize this topic has probably been beaten to death, but I had to put together a presentation for a group of my peers and I thought it might work as a blog post.  Plus, adding it here helps me internalize my thoughts about a topic.  I hope some of it is a useful distillation of the information out there on this huge topic.  If you find it interesting, I highly recommend checking out one of the books I list below.

A Definition:

DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.  It’s also characterized by operations using many of the same techniques as developers.

Think automated infrastructure provisioning.  You’ll frequently hear the phrase “infrastructure as code.”  What that means is that provisioning activities are driven by a recipe that can be treated like a program.  For example, the application Puppet has a concept called “manifests” which are used to create an application and also to determine if the running machines comply to that specification.

The Three Ways:

In “The Phoenix Project,” Gene Kim talks about the Three Ways, methods used to continuously improve IT operations.  These have been taken from manufacturing theories used in many organizations today.  (credit for the images goes to Gene Kim on his https://itrevolution.com/ website)

The First Way


The First Way emphasizes the performance of the entire system.  It also encourages IT to look at Operations as a Customer of Development.  It consists of Dev creating services which are transitioned to Operations and then consumed by the Business.

The Second Way


The Second Way is all about feedback loops.  There should be continuous feedback about the results of the product delivered to Operations by Development.  This enables continuous improvement to be built-in.

The Third Way


The Third Way is about the culture of the organization.  It’s about creating a culture that fosters two things: continual experimentation,  and understanding that repetition and practice is the prerequisite to mastery.  IT can be very resistant to change.  Also failures can result in finger pointing and this can create an “us versus them” environment.  I think this way is probably the hardest to implement, because it can require a real mind shift in the people of the organization.

Common DevOps Practices

Let’s talk about some of the more common practices organizations use to implement a DevOps culture.

Version Control

This is key to the concept I mentioned above around “infrastructure as code.”  You need to have some way to control the configuration of your systems and the best way to do this is some type of version control system.  Many companies are using Git and Github for this, although you might also see systems like svn and cvs.  This is also where products like Puppet and Chef come in, as they provide a way to consume these “recipes” when building and maintaining systems.

Automated Testing

Instrumental in implementing the Second Way, some type of automated testing should be built into an environment so that continual improvements can be realized.  Also, this will help minimize issues creeping into Production.  Some examples of testing frameworks include Pester and Cucumber.  These are both examples of software that is designed to provide BDD, or Behavior-Driven Development.  A good read about what BDD is and why it can help improve your processes and app development is here.  You can also find a good intro into testing methodologies here.


This is almost an obvious one, but the advent of virtualization enabled the implementation of DevOps throughout organizations.  It made it much simpler to deploy systems automatically and based on a configuration described by code.  Systems like containers and Docker have taken this to the next level by abstracting even further from the underlying hardware.  New tools like NSX and network virtualization extend this promise of “infrastructure as code” by allowing Ops to control not only the systems, but also the networks that connect them.

More Reading

Here are some good resources if you want to delve more into the world of DevOps and help improve your environment.

A Rubrik Python Primer


One of my co-workers over at virtuallysober.com recently posted about using Rubrik’s REST API with PowerShell.  As I’ve been working on my Python-fu, I thought I’d piggyback (or steal…) on his idea and do a similar thing with Python.  First, I’ll distill some of the things I’ve learned about consuming RESTful APIs with Python.  Then, I’ll dive into some of the things you might do with our APIs.

RESTful API Primer

I won’t spend a ton of time on this, as there are a lot of good references out there on what a REST API entails.  The first place to start, like most things, is with Wikipedia.  That might be a bit dry, so a less pedantic place to learn about it might be here.  The basics are that you can communicate with a web service using very straightforward commands like GET, PATCH, POST, etc… The big concept is that those commands are stateless, with the command containing all of the information or state to perform the action.  Also, the API will specify something called an endpoint, which is basically a URL that can accept these RESTful commands.  Rubrik makes it nice to determine what those might be by publishing the documentation on the cluster itself:


Talking to a Webserver in Python


First of all, we need a way to talk to the Rubrik.  You can do this in a couple different ways in Python.  The first is the “curl” command which would look something like this:

curl -k -u admin:pass -X GET 'https://<rubrik_ip_address>/api/v1/vmware/vm'

Let’s parse the above command.  We’ve used the “k” flag to bypass an alert about self-signed certs.  Also, we specified the username and password after the “u” flag.  The next thing is we tell the server what HTTP method we’ll use; here we used a GET command.  Then, the actual endpoint is used.  In this case, we’re asking the cluster for a list of all the VMs, which will be returned as an array of key-value pairs.

However, a big problem with this method is that we need to put our password in plaintext in our code.  What if we wanted to create a “token” instead that could be used in other commands.  We need first to get an authorization code from the Rubrik in order to validate our access to the system.  How do we do that?  By hitting another endpoint, of course!

curl -k -u admin:pass -X POST "https://<rubrik_ip_address>/api/v1/session"

The response will be an array containing the session ID, the token, and the User ID.  Then, the token can be extracted from the array and then used in subsequent commands to the system like so:

curl -k -H 'Authorization: Bearer $token_id' -X GET 'https://<rubrik_ip_address>/api/v1/vmware/vm'

The requests library

Curl is one way to access your system, but probably not the most useful.   A better method when you want to use it programmatically is the excellent Requests library in Python.  This is a library that allows your program/script to pass HTTP requests natively and use the data that returns.  The documentation for requests is very good and you can find it here.

Let’s go through a basic example of how you might connect to Rubrik similarly to the above example.  First, we need to import the requests module, then we will create an object that contains the VMs.

import requests
r = requests.get('<rubrik_ip_address>/api/v1/vmware/vm', verify = False, auth =('admin','pass'))

We use the ‘verify = False’ because the system is using a self-signed certificate.  Again, this has the problem of putting the password in the code in plaintext.  We could get around that by encoding the password with the base64 module then passing it into each command.  However, it’s much more useful to authenticate the session and use the token in each of the proceeding commands.

import requests
session = requests.post('<rubrik_ip_address>/api/v1/session', verify = False, auth =('admin','pass'))
session_token = session.json()
authorization = 'Bearer ' + session_token['token']
vm_list = requests.get('<rubrik_ip_address>/api/v1/vmware/vm', verify = False, headers = {'Content-Type': 'application/json', 'Authorization': authorization})
vm_list_json = vm_list.json()

You’ll notice we take the results of the initial POST command and contain them in the ‘token’ object.  Once we’ve done that, we can access values from with that object by referencing the key, in this case our key is ‘token’.  Once we’ve stored the results of our command in the ‘vm_list’ object, we can retrieve information from it by using the same method we retrieved our key – calling keys that are contained within the JSON file.

Learning More

Now, if you’ve read any of my previous posts, you know I’m a relative novice to the world of Python programming.  So, this represents the very basics of connecting to your Rubrik (or any RESTful system, for that matter).  I recommend going into your system and exploring both the documentation and also our explorer, which is based on the Swagger framework.

In future posts, I’ll go into how you might actually use this information in your day-to-day operations and scripts.


Also, if you’d like to learn more about our API and how you might use PowerShell with it, check out my colleague Joshua Stenhouse’s blog at https://virtuallysober.com/2017/05/08/introduction-to-rubrik-rest-apis-using-powershell-swagger/.

Ravello Systems Review

Recently, I was given access to an account at Ravello Systems (full disclosure: this is a free account given to vExperts) and I thought I’d write about my experience. For those of you that don’t know, it’s a front end for deploying workloads in AWS and was bought by Oracle in 2015.

I’ve had an account with them for a while, but really never needed to utilize it due to having some pretty sweet home lab gear provided by my previous job. However, with me going over to Rubrik in March, I obviously had to return that stuff and I’m not sure if I’m going to purchase new gear on my own. It’s just getting so you don’t need a homelab for a lot of things anymore, and tools like Ravello make that possible.

Well, on to the review. The interface is really nice in that it looks like a standard blueprint and has a lot in common with a Visio or Lucidchart drawing. You add components to the design and on the right pane you can configure their settings.


The account comes with various pre-configured VMs, like the one shown above which can be used to install ESXi.  I’m building a vSphere farm in that example.  You do have to provide your own software images and licenses, but they can be uploaded easily.  Once that’s done, you simply connect the ISO to the VM and install ESXi normally.  You can also do some cool things with their import tool, like pulling in running VM images from your existing vSphere environments,  sort of like a V2V converter.

The networking options are fairly robust, as well.  You can configure DHCP or static addresses, as well as control which NICs have external access.

Finally, the entire platform has a REST API available, if you want to automate the provisioning or management of your environments here.  This could be really powerful, as it extends the functionality to any scripts or automation tools you might have.


For a potential homelab / SMB lab use, I think this could be really powerful.  It reduces or eliminates the need to buy gear that will eventually become obsolete (or get taken back by your previous employer!).

A New Adventure

So, yesterday was my first day at Rubrik and I couldn’t be more excited.  I had a great 4 1/2 years at IVOXY in the VAR (value-added reseller, for those of you wondering) space, but figured it was time to go back to an IT vendor.  I couldn’t be more happy so far with the company and the team I’m working with.

If you’re not familiar, Rubrik was founded to look at backup in a different way.  We don’t require difficult installation, or complex policies, or even a full-time backup admin.  With an appliance model, and simple configuration, we can be up and protecting an environment in under an hour.  Let’s see any of the traditional software on hardware options do that!

Anyways, if you want more information about Rubrik and what we do, you can find really good info on our website. Cheers!

Don’t backup, go forward!

Deploying NSX 6.3 in a vSphere 6.5 Environment (Part 2)

Last time, we went through the initial setup steps to get NSX 6.3 deployed in your vSphere 6.5 environment.  Today, we’ll finish all of the initial configuration and get your environment to a place where you can start deploying the tasty bits of NSX, like distributed firewalling.

When you first login with your user, you may see the following error.  This is because you need to explicitly give rights to the NSX installation and NSX Manager.  It is initially only given to the login you used to install the service, generally the administrator@vsphere.local user.


To fix it, login as the user you used to install and go to the Networking and Security section and select NSX Managers:


Once there, select the NSX Manager on the Navigator tab and under Manage select Users.  Add the user you’d like to have access (yourself at least!) to the system and give yourself the appropriate rights.  For my lab, I’ve given myself the Enterprise Administrator role which is the NSX god role.  Then you can log back into your Web Client and you will see the NSX Manager listed and you can continue with configuring NSX.

Select installation under Networking & Security.  Now we’re going to deploy the NSX Controller nodes.  In a production environment, three controllers are deployed for each NSX instance.  As the boys from Monty Python said, “the number shall be three, two is not enough and four shall be right out.”  I’m paraphrasing, obviously, but the gist is that there are exactly 3 NSX Controller nodes in an NSX implementation.  I’m only going to deploy one due to resource limitations, but you can do that in a lab environment.

Click on the green plus symbol to add a controller and fill in the dialog box:


IF you haven’t created any IP Pools yet, you’ll need to do that to continue deploying the controller.


Click on Installation under Networking & Security and then Host Preparation.  Once there, select the cluster you want to install NSX on and under Actions click Install.  As you can see, I’m following VMware’s recommended practice of having a resource and a management cluster.   If you do this in production, you may also have an Edge cluster to hold any Edge devices you deploy.

This is where we install the ESXi VIBs and then complete the configuration of the VXLAN transport network.  If you have problems, as I did, you can follow the steps in this KB: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2075600

I ended up trying to manually install the VIB on each of my hosts using this KB about a problem with vSphere Update Manager: https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2053782.  It still failed, but after patching my ESXi hosts to the latest version using the great instructions Vladan has here I was able to install the VIBs from the GUI.


Once that’s finished, we’ll move on to the creation of the VTEP (VXLAN tunneling endpoint).  This will create a portgroup on the distributed virtual switch that I already created.  Ultimately, it creates a new vmkernel port on each host in that portgroup that the system uses as the VTEP.

You’ll need a larger than standard MTU for this to work correctly, as the VXLAN encapsulation adds bytes to the end of the frame.  The minimum is 1550, although the recommended value is 1600 bytes.  Keep in mind, the underlying network must support the increased value.  When the system asks for IP addresses of the VTEP, I recommend using an IP Pool like we did for the controller(s).

Click on Not Configured under the VXLAN column, choose the VLAN and IP addressing scheme and click Finish:


When it’s done, you should see the following:


At this point, NSX is configured and operating on your cluster.  You can see the portgroup created for the VXLAN traffic by going to Networking and selecting the portgroup on your distributed virtual switch:


Next time, I’ll dive deeper into some of the cooler features of NSX 6.3 and what things you might deploy initially to justify the money you’ll have to spend for the licensing!

Protecting Children with Technology

I’m sure a lot of you have seen the recent testimony by Ashton Kutcher in front of the Senate concerning child exploitation.  If you haven’t, you can see it here: https://www.youtube.com/watch?v=HsgAq72bAoU

It really moved me and got me thinking about what a wonderful use of technology his Thorn project is.  I’ve donated some money and also offered up my services around data center infrastructure.  They’re looking for all different types of technologists, but especially coders who can help disrupt the online activities of these predators.  If you have mad skills in these areas, I highly recommend checking them out and maybe donating some of your time and knowledge to an amazing project.

Here is the link to their site: https://www.wearethorn.org

Deploying NSX 6.3 in a vSphere 6.5 Environment (Part 1)

Deploying NSX 6.3 in a vSphere 6.5 Environment (Part 1):

Today, I’m going to go through the steps for deploying NSX 6.3 in my homelab.  My homelab consists of 4 Intel NUCs, with one of those running in a management cluster.  I only have 1 NIC per host at this point, but I’m thinking of adding a USB NIC to each of the systems in the VM cluster, to give me some more flexibility around networking.  Specifically, I’d like to have a standard switch that has the host management vmkernel address, so I can muck with the networking without taking my hosts offline.

I’m running a mixed environment of vCenter 6.5 and ESXi 6.0 at home and had previously deployed NSX but had to rip it out to upgrade to 6.5.  It was an interesting experience removing NSX and one that you’d probably never do in a production environment.  However, it did demonstrate for me how deeply integrated this is once installed.

Here are the initial steps:

Download the OVA from vmware.com.

Deploy the OVA and give your NSX Manager a name:


Pick a place to put it. I’m putting the VM on my VSAN datastore:


Pick a network for the management interface. This isn’t the NSX networks that will be defined later.  It should be a network that can communicate with the vCenter server.  You’ll set an IP address in the next step:


Set the parameters for the NSX manager. You’ll set IP address, hostname and DNS server here.  Also, set the passwords for both the “admin” user and the “privilege” mode of the CLI.  This is much like the enable mode on a Cisco network device.


Once you’ve set these, click next a couple times and then Finish to start the deployment. It will take few minutes.  When the deployment is finished, power on the VM to complete the initial setup.  If you want, you can watch the boot process with the VM Console and when it’s finished you’ll be ready for the next configuration steps.  When the deployment is finished, the NSX Manager VM should show the login prompt:


Next, login to the web interface of the NSX Manager using the “admin” username and the password you setup in the initial OVA screens. Once you’ve logged in you’ll see the NSX Appliance Management page:


From here, click on “Manage vCenter Registration” and input both the Lookup Service URL and vCenter Server addresses. You’ll be asked to accept certificates in both cases. Accept those and this will register the NSX environment with your vCenter installation.



Also, make sure your NTP settings are correct, by clicking the Manage Appliance Settings button on the home page.  NTP and time in general is VERY important for things like SSO and SSL to work correctly.  One piece of advice I have is that if you’re ever having issues with services not working correctly, or login issues, check the time first.


If all went well, you should see Connected and nice, green circles (and all sysadmins have a Pavlovian desire to see green circles, don’t we!).


Now, you should be able to login to your vSphere Web Client and see the NSX icon showing up as Network and Security on the Home Page. If you don’t see it, logout and log back in.  We all know how much the Web Client likes a Refresh!  Unfortunately, this is one of those areas that isn’t supported in the new HTML5 client, but hopefully that will change in the future as VMware rolls more functionality into that client and eventually (one can hope!) moves us into a Flash-free future.


That completes the initial deployment of the NSX manager.  Your NSX Manager is deployed and registered, but there are a few more steps we need to complete in order to have a fully functional SDN solution.  Next time, we’ll go through the initial configuration of the application, including host preparation and creating the networking requirements.  Also, we’ll need to apply licensing at some point to the installation.