What is DevOps?

I realize this topic has probably been beaten to death, but I had to put together a presentation for a group of my peers and I thought it might work as a blog post.  Plus, adding it here helps me internalize my thoughts about a topic.  I hope some of it is a useful distillation of the information out there on this huge topic.  If you find it interesting, I highly recommend checking out one of the books I list below.

A Definition:

DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.  It’s also characterized by operations using many of the same techniques as developers.

Think automated infrastructure provisioning.  You’ll frequently hear the phrase “infrastructure as code.”  What that means is that provisioning activities are driven by a recipe that can be treated like a program.  For example, the application Puppet has a concept called “manifests” which are used to create an application and also to determine if the running machines comply to that specification.

The Three Ways:

In “The Phoenix Project,” Gene Kim talks about the Three Ways, methods used to continuously improve IT operations.  These have been taken from manufacturing theories used in many organizations today.  (credit for the images goes to Gene Kim on his https://itrevolution.com/ website)

The First Way


The First Way emphasizes the performance of the entire system.  It also encourages IT to look at Operations as a Customer of Development.  It consists of Dev creating services which are transitioned to Operations and then consumed by the Business.

The Second Way


The Second Way is all about feedback loops.  There should be continuous feedback about the results of the product delivered to Operations by Development.  This enables continuous improvement to be built-in.

The Third Way


The Third Way is about the culture of the organization.  It’s about creating a culture that fosters two things: continual experimentation,  and understanding that repetition and practice is the prerequisite to mastery.  IT can be very resistant to change.  Also failures can result in finger pointing and this can create an “us versus them” environment.  I think this way is probably the hardest to implement, because it can require a real mind shift in the people of the organization.

Common DevOps Practices

Let’s talk about some of the more common practices organizations use to implement a DevOps culture.

Version Control

This is key to the concept I mentioned above around “infrastructure as code.”  You need to have some way to control the configuration of your systems and the best way to do this is some type of version control system.  Many companies are using Git and Github for this, although you might also see systems like svn and cvs.  This is also where products like Puppet and Chef come in, as they provide a way to consume these “recipes” when building and maintaining systems.

Automated Testing

Instrumental in implementing the Second Way, some type of automated testing should be built into an environment so that continual improvements can be realized.  Also, this will help minimize issues creeping into Production.  Some examples of testing frameworks include Pester and Cucumber.  These are both examples of software that is designed to provide BDD, or Behavior-Driven Development.  A good read about what BDD is and why it can help improve your processes and app development is here.  You can also find a good intro into testing methodologies here.


This is almost an obvious one, but the advent of virtualization enabled the implementation of DevOps throughout organizations.  It made it much simpler to deploy systems automatically and based on a configuration described by code.  Systems like containers and Docker have taken this to the next level by abstracting even further from the underlying hardware.  New tools like NSX and network virtualization extend this promise of “infrastructure as code” by allowing Ops to control not only the systems, but also the networks that connect them.

More Reading

Here are some good resources if you want to delve more into the world of DevOps and help improve your environment.

Deploying NSX 6.3 in a vSphere 6.5 Environment (Part 2)

Last time, we went through the initial setup steps to get NSX 6.3 deployed in your vSphere 6.5 environment.  Today, we’ll finish all of the initial configuration and get your environment to a place where you can start deploying the tasty bits of NSX, like distributed firewalling.

When you first login with your user, you may see the following error.  This is because you need to explicitly give rights to the NSX installation and NSX Manager.  It is initially only given to the login you used to install the service, generally the administrator@vsphere.local user.


To fix it, login as the user you used to install and go to the Networking and Security section and select NSX Managers:


Once there, select the NSX Manager on the Navigator tab and under Manage select Users.  Add the user you’d like to have access (yourself at least!) to the system and give yourself the appropriate rights.  For my lab, I’ve given myself the Enterprise Administrator role which is the NSX god role.  Then you can log back into your Web Client and you will see the NSX Manager listed and you can continue with configuring NSX.

Select installation under Networking & Security.  Now we’re going to deploy the NSX Controller nodes.  In a production environment, three controllers are deployed for each NSX instance.  As the boys from Monty Python said, “the number shall be three, two is not enough and four shall be right out.”  I’m paraphrasing, obviously, but the gist is that there are exactly 3 NSX Controller nodes in an NSX implementation.  I’m only going to deploy one due to resource limitations, but you can do that in a lab environment.

Click on the green plus symbol to add a controller and fill in the dialog box:


IF you haven’t created any IP Pools yet, you’ll need to do that to continue deploying the controller.


Click on Installation under Networking & Security and then Host Preparation.  Once there, select the cluster you want to install NSX on and under Actions click Install.  As you can see, I’m following VMware’s recommended practice of having a resource and a management cluster.   If you do this in production, you may also have an Edge cluster to hold any Edge devices you deploy.

This is where we install the ESXi VIBs and then complete the configuration of the VXLAN transport network.  If you have problems, as I did, you can follow the steps in this KB: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2075600

I ended up trying to manually install the VIB on each of my hosts using this KB about a problem with vSphere Update Manager: https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2053782.  It still failed, but after patching my ESXi hosts to the latest version using the great instructions Vladan has here I was able to install the VIBs from the GUI.


Once that’s finished, we’ll move on to the creation of the VTEP (VXLAN tunneling endpoint).  This will create a portgroup on the distributed virtual switch that I already created.  Ultimately, it creates a new vmkernel port on each host in that portgroup that the system uses as the VTEP.

You’ll need a larger than standard MTU for this to work correctly, as the VXLAN encapsulation adds bytes to the end of the frame.  The minimum is 1550, although the recommended value is 1600 bytes.  Keep in mind, the underlying network must support the increased value.  When the system asks for IP addresses of the VTEP, I recommend using an IP Pool like we did for the controller(s).

Click on Not Configured under the VXLAN column, choose the VLAN and IP addressing scheme and click Finish:


When it’s done, you should see the following:


At this point, NSX is configured and operating on your cluster.  You can see the portgroup created for the VXLAN traffic by going to Networking and selecting the portgroup on your distributed virtual switch:


Next time, I’ll dive deeper into some of the cooler features of NSX 6.3 and what things you might deploy initially to justify the money you’ll have to spend for the licensing!

Deploying NSX 6.3 in a vSphere 6.5 Environment (Part 1)

Deploying NSX 6.3 in a vSphere 6.5 Environment (Part 1):

Today, I’m going to go through the steps for deploying NSX 6.3 in my homelab.  My homelab consists of 4 Intel NUCs, with one of those running in a management cluster.  I only have 1 NIC per host at this point, but I’m thinking of adding a USB NIC to each of the systems in the VM cluster, to give me some more flexibility around networking.  Specifically, I’d like to have a standard switch that has the host management vmkernel address, so I can muck with the networking without taking my hosts offline.

I’m running a mixed environment of vCenter 6.5 and ESXi 6.0 at home and had previously deployed NSX but had to rip it out to upgrade to 6.5.  It was an interesting experience removing NSX and one that you’d probably never do in a production environment.  However, it did demonstrate for me how deeply integrated this is once installed.

Here are the initial steps:

Download the OVA from vmware.com.

Deploy the OVA and give your NSX Manager a name:


Pick a place to put it. I’m putting the VM on my VSAN datastore:


Pick a network for the management interface. This isn’t the NSX networks that will be defined later.  It should be a network that can communicate with the vCenter server.  You’ll set an IP address in the next step:


Set the parameters for the NSX manager. You’ll set IP address, hostname and DNS server here.  Also, set the passwords for both the “admin” user and the “privilege” mode of the CLI.  This is much like the enable mode on a Cisco network device.


Once you’ve set these, click next a couple times and then Finish to start the deployment. It will take few minutes.  When the deployment is finished, power on the VM to complete the initial setup.  If you want, you can watch the boot process with the VM Console and when it’s finished you’ll be ready for the next configuration steps.  When the deployment is finished, the NSX Manager VM should show the login prompt:


Next, login to the web interface of the NSX Manager using the “admin” username and the password you setup in the initial OVA screens. Once you’ve logged in you’ll see the NSX Appliance Management page:


From here, click on “Manage vCenter Registration” and input both the Lookup Service URL and vCenter Server addresses. You’ll be asked to accept certificates in both cases. Accept those and this will register the NSX environment with your vCenter installation.



Also, make sure your NTP settings are correct, by clicking the Manage Appliance Settings button on the home page.  NTP and time in general is VERY important for things like SSO and SSL to work correctly.  One piece of advice I have is that if you’re ever having issues with services not working correctly, or login issues, check the time first.


If all went well, you should see Connected and nice, green circles (and all sysadmins have a Pavlovian desire to see green circles, don’t we!).


Now, you should be able to login to your vSphere Web Client and see the NSX icon showing up as Network and Security on the Home Page. If you don’t see it, logout and log back in.  We all know how much the Web Client likes a Refresh!  Unfortunately, this is one of those areas that isn’t supported in the new HTML5 client, but hopefully that will change in the future as VMware rolls more functionality into that client and eventually (one can hope!) moves us into a Flash-free future.


That completes the initial deployment of the NSX manager.  Your NSX Manager is deployed and registered, but there are a few more steps we need to complete in order to have a fully functional SDN solution.  Next time, we’ll go through the initial configuration of the application, including host preparation and creating the networking requirements.  Also, we’ll need to apply licensing at some point to the installation.