Step 4: Virtualization engineer controls Day 2 Ops for the Full Stack
Network Admins calm down and look here.
The network admins are not usually in favor of anything that replaces their day to day jobs. If you can show them the finished solution, you may be able to sway them in favour. Make sure to give them a demo of the OMNI plugin running inside vSphere using VxRail SmartFabric enabled switches.
Click into the Omni dashboard.
The OMNI dashboard displays the current information on the VxRail SmartFabric enabled Switches. Here the Virtualization engineer has visibility to the OS10 version and the VLANS currently setup. That’s it? Not sure what NetAdmin Ned was expecting but there is no Network-Chaos-Now button!
Lets give Ned a use-case for SmartFabrics
You can describe a scenario to Ned that is common. It’s time to add a new host to the existing cluster. Normally you would ask Ned to prepare the host network ports. He would label them add all the required VLANS and to make sure the settings match with the existing host ports.
With SmartFabrics no need to bother Ned
Adding hosts to HCI clusters are now done solely by the Virtualization Admin without any need to ask the NetAdmin to prepare host ports. It is all taken care of automatically by the VxRail SmartFabric services and the OMNI plugin.
At this point we have already run the first two steps to get SmartFabric installed, our Switches have SmartFabric services enabled and the VxRail Manager Gui deployment has automatically configured the switch and disabled the CLI. Now we will add the Open Manage Network Integration or SmartFabric OMNI plugin to vSphere, giving the virtualization engineer full visibility and control of the dedicated HCI switch fabric.
Download the OMNI plugin VM
We begin by deploying the OMNI VM ova, which is available from VMware Solution Exchange here. It seems to bounce you now to the Dell Support site where you can get the latest version here. As of June 2019 the latest version is 1.1.18.
Deploy the OVA
These steps are straightforward. Isn’t great that we can do the OVA deployment thru the HTML5 interface? Be sure to match up the source and destination networks correctly at Step 7 in the process. Later on we will configure an ipv4 address on vCenter network and enable ipv6 on the MGMT network.
Power on the VM and connect to a console on the OMNI vm. You need to establish a password here for the admin user. The default username and default password is setup as admin / admin.
Configure the OMNI appliance
Since this is a first run deployment, we are going to select option “0. Full Setup” at the Menu. As you can see you can re-run and do other admin tasks later if required from this interface.
Before we active the first connection, we will setup the Profile name “vCenter Server Network” and assign a valid ipv4 address for the OMNI VM here.
The second connection is for ipv6 discovery on the VxRail Management network, no ipv4 address needed here, set to disabled.
Set IPv6 Config to “Link-Local”.
EXTRA STEP Added Feb. 2020.
Navigate to Routing and select Edit. Select Add.
For “Destination/Prefix” enter “fde1:53ba:e9a0:cccc::/64”.
Leave “Next Hop” empty, Leave”Metric” default of 0. Click OK
Be sure to activate both networks on the last screen.
You will be prompted at the CLI again for NTP, SSL Cert, the appliance ip address, FQDN for VCSA, VC Username and Password, and finally the OMNI ip for REST and REST username and password.
Log in and out of vSphere and the OMNI plugin will appear.
Connect the VxRail nodes to deploy SmartFabric enabled switches. Power on the nodes. Connect the laptop to the first port(port 1 only!) in the SFS switch. This is the “jump port” and will allow access to the default ip address [https://192.168.10.200] of the VxRail Manager VM. The VxRail Manager VM is already deployed in the appliances in factory.
VxRail will deploy SmartFabric now
Since VxRail version 4.7.100 there is now the ability for the GUI driven VxRail Manager install to detect Dell EMC SmartFabric enabled switches. This automates configuration of the switches during the full stack deployment.
Setup the REST user to deploy SmartFabric
When the installer selects to deploy SmartFabric switches the next step will be to configure the Rest User account on the SFS enabled switch. If its a first run, a new password will be created. If you are setting up a second VxRail cluster into an existing SmartFabric, then the current password will be requested. The Rest User account will be documented in the Pre-Install Checklist by the installer.
New option to choose 2-Node cluster
The ability to deploy 2-node clusters is now an option with VxRail 4.7 versions. The GUI install gives you a choice for 2-Node here. For all other VxRail deployments the default option is selected.
The initial cluster deployment needs to be 3 or 4 nodes. Normally the node with lowest serial number wins the VxRail Manager election process. This is signified by the small “house” icon. If you don’t see all the powered on nodes, try a reboot of the missing node and wait. It should be detected by ipv6 multicast using the loudmouth service on the discovery VLAN 3939.
Automate the build with JSON
Remember we use the Pre-Install checklist is used to capture all the required information to make a VxRail deployment in a customer Datacenter go perfectly. It can also be used to generate a JSON file to help eliminate human error during the GUI setup.
Here we have used the JSON file to populate all the fields in the GUI install of VxRail. You only need to verify that everything is correct and provide the passwords.
New external MGMT VLAN option
There is a new option during the VxRail 4.7 version of the install that asks for a Management VLAN ID. The default is 0 to allow for Native VLAN to be used, or you can specify a different public management VLAN here. It should be different to the internal management VLAN used to separate the cluster build. The Internal Management VLAN was the -m option set in Step 1. This is very easy with SmartFabric now, as the VxRail Manager will make the necessary port configuration changes on the SFS enabled switches.
VxRail Manager automates the Switch configuration
There is a new task at the end of the GUI install and just before validation. SmartFabric services will now configure all the required VLANS that were specified, including Management, vMotion, vSAN and VM traffic.
Validation eliminates human error
Trust but verify. Even with the SmartFabric automation, VxRail Manager still performs the important task of validating the install. Validation must pass before the actual deployment begins. This will capture anything amiss such as missing DNS entries, IP address conflicts , missing NTP. Basically anything that could prevent the build from completing successfully.
Once validation has passed there is an opportunity to download a copy of the JSON file (useful if Step-By-Step was selected).
It is now time to kick-off the build. This is a fully automated process. There is zero value in manually setting up VSAN. By automating the process Dell EMC can be certain that every customer install of VxRail clusters are fully aligned to best practice. No snowflakes!
I decided to document the steps I used to enable SmartFabric Services on VxRail. These are partly notes for myself and Customer Solution Centre engineers that will likely need to showcase this capability very soon. The demand for this solution is very high and the customers I have met are impressed by what is now possible (wait until you see the roadmap!).
Note: This is not a guide for end user customers because a lot of what I write about is handled thru our automated deployment appliance; VxRail. A note of thanks to Allan Scott from New York CSC that helped with the first SFS deployment and documentation.
SmartFabric is supported on the 4100 series from Dell EMC. Current models are 10G – S4112 F/T, S4128 F/T or S4148 F/T (25G coming soon). Sales can order these switches to be delivered from factory with OS10 and licenses already applied.
If you need the latest version of OS10 – get it here: force10networks.com , request login thru support page and download 10.4.1.x. Put OS10 .bin file and licence .xml file in a USB drive – insert USB drive into switch.
Cabling the TORs
First cable up ports 29 and 30 – 100GB cables for ISL (VLT).
Next cable up ports 25 and 26 – 100GB cables for Uplink.
Plug laptop into port 1 on switch.
Connect new VxRail appliances in any other port starting at port 2.
Installing or Upgrading OS10 on switch:
This is an optional step. The switches can be ordered and configured in factory, and so should arrive ready to begin at Step 2.
Connect the laptop to the serial port on one of the switches and start putty. Putty settings are 115200, 8, stop, none, none. I used a USB serial port so my COM port was COM3.
Powerup the serial connected switch – and break into ONIE mode by hitting ESC during bootup.
Choose “onie-discovery-stop” from the menu.
At prompt type: fdisk -l
USB Thumbdrive should be /dev/sdb1
mount -t vfat /dev/sdb1 /mnt/usb
Check IOS10 Version & Install License:
show license status (skip next step
if already installed)
My showcase lab is using 40GB uplinks rather than 100GB so I needed to change the profile of the uplinks before applying the VxRail SFS personality. You can skip this step if you are using 100GB links.
switch-port-profile 1/1 profile-2
Switch port profile will be applied only after a save and reload. All
management port configurations will be retained but all other configurations
will be wiped out after the reload.
Enable the VxRail personality:
The SFS personality script is included in OS10. Once applied to each ToR switch, the switches will reboot with SmartFabric Mode enabled and you are now ready to perform a VxRail deployment from the VxRail Manager.
Back in October I asked the question, is HCI networking easy? I stand by my assertion that it is already pretty simple once you understand the converged design for HCI does not require separate physical fabrics. Once you setup your required VLANS and appropriate MTU and multicast for IPv6 you are 90% of the way done. So if it’s so easy already, then why am I so excited about SmartFabric for VxRail appliances? Start up a conversation with Ned the network engineer about the automated deployment and simplified life-cycle management capabilities of SmartFabrics and you will get back a blank dull stare. “That’s just a python script” Ned the Network Admin will say, “Take your fancy sales patter down to the Virtualization guys, we don’t want your kind round here!”
It’s not easy to impress a NetAdmin
Ned has a point I
suppose. The Network Admins job is to move packets reliably from one part of
the network to the other, monitor the network for any problems, and to design
and build future networks. The last
thing Ned needs is dealing with end users complaining about network problems
caused by poorly designed applications (its never the networks fault!). This is
actually the main selling point FOR the use of Smart Fabrics. Let me explain
Before you understand why SmartFabric for VxRail, you have to first understand the reason for choosing VxRail appliances in the first place. I have written a few Blogs on this already here and here . In short it’s an engineered solution for vSAN that comes from Dell EMC fully validated and tested and automatically deployed and updated throughout its life-cycle.
VxRail appliances don’t require a Storage expert or Server guru and it even saves the Virtualization admin from having to spend countless hours reading design and deploy documentation. After deployment is done, day 2 simplified operations begins. As a result maintenance and updates and upgrades are made easy through a single bundle file that covers the entire stack (not just the software on top of somebody else’ hypervisor – and includes the hardware too.
Can SmartFabric simplify HCI?
The last part of the
HCI architecture that needed simplification was the network, so Dell EMC has
had its sites on Ned’s cubicle for a while now. If we are going to provide a
fully automated deployment experience for our customers, it only makes sense to
include the Top of Rack switches that are being used by VxRail appliances. After all why would the network admin want to
be responsible for those HCI host ports anyways? If something goes wrong in the
vSAN stack, Ned doesn’t want to be dragged in to a War Room to defend last
weekend’s network changes. Ned knows the changes the network team made at the
weekend were to the core only.
Now with Smart Fabric for VxRail, Ned can still own the core, and leave the HCI network problems to the Virtualization team. SmartFabric will fully configure a redundant TOR fabric for VxRail, and continue to maintain the network for the life-cycle of the HCI solution. When it’s time to patch the HCI network, SmartFabric will provide a bundle file, and perform a non-disruptive rolling upgrade to the network TORs leaving Ned free to watch old episodes of Futurama. If the HCI team needs to expand their existing VxRail cluster by adding a new node, then SmartFabric will fully automate the changes to the TOR switch, no need for Ned to ever get involved.
Is BYO Networking still an Option?
One of the advantages for VxRail customers has been the fact that it is BYON (Bring Your Own Networking). This means that Dell EMC does not force you to take a switch from their portfolio into your datacenter. For some customers, this would be non-negotiable. They may have standardized on a specific brand and prefer to stay that way, no matter what they run at the Storage or Virtualization layer. VxRail networking is compatible with any modern low latent switch and the introduction of VxRail SmartFabric does NOT mean that the BYON option is no longer a choice. Hopefully the automation that comes with SmartFabric for VxRail will entice some customers to converge the entire HCI stack and give Ned some peace of mind.
Here is a great Blog on vSAN and Network Switch choices from @LostSignal on this Blog: https://blogs.vmware.com/virtualblocks/2019/03/21/designing-vsan-networks-2019-update/
Update! This week I will be meeting Hasan Mansur at the Limerick Customer Solution Center who writes a great Dell EMC networking blog at https://hasanmansur.com/ . Hasan has written two great articles there about SmartFabric Services. Please check it out Part 1 here and Part 2 here.
Even though hyper-converged solutions have been one of the hottest trends in the Datacenter since virtualization, you will still meet traditional architects that are seeing this technology for the first time. Many times the customer will come to the conversation with just the virtualization lead, sometimes they will bring the Storage or Compute team, but often they will forget to tell the Networking team any of their plans (no wonder the network engineer can be so grumpy). This can prove problematic for a networking team that is not familiar with a few of the basic HCI requirements. Continue reading “Is HCI networking easy?”