Logo

Well folks last night I wrote a post about the insight out of the ESX and VC communications. Now I thought let me also put some commercial stuff which will help people to differentiate in between the various flavors of ESX.

Can we call it as flavours... well if you ask a Open Source freak and geek you will get the same answer.

So for me all of the different ESX availability package is something like flavors of it.

Now let me come up with the original topic.

So, I have seen people are very confused about the licensing package in between the various different bottle. Basically Vmware has divided the packaging of their into four flavor now with this new release of vSphere 4.0. Those are as below:

1. Standard
2. Advanced
3. Enterprise
4. Enterprise Plus


Well there are differentiation in between these packaging. Better if you guys can look at the below snap which talked about the differentiation in between the various packaging.





So this talked about the Enterprise Plus too (a new packaged baby). Cool features Vmware added to this brunch are something like this:

> Host Profiles
> DVS (Distributed Virtual Switch)
> vShield Zones
> Vmware Data Recovery
> Thin Provisioning


So now people has a full plate to grasp. Guys pleas hurry. Offer valid till stock lasts. :)

Well Well Well !!!!!

Guys am back from a long vacation (well to me the vacation is the packaging a new thing). Well we can talk about my next venture definitely some time.

Now people were asking me what are you doing these days and I said folks gimme some time so that I can come up with the best shot as usual.

Now I thought letz post this twist. I was bogged down with this since a long time and now got all the big shots in my hand.

Well sorry about a late post. I was completely busy with my new packaging and product and also with the vCenter Orchestrator training.

Now let me start on this. This is pretty big one but a cool stuff.

Issue is as below:
****************
Intermittent ESX host disconnect alarms from the Virtual Center. Typical disconnect/reconnect times are less than 5 minutes.


Action Plan Taken as below:
*************************
Verified/Validate all physical network connections have specific speed/duplex settings. Verified physical switch ports are not showing signs of errors. Verified utilization of switch ports are below maximums. Virtual Center management configuration is per specification. ESX Host servers are configured to matching specifications. Network trace captures have been collected and analyzed.


Action needed finally:
********************
Need to answer the questions below surrounding proper application responses, network transmission patterns, and general what the network patterns should be, and how the disconnect alarms are triggered.


Open Questions:
***************
1. How exactly does the ESX Host determine that it is no longer communicating with the VC server?
2. What is the exact process that restarts ‘hostd’ or other services to re-establish ESX to VC communications?
3. What is the timeline of ESX Host determining there is an issue and the restart of the ‘hostd’ process to re-establish VC communications?
4. What is the normal timeline for VC to ESX Host heartbeats?
5. What is the normal timeline for ESX to VC Heartbeats?
6. Why is there a small and large UDP 902 packet from each ESX host?
7. Why is the small UDP 902 packet always 71 bytes?
8. Why is there variation in the order of the large/small pattern?
9. Why do different ESX hosts use a different large UDP 902 size?
10. Why does the size of the large UDP 902 packet for a given ESX host change?
11. Does the TCP 902 traffic from the VC server play any role in the hostd restart?
12. Why do we see gaps as long as 3,384 seconds with no VC to ESX TCP 902 traffic?


Backup Information provided on the next few pages showing the analysis of the network captures.



The two baseline traces of “normal ESX traffic” provided some interesting data for consideration.

Trace 4506 A:
• 2-hour trace taken on 4/24 from 8:50 AM to 10:50 AM
• Contains UDP/TCP 902 traffic for 11 ESX hosts

Trace 4506 B:
• 2-hour trace taken on 4/24 from 8:50 AM to 10:50 AM
• Contains UDP/TCP 902 traffic for 8 ESX hosts

Here is a summary of the UDP/TCP 902 characteristics seen in these baseline traces, sorted by ESX host IP address:

19 ESX hosts were seen in the two trace files as expected:






Here are some observations for the UDP 902 traffic:

• The UDP 902 traffic is one-way from the ESX host to the VC server as expected
• The delta time between UDP 902 heartbeats is very consistent with no gaps found in either trace file
• The minimum delta is consistently 10.0 seconds and the maximum observed delta was 10.9 seconds
• Each ESX host sends UDP 902 packets with the same two sizes, a large packet and a small packet
• The small UDP 902 packet was always 71 bytes for all 19 ESX hosts
• The large UDP 902 packet varies from 320 bytes minimum to 1,490 bytes maximum, but each host only uses one large size in these traces
• The typical pattern is to alternate large and small UDP 902 packets, but we do see some exceptions as shown below:

Note the double small and double large UDP 902 packets highlighted below:





Here are some follow-up questions for the UDP 902 traffic:

• Why is there a small and large packet from each ESX host?
• Why is the small packet always 71 bytes?
• Why is there variation in the order of the large/small pattern?
• Why do different ESX hosts use a different large size?



For the earlier trace with the ESX host disconnect, we see an interesting pattern of UDP 902 packet sizes:

• For 1,910 seconds from the start of the trace to the gap in UDP 902 traffic, we see a pattern of 71 bytes / 75 bytes
• For 411 seconds during the gap we see no UDP 902 traffic
• For 265 seconds just after the gap, we see a pattern of 71 bytes / 125 bytes
• For 1,015 seconds to the end of the trace, we see a pattern of 71 bytes / 190 bytes

The baseline trace A shows this ESX host sending a pattern of 71 bytes / 710 bytes. We should try to understand the reason for the different sizes.

Note the changes in UDP 902 packet sizes around the gap in traffic:






Here are some observations for the TCP 902 traffic:

• The TCP 902 traffic is initiated by the VC server and is bi-directional as expected
• The number of TCP 902 connections per ESX host varies from 2 to 8 during the 2-hour trace
• There is a wide variation of packets and bytes per ESX host
• There are large time gaps (“max delta” on chart) with no TCP 902 traffic from the VC server

Here are some follow-up questions for the TCP 902 traffic:

• In the earlier “ESX host disconnect” trace, there was a ~310 second gap from the last VC server TCP 902 packet and the syslog message showing a hostd restart
• If the absence of TCP 902 traffic is triggering the hostd restart, the timeout value should be less than 310 seconds
• Note that during the baseline traces, every ESX host had gaps larger than 310 seconds with no triggered hostd restart ß Max gap was 3,384 seconds

We should try to understand how hostd decides to trigger a service restart. The baseline traces make it hard to see how the TCP 902 traffic could be the trigger.



Now this is the time to answer these questions. Look below for the answers.

1. How exactly does the ESX Host determine that it is no longer communicating with the VC server?

Ans. When the ESX Host does not respond back within ~20 seconds from the VC initiated Hostsync process.

2. What is the exact process that restarts 'hostd' or other services to re-establish ESX to VC communications?

Ans. The vmware-watchdog process.

3. What is the timeline of ESX Host determining there is an issue and the restart of the 'hostd' process to re-establish VC communications?

Ans. It checks every minute for hostd to be running, -u 60, and if it finds that it's not running after 5 failed checks, -q 5, it will restart hostd.

4. What is the normal timeline for VC to ESX Host heartbeats?

Ans. Heartbeat is initiated from the ESX Host.

5. What is the normal timeline for ESX to VC Heartbeats?

Ans. Every 10 seconds.

6. Why is there a small and large UDP 902 packet from each ESX host?

Ans. The message sent to VC from vpxa is a serialization of VpxHeartbeatMsg object. However, it's size depends on the quick status size. In the smallest situation, no quick status needs to be sent. The quick status depends on the machine status such as HA.

7. Why is the small UDP 902 packet always 71 bytes?

Ans. See answer to 6 above.

8. Why is there variation in the order of the large/small pattern?

Ans. Depends on running time status of ESX. In case a host keeps adding new VMs or removing new VMs, it will have more status data.

9. Why do different ESX hosts use a different large UDP 902 size?

Ans. It depends on running state of the ESX host.

10. Why does the size of the large UDP 902 packet for a given ESX host change?

Ans. It depends on running state of the ESX host.

11. Does the TCP 902 traffic from the VC server play any role in the hostd restart?

Ans. TCP traffic from VC is used to transport remote RDP call, (in our term, VMOMI call). It does not play a role.

12. Why do we see gaps as long as 3,384 seconds with no VC to ESX TCP 902 traffic?

Ans. When ESX has no status change for example, all VMs are powered off, and no status change on ESX, (both Host or VMs), there is no sync needed between VC and ESX. The only traffic will be udp traffic to VC server to indicate that the Host is alive.

Folks,

Since a long time I was thinking of doing something different. Now this is the time I thought the different that would be best from my side as a versatile attempt is to write something on Microsoft Hyper-V.

So lets look at it how it is making progress and a small intro.

Microsoft Hyper-V Server 2008 provides a simplified, reliable, and optimized virtualization solution, enabling improved server utilization and reduced costs. Since Hyper-V Server is a dedicated stand-alone product, which contains only the Windows Hypervisor, Windows Server driver model and virtualization components, it provides a small footprint and minimal overhead. It easily plugs into customers’ existing IT environments, leveraging their existing patching, provisioning, management, support tools, processes, and skills.

IT Pros can easily to leverage their existing knowledge and skills with Microsoft virtualization products, as well as the collective knowledge of the community, minimizing any learning curve. In addition, with Microsoft providing comprehensive support for Microsoft applications and heterogeneous guest operating systems, customers can virtualize with confidence and peace of mind.

Key Benefits

• Improved server utilization

• Small footprint

• Minimal overhead

When to Use Hyper-V Server 2008

Microsoft Hyper-V Server 2008 is a great choice for customers who want a basic and simplified virtualization solution for consolidating servers as well as for development and test environments. Hyper-V Server 2008 only offers the most basic of virtualization features, making it ideal for:

• Test and Development

• Basic Server Consolidation

• Branch Office Consolidation

• Hosted Desktop Virtualization (VDI)

Customers who require richer and more robust virtualization features, such as Quick Migration, multi-site clustering, large memory support (greater than 32 GB of RAM), and more than four processors on the host server, should use Windows Server 2008. Windows Server 2008 provides business continuity, disaster recovery, greater scalability for consolidating large workloads, and flexible and cost-effective virtualization rights (one free virtual instance for Standard Edition, four free virtual instances for Enterprise Editions, and unlimited virtual instances for Datacenter Edition with the purchase of a license of Windows Server 2008).

The following table outlines which Hyper-V–enabled product would suit your needs:

Virtualization Needs	Microsoft Hyper-V Server 2008	Windows Server 2008 Standard	Windows Server 2008 Enterprise	Windows Server 2008 Datacenter
Server Consolidation
Test and Development
Mixed OS Virtualization (Linux and Windows)
Local Graphical User Interface
High Availability—Clustering
Quick Migration
Large Memory Support (Host OS) > 32 GB RAM
Support for > 4 Processors (Host OS)
Ability to Add Additional Server Roles
Guest Virtualization Rights Included in Host Server License	None—Each Windows Guest VM Requires a License	1 Physical + 1 VM*	1 Physical + 4 VMs*	1 Physical + Unlimited VMs (Free)

Guys,

A heads up for those who want to be on track with the changing of technology space from time to time.

So this post if for all who fell frightened that who is who and what is what in the next release. Also this is a complete detail of on which what is going to be supported.





* Although these products will be compatible with vSphere in 2H09, you should start planning for the upgrade today.


To find out more, please visit the vSphere Upgrade Center at http://www.vmware.com/go/vsphere-upgrade-center.


** vCenter Server Heartbeat 5.5 becomes compatible with vCenter Server 4.0 with the release of Heartbeat 5.5 U1.


*** VMware Consolidated Backup 1.5 becomes compatible with vSphere 4.0 with the release of VCB 1.5 U1.


**** VMware View Manager and VMware View Composer are compatible with ESXi U3 and later.

Hi Friends,

From my personal experiences at the support level I have seen that having the Debugging level to a great extent will always pay us back. So at any time changing the debugging level to the highest will give us a clear picture of what exactly is going on.

So here we go. We know the NFC (Network File Copy) is a new baby in Vmware Virtual Infrastructure world. This has a great potential too at the same time. So this also has a debugging level set from the VPXA and VPXD stand point.

So we need to increase the logging level to the max it is available to get to know what is under table.

Complete the following steps to setup network file copy (nfc) debugging:
Setup verbose logging at the host (/etc/opt/vmware/vpxa/vpxa.cfg):
Add the following to the vpxa.cfg file:

"nfc"
"loglevel" debug <==change loglevel to "debug"; it's "error" by default "/loglevel"


Restart vpxa with the following:
# service vmware-vpxa restart

Setup verbose logging at the VC Server:

To do this, edit the vpxd.cfg file located at %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

WARNING: Make a backup copy of the file before editing.

Edit the file with Notepad or any text editor.
Add the following to vpxd.cfg between the "config" stanza. "nfc" and "log" elements are going to be added as children of the "config" element.

"config"
...
"nfc"
"loglevel"debug"/loglevel"
"/nfc"

"log"
"level"trivia"/level"
"maxfilenum"30"/maxfilenum"
"/log"
...
"/config"


This will setup nfc debug, change the loglevel to trivia and retain 30 Files instead of the default 10. The trace will log SQL statements and connections to help diagnose the issue. The log level must be set to verbose or trivia for this to work.

This cannot all be done in the VI client GUI , so it will have to be done as shown previously.
NOTE: These should be removed when finished (copy the saved file back over them) , to avoid confusion with the GUI setup.

After making the changes restart the VC service (VC server reads vpxd.cfg on startup, so it needs to be restarted).

On ESX 3, VirtualMachine(VM) has one or more VMDK files (extension .vmdk) and one or more flat vmdk files (last characters flat.vmdk ).In some cases, you may corrupt, lose, or accidentally delete your VMDK files. The VMDK’s contain metadata for the flat.vmdk files. Without VMDK’s, you cannot load the flat.vmdk-files.


Consequence: You cannot load the VM in VirtualCenter and cannot start the VM.

There are two ways to recreate the vmdk files (.vmdk):

I. Creating a new virtual disk.

* Create a new virtual disk of the same size as old ones for the "VM A" as vmA1-flat.vmdk which in turn will create a new vmdk descriptor file and refer that file for the old one.

Command:

#vmkfstools -c 8192m /vmfs/volumes/100GB-Tru64-EVA/vmA/vmA1.vmdk
where, 8192MB is the virtual disk size and vmA1.vmdk is the descriptor file for the newly created disk "vmA1-flat.vmdk".

Edit the VMDK’s with a text editor(vi/nano):

This will now refer to the vmA1, but it should refer to you're the old one, vmA. You need to replace the vmA1 descriptor file with the correct file names; that is, vmA.

For example:
# mv vmA1.vmdk vmA.
# vi vmA.vmdk

The file will look like:
# Disk DescriptorFile
version=1
CID=6479ab28
parentCID=ffffffff
createType="vmfs"

# Extent description

RW 4194304 VMFS "vmA1-flat.vmdk"

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.uuid = "60 00 C2 9a dc 6c 31 eb-81 6f f1 a1 ca 2d 7b 37"
ddb.geometry.cylinders = "261"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

Edit the line: RW 4194304 VMFS "vmA1-flat.vmdk" TO RW 4194304 VMFS "vmA-flat.vmdk"

* Save the file.
* Load the VM from the VirtualCenter; that is, Power ON VM A.

2. Creating a new virtual machine.
A. Determine the hard disk sizes of your original VM. (Lets call this VM as “VM A”).
B. Create a new VM (called “VM B”) from VirtualCenter with the same number of hard disks as the old VM, and the exact same sizes.

Steps:

a) Login to VirtualCenter. Select the ESX host.
b) Select "CREATE A NEW VIRTUAL MACHINE". A "New VM wizard" will appear.Select "VM Configuration" as "Typical". Click NEXT.
c) Give a name to the VM; for eg: vmB. Click NEXT.
d) Select the datastore and click NEXT.
e) Select the Guest OS as the VM A "vmA" was installed with. Click NEXT.
f) Select the no. of virtual processors. Click NEXT.
g) Select the memory as required, click NEXT.
h) Create Network Connections as required, click NEXT.I) Select the Virtual disk size same as "VM A". Click NEXT.
j) Click FINISH.


3. After the VM B has been created, use Putty (or a similar tool) to navigate to the ESX server. Then navigate to the location/directory where your VM B is stored. For example:#cd /vmfs/volumes/100GB-Tru64-EVA/vmB


4. Copy all VMDK’s (not the flat ones, but only the metadata files) to VM A directory. The filesize of your META-data files should be a few KB. For example: # cp vmB.vmdk /vmfs/volumes/100GB-Tru64-EVA/vmA/


5. Navigate to VM A directory where the *.VMDK file/s are copied. Edit the VMDK’s with a text editor(vi/nano): They will now refer to the VM B, but they should refer to your old VM A. You should replace the VM B filenames with the correct file names; that is, VM A.

For example:
# mv vmB.vmdk vmA.vmdk
# vi vmA.vmdk

The file will look like:

# Disk DescriptorFile
version=1
CID=6479ab28
parentCID=ffffffff
createType="vmfs"
# Extent description
RW 4194304 VMFS "vmB-flat.vmdk"

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.uuid = "60 00 C2 9a dc 6c 31 eb-81 6f f1 a1 ca 2d 7b 37"
ddb.geometry.cylinders = "261"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

Edit the line:
RW 4194304 VMFS "vmB-flat.vmdk" TO RW 4194304 VMFS "vmA-flat.vmdk"

5. Save the file. 6. Load the VM from the VirtualCenter; that is, Power ON VM A.You should now be able to Power ON the VM A and recover the data.