Saturday, October 4, 2014

Whats new in vSphere 5.5 ?

Whats new in vSphere 5.5?
  • Hot-Pluggable SSD PCI Express (PCIe) Devices - ability to hot-add or hot-remove the solid-state disks.
  • Support for Reliable Memory Technology - ESXi runs on Memory, if error occurs, it crashes and VMs too. To protect against memory errors, ESXi takes advantage of hardware vendor enable Reliable memory technology.
  • Enhancements for CPU C-States.
  • Storage - Support for 62TB VMDK- vSphere 5.5 increases the maximum size of a virtual machine disk file (VMDK) to 62TB (note the maximum VMFS volume size is 64TB where the max VMDK file size is 62TB).  The maximum size for a Raw Device Mapping (RDM) has also been increased to 62TB.
  • 16GB E2E support.
  • Expanded vGPU Support - 5.1 was limited to Nvidia - now supports Nvidia & AMD GPUs.
  • Doubled Host-Level Configuration Maximums.
  • 16Gb End-to-End Support – In vSphere 5.5 16Gb end-to-end FC support is now available.  Both the HBAs and array controllers can run at 16Gb as long as the FC switch between the initiator and target supports it.
  • Graphics acceleration now possible on Linux Guest OS.
  • vSphere App HA - works in conjunction with vSphere HA monitoring and VM monitoring to improve application up-time. can be configured to restart application service when issue is detected. Can also reset VM if Application fails to start.
  • For hosts with different CPU vendors in a cluster:
    • Per-Virtual-Machine CPU masking - hide or show NX/XD bit (No Execute / Execute Disable)
    • VMware Enhanced vMotion compatibility - On the hardware side - Intel & AMD put functions in the CPUs that would allow them to modify the CPU ID value returned by the CPUs. Intel calls this functionality as FlexMigration. AMD - embedded this into the AMD-V virtualization extenstions. On Software side, VMware created s/w that takes advantage of this hardware functionality to create a common CPU ID baseline for all servers within the cluster. Introduced in ESX/ESXi 3.5 Update2.

How Storage vMotion works ?

How Storage vMotion works?

Storage vMotion enables live migration of running virtual machine disk files from one storage location to another with no downtime or service disruption.

  • This simplifies storage array migration or storage upgrades.
  • Dynamically optimize storage I/O performance.
  • Efficiently utilize storage and manage capacity.
  • Manually balances the storage load.

Storage vMotion process:

  1. vSphere copies the non-volatile files that make up a VM: vmx, swp, logs & snapshots.
  2. vSphere starts a ghost or shadow VM on the destination datastore. Because the ghost VM does not yet have a virtual disk (that hasn't been copied over yet), it sits idle waiting for its virtual disks.
  3. Storage vMotion first creates a destination disk. Then a mirror device - a new driver that mirrors I/Os between source & destination.
  4. I/O mirroring in place, vSphere makes a single-pass copy of virtual disks from source to destination. As the changes are made to the source, the mirror driver ensures that changes are also reflected at the destination.
  5. When the virtual disk copy completes, vSphere quickly suspends & resumes in order to transfer control over to the ghost VM on the destination datastore.
  6. Files on the source datastore are deleted.

Virtual mode RDM storage vMotion - if you want to migrate only the vmdk mapping file - select “Same Format as source”.

Physical mode RDMs are not affected.

How VMware DRS works ?

How VMware DRS works?

VMware DRS aggregates the computing capacity across a collection of servers and intelligently allocates the available resources among the virtual machines based on predefined rules. When the virtual machine experiences increased load, DRS evaluates its priority.

VMware DRS allows you to control the placement of virtual machines on the hosts within the cluster by using affinity rules. By default, VMware DRS checks every 5mins to see if the cluster's workload is balanced. DRS is needed to be enabled for resource pools to be created.

DRS is invoked by certain actions in the cluster
  • adding or removing the ESXi host
  • changing resource settings on the VM

Automatic DRS mode determines the best possible distribution of virtual machines and the manual DRS mode provides recommendation for optimal placement of the virtual machines and leaves it the system administrator to decide.

Manual – every time you power on the VM, the cluster prompts you to select the ESXi host where the VM should be hosted. Recommends migration

Partially Automatic – every time you power on the VM, the cluster DRS automatically selects the ESXi host & Recommends migration

Fully Automatic – every time you power on the VM, the cluster DRS automatically selects the ESXi host & migration. Scaled from Conservative to Aggressive
  • Apply priority 1 recommendations - affinity rules & host maintenance
  • Apply priority 1 & 2 recommendations - promise significant improvement to cluster load balance
  • Apply priority 1, 2 & 3 recommendations - promise at-least good improvement to cluster load balance
  • Apply priority 1, 2, 3 & 4 recommendations - promise moderate improvement to cluster load balance
  • Apply all recommendations - promise even a slight improvement to cluster load balance. 

There are three major elements here:
  1. Migration Threshold
  2. Target host load standard deviation
  3. Current host load standard deviation

When you change the “Migration Threshold” the value of the “Target host load standard deviation” will also change. Two host cluster with threshold set to three has a THLSD of 0.2, a three host cluster has a THLSD of 0.163.

While the cluster is imbalanced (Current host load standard deviation > Target host load standard deviation) select a VM to migrate based on specific criteria and simulate a move and re-compute the “Current host load standard deviation” and add to the migration recommendation list. If the cluster is still imbalanced (Current host load standard deviation > Target host load standard deviation) repeat procedure.

How does DRS selects the best VM to move?

For each VM check if a VMotion to each of the hosts which are less utilized than source host would result in a less imbalanced cluster and meets the Cost Benefit and Risk Analysis criteria.

How VMware vMotion works ?

How VMware vMotion works ?

VMware vMotion enables live migration of running virtual machines from one physical to another with zero downtime, continuous service availability & complete transaction integrity. This feature improves availability of conducting maintenance without disrupting business.

VMware vMotion is enabled by three underlying technologies:

1)  Entire state of virtual machine is encapsulated by set of files stored on shared storage

2)  Active memory page & system state of virtual machine (preCopy) is rapidly transferred over high-speed vMotion network allowing to switch from source host to destination host. Keeps track of on-going memory transaction in a memory bitmap. Once entire memory & system state are copied to destination host, the source virtual machine is Quiesced. Memory bitmap does not have contents of memory; instead it has addresses of that memory (also called dirty memory). Target host reads the addresses in the memory bitmap file and requests the contents of the addresses from the source host. After copying the bitmap to target host, the virtual machine resumes on the target ESX host. The entire process takes < 2 seconds.

3)   The network is also virtualized, ensuring even after the migration virtual machine network identity & network connections are preserved. VMware vMotion manages Virtual MAC. Once the destination machine is activated, vMotion sends RARP message to the physical switch to ensure that it is aware of the new physical location of the virtual MAC address. After virtual machine successfully operating on the target host, memory on the source host is deleted.

VMware vMotion also migrates resource allocation (CPU & memory) from one host to another. On a bad day, ping -t may result in loss of one ping packet. Applications can withstand loss of more than a packet or two.

VMware vMotion requirements:

1) Shared storage

2) Gigabit Ethernet NIC with VMkernel port defined for vMotion on each ESXi hosts

A Successful vMotion relies on the following host conditions:

1) Source & destination host must be configured with identical virtual switches with vMotion enabled port group.

2) All port groups to which virtual machine being migrated must exists on both ESXi - case sensitive, VLANs.

3) Processors must be compatible.

A successful vMotion relies on the following virtual machine conditions:

1) Virtual machine must not be connected to any physical device available only to one ESXi host.

2) Virtual machine must not be connected to an internal-only virtual switch.

3) Virtual machine must not have CPU affinity set to specific CPU.

4) Virtual machine must have all the files on VMFS or NFS datastore accessible to both ESXi hosts.

High priority migration does not proceed if the resources aren’t available to be reserved for the migration. Standard priority migration might proceed slowly and might fail to complete if enough resources aren't available.

At 14%, a pause occurs while the hosts establish communications and gather information for pages in memory to be migrated. 

And at 65%, another pause occurs when the source virtual machine is quiesced and dirty memory pages are fetched from the source host.

Sometimes, the vMotion progress action fails at certain percentage. Below are the reasons for the vMotion failure at certain percentage:

1)      9% - issue with the ESXi NICs – upgrade the NIC firmware to resolve the issue

2)      10% - datastore of the VM mounted was mounted read-only – needed read/write access

3)      10% - log.rotateSize value in the virtual machine's .vmx file is set to a very low value, it causes the vmware.log file to rotate so quickly that by the time the destination host is requesting the vmware.log file's VMFS lock. The destination host is then unable to acquire a proper file lock, and this causes the vMotion migration failure.

4)      14% - fails if there are multiple VMkernel ports in the same network (or) incorrect VMkernal interfaces selected for vMotion.

5)      78% - NFS storage path UUID mismatch

6)      82% - caused by incorrect datastore information or out-of-date virtual hardware.

7)      90% - attempting to vMotion 64-bit virtual machines from an ESX/ESXi host with VT enabled in the BIOS to an ESX/ESXi host with VT not enabled in the BIOS.

8)      90% - occurs if the host Migrate.MemChksum value is set to 1, but the other hosts are set to 0.

Tuesday, September 30, 2014

vCenter server service not starting

vCenter server service not starting


1)    vCenter server service not starting (or) keeps starting and stopping frequently.

2)    Application event log – Microsoft SQL Server Native client could not allocate space for the object <object_name> in database <database_name> because PRIMARY file group is full. Create disk space by deleting uneeded files, dropping objects in the filegroup, adding additional files to the filegroup or setting autogrowth on for existing files in the filegroup.


VCenter server having its database running on the local server Microsoft SQL Express edition.

1) SQL Express edition has a limitation of 10GB per database.
2) There are many reasons for vCenter server service not starting and this is one of the reasons.


VMware vCenter Server stores performance, tasks & event logs data in the vCenter Server database. Over time, data collection results in growth of the database files and a mechanism is needed to shrink these files

VMware vCenter Server has a Database Retention Policy setting that allows you to specify when vCenter Server tasks and events should be deleted. Because this setting does not affect performance data records, it is still possible to purge or shrink old records from the database.

To access the Database Retention Policy setting in the vSphere Client, click Administration > vCenter Server Settings > Database Retention Policy.

Due to this, if vCenter database has reached the limit and database free space is very low, VMware vCenter server service fails to starts (or) keeps starting and stopping frequently.


1) Stop the vCenter Server services if it keeps starting and stopping frequently.

2) Open Microsoft SQL Server Management Studio.
a.    Server type – Database Engine
b.    Server name – <database_server_name>\<database_name>

c.    Authentication – Windows Authentication

3)    You can check the database free space, by connecting to the database using Microsoft SQL Management Studio. Select the database -> click properties -> Select "General". You can see the space available in the database (as shown below).

4) Before purging the unwanted data (Event logs) from the database, we need to take a backup of the vCenter database.

5) Right click “vcenter_database” -> click “New Query” -> Query window opens.

 6) To check the database table sizes, use the following SQL query.

SELECT [Table Name],
(SELECT rows FROM sysindexes s WHERE s.indid < 2 AND = OBJECT_ID(a.[Table Name]))
AS [Row count], [Total space used (MB)] FROM 
        SELECT        QUOTENAME(USER_NAME(o.uid)) + '.' + QUOTENAME(OBJECT_NAME( AS [Table Name],
                CONVERT(numeric(15,2),(((CONVERT(numeric(15,2),SUM(i.reserved)) * (SELECT low
FROM master.dbo.spt_values (NOLOCK) WHERE number = 1 AND type = 'E')) / 1024.)/1024.)) AS [Total space used (MB)]
        FROM        sysindexes i (NOLOCK)
                        INNER JOIN
                sysobjects o (NOLOCK)
       = AND
                ((o.type IN ('U', 'S')) OR o.type = 'U') AND
                (OBJECTPROPERTY(, 'IsMSShipped') = 0)
        WHERE        indid IN (0, 1, 255)
                ) as a
ORDER BY        [Total space used (MB)] DESC

7) Copy the above SQL query to the query window -> select all the query (use Ctrl + a) -> click F5.

8) Query output shows as below. The table using maximum space is shown on the top. Here “VPX_EVENT_ARG” and “VPX_EVENT” uses maximum space (approx.. 6GB out of 10GB database size)

9) We can truncate the event logs to free up the database space.

10)Execute the SQL query shown below – this will delete all the Tasks & events in the vCenter.

truncate table VPX_TASK
truncate table VPX_ENTITY_LAST_EVENT
truncate table VPX_EVENT
truncate table VPX_EVENT_ARG
alter table VPX_EVENT_ARG add
constraint FK_VPX_EVENT_ARG_REF_EVENT foreign key(EVENT_ID) references VPX_EVENT (EVENT_ID) on delete cascade, constraint FK_VPX_EVENT_ARG_REF_ENTITY foreign key (OBJ_TYPE) references VPX_OBJECT_TYPE (ID)
alter table VPX_ENTITY_LAST_EVENT add
constraint FK_VPX_LAST_EVENT_EVENT foreign key(LAST_EVENT_ID) references VPX_EVENT (EVENT_ID) on delete cascade

11) Copy the above SQL query to the query window -> select all the query (use Ctrl + a) -> click F5.

12) Query output shows as below. “VPX_EVENT_ARG” and “VPX_EVENT” data from the vCenter database got cleared.

13) Verify the database free space, by connecting to the database using Microsoft SQL Management Studio. Select the database -> click properties -> Select "General". You can see the free space available in the database.

14) Start the “VMware vCenter Server Service”.

Monday, March 3, 2014

vCenter Operations Manager (vcops) installation steps

vCenter Operations Manager (vcops) installation steps

1) Download the vApp from

2) Using the vSphere Client, connect to the vCenter Server with the cluster where you are installing the vApp.

3) Set up the IP Pool:
    1. Select the DataCenter which has the cluster where the vApp is to be hosted, and click the IP Pools tab.
    2. Click Add... at the top of the page. A dialog box opens.
    1. Enter a name for the IP Pool. (For example, vCOps.)
    2. Enter the subnet (not the subnet mask) that the vCenter Server resides on.
    3. Enter the subnet mask as a bit. (For example, a mask of is equivalent to a /24 network.)
    4. Enter the Gateway IP.
    5. Do not enable the IP Pool, as this is not required for proper use.

    1. Click the DNS tab and enter the appropriate DNS information for this network.

      Note: You can enter multiple DNS servers by separating them with a comma.

    1. Click the Associations tab and select the Port Group the vCenter Server is attached to, or one that can easily reach vCenter.
    2. Click OK.

4) Deploy the vApp:
    1. In the vSphere Client, click File > Deploy OVF Template...
    2. Follow the prompts in the Deploy OVF Template wizard.

      Note: When specifying a disk format, VMware recommends using Thick Provisioned Eager Zeroed in vSphere 5.

    1. When specifying an IP allocation scheme, select Fixed (recommended) or DHCP.

      NoteFixed IP allocation requires you to provide two IP addresses for the two virtual machines in the vApp. DHCP allocation requires that you enable DHCP in your IP pool. Transient is not supported at this time.
    2. Click Finish and wait for the deployment process to complete.
5) (Optional) Set up a DRS affinity rule:

Note: These steps may be necessary for distributed vSwitch communication issues. Skip these steps unless you experience difficulties with initialization.
    1. Right-click the cluster which will host the vApp, and select Edit Settings.
    2. Click Rules.
    3. Click Add.
    4. Enter a name for the Rule. (For example, vCOps.)
    5. Select Keep Virtual Machines Together.
    6. Click Add.
    7. Select the UI VM and Analytics VM virtual machines.
    8. Click OK.
6) Power on the vApp.

7) Set up vCenter monitoring:
    1. Browse to http://IP_address/admin, where IP_address is the IP address or fully qualified domain name (FQDN) of the vApp.

      Note: You can confirm this address on the console of the UI virtual machine.

    1. Update the administrator password ("admin") which provides access to the Administration Portal and SSH access to the vApp.

      Note: The password requires a minimum of eight characters that include at least one letter and one digit or special character.
    2. Update the root password ("root") for the operation system of the vApp.

      Note: The admin password and root password are different. Please document your root password.

    1. Enter a human-friendly name for the vCenter Server system.
    2. Enter the FQDN (recommended) or IP address of the vCenter Server system to collect information from and monitor.
    3. Enter the registration credentials for vCenter Operations Manager to use when connecting to the vCenter Server.

      Note: The user credentials you provide must have administrator privileges within vCenter. VMware recommends the use of a service account.
    4. (Optional) Enter the collection credentials for vCenter Operations Manager to use for collecting data from vCenter Server.

      Note: Using separate collection credentials can be used to increase security, and only require read-only privileges within vCenter.

    1. Click Test to verify that vCenter Operations Manager can connect to the vCenter Server.

      Note: If this step fails, verify that you are able to log into the vCenter Server using the credentials used in step f.
    2. (Optional) If you have linked vCenter Server systems, select the appropriate members of the linked group to register with, and provide a name for each system. Each vCenter Server must be registered individually.

      Note: You can register vCenter Operations Manager with a subset of vCenter Server systems for scalability or inventory management purposes.
    3. (Optional) If vCenter Operations detects that you have vCenter Operations 1.x or Capacity IQ 1.x installed, provide the appropriate credentials to import the data from the older installations.

      Note: This can only be performed during installation, and will not be presented in the Admin page after the installation. If this information must be imported post-installation, you must deploy a new vApp to do so.

8) After the installation is complete, browse to http://IP_address, where IP_address is the IP address or fully qualified domain name (FQDN) of the vApp.

9) To license vCenter Operations Manager:
    1. Using the vSphere Client, connect to the vCenter Server with the cluster where you are installing the vApp.
    2. Go to Home > Licensing.
    3. Verify that a vCenter Operations asset is listed.
    4. Click Manage vSphere Licenses.
    5. Enter the License in the top field and click Next.
    6. Click the Solutions tab and select vCenter Operations.

      Note: If the solution is not visible, toggle Show all at the top of the list.
    7. Click Next.
    8. Click Next.
    9. Click Finish.