vSphere HA configuration fails with error “vSphere HA agent for this host has an error: vSphere HA agent cannot be installed or configured”

If you reconfigure on cluster level (disable/enable HA) OR host lever (Right Click on Host \ Select Reconfigure for vSphere HA), following event is seen.

The object 'Vim.Datastore.datastore-XXXX has already been deleted OR has not been completely created.

You may see following lines in VPXD.LOG.

2019-02-05T10:42:20.182-08:00 error vpxd[07320] [Originator@6876 sub=DAS opID=HostHAOpsActionCommand-applyOnMultiEntity-5316758-ngc:70749342-ea-01] EnableDAS failed on host [vim.HostSystem:host-2728,ils-cls1-esxi-02.coloils.local]: class Vmomi::Fault::ManagedObjectNotFound::Exception(vmodl.fault.ManagedObjectNotFound)
--> [context]zKq7AVMEAAAAAEklWwAjdnB4ZAAASi0fdm1hY29yZS5kbGwAAACHBgDesAYAtEECAd1KAHZtb21pLmRsbAABfioBAsVqPXZweGQuZXhlAAKXfT0CoAMjAvv6IgK/UUoC1xRNAj08SwJKnUsChJdLAl+pCwLB3wsCraQLAozACwIMiUsC/B0wA9FUBHZpbS10eXBlcy5kbGwAARfZBAIW9QwCX6kLAsHfCwKtpAsChswLAKtoGACceBgAiQsiBH9PAk1TVkNSMTIwLmRsbAAEJlECBdITAEtFUk5FTDMyLkRMTAAG9FQBbnRkbGwuZGxsAA==[/context]
2019-02-05T10:42:20.204-08:00 info vpxd[07320] [Originator@6876 sub=MoHost opID=HostHAOpsActionCommand-applyOnMultiEntity-5316758-ngc:70749342-ea-01] [HostMo::UpdateDasState] VC state for host host-2728 (uninitialized -> init error), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
2019-02-05T10:42:20.231-08:00 info vpxd[07320] [Originator@6876 sub=DAS opID=HostHAOpsActionCommand-applyOnMultiEntity-5316758-ngc:70749342-ea-01] [VpxdDasConfigLRO::Cleanup] Number of unprotected vms: 11
2019-02-05T10:42:20.231-08:00 info vpxd[07320] [Originator@6876 sub=vpxLro opID=HostHAOpsActionCommand-applyOnMultiEntity-5316758-ngc:70749342-ea-01] [VpxLRO] -- FINISH lro-53671774
2019-02-05T10:42:20.231-08:00 info vpxd[07320] [Originator@6876 sub=Default opID=HostHAOpsActionCommand-applyOnMultiEntity-5316758-ngc:70749342-ea-01] 
[VpxLRO] -- ERROR lro-53671774 -- -- DasConfig.ConfigureHost: vmodl.fault.ManagedObjectNotFound:
--> Result:
--> (vmodl.fault.ManagedObjectNotFound) {
--> faultCause = (vmodl.MethodFault) null, 
--> faultMessage = <unset>, 
--> obj = 'vim.Datastore:0c057a11-f239-452b-9ccc-113fd7f52161:datastore-3958'
--> msg = ""
--> }
--> Args:
--> 
2019-02-05T10:42:20.232-08:00 info vpxd[07320] [Originator@6876 sub=vpxLro opID=HostHAOpsActionCommand-applyOnMultiEntity-5316758-ngc:70749342-ea] [VpxLRO] -- FINISH task-135218
2019-02-05T10:42:20.232-08:00 info vpxd[07320] [Originator@6876 sub=Default opID=HostHAOpsActionCommand-applyOnMultiEntity-5316758-ngc:70749342-ea] 
[VpxLRO] -- ERROR task-135218 -- host-2728 -- vim.HostSystem.reconfigureDAS: vmodl.fault.ManagedObjectNotFound:
--> Result:
--> (vmodl.fault.ManagedObjectNotFound) {
--> faultCause = (vmodl.MethodFault) null, 
--> faultMessage = <unset>, 
--> obj = 'vim.Datastore:0c057a11-f239-452b-9ccc-113fd7f52161:datastore-3958'
--> msg = ""
--> }
--> Args:
-->

This problem may happen because of stale cache entries of data stores configured for heartbeat. To resolve this issue restart VPXD service on vCenter.

service-control --stop vpxd
service-control --start vpxd

Refer KB for more details on vCenter service stop/start.

Then reconfigure HA, if multiple host affected then disable/enable vSphere HA on cluster level, if single host is affected then reconfigure on host level (Right Click on Host select option Reconfigure for vSphere HA)

#datastore, #error, #vmware, #vsphereha

All the options of update manager for vCenter web client display error “interface com.vmware.vim.binding.integrity.VcIntegrity is not visible from class loader”

This particular error is seen with vCenter 6.5 however other version may be affected.

um.jpg

You may see following lines in vSphere-Client.virgo.log

[WARN ] http-bio-9090-exec-10 org.springframework.flex.core.DefaultExceptionLogger The following exception occurred during request processing by the BlazeDS MessageBroker and will be serialized back to the client: flex.messaging.MessageException: The supplied destination id is not registered with any service.
at flex.messaging.MessageBroker.routeMessageToService(MessageBroker.java:1477)
at flex.messaging.endpoints.AbstractEndpoint.serviceMessage(AbstractEndpoint.java:1046)
at flex.messaging.endpoints.AbstractEndpoint$$FastClassByCGLIB$$1a3ef066.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.flex.core.MessageInterceptionAdvice.invoke(MessageInterceptionAdvice.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:573)
at flex.messaging.endpoints.AMFEndpoint$$EnhancerByCGLIB$$4442b278.serviceMessage(<generated>)
at flex.messaging.endpoints.amf.MessageBrokerFilter.invoke(MessageBrokerFilter.java:101)

 

To resolve this issue, take SSH access to vCenter server. Stop and start vmware-updatemgr & vsphere-client services.

service-control --stop vmware-updatemgr
service-control --stop vsphere-client

service-control --start vmware-updatemgr
service-control --start vsphere-client

Refer KB for more details on vCenter service stop/start.

#update-manager, #vcenter, #vmware

Clarification on ESXi Hardware sensor shows as ‘Unknown’

Since the ESXi version 6.5 Update2/6.7, you may see some of hardware sensor may show as Unknown.

HardwareSensor

VMware accept this issue as part of KB https://kb.vmware.com/kb/57171 and mentioned it as expected behavior.

To give more clarification, the sensor status presented in version prior to 6.5 U2/6.7 used to show green for any status that fell under the list category that could not be read OR understand. It used to show green for anything that is actually unknown. This behavior got changed in 6.5 U2 and 6.7 to show as unknown wherever the values cannot be read.

You can contact your hardware vendor to know why ESXi unable to read certain sensor that showing as unknown.

This behavior is because of defect fix as part of KB

VMware team explained the changes to Hardware Health Monitoring in vSphere Stack https://videos.vmworld.com/global/2018/videoplayer/23425

#esxi, #sensor, #unknown, #vmware

vCenter service ‘VPXD’ fails to start with error “No space left on device”’

When you start the service manually using following command it gives error.

Syntax

service-control --start vmware-vpxd

Error

Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False
2019-02-08T22:44:52.954Z   Service vpxd state STOPPED
Error executing start on service vpxd. Details {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Service-control failed. Error {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

Error in VPXD log


2019-02-08T22:44:53.608Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Starting VMware VirtualCenter 6.5.0 build-8024368
2019-02-08T22:44:53.608Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Log directory: /var/log/vmware/vpxd.
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=Main] Account name: root
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxUtil] [LoadMachineInstanceUuid] Local instance UUID: 6f0cbee2-56b1-43d3-b13a-348208627e07
2019-02-08T22:44:53.614Z error vpxd[7FB5996C4800] [Originator@6876 sub=Main] Init failed. Exception: N7Vmacore15SystemExceptionE(No space left on device)
--> [context]zKq7AVECAAAAADBxegAOdnB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAC8HisAtJgaAH6mGgG4w6d2cHhkAAEBxqcBCsinAeEoVQG9+1QBap9TAuAFAmxpYmMuc28uNgABdZdT[/context]
2019-02-08T22:44:53.615Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
2019-02-08T22:44:53.615Z info vpxd[7FB5996C4800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd.
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=CryptUtil] [static bool Vpx::Common::CryptUtil::UnmungePasswordToBuffer(VpxDecrypter*, const string&, char*, size_t)] invalid decrypter
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 00000:0
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType]: Database error: ODBC error: (00000) -
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Error getting configuration info from the database
2019-02-08T22:44:53.617Z warning vpxd[7FB5996C4800] [Originator@6876 sub=Main] Database not initialized. Nothing to unlock
2019-02-08T22:44:53.617Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

This issue may caused because of Inodes are full on vCenter appliance. More details on Inodes

To see current storage usage of iNodes, run following command.


root@vCenterServer [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 712704        0  100% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core

This command will give you iNode counts

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

sample output


    875 /usr/bin
   1417 /opt/vmware/lib/python2.7/test
   1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
   5362 /usr/share/man/man3
607550 /var/spool/mqueue

Go to path where you see most of the inodes, then get file count.

[/var/spool/mqueue ]# ls -ltrh | wc
607551 5467952 35237911

Delete inodes, take snapshot of vCenter VM for safer side.

find /var/spool/mqueue/ -type f -print0 | xargs  -0 rm -f

Verify space for inodes again using command ‘df -i

# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 105154   607550   15% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core

 

#linux, #vmware, #vpxd

How to join VMware ESX / VCenter to Active Directory domain and manage using domain account.

Most of organization uses AD infrastructure for authentication and administrator the resources of organization. ESX servers and VSphere can also be joined to AD domain and administrator using domain account.

Here are the steps to do the same.

Confirm the appropriate DNS address & domain name is configured for ESX.  Login to ESX using VSphere client, select Configuration tab, select ‘DNS and routing’. If correct DNS IP is not configured then click on ‘Properties’ and save valid details.

1.jpg

Click on Authentication Services, select Properties to edit current settings, select directory service type as Active Directory. Type domain name then join to domain by giving domain admin credential.

2.jpg

Once ESX joined in the domain, you should be able to see computer account listed in ADUC (Active directory user’s & computer)

3.jpg

Now create a group called ‘VMWare Admin’ & a user ‘VAdmin’, this user will be the member of ‘VMware admin’ group.

4.jpg

In your VSphere client, select Permission tab, right click on empty space select ‘Add Permission’.

5.jpg

Click on ADD, from drop down menu select the domain, you should be able to see AD objects, select VMware admin group.

6.jpg

Select the role you wish to give to map with AD group ‘VMware admin’.

7.jpg

Once it added successfully, you should be able to login in ESX Vsphere client using domain credential. You can check box ‘use windows session credential’ if you wants to login using current windows login.

8.jpg

If you are managing ESX using VMWare VCenter then open VCenter page, go to ‘Administration’ page, select ‘Configuration’, In ‘Identity source’ option select ‘Active Directory (integrated windows authentication)’, Type appropriate domain name and click on OK.

9.jpg

Click on ‘Global Permission’ Tab, click on ‘+’ option then Add AD group ‘VMware admin’.

11.jpg

Select the role you wish to map with to ‘VMware admin’ AD group.

12.jpg

Now you should be able to login to VCenter using windows & AD authentication.

13.jpg

#active-directory, #vmware, #windows

How to create ISCSI shared disk in windows server for ESX VSphere environment.

This my first blog for ESX/VSphere where the requirement was to create ISCSI data store for HA & DRS (High Availability & Distributed Resource Scheduler).  We created storage from windows server instead of having a dedicated storage appliance/VM. The purpose of such configuration is only for learning & testing the HA/DRS activity.

This configuration is designed on windows 2012 R2 with ESX 6.0.

On windows server, install ‘File and ISCSI services’ and make sure ‘ISCSI Target server’ is selected.

1.jpg

Once the role is installed then select ‘File and storage services’ feature and then ‘ISCSI’.

Right click on empty space and select ‘New ISCSI Virtual Disk’ as shown in above example. In my screenshot one of the volume is created and I am creating a new one.

2.jpg

Now select the partition where you wish to place the Virtual Disk.

3.jpg

Give the appropriate Name and Description and confirm the path of SCSI target.

4.jpg

Give the appropriate size & select type of disk as mentioned below. In my example I have selected ‘Dynamically expending’ as I don’t need to worry about space.

5.jpg

You need to create target that will be the list of ESX servers participating in HA/DRS and access VHD using ISCSI protocol.

6.jpg

In my example, I have used DNS name of ESX servers as initiator but it can also be access using IQN, IP address OR MAC address.

16.jpg

In following screenshot, there are two ESX servers are added but you have even more.

8.jpg

Select type of authentication, I am leaving it blank to avoid confusion.

9.jpg

In next page, you can confirm the setting you have selected and the result of different section will be available for you.10.jpg

Now the shared ISCSI disk is ready, you can add this in ESX server using VSphere console. Select ESX server \ Manage \Storage \ Storage Adaptor \Target \ Add Target.

11.jpg

Rescan storage so all newly attache drives are visible.

12.jpg

Now you should be able to see the path of all available SCSI share disk.

13.jpg

SCSI disk will also be available as storage devices.

14.jpg

In your windows server, you would notice the target status as ‘connected’.

15.jpg

#disaster-recovery, #esx, #scsi, #storage, #vmware