vMotion may fail because of TegileNasPlugin installed on ESXi host.

You may see following lines in VMware.log of the VM that was attempted to migrate.

2019-09-12T18:05:51.309Z| Worker#1| I125: DISK: OPEN scsi1:0 '/vmfs/volumes/b57aef17-591cfc64/NAS1/NAS1_1.vmdk' persistent R[]
2019-09-12T18:05:51.315Z| Worker#2| I125: TegileNasPlugin: StartSession: Req for 10.1.192.42 /export/ESX_5/ZEBI_ESX4 /vmfs/volumes/b57aef17-591cfc64 NFS 1
2019-09-12T18:05:51Z[+0.000]| Worker#2| W115: Caught signal 11 -- tid 190820 (addr 6379C5FA4C)
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: rip 0x6379c5fa4c rsp 0x637d6bec18 rbp 0x637d6bec50
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: rax 0x2f rbx 0x6338557110 rcx 0x20 rdx 0x656d756c rsi 0x6338572fa0 rdi 0x6f762f73666d762f
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125:         r8 0x0 r9 0x2a r10 0x1999999999999999 r11 0x656d756c r12 0x6338572fa0 r13 0x637d6becb4 r14 0x637d6becb0 r15 0x6338505c00
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: stack 637D6BEC18 : 0x000000637c77c4c1 0x0000000000000000
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: stack 637D6BEC28 : 0x1e840cb0799abf47 0x0000006338572fa0
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace:
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace[0] 000000637d6be690 rip=00000063370c39c7 rbx=00000063370c34c0 rbp=000000637d6be6b0 r12=0000000000000000 r13=0000006337d64d01 r14=000000000000000a r15=000000637d6bec98
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace[1] 000000637d6be6c0 rip=00000063372f64c0 rbx=000000637d6bec98 rbp=000000637d6be8d0 r12=000000000000000b r13=0000006337d64d01 r14=000000000000000a r15=000000637d6bec98
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace[2] 000000637d6be8e0 rip=00000063372f68c4 rbx=0000000000000008 rbp=000000637d6be930 r12=000000637d6c0538 r13=0000006337d64dc8 r14=0000006337d64dc0 r15=000000000000000b
2019-09-12T18:05:51.316Z| Worker#3| I125: TegileNasPlugin: StartSession: Req for 10.1.192.42 /export/ESX_5/ZEBI_ESX4 /vmfs/volumes/b57aef17-591cfc64 NFS 1
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: Backtrace[3] 000000637d6be940 rip=000000000038600f rbx=0000006338557110 rbp=000000637d6beb80 r12=000000637d6be9c0 r13=000000637d6becb4 r14=000000637d6becb0 r15=0000006338505c00

 

VMkernel.log (source ESXi host).

2019-09-12T18:05:56.302Z cpu29:81337)WARNING: Migrate: 273: 2017306294537764927 S: Failed: Failed to resume virtual machine (0xbad0044) @0x418018883122
2019-09-12T18:05:56.302Z cpu29:81337)VMotionRecv: 3733: 2017306294537764927 S: Error handling message: Connection reset by peer
2019-09-12T18:05:56.312Z cpu27:76130)WARNING: Migrate: 6279: 2017306294537764927 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
2019-09-12T18:05:56.343Z cpu44:76290)CBT: 1341: Created device 20d0803-cbt for cbt driver with filehandle 34408451

 

VMWarning.log (source ESXi host).

2019-09-12T14:38:17.513Z cpu10:77420)WARNING: CBT: 1133: Unsupported ioctl 62
2019-09-12T18:05:56.302Z cpu29:81337)WARNING: Migrate: 273: 2017306294537764927 S: Failed: Failed to resume virtual machine (0xbad0044) @0x418018883122
2019-09-12T18:05:56.312Z cpu27:76130)WARNING: Migrate: 6279: 2017306294537764927 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
2019-09-12T18:05:56.343Z cpu44:76290)WARNING: CBT: 1133: Unsupported ioctl 63

 

VMKernel.log (destination ESXi host).

2019-09-12T18:05:56.337Z cpu22:190653)Hbr: 3489: Migration end received (worldID=190654) (migrateType=1) (event=1) (isSource=0) (sharedConfig=1)
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 6749: 2017306294537764927 D: Migration cleanup initiated, the VMX has exited unexpectedly. Check the VMX log for more details.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 273: 2017306294537764927 D: Failed: Migration determined a failure by the VMX (0xbad0092) @0x4180296b3091
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: VMotionUtil: 7649: 2017306294537764927 D: timed out waiting 0 ms to transmit data.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: World: vm 190653: 3566: VMMWorld group leader = 190654, members = 2
2019-09-12T18:05:56.337Z cpu32:190660)VMotionUtil: 7552: 2017306294537764927 D: Socket 0x430a8b349c00 sendSocket pending: 563164/563272 snd 0 rcv
2019-09-12T18:05:56.525Z cpu14:65689)CBT: 1376: Destroying device 5920950-cbt for cbt driver with filehandle 93456720

 

VObd.log (destination ESXi host).

2019-09-12T18:05:56.333Z: [UserWorldCorrelator] 244452187008us: [vob.uw.core.dumped] /bin/vmx(190653) /var/core/vmx-zdump.001
2019-09-12T18:05:56.333Z: [UserWorldCorrelator] 244476489345us: [esx.problem.application.core.dumped] An application (/bin/vmx) running on ESXi host has crashed (2 time(s) so far). A core file may have been created at /var/core/vmx-zdump.001.

 

VMKwarning.log (destination ESXi host).

2019-09-12T18:05:51.304Z cpu22:190653)WARNING: CBT: 1133: Unsupported ioctl 62
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 6749: 2017306294537764927 D: Migration cleanup initiated, the VMX has exited unexpectedly. Check the VMX log for more details.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 273: 2017306294537764927 D: Failed: Migration determined a failure by the VMX (0xbad0092) @0x4180296b3091
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: VMotionUtil: 7649: 2017306294537764927 D: timed out waiting 0 ms to transmit data.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: World: vm 190653: 3566: VMMWorld group leader = 190654, members = 2

 

This is caused because of TegileNasPlugin installed on ESXI host. Make sure you connect with NAS vendor

 

#tegilenasplugin, #vcenter, #vmotion, #vmware-log

Unable to take ESXi configuration backup using PowerShell.

As per the KB you can take configuration backup of ESXI so it can be restore if something goes wrong during maintenance/upgrade/reinstall kind of action. Following PowerShell can be use from VMware CLI.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\backup

 

However sometime this command may fail with following error.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\temp
An error occurred while sending the request.
At line:1 char:1
+ Get-VMHostFirmware -VMHost esxi-1.domain.local -BackupConfiguration -D ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], ViError
    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_DownloadError,VMware.VimAutomation.ViCore.Cmdlets.
   Commands.Host.GetVMHostFirmware

 

You may see following lines in vpxd.log (vCenter) and hostd.log (ESXi host).

Hostd.log

2019-09-13T02:26:47.117Z info hostd[2098523] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424
2019-09-13T02:26:47.119Z info hostd[2099187] [Originator@6876 sub=SysCommandPosix opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] ForkExec(/sbin/firmwareConfig.sh) 2266493
2019-09-13T02:26:48.393Z info hostd[2099187] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424 Status success [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

VPXD.log

2019-09-13T02:28:26.134Z info vpxd[05841] [Originator@6876 sub=vpxLro opID=57b9c4a1] [VpxLRO] -- BEGIN lro-838226 -- firmwareSystem-161 -- vim.host.FirmwareSystem.backupConfiguration -- 521f50b8-5645-404f-11f8-f44099740a62(524a10f8-512b-637a-60ad-fb0b1d7510b6)
2019-09-13T02:28:29.300Z error vpxd[18796] [Originator@6876 sub=HostPicker opID=sps-Main-533116-133-60] [PickDoWork] Couldn't find any candidate host that satisfies all constraints

 

This particular problem comes when the port 80 (http) is blocked from your workstation to ESXi.  Following PS can be use to check the connectivity.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
WARNING: TCP connect to esxi-1.domain.local:80 failed
ComputerName           : esxi-1.domain.local
RemoteAddress          : 192.168.0.81
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          : 192.168.0.10
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : False

 

Verify if the firewall inside the ESXi OR external firewall blocking that communication.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access'
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   False   80                            TCP

 

If this is disabled then enabled using following command.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access' | Set-VMHostFirewallException -Enabled $True
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   True    80                            TCP

 

A successful connection should show like below.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
ComputerName           : esxi-1.domain.local
RemoteAddress          : 192.168.0.81
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          : 192.168.0.10
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : True

 

Make sure the IP of your workstation is present in allowed list. That can be done using the vCenter OR ESXi web console.

allowedip.jpg

vCenter Appliance Network configuration change via command line.

Most of the system configuration such as changing the hostname, IP, DNS..etc can be done using VAMI interface (https://vCenterIPorFQDN:5480) however if you see the changes are not taking affect, e.g. changing the DNS IP then following command can be used.

  • Enable SSH using VAMI interface, select Access tab, click Edit and Enable SSH login.
  • Take SSH session via putty to connect vCenter Appliance.
  • Use following command and change required configuration change.
root@vcsa[~]# /opt/vmware/share/vami/vami_config_net
Main Menu
0)      Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0

 

Here are some samples:-

  • Review current configuration.
Enter a menu number [0]: 0
Network Configuration for eth0
IPv4 Address:   192.168.0.51
Netmask:        255.255.255.0
IPv6 Address:
Prefix:
Global Configuration
IPv4 Gateway:   192.168.0.1
IPv6 Gateway:
Hostname:       vcsa1.lab.org
DNS Servers:    127.0.0.1, 192.168.0.10
Domain Name:
Search Path:
Proxy Server:

Whenever you make any changes you will be prompted for following warning message.

Warning: if any of the interfaces for this VM use DHCP, the Hostname, DNS, and Gateway parameters will be overwritten by information from the DHCP server. Type Ctrl-C to go back to the Main Menu

  • Changing hostname
Enter a menu number [0]: 3
New hostname [vcsa1.lab.org]:vcsa2.lab.org

  • Changing DNS server and Domain name.
Enter a menu number [0]: 4
DNS Server 1 [127.0.0.1]: 192.168.0.10
DNS Server 2 (optional) [192.168.0.10]: 192.168.0.11
Domain Name (optional) []: lab.org
Search Path (space separated) (optional) []: lab.org
DNS server settings updated

  • Changing proxy address of vCenter appliance
Enter a menu number [0]: 5
Is an IPv4 proxy server necessary to reach the Internet? y/n [n]: y
Proxy Server (http:// will be auto prepended) []: proxy1.lab.org
Proxy Port []: 8080

  • Changing IP address.
Enter a menu number [0]: 6
Configure an IPv6 address for eth0? y/n [n]: n
Configure an IPv4 address for eth0? y/n [n]: y
Use a DHCPv4 Server instead of a static IPv4 address? y/n [n]: n
IPv4 Address [192.168.0.51]: 192.168.0.51
Netmask [255.255.255.0]: 255.255.255.0
IPv4 Address:   192.168.0.51
Netmask:        255.255.255.0

Is this correct? y/n [y]: y

Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 1
Network parameters successfully changed to requested values

Un-handled Exception with ESXi UI

Following two problems are seen with ESXi version 6.7 Update 2 (Build 13006603).

  • Using host UI (https://HOSTNameORIP/ui ) when you click on Storage then Adapter then  following unhandled exception comes. This stop configuring storage related activity from ESXi.
Unhandled exception
-----------
Unfortunately, we hit an error that we weren't expecting.
The client may continue working, but at this point,
we recommend refreshing your browser and submitting a bug report.
Press the Esc key to hide this dialog and continue without refreshing

 

Clicking on Details shows following lines.

Cause: Possibly unhandled rejection: {}
Version: 1.33.3
Build: 12923304
ESXi: 6.7.0
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Exception stack:
this.toAddress64@https://192.168.100.11/ui/scripts/main.js:422:206
updateAdapter@https://192.168.100.11/ui/scripts/main.js:365:26206
@https://192.168.100.11/ui/scripts/main.js:365:26542
k/<@https://192.168.100.11/ui/scripts/main.js:324:23176
$digest@https://192.168.100.11/ui/scripts/main.js:324:28780
$evalAsync/<@https://192.168.100.11/ui/scripts/main.js:324:30503
e@https://192.168.100.11/ui/scripts/main.js:323:10071
tg/h.defer/c<@https://192.168.100.11/ui/scripts/main.js:323:11522

 

  • Using host UI when you import virtual machine using OVF template following stack is seen.
TypeError: Cannot read property 'keyValue' of undefined
    at updateSummaryPortlet (https://192.168.10.11/ui/scripts/main.js:375:415)
    at $scope.wizardOptions.onFinish (https://192.168.10.11/ui/scripts/main.js:375:5968)
    at https://192.168.10.11/ui/scripts/main.js:324:23176
    at m.$digest (https://192.168.10.11/ui/scripts/main.js:324:28780)
    at https://192.168.10.11/ui/scripts/main.js:324:30503
    at e (https://192.168.10.11/ui/scripts/main.js:323:10071)
    at https://192.168.10.11/ui/scripts/main.js:323:11522

 

VMkernel.log shows following lines.

2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)World: 11943: VC opID esxui-4004-bd50 maps to vmkernel opID 27da76af
2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)NVDManagement: 1461: No nvdimms found on the system

 

Clicking on Reload option log out ESXi web UI and you have to login to UI again.

This particular problem is fixed in ESXi build number 13981272 (ESXi 6.7 EP 10) OR later version. Refer KB  to know more about ESXi build version.
ESXi Patch download link
Select ESXi and version 6.7 then search to get latest patch.

The workaround is to press escape after opening the stack OR to import OVF using vCenter UI if host is part of vCenter, same for changing storage settings. 

VMware Appliance Monitoring Service (vmware-statsmonitor) doesn’t start.

After reboot vmware-statsmonitor service doesn’t start automatically. In some situation, this service even doesn’t start manually and fail with following error.

root@buildvcenter [ ~ ]# service-control --start vmware-statsmonitor
Operation not cancellable. Please wait for it to finish...
Performing start operation on service statsmonitor...
channel 3: open failed: administratively prohibited: open failed
Error executing start on service statsmonitor. Details {
    "detail": [
        {
            "args": [
                "statsmonitor"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'statsmonitor'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "problemId": null,
    "resolution": null,
    "componentKey": null
}

 

Following line can be seen in Vmon log (/var/log/vmware/vmon/vmon-syslog.log)

2019-08-08T09:26:42.809220-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:42.809574-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:45.810708-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:51.166333-07:00 warning vmon   Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /var/vmware/applmgmt/statsmonitor_health.xml. Error: [Errno 2] No such file or directory: '/var/vmware/applmgmt/statsmonitor_health.xml'
2019-08-08T09:26:51.166701-07:00 warning vmon
2019-08-08T09:26:51.194457-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:51.194832-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:54.195953-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:58.391456-07:00 notice vmon   Service start operation timed out.
2019-08-08T09:26:58.394656-07:00 notice vmon   Cancelling execution of pid 38052
2019-08-08T09:26:58.395009-07:00 warning vmon   Found empty StopSignal parameter in config file. Defaulting to SIGTERM

This issue happens because of startup delay and can be fix by following steps:-

  • Take snapshot of vCenter to be in safer side.
  • Take SSH to VCSA using root login.
  • Modify statsmonitor service config for vMon to set higher start up timeout:
sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
kill -HUP $(cat /var/run/vmon.pid)
  • Stop and start stats-monitor service explicitly.
/usr/lib/vmware-vmon/vmon-cli -k statsmonitor
/usr/lib/vmware-vmon/vmon-cli -i statsmonitor
  • Then restart vCenter to see if that service start automatically

Remote access for ESXi local user account ‘root’ has been locked for XXXX seconds after XXXX failed login attempts.

Due to consistent wrong password by application OR user, root a/c of ESXi may get locked out. This prevent any further login externally to ESXi host.

This happens most likely due to backup/monitoring application that has root user a/c configured the operation.

To solve this issue.

  • Take Console access (iDRAC/ILO/KVM..etc depending on hardware).
  • Press F2 to customize the system. Log in as root.
  • Use the Up/Down arrows to navigate to Troubleshooting Options > Enable ESXi Shell.
  • Press ALT+F1, login as root. Run the following commands to show number of failed attempts:
pam_tally2 --user root
  • Run the following command to unlock the root account:
pam_tally2 --user root --reset

 

Now you should be able to login to ESXi using root a/. You can review hostd log to find out from where failed login coming.

less /var/log/hostd.log |grep -i 'password'

2019-07-31T17:08:51.735Z info hostd[2099345] [Originator@6876 sub=Default 2019-07-31T20:51:07.055Z warning hostd[2205446] [Originator@6876 sub=Default opID=esxui-fca4-b52f] Rejected password for user root from 192.168.214.108
2019-07-31T20:51:11.056Z verbose hostd[2205444] [Originator@6876 sub=Solo.Vmomi] Arg password:
2019-07-31T20:51:39.634Z warning hostd[2099346] [Originator@6876 sub=Default opID=esxui-c6d2-b54a] Rejected password for user root from 192.168.214.108

 

Based on IP address check if any application configured with incorrect root password.

Please note in ESXi 6.5/6.7, invalid login may cause host to be unresponsive. Review KB  for proactive steps.

 

Unable to open vCenter 6.5 flash based web client

When you open web client following pop-up error is seen.

Cannot navigate to the desired location.
Error details: An error occurred while activating extension vsphere.core.viTree.hostsAndClustersView.
Invalid domain view id: vsphere.core.viTree.hostsAndClustersView"

When we click on Yes then you see following stack.

Error: Invalid domain view id: vsphere.core.viTree.hostsAndClustersView
at com.vmware.vsphere.client.views.app::AppViewMediator/showDomainView()
at com.vmware.vsphere.client.views.app::AppViewMediator/activateExtensionInternal()
at MethodInfo-1531()
at com.vmware.vsphere.client.views.app::AppViewMediator/onUiReady()
at EventFunctor/notifyTarget()
at EventFunctor/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.frinje::EventBus/dispatchEvent()
at com.vmware.frinje::EventBus/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.docking::DockableUiManager/onLayoutDataRetrieved()
at com.vmware.docking::DockableUiManager/onUserDataRetrieved()
at EventFunctor/notifyTarget()
at EventFunctor/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.frinje::EventBus/dispatchEvent()
at com.vmware.frinje::EventBus/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at Function/&lt;anonymous&gt;()
at Function/&lt;anonymous&gt;()
at Function/&lt;anonymous&gt;()
at com.vmware.flexutil::MxmlDeserializer/onThreadTerminated()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.flexutil::PseudoThread/terminate()
at com.vmware.flexutil::PseudoThread/onTimer()
at flash.utils::Timer/tick()

This does not allow web client to load. However HTML based web client works fine.
Following lines can be seen in vsphere_client_virgo.log (/var/log/vmware/vsphere-client/logs)

[2019-06-27T19:46:51.237Z] [ERROR] http-bio-9090-exec-3 70000894 100040 200011 com.vmware.vise.util.logging.LogServiceImpl [Flex|AppErrorHandler]Unable to load resource module from /veeam-zip-ui/locales/veeam-zip-ui-resources-en_US.swf
Error: Unable to load resource module from /veeam-zip-ui/locales/veeam-zip-ui-resources-en_US.swf
at MethodInfo-666()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfoProxy/moduleEventHandler()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfo/errorHandler()
[2019-06-27T19:46:52.384Z] [INFO ] http-bio-9090-exec-10 org.springframework.flex.servlet.MessageBrokerHandlerAdapter Channel endpoint amf received request.
[2019-06-27T19:46:52.514Z] [INFO ] http-bio-9090-exec-6 org.springframework.flex.servlet.MessageBrokerHandlerAdapter Channel endpoint amf received request.
[2019-06-27T19:46:52.515Z] [INFO ] http-bio-9090-exec-6 com.vmware.vise.util.session.SessionUtil Generated hashed session id: 100041
[2019-06-27T19:46:52.515Z] [INFO ] http-bio-9090-exec-6 com.vmware.vise.util.session.SessionUtil Mapping: /vsphere-client/telemetry-ui: sessionid to -&gt; 100041 with clientId: 200011
[2019-06-27T19:46:52.515Z] [INFO ] http-bio-9090-exec-6 70000895 100041 200011 com.vmware.vise.util.session.SessionUtil Associated sessionId 100041 with clientId 200011. Context path: /vsphere-client/telemetry-ui
[2019-06-27T19:46:52.665Z] [INFO ] http-bio-9090-exec-3 70000896 100040 200011 org.springframework.flex.servlet.MessageBrokerHandlerAdapter Channel endpoint amf received request.
[2019-06-27T19:46:52.666Z] [ERROR] http-bio-9090-exec-3 70000896 100040 200011 com.vmware.vise.util.logging.LogServiceImpl [Flex|AppErrorHandler]Unable to load resource module from /veeam-backup-ui/locales/VeeamBackupUi-en_US.swf
Error: Unable to load resource module from /veeam-backup-ui/locales/VeeamBackupUi-en_US.swf
at MethodInfo-666()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfoProxy/moduleEventHandler()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfo/errorHandler()

Potential solution:

  • AS per the error this is being caused by VEEAM backup client plugin. Veeam support may fix this issue by reinstalling the plugin. Most the common troubleshooting is mentioned in VEEAM KB
  • Some of the cases it is observed this error may cause because of workstation locale (language) is not English, in that case vCenter URL https://vCenter_IP/vsphere-client/?locale=en_US&csp with English can be use (just change vCenterIP OR Name of your vCenter).

Due to 3party plugin, many time you may encounter timeout while loading the inventory that takes more than default 120 seconds. When you expend inventory such datacenter/cluster or any other folder then at the bottom of the page you may see following warning.

The query execution timed out because of a back-end data adapter com.ni.eseries.vcenterWebClientui.ESeriesDataAdapter' which took more than 120 seconds.

At the same time vsphere_client_virgo.log (/var/log/vmware/vsphere-client/logs/ ) shows following lines.

[2019-06-14T08:15:10.716-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out Backend -- Find Extension --Key:com.netapp.santricity.vmware.vcenter.plugin
[2019-06-14T08:15:10.718-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out Backend -- Extension with Key com.netapp.santricity.vmware.vcenter.plugin Found
[2019-06-14T08:15:10.718-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out Backend -- Setup server URLs -- ReST:https://10.160.16.251:8084 Plugin:https://10.0.0.121:8084/vcenter2 Date:Fri Jun 14 08:15:10 AKDT 2019
[2019-06-14T08:15:10.718-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out VCenterPluginRest -- New Loaded Server URL: 'https://10.0.0.121:8084/vcenter2/vcenterRest' with date of Fri Jun 14 08:15:10 AKDT 2019
[2019-06-14T08:15:12.833-08:00] [WARN ] http-bio-9090-exec-12 com.vmware.vise.util.concurrent.ExecutorUtil Task 'Adapter com.vmware.opsmgmt.client.alarms.impl.AlarmIssueAdapter' timed out and was cancelled.
[2019-06-14T08:15:12.838-08:00] [ERROR] data-service-pool-796 70001202 100084 200043 com.vmware.opsmgmt.client.alarms.util.EntityNameRetriever Error retrieving entity names com.vmware.vise.data.query.DataServiceException: The data service execution was interrupted

[2019-06-14T08:15:12.838-08:00] [INFO ] data-service-pool-761 com.vmware.vise.util.session.SessionUtil Generated hashed session id: 100085
[2019-06-14T08:15:12.842-08:00] [INFO ] data-service-pool-761 com.vmware.vise.util.session.SessionUtil Mapping: ds/sdk: sessionid to -&gt; 100085 with clientId: 200043
[2019-06-14T08:15:12.848-08:00] [ERROR] data-service-pool-796 70001202 100084 200043 com.vmware.opsmgmt.client.alarms.util.EntityNameRetriever Error retrieving entity names com.vmware.vise.data.query.DataServiceException: The data service execution was interrupted
at com.vmware.vise.data.query.impl.DataServiceImpl.getResults(DataServiceImpl.java:364)
at com.vmware.vise.data.query.impl.DataServiceImpl.getResponse(DataServiceImpl.java:196)
at com.vmware.vise.data.query.impl.DataServiceImpl.getData(DataServiceImpl.java:171)
at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Here you can see the timeout issue is being caused by NetApp plugin. We can increase the default timeout as per KB OR contact NetAPP vendor to fix the issue with their add-in.

If you feel these issues are causing because of some of unwanted plugin installed in past then remove it using mob page, Refer KB