vMotion may fail because of TegileNasPlugin installed on ESXi host.

You may see following lines in VMware.log of the VM that was attempted to migrate.

2019-09-12T18:05:51.309Z| Worker#1| I125: DISK: OPEN scsi1:0 '/vmfs/volumes/b57aef17-591cfc64/NAS1/NAS1_1.vmdk' persistent R[]
2019-09-12T18:05:51.315Z| Worker#2| I125: TegileNasPlugin: StartSession: Req for 10.1.192.42 /export/ESX_5/ZEBI_ESX4 /vmfs/volumes/b57aef17-591cfc64 NFS 1
2019-09-12T18:05:51Z[+0.000]| Worker#2| W115: Caught signal 11 -- tid 190820 (addr 6379C5FA4C)
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: rip 0x6379c5fa4c rsp 0x637d6bec18 rbp 0x637d6bec50
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: rax 0x2f rbx 0x6338557110 rcx 0x20 rdx 0x656d756c rsi 0x6338572fa0 rdi 0x6f762f73666d762f
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125:         r8 0x0 r9 0x2a r10 0x1999999999999999 r11 0x656d756c r12 0x6338572fa0 r13 0x637d6becb4 r14 0x637d6becb0 r15 0x6338505c00
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: stack 637D6BEC18 : 0x000000637c77c4c1 0x0000000000000000
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: SIGNAL: stack 637D6BEC28 : 0x1e840cb0799abf47 0x0000006338572fa0
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace:
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace[0] 000000637d6be690 rip=00000063370c39c7 rbx=00000063370c34c0 rbp=000000637d6be6b0 r12=0000000000000000 r13=0000006337d64d01 r14=000000000000000a r15=000000637d6bec98
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace[1] 000000637d6be6c0 rip=00000063372f64c0 rbx=000000637d6bec98 rbp=000000637d6be8d0 r12=000000000000000b r13=0000006337d64d01 r14=000000000000000a r15=000000637d6bec98
2019-09-12T18:05:51Z[+0.001]| Worker#2| I125: Backtrace[2] 000000637d6be8e0 rip=00000063372f68c4 rbx=0000000000000008 rbp=000000637d6be930 r12=000000637d6c0538 r13=0000006337d64dc8 r14=0000006337d64dc0 r15=000000000000000b
2019-09-12T18:05:51.316Z| Worker#3| I125: TegileNasPlugin: StartSession: Req for 10.1.192.42 /export/ESX_5/ZEBI_ESX4 /vmfs/volumes/b57aef17-591cfc64 NFS 1
2019-09-12T18:05:51Z[+0.000]| Worker#2| I125: Backtrace[3] 000000637d6be940 rip=000000000038600f rbx=0000006338557110 rbp=000000637d6beb80 r12=000000637d6be9c0 r13=000000637d6becb4 r14=000000637d6becb0 r15=0000006338505c00

 

VMkernel.log (source ESXi host).

2019-09-12T18:05:56.302Z cpu29:81337)WARNING: Migrate: 273: 2017306294537764927 S: Failed: Failed to resume virtual machine (0xbad0044) @0x418018883122
2019-09-12T18:05:56.302Z cpu29:81337)VMotionRecv: 3733: 2017306294537764927 S: Error handling message: Connection reset by peer
2019-09-12T18:05:56.312Z cpu27:76130)WARNING: Migrate: 6279: 2017306294537764927 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
2019-09-12T18:05:56.343Z cpu44:76290)CBT: 1341: Created device 20d0803-cbt for cbt driver with filehandle 34408451

 

VMWarning.log (source ESXi host).

2019-09-12T14:38:17.513Z cpu10:77420)WARNING: CBT: 1133: Unsupported ioctl 62
2019-09-12T18:05:56.302Z cpu29:81337)WARNING: Migrate: 273: 2017306294537764927 S: Failed: Failed to resume virtual machine (0xbad0044) @0x418018883122
2019-09-12T18:05:56.312Z cpu27:76130)WARNING: Migrate: 6279: 2017306294537764927 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
2019-09-12T18:05:56.343Z cpu44:76290)WARNING: CBT: 1133: Unsupported ioctl 63

 

VMKernel.log (destination ESXi host).

2019-09-12T18:05:56.337Z cpu22:190653)Hbr: 3489: Migration end received (worldID=190654) (migrateType=1) (event=1) (isSource=0) (sharedConfig=1)
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 6749: 2017306294537764927 D: Migration cleanup initiated, the VMX has exited unexpectedly. Check the VMX log for more details.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 273: 2017306294537764927 D: Failed: Migration determined a failure by the VMX (0xbad0092) @0x4180296b3091
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: VMotionUtil: 7649: 2017306294537764927 D: timed out waiting 0 ms to transmit data.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: World: vm 190653: 3566: VMMWorld group leader = 190654, members = 2
2019-09-12T18:05:56.337Z cpu32:190660)VMotionUtil: 7552: 2017306294537764927 D: Socket 0x430a8b349c00 sendSocket pending: 563164/563272 snd 0 rcv
2019-09-12T18:05:56.525Z cpu14:65689)CBT: 1376: Destroying device 5920950-cbt for cbt driver with filehandle 93456720

 

VObd.log (destination ESXi host).

2019-09-12T18:05:56.333Z: [UserWorldCorrelator] 244452187008us: [vob.uw.core.dumped] /bin/vmx(190653) /var/core/vmx-zdump.001
2019-09-12T18:05:56.333Z: [UserWorldCorrelator] 244476489345us: [esx.problem.application.core.dumped] An application (/bin/vmx) running on ESXi host has crashed (2 time(s) so far). A core file may have been created at /var/core/vmx-zdump.001.

 

VMKwarning.log (destination ESXi host).

2019-09-12T18:05:51.304Z cpu22:190653)WARNING: CBT: 1133: Unsupported ioctl 62
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 6749: 2017306294537764927 D: Migration cleanup initiated, the VMX has exited unexpectedly. Check the VMX log for more details.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: Migrate: 273: 2017306294537764927 D: Failed: Migration determined a failure by the VMX (0xbad0092) @0x4180296b3091
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: VMotionUtil: 7649: 2017306294537764927 D: timed out waiting 0 ms to transmit data.
2019-09-12T18:05:56.337Z cpu22:190653)WARNING: World: vm 190653: 3566: VMMWorld group leader = 190654, members = 2

 

This is caused because of TegileNasPlugin installed on ESXI host. Make sure you connect with NAS vendor

 

#tegilenasplugin, #vcenter, #vmotion, #vmware-log

Unable to take ESXi configuration backup using PowerShell.

As per the KB you can take configuration backup of ESXI so it can be restore if something goes wrong during maintenance/upgrade/reinstall kind of action. Following PowerShell can be use from VMware CLI.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\backup

 

However sometime this command may fail with following error.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\temp
An error occurred while sending the request.
At line:1 char:1
+ Get-VMHostFirmware -VMHost esxi-1.domain.local -BackupConfiguration -D ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], ViError
    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_DownloadError,VMware.VimAutomation.ViCore.Cmdlets.
   Commands.Host.GetVMHostFirmware

 

You may see following lines in vpxd.log (vCenter) and hostd.log (ESXi host).

Hostd.log

2019-09-13T02:26:47.117Z info hostd[2098523] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424
2019-09-13T02:26:47.119Z info hostd[2099187] [Originator@6876 sub=SysCommandPosix opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] ForkExec(/sbin/firmwareConfig.sh) 2266493
2019-09-13T02:26:48.393Z info hostd[2099187] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424 Status success [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

VPXD.log

2019-09-13T02:28:26.134Z info vpxd[05841] [Originator@6876 sub=vpxLro opID=57b9c4a1] [VpxLRO] -- BEGIN lro-838226 -- firmwareSystem-161 -- vim.host.FirmwareSystem.backupConfiguration -- 521f50b8-5645-404f-11f8-f44099740a62(524a10f8-512b-637a-60ad-fb0b1d7510b6)
2019-09-13T02:28:29.300Z error vpxd[18796] [Originator@6876 sub=HostPicker opID=sps-Main-533116-133-60] [PickDoWork] Couldn't find any candidate host that satisfies all constraints

 

This particular problem comes when the port 80 (http) is blocked from your workstation to ESXi.  Following PS can be use to check the connectivity.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
WARNING: TCP connect to esxi-1.domain.local:80 failed
ComputerName           : esxi-1.domain.local
RemoteAddress          : 192.168.0.81
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          : 192.168.0.10
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : False

 

Verify if the firewall inside the ESXi OR external firewall blocking that communication.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access'
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   False   80                            TCP

 

If this is disabled then enabled using following command.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access' | Set-VMHostFirewallException -Enabled $True
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   True    80                            TCP

 

A successful connection should show like below.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
ComputerName           : esxi-1.domain.local
RemoteAddress          : 192.168.0.81
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          : 192.168.0.10
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : True

 

Make sure the IP of your workstation is present in allowed list. That can be done using the vCenter OR ESXi web console.

allowedip.jpg

vCenter Appliance Network configuration change via command line.

Most of the system configuration such as changing the hostname, IP, DNS..etc can be done using VAMI interface (https://vCenterIPorFQDN:5480) however if you see the changes are not taking affect, e.g. changing the DNS IP then following command can be used.

  • Enable SSH using VAMI interface, select Access tab, click Edit and Enable SSH login.
  • Take SSH session via putty to connect vCenter Appliance.
  • Use following command and change required configuration change.
root@vcsa[~]# /opt/vmware/share/vami/vami_config_net
Main Menu
0)      Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0

 

Here are some samples:-

  • Review current configuration.
Enter a menu number [0]: 0
Network Configuration for eth0
IPv4 Address:   192.168.0.51
Netmask:        255.255.255.0
IPv6 Address:
Prefix:
Global Configuration
IPv4 Gateway:   192.168.0.1
IPv6 Gateway:
Hostname:       vcsa1.lab.org
DNS Servers:    127.0.0.1, 192.168.0.10
Domain Name:
Search Path:
Proxy Server:

Whenever you make any changes you will be prompted for following warning message.

Warning: if any of the interfaces for this VM use DHCP, the Hostname, DNS, and Gateway parameters will be overwritten by information from the DHCP server. Type Ctrl-C to go back to the Main Menu

  • Changing hostname
Enter a menu number [0]: 3
New hostname [vcsa1.lab.org]:vcsa2.lab.org

  • Changing DNS server and Domain name.
Enter a menu number [0]: 4
DNS Server 1 [127.0.0.1]: 192.168.0.10
DNS Server 2 (optional) [192.168.0.10]: 192.168.0.11
Domain Name (optional) []: lab.org
Search Path (space separated) (optional) []: lab.org
DNS server settings updated

  • Changing proxy address of vCenter appliance
Enter a menu number [0]: 5
Is an IPv4 proxy server necessary to reach the Internet? y/n [n]: y
Proxy Server (http:// will be auto prepended) []: proxy1.lab.org
Proxy Port []: 8080

  • Changing IP address.
Enter a menu number [0]: 6
Configure an IPv6 address for eth0? y/n [n]: n
Configure an IPv4 address for eth0? y/n [n]: y
Use a DHCPv4 Server instead of a static IPv4 address? y/n [n]: n
IPv4 Address [192.168.0.51]: 192.168.0.51
Netmask [255.255.255.0]: 255.255.255.0
IPv4 Address:   192.168.0.51
Netmask:        255.255.255.0

Is this correct? y/n [y]: y

Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 1
Network parameters successfully changed to requested values

Configure LDAPS authentication for vCenter Server.

Following steps can help to configure Active directory LDAPs Authentication for vCenter servers.

Step 1

Note down the DC (Domain controller) assigned with LDAP. If you want to know all domain controllers following windows command can be used. This can be executed from any windows machine that is joined to AD domain.

nltest /dclist:DomainName

Step 2

Select one of the Domain controller that is configured as LDAP identity source. Login to vCenter appliance using SSH session (use Putty/Terminal access) to get LDAP certificate from DC.

openssl s_client -connect DC1.ad.local:636 -showcerts

Replace DC1.ad.local as the domain controller of your environment. The top most certificate in this chain is the certificate of the domain controller.

-----BEGIN CERTIFICATE-----
MIIC4DCCAcigAwIBAgIQJ3hiT2fQzIBLYFPywfvCgjANBgkqhkiG9w0BAQUFADAZ
MwMDAwMDBaMBkxFzAVBgNVBAMTDmFkLmdzc2xhYnMub3JnMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7JFQshqvAH+bsej+FE6IYf3LA38EpMmnsCJV
nvvX1RXoHs5tr8iwbm6fMggRHZA8jHY3Z/wnLkh1Ct+8MylrGVRL4MB1bXeSH7MT
TTCMCI/ikokCO6vkVlG1RP/YcMOIUCLERsgJiZ8qCEZYLdw8ioZuA1kaGQkiJRy8
KZI5lz4nqV9owks1e4TW5TtCTDqorYxBz2x2PsZLTih/fgLf9kRr0QUHc/f8TMuI
3LWdGdodxUKKAP7cHU5awhsOdiDjqWEuYA4gioog0Dd9sE111JvPP0opSPMgnMpf
CWOc04z8dqkR15BChG36Gvgqqbnf77vknDe1RgkFhyK6GjKGTQIDAQABoyQwIjAL
BgNVHQ8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQEFBQAD
ggEBAC8sNBB5e5WffE9VjU5zcDqvOQqE24XD1bdFeKW/ud6aYwmF5YV4wFpEGkA9
AnmCDTsxtHiRytwnN8uGll9acBCs8VQaB1HZ33GxdzNfIgtCq4XPlhHrO1+YU3+g
bez2zI5TKVnm2XE4mpwyZHSbbiXzh2SbAQI1QTde9slTFTkib0HsMZYxBE5Xsgdq
RXUX6xvU2sMbHevj13zkGfoF71T72ddq78LTCbrX3EU0jYbHhrKTqRc6qHAv9fz4
2z8xKysVs+CCx8g+qEm+igMxb9/XdA2HUOA8l+NDlH/qS78e9ty0XNayl8ZC/7bZ
cKk5wfWIbFHIIBMbl7PY2eaQK8c=
-----END CERTIFICATE-----

Copy the complete string including —–BEGIN CERTIFICATE—– until (including) —–END CERTIFICATE—– into a text file. Remove any additional characters after —–END CERTIFICATE—–. Save that content into Notepad file and save file with as cer extension (e.g. ldap_dc.cer).

 

Step 3

Open vCenter web client (HTML/Flash). Go to \Home\Administration \ Configuration under Single Sign-on\ click on + sign \ select Active directory as an LDAP server

Give the appropriate name following options.

vCenter 6.0

Name = domain name
Base DN for users: dc=domainname,dc=local
(This option to search user's in specific organization unit OR container of AD.)
Domain name: domainname.local
Domain alias: domainname
Base DN for groups: dc=domainname,dc=local
(This option to search AD group's in a specific organization unit OR container of AD.)
Primary server URL: ldaps://DC1.ad.local:636
(You can mentioned domain instead of specific DC if all your domain controller configured to use SSL for LDAP.)
Secondary server URL:ldaps://DC2.ad.local:636
(This is optional)

 

ldap_6.0.jpg

vCenter 6.5/6.7

Name = domain name
Base DN for users: dc=domainname,dc=local
Base DN for groups: dc=domainname,dc=local
Domain name: domainname.local
Domain alias: domainname
User name = adminuser@domain.local
Password ****

When you select Connect to any domain controller in the domain then vCenter connects to DC that is acting as primary domain controller (PDC). NLTest output will tell you the current primary domain controller. This option may not work for version prior to 6.7 U1 OR 6.5 U2D due to known issue. Refer The workaround is to download LDAP certificate for all DC’s (DC list can be obtain from NSTest as mentioned in point 1) then provide certificate in next step of configuration.

You also have ability to specify primary and second LDAP servers.

Primary server URL: ldaps://DC1.ad.local:636
Secondary server URL:ldaps://DC2.ad.local:636

ldap_6.7.jpg

In the next screen, upload the certificate downloaded in step2

ldap_6.7_2.jpg

If all the configuration is correct then Active Directory as an LDAP server should be added without any issue.

Un-handled Exception with ESXi UI

Following two problems are seen with ESXi version 6.7 Update 2 (Build 13006603).

  • Using host UI (https://HOSTNameORIP/ui ) when you click on Storage then Adapter then  following unhandled exception comes. This stop configuring storage related activity from ESXi.
Unhandled exception
-----------
Unfortunately, we hit an error that we weren't expecting.
The client may continue working, but at this point,
we recommend refreshing your browser and submitting a bug report.
Press the Esc key to hide this dialog and continue without refreshing

 

Clicking on Details shows following lines.

Cause: Possibly unhandled rejection: {}
Version: 1.33.3
Build: 12923304
ESXi: 6.7.0
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Exception stack:
this.toAddress64@https://192.168.100.11/ui/scripts/main.js:422:206
updateAdapter@https://192.168.100.11/ui/scripts/main.js:365:26206
@https://192.168.100.11/ui/scripts/main.js:365:26542
k/<@https://192.168.100.11/ui/scripts/main.js:324:23176
$digest@https://192.168.100.11/ui/scripts/main.js:324:28780
$evalAsync/<@https://192.168.100.11/ui/scripts/main.js:324:30503
e@https://192.168.100.11/ui/scripts/main.js:323:10071
tg/h.defer/c<@https://192.168.100.11/ui/scripts/main.js:323:11522

 

  • Using host UI when you import virtual machine using OVF template following stack is seen.
TypeError: Cannot read property 'keyValue' of undefined
    at updateSummaryPortlet (https://192.168.10.11/ui/scripts/main.js:375:415)
    at $scope.wizardOptions.onFinish (https://192.168.10.11/ui/scripts/main.js:375:5968)
    at https://192.168.10.11/ui/scripts/main.js:324:23176
    at m.$digest (https://192.168.10.11/ui/scripts/main.js:324:28780)
    at https://192.168.10.11/ui/scripts/main.js:324:30503
    at e (https://192.168.10.11/ui/scripts/main.js:323:10071)
    at https://192.168.10.11/ui/scripts/main.js:323:11522

 

VMkernel.log shows following lines.

2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)World: 11943: VC opID esxui-4004-bd50 maps to vmkernel opID 27da76af
2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)NVDManagement: 1461: No nvdimms found on the system

 

Clicking on Reload option log out ESXi web UI and you have to login to UI again.

This particular problem is fixed in ESXi build number 13981272 (ESXi 6.7 EP 10) OR later version. Refer KB  to know more about ESXi build version.
ESXi Patch download link
Select ESXi and version 6.7 then search to get latest patch.

The workaround is to press escape after opening the stack OR to import OVF using vCenter UI if host is part of vCenter, same for changing storage settings. 

VMware Appliance Monitoring Service (vmware-statsmonitor) doesn’t start.

After reboot vmware-statsmonitor service doesn’t start automatically. In some situation, this service even doesn’t start manually and fail with following error.

root@buildvcenter [ ~ ]# service-control --start vmware-statsmonitor
Operation not cancellable. Please wait for it to finish...
Performing start operation on service statsmonitor...
channel 3: open failed: administratively prohibited: open failed
Error executing start on service statsmonitor. Details {
    "detail": [
        {
            "args": [
                "statsmonitor"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'statsmonitor'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "problemId": null,
    "resolution": null,
    "componentKey": null
}

 

Following line can be seen in Vmon log (/var/log/vmware/vmon/vmon-syslog.log)

2019-08-08T09:26:42.809220-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:42.809574-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:45.810708-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:51.166333-07:00 warning vmon   Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /var/vmware/applmgmt/statsmonitor_health.xml. Error: [Errno 2] No such file or directory: '/var/vmware/applmgmt/statsmonitor_health.xml'
2019-08-08T09:26:51.166701-07:00 warning vmon
2019-08-08T09:26:51.194457-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:51.194832-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:54.195953-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:58.391456-07:00 notice vmon   Service start operation timed out.
2019-08-08T09:26:58.394656-07:00 notice vmon   Cancelling execution of pid 38052
2019-08-08T09:26:58.395009-07:00 warning vmon   Found empty StopSignal parameter in config file. Defaulting to SIGTERM

This issue happens because of startup delay and can be fix by following steps:-

  • Take snapshot of vCenter to be in safer side.
  • Take SSH to VCSA using root login.
  • Modify statsmonitor service config for vMon to set higher start up timeout:
sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
kill -HUP $(cat /var/run/vmon.pid)
  • Stop and start stats-monitor service explicitly.
/usr/lib/vmware-vmon/vmon-cli -k statsmonitor
/usr/lib/vmware-vmon/vmon-cli -i statsmonitor
  • Then restart vCenter to see if that service start automatically

Unable to clone windows VM using vCenter.

You may see following task and event.

Task:

An error occurred while quiescing the virtual machine. See the virtual machine's event log for details
An error occurred while taking a snapshot: Failed to quiesce the virtual machine
An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

 

Events:

Warning message on VMNAME on HOSTFQDN in Datacenter: The guest OS has reported an error during quiescing. The error code was: 5 The error message was: 'VssSyncStart' operation failed: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (0x80070422)

 

Similar error may come during backup/snapshot of virtual machine.

During cloning process, vSphere take snapshot with option “Quiesce Guest File System” for windows VM internally. During clone OR snapshot (when option “Quiesce Guest File System” is selected) OR back above error message can be seen.

What is Quiesce?

VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups/snapshot. This process might include such operations as flushing dirty buffers from the operating system’s in-memory cache to disk, or other higher-level application-specific tasks. Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup. Quiescing is not necessary for memory snapshots; it is used primarily for backups.

Here is the solution that should work in most of the situation.

  • Take console/ RDP to windows virtual machine. Open services.msc.
  • Ensure that Virtual Disk service is started and startup type is Automatic.
  • Ensure that VMware snapshot provider service is stopped and Disabled.
  • Ensure that VMware Tools service is running.
  • Ensure that Volume Shadow Copy service started and start up type is Automatic.
  • Ensure that the VMware tools version is up to date.

Clone the VM (OR run test Backup job or take a quiesced snapshot) using the vSphere Client.