Unable to take ESXi configuration backup using PowerShell.

As per the KB you can take configuration backup of ESXI so it can be restore if something goes wrong during maintenance/upgrade/reinstall kind of action. Following PowerShell can be use from VMware CLI.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\backup


However sometime this command may fail with following error.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\temp
An error occurred while sending the request.
At line:1 char:1
+ Get-VMHostFirmware -VMHost esxi-1.domain.local -BackupConfiguration -D ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], ViError
    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_DownloadError,VMware.VimAutomation.ViCore.Cmdlets.


You may see following lines in vpxd.log (vCenter) and hostd.log (ESXi host).


2019-09-13T02:26:47.117Z info hostd[2098523] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424
2019-09-13T02:26:47.119Z info hostd[2099187] [Originator@6876 sub=SysCommandPosix opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] ForkExec(/sbin/firmwareConfig.sh) 2266493
2019-09-13T02:26:48.393Z info hostd[2099187] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424 Status success [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):


2019-09-13T02:28:26.134Z info vpxd[05841] [Originator@6876 sub=vpxLro opID=57b9c4a1] [VpxLRO] -- BEGIN lro-838226 -- firmwareSystem-161 -- vim.host.FirmwareSystem.backupConfiguration -- 521f50b8-5645-404f-11f8-f44099740a62(524a10f8-512b-637a-60ad-fb0b1d7510b6)
2019-09-13T02:28:29.300Z error vpxd[18796] [Originator@6876 sub=HostPicker opID=sps-Main-533116-133-60] [PickDoWork] Couldn't find any candidate host that satisfies all constraints


This particular problem comes when the port 80 (http) is blocked from your workstation to ESXi.  Following PS can be use to check the connectivity.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
WARNING: TCP connect to esxi-1.domain.local:80 failed
ComputerName           : esxi-1.domain.local
RemoteAddress          :
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          :
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : False


Verify if the firewall inside the ESXi OR external firewall blocking that communication.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access'
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   False   80                            TCP


If this is disabled then enabled using following command.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access' | Set-VMHostFirewallException -Enabled $True
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   True    80                            TCP


A successful connection should show like below.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
ComputerName           : esxi-1.domain.local
RemoteAddress          :
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          :
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : True


Make sure the IP of your workstation is present in allowed list. That can be done using the vCenter OR ESXi web console.



vCenter Appliance Network configuration change via command line.

Most of the system configuration such as changing the hostname, IP, DNS..etc can be done using VAMI interface (https://vCenterIPorFQDN:5480) however if you see the changes are not taking affect, e.g. changing the DNS IP then following command can be used.

  • Enable SSH using VAMI interface, select Access tab, click Edit and Enable SSH login.
  • Take SSH session via putty to connect vCenter Appliance.
  • Use following command and change required configuration change.
root@vcsa[~]# /opt/vmware/share/vami/vami_config_net
Main Menu
0)      Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0


Here are some samples:-

  • Review current configuration.
Enter a menu number [0]: 0
Network Configuration for eth0
IPv4 Address:
IPv6 Address:
Global Configuration
IPv4 Gateway:
IPv6 Gateway:
Hostname:       vcsa1.lab.org
DNS Servers:,
Domain Name:
Search Path:
Proxy Server:

Whenever you make any changes you will be prompted for following warning message.

Warning: if any of the interfaces for this VM use DHCP, the Hostname, DNS, and Gateway parameters will be overwritten by information from the DHCP server. Type Ctrl-C to go back to the Main Menu

  • Changing hostname
Enter a menu number [0]: 3
New hostname [vcsa1.lab.org]:vcsa2.lab.org

  • Changing DNS server and Domain name.
Enter a menu number [0]: 4
DNS Server 1 []:
DNS Server 2 (optional) []:
Domain Name (optional) []: lab.org
Search Path (space separated) (optional) []: lab.org
DNS server settings updated

  • Changing proxy address of vCenter appliance
Enter a menu number [0]: 5
Is an IPv4 proxy server necessary to reach the Internet? y/n [n]: y
Proxy Server (http:// will be auto prepended) []: proxy1.lab.org
Proxy Port []: 8080

  • Changing IP address.
Enter a menu number [0]: 6
Configure an IPv6 address for eth0? y/n [n]: n
Configure an IPv4 address for eth0? y/n [n]: y
Use a DHCPv4 Server instead of a static IPv4 address? y/n [n]: n
IPv4 Address []:
Netmask []:
IPv4 Address:

Is this correct? y/n [y]: y

Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 1
Network parameters successfully changed to requested values

Configure LDAPS authentication for vCenter Server.

Following steps can help to configure Active directory LDAPs Authentication for vCenter servers.

Step 1

Note down the DC (Domain controller) assigned with LDAP. If you want to know all domain controllers following windows command can be used. This can be executed from any windows machine that is joined to AD domain.

nltest /dclist:DomainName

Step 2

Select one of the Domain controller that is configured as LDAP identity source. Login to vCenter appliance using SSH session (use Putty/Terminal access) to get LDAP certificate from DC.

openssl s_client -connect DC1.ad.local:636 -showcerts

Replace DC1.ad.local as the domain controller of your environment. The top most certificate in this chain is the certificate of the domain controller.


Copy the complete string including —–BEGIN CERTIFICATE—– until (including) —–END CERTIFICATE—– into a text file. Remove any additional characters after —–END CERTIFICATE—–. Save that content into Notepad file and save file with as cer extension (e.g. ldap_dc.cer).


Step 3

Open vCenter web client (HTML/Flash). Go to \Home\Administration \ Configuration under Single Sign-on\ click on + sign \ select Active directory as an LDAP server

Give the appropriate name following options.

vCenter 6.0

Name = domain name
Base DN for users: dc=domainname,dc=local
(This option to search user's in specific organization unit OR container of AD.)
Domain name: domainname.local
Domain alias: domainname
Base DN for groups: dc=domainname,dc=local
(This option to search AD group's in a specific organization unit OR container of AD.)
Primary server URL: ldaps://DC1.ad.local:636
(You can mentioned domain instead of specific DC if all your domain controller configured to use SSL for LDAP.)
Secondary server URL:ldaps://DC2.ad.local:636
(This is optional)



vCenter 6.5/6.7

Name = domain name
Base DN for users: dc=domainname,dc=local
Base DN for groups: dc=domainname,dc=local
Domain name: domainname.local
Domain alias: domainname
User name = adminuser@domain.local
Password ****

When you select Connect to any domain controller in the domain then vCenter connects to DC that is acting as primary domain controller (PDC). NLTest output will tell you the current primary domain controller. This option may not work for version prior to 6.7 U1 OR 6.5 U2D due to known issue. Refer The workaround is to download LDAP certificate for all DC’s (DC list can be obtain from NSTest as mentioned in point 1) then provide certificate in next step of configuration.

You also have ability to specify primary and second LDAP servers.

Primary server URL: ldaps://DC1.ad.local:636
Secondary server URL:ldaps://DC2.ad.local:636


In the next screen, upload the certificate downloaded in step2


If all the configuration is correct then Active Directory as an LDAP server should be added without any issue.

Un-handled Exception with ESXi UI

Following two problems are seen with ESXi version 6.7 Update 2 (Build 13006603).

  • Using host UI (https://HOSTNameORIP/ui ) when you click on Storage then Adapter then  following unhandled exception comes. This stop configuring storage related activity from ESXi.
Unhandled exception
Unfortunately, we hit an error that we weren't expecting.
The client may continue working, but at this point,
we recommend refreshing your browser and submitting a bug report.
Press the Esc key to hide this dialog and continue without refreshing


Clicking on Details shows following lines.

Cause: Possibly unhandled rejection: {}
Version: 1.33.3
Build: 12923304
ESXi: 6.7.0
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Exception stack:


  • Using host UI when you import virtual machine using OVF template following stack is seen.
TypeError: Cannot read property 'keyValue' of undefined
    at updateSummaryPortlet (
    at $scope.wizardOptions.onFinish (
    at m.$digest (
    at e (


VMkernel.log shows following lines.

2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)World: 11943: VC opID esxui-4004-bd50 maps to vmkernel opID 27da76af
2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)NVDManagement: 1461: No nvdimms found on the system


Clicking on Reload option log out ESXi web UI and you have to login to UI again.

This particular problem is fixed in ESXi build number 13981272 (ESXi 6.7 EP 10) OR later version. Refer KB  to know more about ESXi build version.
ESXi Patch download link
Select ESXi and version 6.7 then search to get latest patch.

The workaround is to press escape after opening the stack OR to import OVF using vCenter UI if host is part of vCenter, same for changing storage settings. 

VMware Appliance Monitoring Service (vmware-statsmonitor) doesn’t start.

After reboot vmware-statsmonitor service doesn’t start automatically. In some situation, this service even doesn’t start manually and fail with following error.

root@buildvcenter [ ~ ]# service-control --start vmware-statsmonitor
Operation not cancellable. Please wait for it to finish...
Performing start operation on service statsmonitor...
channel 3: open failed: administratively prohibited: open failed
Error executing start on service statsmonitor. Details {
    "detail": [
            "args": [
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'statsmonitor'",
            "translatable": "An error occurred while starting service '%(0)s'"
    "problemId": null,
    "resolution": null,
    "componentKey": null


Following line can be seen in Vmon log (/var/log/vmware/vmon/vmon-syslog.log)

2019-08-08T09:26:42.809220-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:42.809574-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:45.810708-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:51.166333-07:00 warning vmon   Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /var/vmware/applmgmt/statsmonitor_health.xml. Error: [Errno 2] No such file or directory: '/var/vmware/applmgmt/statsmonitor_health.xml'
2019-08-08T09:26:51.166701-07:00 warning vmon
2019-08-08T09:26:51.194457-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:51.194832-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:54.195953-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:58.391456-07:00 notice vmon   Service start operation timed out.
2019-08-08T09:26:58.394656-07:00 notice vmon   Cancelling execution of pid 38052
2019-08-08T09:26:58.395009-07:00 warning vmon   Found empty StopSignal parameter in config file. Defaulting to SIGTERM

This issue happens because of startup delay and can be fix by following steps:-

  • Take snapshot of vCenter to be in safer side.
  • Take SSH to VCSA using root login.
  • Modify statsmonitor service config for vMon to set higher start up timeout:
sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
kill -HUP $(cat /var/run/vmon.pid)
  • Stop and start stats-monitor service explicitly.
/usr/lib/vmware-vmon/vmon-cli -k statsmonitor
/usr/lib/vmware-vmon/vmon-cli -i statsmonitor
  • Then restart vCenter to see if that service start automatically

Unable to clone windows VM using vCenter.

You may see following task and event.


An error occurred while quiescing the virtual machine. See the virtual machine's event log for details
An error occurred while taking a snapshot: Failed to quiesce the virtual machine
An error occurred while saving the snapshot: Failed to quiesce the virtual machine.



Warning message on VMNAME on HOSTFQDN in Datacenter: The guest OS has reported an error during quiescing. The error code was: 5 The error message was: 'VssSyncStart' operation failed: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (0x80070422)


Similar error may come during backup/snapshot of virtual machine.

During cloning process, vSphere take snapshot with option “Quiesce Guest File System” for windows VM internally. During clone OR snapshot (when option “Quiesce Guest File System” is selected) OR back above error message can be seen.

What is Quiesce?

VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups/snapshot. This process might include such operations as flushing dirty buffers from the operating system’s in-memory cache to disk, or other higher-level application-specific tasks. Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup. Quiescing is not necessary for memory snapshots; it is used primarily for backups.

Here is the solution that should work in most of the situation.

  • Take console/ RDP to windows virtual machine. Open services.msc.
  • Ensure that Virtual Disk service is started and startup type is Automatic.
  • Ensure that VMware snapshot provider service is stopped and Disabled.
  • Ensure that VMware Tools service is running.
  • Ensure that Volume Shadow Copy service started and start up type is Automatic.
  • Ensure that the VMware tools version is up to date.

Clone the VM (OR run test Backup job or take a quiesced snapshot) using the vSphere Client.


Remote access for ESXi local user account ‘root’ has been locked for XXXX seconds after XXXX failed login attempts.

Due to consistent wrong password by application OR user, root a/c of ESXi may get locked out. This prevent any further login externally to ESXi host.

This happens most likely due to backup/monitoring application that has root user a/c configured the operation.

To solve this issue.

  • Take Console access (iDRAC/ILO/KVM..etc depending on hardware).
  • Press F2 to customize the system. Log in as root.
  • Use the Up/Down arrows to navigate to Troubleshooting Options > Enable ESXi Shell.
  • Press ALT+F1, login as root. Run the following commands to show number of failed attempts:
pam_tally2 --user root
  • Run the following command to unlock the root account:
pam_tally2 --user root --reset


Now you should be able to login to ESXi using root a/. You can review hostd log to find out from where failed login coming.

less /var/log/hostd.log |grep -i 'password'

2019-07-31T17:08:51.735Z info hostd[2099345] [Originator@6876 sub=Default 2019-07-31T20:51:07.055Z warning hostd[2205446] [Originator@6876 sub=Default opID=esxui-fca4-b52f] Rejected password for user root from
2019-07-31T20:51:11.056Z verbose hostd[2205444] [Originator@6876 sub=Solo.Vmomi] Arg password:
2019-07-31T20:51:39.634Z warning hostd[2099346] [Originator@6876 sub=Default opID=esxui-c6d2-b54a] Rejected password for user root from


Based on IP address check if any application configured with incorrect root password.

Please note in ESXi 6.5/6.7, invalid login may cause host to be unresponsive. Review KB  for proactive steps.