Unable to take ESXi configuration backup using PowerShell.

As per the KB you can take configuration backup of ESXI so it can be restore if something goes wrong during maintenance/upgrade/reinstall kind of action. Following PowerShell can be use from VMware CLI.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\backup

 

However sometime this command may fail with following error.

PS C:\> Get-VMHostFirmware -VMHost esxi-1.gsslabs.org -BackupConfiguration -DestinationPath c:\temp
An error occurred while sending the request.
At line:1 char:1
+ Get-VMHostFirmware -VMHost esxi-1.domain.local -BackupConfiguration -D ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], ViError
    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_DownloadError,VMware.VimAutomation.ViCore.Cmdlets.
   Commands.Host.GetVMHostFirmware

 

You may see following lines in vpxd.log (vCenter) and hostd.log (ESXi host).

Hostd.log

2019-09-13T02:26:47.117Z info hostd[2098523] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424
2019-09-13T02:26:47.119Z info hostd[2099187] [Originator@6876 sub=SysCommandPosix opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] ForkExec(/sbin/firmwareConfig.sh) 2266493
2019-09-13T02:26:48.393Z info hostd[2099187] [Originator@6876 sub=Vimsvc.TaskManager opID=8b66f7f-a7-1965 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask--vim.host.FirmwareSystem.backupConfiguration-3991232424 Status success [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

VPXD.log

2019-09-13T02:28:26.134Z info vpxd[05841] [Originator@6876 sub=vpxLro opID=57b9c4a1] [VpxLRO] -- BEGIN lro-838226 -- firmwareSystem-161 -- vim.host.FirmwareSystem.backupConfiguration -- 521f50b8-5645-404f-11f8-f44099740a62(524a10f8-512b-637a-60ad-fb0b1d7510b6)
2019-09-13T02:28:29.300Z error vpxd[18796] [Originator@6876 sub=HostPicker opID=sps-Main-533116-133-60] [PickDoWork] Couldn't find any candidate host that satisfies all constraints

 

This particular problem comes when the port 80 (http) is blocked from your workstation to ESXi.  Following PS can be use to check the connectivity.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
WARNING: TCP connect to esxi-1.domain.local:80 failed
ComputerName           : esxi-1.domain.local
RemoteAddress          : 192.168.0.81
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          : 192.168.0.10
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : False

 

Verify if the firewall inside the ESXi OR external firewall blocking that communication.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access'
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   False   80                            TCP

 

If this is disabled then enabled using following command.

PS C:\> Get-VMHostFirewallException -VMHost esxi-1.domain.local -Name 'vSphere Web access' | Set-VMHostFirewallException -Enabled $True
Name                 Enabled IncomingPorts  OutgoingPorts  Protocols  ServiceRunning
----                 ------- -------------  -------------  ---------  --------------
vSphere Web Access   True    80                            TCP

 

A successful connection should show like below.

PS C:\> Test-NetConnection -Port 80 -ComputerName esxi-1.domain.local
ComputerName           : esxi-1.domain.local
RemoteAddress          : 192.168.0.81
RemotePort             : 80
InterfaceAlias         : Ethernet0
SourceAddress          : 192.168.0.10
PingSucceeded          : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : True

 

Make sure the IP of your workstation is present in allowed list. That can be done using the vCenter OR ESXi web console.

allowedip.jpg

Advertisements

vCenter Appliance Network configuration change via command line.

Most of the system configuration such as changing the hostname, IP, DNS..etc can be done using VAMI interface (https://vCenterIPorFQDN:5480) however if you see the changes are not taking affect, e.g. changing the DNS IP then following command can be used.

  • Enable SSH using VAMI interface, select Access tab, click Edit and Enable SSH login.
  • Take SSH session via putty to connect vCenter Appliance.
  • Use following command and change required configuration change.
root@vcsa[~]# /opt/vmware/share/vami/vami_config_net
Main Menu
0)      Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0

 

Here are some samples:-

  • Review current configuration.
Enter a menu number [0]: 0
Network Configuration for eth0
IPv4 Address:   192.168.0.51
Netmask:        255.255.255.0
IPv6 Address:
Prefix:
Global Configuration
IPv4 Gateway:   192.168.0.1
IPv6 Gateway:
Hostname:       vcsa1.lab.org
DNS Servers:    127.0.0.1, 192.168.0.10
Domain Name:
Search Path:
Proxy Server:

Whenever you make any changes you will be prompted for following warning message.

Warning: if any of the interfaces for this VM use DHCP, the Hostname, DNS, and Gateway parameters will be overwritten by information from the DHCP server. Type Ctrl-C to go back to the Main Menu

  • Changing hostname
Enter a menu number [0]: 3
New hostname [vcsa1.lab.org]:vcsa2.lab.org

  • Changing DNS server and Domain name.
Enter a menu number [0]: 4
DNS Server 1 [127.0.0.1]: 192.168.0.10
DNS Server 2 (optional) [192.168.0.10]: 192.168.0.11
Domain Name (optional) []: lab.org
Search Path (space separated) (optional) []: lab.org
DNS server settings updated

  • Changing proxy address of vCenter appliance
Enter a menu number [0]: 5
Is an IPv4 proxy server necessary to reach the Internet? y/n [n]: y
Proxy Server (http:// will be auto prepended) []: proxy1.lab.org
Proxy Port []: 8080

  • Changing IP address.
Enter a menu number [0]: 6
Configure an IPv6 address for eth0? y/n [n]: n
Configure an IPv4 address for eth0? y/n [n]: y
Use a DHCPv4 Server instead of a static IPv4 address? y/n [n]: n
IPv4 Address [192.168.0.51]: 192.168.0.51
Netmask [255.255.255.0]: 255.255.255.0
IPv4 Address:   192.168.0.51
Netmask:        255.255.255.0

Is this correct? y/n [y]: y

Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 1
Network parameters successfully changed to requested values

Un-handled Exception with ESXi UI

Following two problems are seen with ESXi version 6.7 Update 2 (Build 13006603).

  • Using host UI (https://HOSTNameORIP/ui ) when you click on Storage then Adapter then  following unhandled exception comes. This stop configuring storage related activity from ESXi.
Unhandled exception
-----------
Unfortunately, we hit an error that we weren't expecting.
The client may continue working, but at this point,
we recommend refreshing your browser and submitting a bug report.
Press the Esc key to hide this dialog and continue without refreshing

 

Clicking on Details shows following lines.

Cause: Possibly unhandled rejection: {}
Version: 1.33.3
Build: 12923304
ESXi: 6.7.0
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Exception stack:
this.toAddress64@https://192.168.100.11/ui/scripts/main.js:422:206
updateAdapter@https://192.168.100.11/ui/scripts/main.js:365:26206
@https://192.168.100.11/ui/scripts/main.js:365:26542
k/<@https://192.168.100.11/ui/scripts/main.js:324:23176
$digest@https://192.168.100.11/ui/scripts/main.js:324:28780
$evalAsync/<@https://192.168.100.11/ui/scripts/main.js:324:30503
e@https://192.168.100.11/ui/scripts/main.js:323:10071
tg/h.defer/c<@https://192.168.100.11/ui/scripts/main.js:323:11522

 

  • Using host UI when you import virtual machine using OVF template following stack is seen.
TypeError: Cannot read property 'keyValue' of undefined
    at updateSummaryPortlet (https://192.168.10.11/ui/scripts/main.js:375:415)
    at $scope.wizardOptions.onFinish (https://192.168.10.11/ui/scripts/main.js:375:5968)
    at https://192.168.10.11/ui/scripts/main.js:324:23176
    at m.$digest (https://192.168.10.11/ui/scripts/main.js:324:28780)
    at https://192.168.10.11/ui/scripts/main.js:324:30503
    at e (https://192.168.10.11/ui/scripts/main.js:323:10071)
    at https://192.168.10.11/ui/scripts/main.js:323:11522

 

VMkernel.log shows following lines.

2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)World: 11943: VC opID esxui-4004-bd50 maps to vmkernel opID 27da76af
2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)NVDManagement: 1461: No nvdimms found on the system

 

Clicking on Reload option log out ESXi web UI and you have to login to UI again.

This particular problem is fixed in ESXi build number 13981272 (ESXi 6.7 EP 10) OR later version. Refer KB  to know more about ESXi build version.
ESXi Patch download link
Select ESXi and version 6.7 then search to get latest patch.

The workaround is to press escape after opening the stack OR to import OVF using vCenter UI if host is part of vCenter, same for changing storage settings. 

VMware Appliance Monitoring Service (vmware-statsmonitor) doesn’t start.

After reboot vmware-statsmonitor service doesn’t start automatically. In some situation, this service even doesn’t start manually and fail with following error.

root@buildvcenter [ ~ ]# service-control --start vmware-statsmonitor
Operation not cancellable. Please wait for it to finish...
Performing start operation on service statsmonitor...
channel 3: open failed: administratively prohibited: open failed
Error executing start on service statsmonitor. Details {
    "detail": [
        {
            "args": [
                "statsmonitor"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'statsmonitor'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "problemId": null,
    "resolution": null,
    "componentKey": null
}

 

Following line can be seen in Vmon log (/var/log/vmware/vmon/vmon-syslog.log)

2019-08-08T09:26:42.809220-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:42.809574-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:45.810708-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:51.166333-07:00 warning vmon   Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /var/vmware/applmgmt/statsmonitor_health.xml. Error: [Errno 2] No such file or directory: '/var/vmware/applmgmt/statsmonitor_health.xml'
2019-08-08T09:26:51.166701-07:00 warning vmon
2019-08-08T09:26:51.194457-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:51.194832-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:54.195953-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:58.391456-07:00 notice vmon   Service start operation timed out.
2019-08-08T09:26:58.394656-07:00 notice vmon   Cancelling execution of pid 38052
2019-08-08T09:26:58.395009-07:00 warning vmon   Found empty StopSignal parameter in config file. Defaulting to SIGTERM

This issue happens because of startup delay and can be fix by following steps:-

  • Take snapshot of vCenter to be in safer side.
  • Take SSH to VCSA using root login.
  • Modify statsmonitor service config for vMon to set higher start up timeout:
sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
kill -HUP $(cat /var/run/vmon.pid)
  • Stop and start stats-monitor service explicitly.
/usr/lib/vmware-vmon/vmon-cli -k statsmonitor
/usr/lib/vmware-vmon/vmon-cli -i statsmonitor
  • Then restart vCenter to see if that service start automatically

Remote access for ESXi local user account ‘root’ has been locked for XXXX seconds after XXXX failed login attempts.

Due to consistent wrong password by application OR user, root a/c of ESXi may get locked out. This prevent any further login externally to ESXi host.

This happens most likely due to backup/monitoring application that has root user a/c configured the operation.

To solve this issue.

  • Take Console access (iDRAC/ILO/KVM..etc depending on hardware).
  • Press F2 to customize the system. Log in as root.
  • Use the Up/Down arrows to navigate to Troubleshooting Options > Enable ESXi Shell.
  • Press ALT+F1, login as root. Run the following commands to show number of failed attempts:
pam_tally2 --user root
  • Run the following command to unlock the root account:
pam_tally2 --user root --reset

 

Now you should be able to login to ESXi using root a/. You can review hostd log to find out from where failed login coming.

less /var/log/hostd.log |grep -i 'password'

2019-07-31T17:08:51.735Z info hostd[2099345] [Originator@6876 sub=Default 2019-07-31T20:51:07.055Z warning hostd[2205446] [Originator@6876 sub=Default opID=esxui-fca4-b52f] Rejected password for user root from 192.168.214.108
2019-07-31T20:51:11.056Z verbose hostd[2205444] [Originator@6876 sub=Solo.Vmomi] Arg password:
2019-07-31T20:51:39.634Z warning hostd[2099346] [Originator@6876 sub=Default opID=esxui-c6d2-b54a] Rejected password for user root from 192.168.214.108

 

Based on IP address check if any application configured with incorrect root password.

Please note in ESXi 6.5/6.7, invalid login may cause host to be unresponsive. Review KB  for proactive steps.

 

Unable to open vCenter 6.5 flash based web client

When you open web client following pop-up error is seen.

Cannot navigate to the desired location.
Error details: An error occurred while activating extension vsphere.core.viTree.hostsAndClustersView.
Invalid domain view id: vsphere.core.viTree.hostsAndClustersView"

When we click on Yes then you see following stack.

Error: Invalid domain view id: vsphere.core.viTree.hostsAndClustersView
at com.vmware.vsphere.client.views.app::AppViewMediator/showDomainView()
at com.vmware.vsphere.client.views.app::AppViewMediator/activateExtensionInternal()
at MethodInfo-1531()
at com.vmware.vsphere.client.views.app::AppViewMediator/onUiReady()
at EventFunctor/notifyTarget()
at EventFunctor/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.frinje::EventBus/dispatchEvent()
at com.vmware.frinje::EventBus/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.docking::DockableUiManager/onLayoutDataRetrieved()
at com.vmware.docking::DockableUiManager/onUserDataRetrieved()
at EventFunctor/notifyTarget()
at EventFunctor/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.frinje::EventBus/dispatchEvent()
at com.vmware.frinje::EventBus/onEvent()
at flash.events::EventDispatcher/dispatchEvent()
at Function/&lt;anonymous&gt;()
at Function/&lt;anonymous&gt;()
at Function/&lt;anonymous&gt;()
at com.vmware.flexutil::MxmlDeserializer/onThreadTerminated()
at flash.events::EventDispatcher/dispatchEvent()
at com.vmware.flexutil::PseudoThread/terminate()
at com.vmware.flexutil::PseudoThread/onTimer()
at flash.utils::Timer/tick()

This does not allow web client to load. However HTML based web client works fine.
Following lines can be seen in vsphere_client_virgo.log (/var/log/vmware/vsphere-client/logs)

[2019-06-27T19:46:51.237Z] [ERROR] http-bio-9090-exec-3 70000894 100040 200011 com.vmware.vise.util.logging.LogServiceImpl [Flex|AppErrorHandler]Unable to load resource module from /veeam-zip-ui/locales/veeam-zip-ui-resources-en_US.swf
Error: Unable to load resource module from /veeam-zip-ui/locales/veeam-zip-ui-resources-en_US.swf
at MethodInfo-666()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfoProxy/moduleEventHandler()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfo/errorHandler()
[2019-06-27T19:46:52.384Z] [INFO ] http-bio-9090-exec-10 org.springframework.flex.servlet.MessageBrokerHandlerAdapter Channel endpoint amf received request.
[2019-06-27T19:46:52.514Z] [INFO ] http-bio-9090-exec-6 org.springframework.flex.servlet.MessageBrokerHandlerAdapter Channel endpoint amf received request.
[2019-06-27T19:46:52.515Z] [INFO ] http-bio-9090-exec-6 com.vmware.vise.util.session.SessionUtil Generated hashed session id: 100041
[2019-06-27T19:46:52.515Z] [INFO ] http-bio-9090-exec-6 com.vmware.vise.util.session.SessionUtil Mapping: /vsphere-client/telemetry-ui: sessionid to -&gt; 100041 with clientId: 200011
[2019-06-27T19:46:52.515Z] [INFO ] http-bio-9090-exec-6 70000895 100041 200011 com.vmware.vise.util.session.SessionUtil Associated sessionId 100041 with clientId 200011. Context path: /vsphere-client/telemetry-ui
[2019-06-27T19:46:52.665Z] [INFO ] http-bio-9090-exec-3 70000896 100040 200011 org.springframework.flex.servlet.MessageBrokerHandlerAdapter Channel endpoint amf received request.
[2019-06-27T19:46:52.666Z] [ERROR] http-bio-9090-exec-3 70000896 100040 200011 com.vmware.vise.util.logging.LogServiceImpl [Flex|AppErrorHandler]Unable to load resource module from /veeam-backup-ui/locales/VeeamBackupUi-en_US.swf
Error: Unable to load resource module from /veeam-backup-ui/locales/VeeamBackupUi-en_US.swf
at MethodInfo-666()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfoProxy/moduleEventHandler()
at flash.events::EventDispatcher/dispatchEvent()
at ModuleInfo/errorHandler()

Potential solution:

  • AS per the error this is being caused by VEEAM backup client plugin. Veeam support may fix this issue by reinstalling the plugin. Most the common troubleshooting is mentioned in VEEAM KB
  • Some of the cases it is observed this error may cause because of workstation locale (language) is not English, in that case vCenter URL https://vCenter_IP/vsphere-client/?locale=en_US&csp with English can be use (just change vCenterIP OR Name of your vCenter).

Due to 3party plugin, many time you may encounter timeout while loading the inventory that takes more than default 120 seconds. When you expend inventory such datacenter/cluster or any other folder then at the bottom of the page you may see following warning.

The query execution timed out because of a back-end data adapter com.ni.eseries.vcenterWebClientui.ESeriesDataAdapter' which took more than 120 seconds.

At the same time vsphere_client_virgo.log (/var/log/vmware/vsphere-client/logs/ ) shows following lines.

[2019-06-14T08:15:10.716-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out Backend -- Find Extension --Key:com.netapp.santricity.vmware.vcenter.plugin
[2019-06-14T08:15:10.718-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out Backend -- Extension with Key com.netapp.santricity.vmware.vcenter.plugin Found
[2019-06-14T08:15:10.718-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out Backend -- Setup server URLs -- ReST:https://10.160.16.251:8084 Plugin:https://10.0.0.121:8084/vcenter2 Date:Fri Jun 14 08:15:10 AKDT 2019
[2019-06-14T08:15:10.718-08:00] [INFO ] data-service-pool-791 70000774 100041 200031 System.out VCenterPluginRest -- New Loaded Server URL: 'https://10.0.0.121:8084/vcenter2/vcenterRest' with date of Fri Jun 14 08:15:10 AKDT 2019
[2019-06-14T08:15:12.833-08:00] [WARN ] http-bio-9090-exec-12 com.vmware.vise.util.concurrent.ExecutorUtil Task 'Adapter com.vmware.opsmgmt.client.alarms.impl.AlarmIssueAdapter' timed out and was cancelled.
[2019-06-14T08:15:12.838-08:00] [ERROR] data-service-pool-796 70001202 100084 200043 com.vmware.opsmgmt.client.alarms.util.EntityNameRetriever Error retrieving entity names com.vmware.vise.data.query.DataServiceException: The data service execution was interrupted

[2019-06-14T08:15:12.838-08:00] [INFO ] data-service-pool-761 com.vmware.vise.util.session.SessionUtil Generated hashed session id: 100085
[2019-06-14T08:15:12.842-08:00] [INFO ] data-service-pool-761 com.vmware.vise.util.session.SessionUtil Mapping: ds/sdk: sessionid to -&gt; 100085 with clientId: 200043
[2019-06-14T08:15:12.848-08:00] [ERROR] data-service-pool-796 70001202 100084 200043 com.vmware.opsmgmt.client.alarms.util.EntityNameRetriever Error retrieving entity names com.vmware.vise.data.query.DataServiceException: The data service execution was interrupted
at com.vmware.vise.data.query.impl.DataServiceImpl.getResults(DataServiceImpl.java:364)
at com.vmware.vise.data.query.impl.DataServiceImpl.getResponse(DataServiceImpl.java:196)
at com.vmware.vise.data.query.impl.DataServiceImpl.getData(DataServiceImpl.java:171)
at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Here you can see the timeout issue is being caused by NetApp plugin. We can increase the default timeout as per KB OR contact NetAPP vendor to fix the issue with their add-in.

If you feel these issues are causing because of some of unwanted plugin installed in past then remove it using mob page, Refer KB

VPXD service crashes with “404 Not Found”

Following lines can be seen in logs.

VPXD.log

2019-06-18T14:11:25.894-05:00 warning vpxd[7FE27C9F37A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Vmomi::Fault::SystemError while trying to connect to SSO Admin server: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found)
2019-06-18T14:11:25.894-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reached. Giving up ...
2019-06-18T14:11:25.895-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found).
2019-06-18T14:11:25.895-05:00 warning vpxd[7FE27C9F37A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 90081 ms
2019-06-18T14:11:25.895-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)
--&gt; Backtrace:
--&gt;
--&gt; [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-8803875, tag: vpxd
--&gt; backtrace[00] libvmacore.so[0x003C5FC4]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--&gt; backtrace[01] libvmacore.so[0x001F0743]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref&lt;Vmacore::System::Backtrace&gt;&amp;)
--&gt; backtrace[02] libvmacore.so[0x0019A69D]: Vmacore::Throwable::Throwable(std::string const&amp;)
--&gt; backtrace[03] vpxd[0x00BD108E]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&amp;)
--&gt; backtrace[04] vpxd[0x00BCEB0A]
--&gt; backtrace[05] vpxd[0x00BBADD0]
--&gt; backtrace[06] vpxd[0x00AF9199]
--&gt; backtrace[07] libc.so.6[0x0001EC36]
--&gt; backtrace[08] vpxd[0x00AF8BFD]
--&gt; [backtrace end]
--&gt;
2019-06-18T14:11:25.896-05:00 warning vpxd[7FE27C9F37A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 94885 ms
2019-06-18T14:11:25.897-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...
2019-06-18T14:11:25.897-05:00 info vpxd[7FE27C9F37A0] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-06-18T14:13:25.898-05:00 info vpxd[7FE27C9F37A0] [Originator@6876 sub=Default] Forcing shutdown of VMware Virtu

SSOAdminserver.log

[2019-06-18T11:27:22.922-05:00 localhost-startStop-1 opId= ERROR com.vmware.identity.admin.server.impl.AboutInfoFactory] Failed to connect to IDM. Attempt [18] of [20]
com.vmware.identity.admin.server.ims.AdminServerException: Failed to query cluster id
    at com.vmware.identity.admin.server.impl.AboutInfoFactory.getClusterId(AboutInfoFactory.java:148)
    at com.vmware.identity.admin.server.impl.AboutInfoFactory.createAboutInfo(AboutInfoFactory.java:104)

[2019-06-18T11:30:22.956-05:00 localhost-startStop-1 opId= ERROR org.springframework.web.context.ContextLoader] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfo' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Cannot create inner bean 'ssoServerAboutInfoFactory$created #6ae84cfa' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfoFactory$created#6ae84cfa' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.sso.AboutInfo]: Factory method 'createAboutInfo' threw exception; nested exception is com.vmware.identity.admin.server.ims.AdminServerException: Failed to connect to IDM after [20] attempts

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfoFactory$created#6ae84cfa' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.sso.AboutInfo]: Factory method 'createAboutInfo' threw exception; nested exception is com.vmware.identity.admin.server.ims.AdminServerException: Failed to connect to IDM after [20] attempts

Vmware-sts-Idmd.log

[2019-06-18T11:00:23.288-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 INFO ] [ServerUtils] Waiting for vmware directory to come up - Retry 11
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 WARN ] [LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.LinuxLdapClientLibrary, error code: 49
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 WARN ] [ServerUtils] cannot bind connection: [ldap://localhost:389, cn=vcsa.domain.local,ou=Domain Controllers,DC=vsphere,DC=local]
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 ERROR] [ServerUtils] cannot establish connection with uri: [ldap://localhost:389]
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 ERROR] [ServerUtils] Failed to connect to vmware directory.

vmdird-syslog.log

2019-06-18T15:01:27.010315-05:00 err vmdird t@140552473413376: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
2019-06-18T15:01:27.010646-05:00 err vmdird t@140552473413376: VmDirSendLdapResult: Request (Bind), Error (49), Message ((49)(SASL step failed.)), (0) socket ([35] 127.0.0.1:389&lt;-127.0.0.1:56831)
2019-06-18T15:01:27.010917-05:00 err vmdird t@140552473413376: Bind Request Failed ([35] 127.0.0.1:389&lt;-127.0.0.1:56831) error 49: Protocol version: 3, Bind DN: "cn= vcsa.domain.local,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL

Above issue is mostly seen with vCenter 6.0 appliance. This happens because vCenter services unable to trust the computer a/c password created for background communication. Detailed steps are mentioned in KB

Here are the steps vCenter appliance.

  • Take snapshot of vCenter appliance virtual machine.
  • Take SSH to vCenter VM.
  • Run vdcadmintool
/usr/lib/vmware-vmdir/bin/vdcadmintool
  • Select option 3

Note
Reset the password of the DN (distinguish name) coming in vmdird-syslog.log, for e.g. the DN as per above error is cn= vcsa.domain.local,ou=Domain Controllers,dc=vsphere,dc=local so the UPN (User principal name) will be vcsa.domain.local@vsphere.local
Write down the system generated password. The password shouldn’t contain @, \ or ” , if above character is included then run the step3 again.

  • Run these commands one by one to update the password:
/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "new password"
quit
  • Stop all vCenter services then start.
service-control --stop vpxd
service-control --start vpxd

Additional Note:-

Most of the time vdcadmintool generate 20 character long password. In rarest situation If the number of chararters goes beyond 20 character and save the password in registry then VMDIRD service wouldn’t start and following lines can be seen.

2019-06-24T15:26:45.967796-05:00 info vmdird  t@140438655727360: SASL2PATH=/opt/likewise/lib64/sasl2:/usr/lib/vmware-vmdir/lib64/sasl2
2019-06-24T15:26:45.968238-05:00 info vmdird  t@140438655727360: Use default max-database-size 21474836480
2019-06-24T15:26:45.968416-05:00 info vmdird  t@140438655727360: mdb stats: last_pgno 1992, max_pgs 5242880
2019-06-24T15:26:45.996245-05:00 info vmdird  t@140438655727360: Startup schema instance (0x768c10)
2019-06-24T15:26:45.997791-05:00 info vmdird  t@140438655727360: Next available local USN: 9200
2019-06-24T15:26:45.999521-05:00 err vmdird  t@140438655727360: VmDirReadDCAccountPassword failed with error code: 87
2019-06-24T15:26:45.999589-05:00 err vmdird  t@140438655727360: LoadServerGlobals: (87)

At the same vdcadmintool will stop generating the password and fails with same error “VmDirReadDCAccountPassword failed with error code: 87”. This happens vdcadmintool may generate maximum 2,147,483,647 characters, but vmdird supports maximum 20 letters only. The workaround is save some incorrect password less than 21 in registry then stop / start vCenter services to bring VMDIRD service up.

/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "wrong passowrd"
quit

#6ae84cfa