VPXD service crashes with “404 Not Found”

Following lines can be seen in logs.

VPXD.log

2019-06-18T14:11:25.894-05:00 warning vpxd[7FE27C9F37A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Vmomi::Fault::SystemError while trying to connect to SSO Admin server: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found)
2019-06-18T14:11:25.894-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reached. Giving up ...
2019-06-18T14:11:25.895-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found).
2019-06-18T14:11:25.895-05:00 warning vpxd[7FE27C9F37A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 90081 ms
2019-06-18T14:11:25.895-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)
--> Backtrace:
-->
--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-8803875, tag: vpxd
--> backtrace[00] libvmacore.so[0x003C5FC4]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001F0743]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x0019A69D]: Vmacore::Throwable::Throwable(std::string const&)
--> backtrace[03] vpxd[0x00BD108E]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&)
--> backtrace[04] vpxd[0x00BCEB0A]
--> backtrace[05] vpxd[0x00BBADD0]
--> backtrace[06] vpxd[0x00AF9199]
--> backtrace[07] libc.so.6[0x0001EC36]
--> backtrace[08] vpxd[0x00AF8BFD]
--> [backtrace end]
-->
2019-06-18T14:11:25.896-05:00 warning vpxd[7FE27C9F37A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 94885 ms
2019-06-18T14:11:25.897-05:00 error vpxd[7FE27C9F37A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...
2019-06-18T14:11:25.897-05:00 info vpxd[7FE27C9F37A0] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-06-18T14:13:25.898-05:00 info vpxd[7FE27C9F37A0] [Originator@6876 sub=Default] Forcing shutdown of VMware Virtu

SSOAdminserver.log

[2019-06-18T11:27:22.922-05:00 localhost-startStop-1 opId= ERROR com.vmware.identity.admin.server.impl.AboutInfoFactory] Failed to connect to IDM. Attempt [18] of [20]
com.vmware.identity.admin.server.ims.AdminServerException: Failed to query cluster id
    at com.vmware.identity.admin.server.impl.AboutInfoFactory.getClusterId(AboutInfoFactory.java:148)
    at com.vmware.identity.admin.server.impl.AboutInfoFactory.createAboutInfo(AboutInfoFactory.java:104)

[2019-06-18T11:30:22.956-05:00 localhost-startStop-1 opId= ERROR org.springframework.web.context.ContextLoader] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfo' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Cannot create inner bean 'ssoServerAboutInfoFactory$created #6ae84cfa' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfoFactory$created#6ae84cfa' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.sso.AboutInfo]: Factory method 'createAboutInfo' threw exception; nested exception is com.vmware.identity.admin.server.ims.AdminServerException: Failed to connect to IDM after [20] attempts

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfoFactory$created#6ae84cfa' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.sso.AboutInfo]: Factory method 'createAboutInfo' threw exception; nested exception is com.vmware.identity.admin.server.ims.AdminServerException: Failed to connect to IDM after [20] attempts

Vmware-sts-Idmd.log

[2019-06-18T11:00:23.288-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 INFO ] [ServerUtils] Waiting for vmware directory to come up - Retry 11
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 WARN ] [LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.LinuxLdapClientLibrary, error code: 49
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 WARN ] [ServerUtils] cannot bind connection: [ldap://localhost:389, cn=vcsa.domain.local,ou=Domain Controllers,DC=vsphere,DC=local]
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 ERROR] [ServerUtils] cannot establish connection with uri: [ldap://localhost:389]
[2019-06-18T11:00:33.292-05:00           4b3cb569-e80c-4702-9505-804a4e0f86d8 ERROR] [ServerUtils] Failed to connect to vmware directory.

vmdird-syslog.log

2019-06-18T15:01:27.010315-05:00 err vmdird t@140552473413376: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
2019-06-18T15:01:27.010646-05:00 err vmdird t@140552473413376: VmDirSendLdapResult: Request (Bind), Error (49), Message ((49)(SASL step failed.)), (0) socket ([35] 127.0.0.1:389<-127.0.0.1:56831)
2019-06-18T15:01:27.010917-05:00 err vmdird t@140552473413376: Bind Request Failed ([35] 127.0.0.1:389<-127.0.0.1:56831) error 49: Protocol version: 3, Bind DN: "cn= vcsa.domain.local,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL

Above issue is mostly seen with vCenter 6.0 appliance. This happens because vCenter services unable to trust the computer a/c password created for background communication. Detailed steps are mentioned in KB

Here are the steps vCenter appliance.

  • Take snapshot of vCenter appliance virtual machine.
  • Take SSH to vCenter VM.
  • Run vdcadmintool
/usr/lib/vmware-vmdir/bin/vdcadmintool
  • Select option 3

Note
Reset the password of the DN (distinguish name) coming in vmdird-syslog.log, for e.g. the DN as per above error is cn= vcsa.domain.local,ou=Domain Controllers,dc=vsphere,dc=local so the UPN (User principal name) will be vcsa.domain.local@vsphere.local
Write down the system generated password. The password shouldn’t contain @, \ or ” , if above character is included then run the step3 again.

  • Run these commands one by one to update the password:
/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "new password"
quit
  • Stop all vCenter services then start.
service-control --stop vpxd
service-control --start vpxd

Additional Note:-

Most of the time vdcadmintool generate 20 character long password. In rarest situation If the number of chararters goes beyond 20 character and save the password in registry then VMDIRD service wouldn’t start and following lines can be seen.

2019-06-24T15:26:45.967796-05:00 info vmdird  t@140438655727360: SASL2PATH=/opt/likewise/lib64/sasl2:/usr/lib/vmware-vmdir/lib64/sasl2
2019-06-24T15:26:45.968238-05:00 info vmdird  t@140438655727360: Use default max-database-size 21474836480
2019-06-24T15:26:45.968416-05:00 info vmdird  t@140438655727360: mdb stats: last_pgno 1992, max_pgs 5242880
2019-06-24T15:26:45.996245-05:00 info vmdird  t@140438655727360: Startup schema instance (0x768c10)
2019-06-24T15:26:45.997791-05:00 info vmdird  t@140438655727360: Next available local USN: 9200
2019-06-24T15:26:45.999521-05:00 err vmdird  t@140438655727360: VmDirReadDCAccountPassword failed with error code: 87
2019-06-24T15:26:45.999589-05:00 err vmdird  t@140438655727360: LoadServerGlobals: (87)

At the same vdcadmintool will stop generating the password and fails with same error “VmDirReadDCAccountPassword failed with error code: 87”. This happens vdcadmintool may generate maximum 2,147,483,647 characters, but vmdird supports maximum 20 letters only. The workaround is save some incorrect password less than 21 in registry then stop / start vCenter services to bring VMDIRD service up.

/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "wrong passowrd"
quit
Advertisements

#6ae84cfa