Windows vCenter VPXD service keep crashing

This particular problem is seen with windows vCenter 6.0.x. However this may be applicable for other version of vCenter on Windows environment.

While accessing the vCenter web console, following popup error comes.

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0000000001F44E10] _serverNamespace = / _isRedirect = false _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

 

VPXD service stop after couple of minutes

C:\Program Files\VMware\vCenter Server\bin>service-control --status vpxd
INFO:root:Service: vpxd, Action: status
Service: vpxd, Action: status
2019-04-10T21:43:03.782Z   State for service vpxd: STOPPED
INFO:root:Stopped:
 vpxd (VMware vCenter Server)
Stopped:
 vpxd (VMware vCenter Server)

 

You may see following line in VPXD log (%ALLUSERSPROFILE%\VMWare\vCenterServer\logs)

2019-04-10T18:07:05.484-04:00 info vpxd[12260] [Originator@6876 sub=vpxdVdb] Registry Item DB 5 value is ''
2019-04-10T18:07:05.484-04:00 info vpxd[12260] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 1000 transactions for 1 GB RAM dedicated to vpxd.
2019-04-10T18:07:05.484-04:00 info vpxd[12260] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: vCenter Server with username sa
2019-04-10T18:07:05.617-04:00 info vpxd[12260] [Originator@6876 sub=Default] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB
2019-04-10T18:07:05.705-04:00 info vpxd[12260] [Originator@6876 sub=DbSchemaHash] [VpxdDbSchemaHash::GetSchemaFilename]: sqlDir: C:\Program Files\VMware\vCenter Server\vpxd\sql\
2019-04-10T18:07:05.705-04:00 info vpxd[12260] [Originator@6876 sub=DbSchemaHash] [VpxdDbSchemaHash::CheckSchema]: schema file: C:\Program Files\VMware\vCenter Server\vpxd\sql\VCDB_mssql.sql
2019-04-10T18:07:05.712-04:00 info vpxd[12260] [Originator@6876 sub=Default] Looking for vpx result filter at C:\Program Files\VMware\vCenter Server\vpxd\vpxResultFilter.xml
2019-04-10T18:07:05.743-04:00 info vpxd[12260] [Originator@6876 sub=QueryServiceProvider] Initializing QueryServiceProviderHelperImpl
2019-04-10T18:07:05.745-04:00 info vpxd[12260] [Originator@6876 sub=MoServiceDirectory] [ServiceDirectory] Local instance UUID: 47A835EA-0A6D-481C-8A96-1E5551262865
2019-04-10T18:07:05.747-04:00 info vpxd[12260] [Originator@6876 sub=MoServiceDirectory] [RegisterBuiltinServiceEndpoints] Found service endpoint: qs-endpoint.xml
2019-04-10T18:07:05.749-04:00 error vpxd[12260] [Originator@6876 sub=MoServiceDirectory] [ServiceDirectory] Local instance not found, url not known
2019-04-10T18:07:05.754-04:00 error vpxd[12260] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument

2019-04-10T18:09:36.253-04:00 error vpxd[10808] [Originator@6876 sub=SoapAdapter.HTTPService] Failed to read request; stream: <io_obj p:0x0000000010250e28, h:3252, <TCP '127.0.0.1:8085'>, <TCP '127.0.0.1:49451'>>, error: class Vmacore::SystemException(An existing connection was forcibly closed by the remote host)

2019-04-10T18:10:01.872-04:00 error vpxd[13252] [Originator@6876 sub=SoapAdapter.HTTPService] Failed to read request; stream: <io_obj p:0x0000000010223f58, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: class Vmacore::TimeoutException(Operation timed out)
2019-04-10T18:10:02.125-04:00 info vpxd[10840] [Originator@6876 sub=Default opID=HB-host-201@77173-3e87a50e] CoreDump: Writing minidump
2019-04-10T18:10:03.200-04:00 panic vpxd[10840] [Originator@6876 sub=Default opID=HB-host-201@77173-3e87a50e] 

Panic: Win32 exception: Stack overflow (0xc00000fd)
rip: 0000000050194ad6 rsp: 000000000c3d9410 rbp: 000000000c3d1ac0
rax: 0000000000007a68 rbx: 0000000005b9f0f0 rcx: 0000000005b9f0f0
rdx: 0000000002fff1a0 rdi: 0000000002ffee30 rsi: 0000000002ffee30
r8:  0000000005b9f678 r9:  000000000c3d94b0 r10: 000000000c3d1000
r11: 000000000c3d3000 r12: 0000000000000002 r13: 0000000000000000
r14: 0000000002fff010 r15: 0000000000000000
Backtrace:
[backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-5183551, tag: vpxd
backtrace[25] ODBC32.dll[0x0007ED3D]
backtrace[26] vpxd.exe[0x00037067]
backtrace[27] vpxd.exe[0x000205B9]
backtrace[28] vpxd.exe[0x000143C1]
backtrace[29] vpxd.exe[0x00748029]
backtrace[30] vpxd.exe[0x00778FBF]
backtrace[58] vpxd.exe[0x0077D136]
backtrace[59] vpxd.exe[0x0077D398]
backtrace[60] vpxd.exe[0x0077BFE5]
backtrace[61] vpxd.exe[0x0077D136]
[backtrace end]
2019-04-10T18:10:03.222-04:00 info vpxd[10840] [Originator@6876 sub=SupportMgr opID=HB-host-201@77173-3e87a50e] Wrote uptime information

 

Some of the cases you may see VPXD crash immediately after restart of VPXD service startup

Log Name:      Application
Source:        Application Error
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Description:
Faulting application vpxd.exe, version 0.0.0.0, time stamp 0x56deb11d, faulting module dbghelp.dll, version 6.0.6001.18000, time stamp 0x4791acb8, exception code 0xc0000005, fault offset 0x0000000000001e44, process id 0x4994, application start time 0x01d53da6d9210c51.

 

In order to solve this issue.

  • Take backup of file C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpxd\vpxd.cfg
  • Add following line in vpxd.cfg under <threadPool> section

<ThreadStackSizeKb>1024</ThreadStackSizeKb>

  • Start VPXD service
C:\Program Files\VMware\vCenter Server\bin>service-control --start vpxd
Advertisements