Unable to start vCenter Update manager service.

While clicking on update manager tab, we see error

“Problem occurred while connecting to the Update Manager server. See the vSphere Web Client logs for more details.”

You may see following line when we start update manager service. Additionally it take long time to start the service.

root@VCSA[~]#service-control --start vmware-updatemgr
Operation not cancellable. Please wait for it to finish...
Performing start operation on service updatemgr...
Error executing start on service updatemgr. Details {
    "resolution": null,
    "detail": [
        {
            "localized": "An error occurred while starting service 'updatemgr'",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "updatemgr"
            ],
            "id": "install.ciscommon.service.failstart"
        }
    ],
    "componentKey": null,
    "problemId": null
}

vum-server.log (/var/log/vmware/vmware-updatemgr) does not update however we see following keep repeating lines in refreshCerts-utility.log

[2019-05-08 21:09:31,605 WARNING] Attempt to login into VC failed...
[2019-05-08 21:09:31,605 INFO] ... Waiting for a minute
[2019-05-08 21:10:31,662 INFO] ... Retrying again
[2019-05-08 21:10:31,663 INFO] command: ['/usr/lib/vmware-vmafd/bin/vmafd-cli', 'get-ls-location', '--server-name', 'localhost']
[2019-05-08 21:10:31,704 INFO] rc: 0, stdout: https://VCSA.ne.local/lookupservice/sdk
, stderr:
[2019-05-08 21:10:31,704 INFO] lsUrl: https://VCSA.ne.local/lookupservice/sdk
[2019-05-08 21:10:31,727 INFO] Querying LS for Local vCenter Server.
[2019-05-08 21:10:31,783 INFO] Found VC(https://VCSA.ne.local:443/sdk)
[2019-05-08 21:10:31,842 ERROR] VC login failure. Exception is (vim.fault.NoPermission) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Permission to perform this operation was denied.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   object = 'vim.Folder:group-d1',
   privilegeId = 'System.View'
}

At the same time SPS service fails with following error
/var/log/vmware/vmware-sps/sps.log

2019-05-09T20:35:07.456Z [main] ERROR opId=sps-Main-433417-209 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.
2019-05-09T20:35:07.457Z [main] ERROR opId=sps-Main-433417-209 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53)
at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:159)
at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:129)
at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:34)
at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:118)
at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:106)
at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:188)
at com.vmware.sps.StorageMain.main(StorageMain.java:67)
Caused by: (vim.fault.NoPermission) {
faultCause = null,
faultMessage = null,
object = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = 6ae1dbea-083e-4a92-a7b4-a5f2672f8d8c,
privilegeId = System.View
}

This issue can be fixed by following steps.

1. Take snapshot of vCenter (& PSC if external).
2. Connect to vCenter database.

root@VCSA[~]#/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

3. Run following SQL query to get SSO admin information.

VCDB=# Select id, principal, role_id, entity_id, flag, surr_key from vpx_access;

4. You may see multiple entries for SSO administrator account. You may need to delete duplicate entries.

id | principal | role_id | entity_id | flag | surr_key
------+------------------------+---------+-----------+------+----------
1 | VSPHERE.LOCAL\Administrator | -1 | 1 | 1 | 1
410 | VSPHERE.LOCAL\AdminCDW | -1 | 30 | 1 | 6
411 | VSPHERE.LOCAL\AdminCDW | -1 | 48 | 1 | 7
412 | VSPHERE.LOCAL\AdminCDW | -1 | 36 | 1 | 8
706 | VSPHERE.LOCAL\Administrator | -1 | 20495 | 1 | 10
1201 | VSPHERE.LOCAL\Administrator | -1 | 1 | 1 | 15
(6 rows)

5. Stop VPXD service then these stale entries

root@vcsa1 [~]# service-control --stop vmware-vpxd

6. Here we need to delete entries where id is 706,1201

VCDB=# Delete From vpx_access WHERE id in (706,1201);

7. Start VPXD service.

root@vcsa1 [~]# service-control --start vmware-vpxd
Advertisements