Configure LDAPS authentication for vCenter Server.

Following steps can help to configure Active directory LDAPs Authentication for vCenter servers.

Step 1

Note down the DC (Domain controller) assigned with LDAP. If you want to know all domain controllers following windows command can be used. This can be executed from any windows machine that is joined to AD domain.

nltest /dclist:DomainName

Step 2

Select one of the Domain controller that is configured as LDAP identity source. Login to vCenter appliance using SSH session (use Putty/Terminal access) to get LDAP certificate from DC.

openssl s_client -connect DC1.ad.local:636 -showcerts

Replace DC1.ad.local as the domain controller of your environment. The top most certificate in this chain is the certificate of the domain controller.

-----BEGIN CERTIFICATE-----
MIIC4DCCAcigAwIBAgIQJ3hiT2fQzIBLYFPywfvCgjANBgkqhkiG9w0BAQUFADAZ
MwMDAwMDBaMBkxFzAVBgNVBAMTDmFkLmdzc2xhYnMub3JnMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7JFQshqvAH+bsej+FE6IYf3LA38EpMmnsCJV
nvvX1RXoHs5tr8iwbm6fMggRHZA8jHY3Z/wnLkh1Ct+8MylrGVRL4MB1bXeSH7MT
TTCMCI/ikokCO6vkVlG1RP/YcMOIUCLERsgJiZ8qCEZYLdw8ioZuA1kaGQkiJRy8
KZI5lz4nqV9owks1e4TW5TtCTDqorYxBz2x2PsZLTih/fgLf9kRr0QUHc/f8TMuI
3LWdGdodxUKKAP7cHU5awhsOdiDjqWEuYA4gioog0Dd9sE111JvPP0opSPMgnMpf
CWOc04z8dqkR15BChG36Gvgqqbnf77vknDe1RgkFhyK6GjKGTQIDAQABoyQwIjAL
BgNVHQ8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQEFBQAD
ggEBAC8sNBB5e5WffE9VjU5zcDqvOQqE24XD1bdFeKW/ud6aYwmF5YV4wFpEGkA9
AnmCDTsxtHiRytwnN8uGll9acBCs8VQaB1HZ33GxdzNfIgtCq4XPlhHrO1+YU3+g
bez2zI5TKVnm2XE4mpwyZHSbbiXzh2SbAQI1QTde9slTFTkib0HsMZYxBE5Xsgdq
RXUX6xvU2sMbHevj13zkGfoF71T72ddq78LTCbrX3EU0jYbHhrKTqRc6qHAv9fz4
2z8xKysVs+CCx8g+qEm+igMxb9/XdA2HUOA8l+NDlH/qS78e9ty0XNayl8ZC/7bZ
cKk5wfWIbFHIIBMbl7PY2eaQK8c=
-----END CERTIFICATE-----

Copy the complete string including —–BEGIN CERTIFICATE—– until (including) —–END CERTIFICATE—– into a text file. Remove any additional characters after —–END CERTIFICATE—–. Save that content into Notepad file and save file with as cer extension (e.g. ldap_dc.cer).

 

Step 3

Open vCenter web client (HTML/Flash). Go to \Home\Administration \ Configuration under Single Sign-on\ click on + sign \ select Active directory as an LDAP server

Give the appropriate name following options.

vCenter 6.0

Name = domain name
Base DN for users: dc=domainname,dc=local
(This option to search user's in specific organization unit OR container of AD.)
Domain name: domainname.local
Domain alias: domainname
Base DN for groups: dc=domainname,dc=local
(This option to search AD group's in a specific organization unit OR container of AD.)
Primary server URL: ldaps://DC1.ad.local:636
(You can mentioned domain instead of specific DC if all your domain controller configured to use SSL for LDAP.)
Secondary server URL:ldaps://DC2.ad.local:636
(This is optional)

 

ldap_6.0.jpg

vCenter 6.5/6.7

Name = domain name
Base DN for users: dc=domainname,dc=local
Base DN for groups: dc=domainname,dc=local
Domain name: domainname.local
Domain alias: domainname
User name = adminuser@domain.local
Password ****

When you select Connect to any domain controller in the domain then vCenter connects to DC that is acting as primary domain controller (PDC). NLTest output will tell you the current primary domain controller. This option may not work for version prior to 6.7 U1 OR 6.5 U2D due to known issue. Refer The workaround is to download LDAP certificate for all DC’s (DC list can be obtain from NSTest as mentioned in point 1) then provide certificate in next step of configuration.

You also have ability to specify primary and second LDAP servers.

Primary server URL: ldaps://DC1.ad.local:636
Secondary server URL:ldaps://DC2.ad.local:636

ldap_6.7.jpg

In the next screen, upload the certificate downloaded in step2

ldap_6.7_2.jpg

If all the configuration is correct then Active Directory as an LDAP server should be added without any issue.

Unable to clone windows VM using vCenter.

You may see following task and event.

Task:

An error occurred while quiescing the virtual machine. See the virtual machine's event log for details
An error occurred while taking a snapshot: Failed to quiesce the virtual machine
An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

 

Events:

Warning message on VMNAME on HOSTFQDN in Datacenter: The guest OS has reported an error during quiescing. The error code was: 5 The error message was: 'VssSyncStart' operation failed: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (0x80070422)

 

Similar error may come during backup/snapshot of virtual machine.

During cloning process, vSphere take snapshot with option “Quiesce Guest File System” for windows VM internally. During clone OR snapshot (when option “Quiesce Guest File System” is selected) OR back above error message can be seen.

What is Quiesce?

VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups/snapshot. This process might include such operations as flushing dirty buffers from the operating system’s in-memory cache to disk, or other higher-level application-specific tasks. Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup. Quiescing is not necessary for memory snapshots; it is used primarily for backups.

Here is the solution that should work in most of the situation.

  • Take console/ RDP to windows virtual machine. Open services.msc.
  • Ensure that Virtual Disk service is started and startup type is Automatic.
  • Ensure that VMware snapshot provider service is stopped and Disabled.
  • Ensure that VMware Tools service is running.
  • Ensure that Volume Shadow Copy service started and start up type is Automatic.
  • Ensure that the VMware tools version is up to date.

Clone the VM (OR run test Backup job or take a quiesced snapshot) using the vSphere Client.

 

How to restore accidentally deleted one note files.

Due to any unwanted circumstance such as incorrect migration, space issue.. etc, you may lose one note files. Fortunately Microsoft one note have automatic backup (out of your regular backup application) that keep certain numbers of files till the retention you have specified.

These settings can be configure from one note application itself.

Open one note, select File from menu list, select Options then select Save & Backup.

onenote_issue.jpg

By default the backup location is typically configured to user profile unless if not change by user/administrator.

e.g.

C:\Users\<UserAlias>\AppData\Local\Microsoft\OneNote\<OneNoteVersion>\Backup

backup_copy.jpg

 

From this location you should see different section and backup copies. Select latest copy then open in one note then right click select Move or Copy option OR you can merge with any existing one note section.

copymove.jpg

Life saver option who use one note

vCenter backup via vCenter Appliance Management interface (VAMI) to SMB fails

Following error can be seen when you edit Backup schedule.

Error in method invocation module ‘util.Messages’ has no attribute ‘ScheduleLocationDoesNotExist’

schedule.jpg

Backup now (Immediate backup) fails with error. “SMB location is invalid”

backupnow.jpg

Following lines can be seen in applmgmt.log (/var/log/vmware/applmgmt/)

2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Failed to mount the cifs share //ad.gsslabs.org/ at /storage/remote/backup/cifs/fs.labs.org/h3CYi0qm/g5dWY4YQ; Err: rc=32, stdOut:, stdErr: mount error(112): Host is down
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Couldn't mount the cifs share //ad.gsslabs.org/ at /storage/remote/backup/cifs/fs.labs.org/h3CYi0qm/g5dWY4YQ
2019-06-07T22:12:28.689 [15209]ERROR:vmware.appliance.vapi.impl:pint.Error('com.vmware.applmgmt.err_invalid_remote_loc', '%(0)s location is invalid.', **{'args': LocationType(string='SMB')})

This issue happens when SMB1 is disabled on File Server OR blocked in network. If you are using windows as file server then run following PowerShell to see current status of SMB version 1 & 2.

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol

Typical Output

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol
EnableSMB1Protocol EnableSMB2Protocol
------------------ ------------------
             False               True

Enabling SMB1 version may help to fix this issue.

Set-SmbServerConfiguration -EnableSMB1Protocol $true

Typical Output

Confirm, Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Server Configuration'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

Please note, if SMB1 is disabled deliberately due to security concern as it considered  weaker protocol compare to SMB2/3 then use alternate protocol (HTTP, FTP, NFS..ETC) to take vCenter backup via VAMI.

vCenter upgrade OR un-installation of version 6.5 may fail on windows server

During upgrade from any existing supported version OR un installation of windows based vCenter fails with following error.

VMware Message Bus Configuration service failed stop.
Couldn't install the message bus configuration service, due to the following reason: traceback
File \Program Files\VMware\vCenter Server\Firstboot\mbcs_firstboot.py. Line 276.
..
..
The directory is not empty

vCenter_upgrade_failed

This issue comes because stale directory known as tomcat.8080 create at various location inside vCenter server that cause upgrade, reinstallation and various different vCenter operation.

To solve this issue, open vCenter installation directory and search tomcat.8080

vCenter_search.jpg

Typical location of this directory

\Program Files\VMware\vCenter Server\mbcs\tomcat.8080
\Program Files\VMware\vCenter Server\vmon\tomcat.8080
\Program Files\VMware\vCenter Server\vsm\wrapper\tomcat.8080

Take backup of all these directory and delete.

Attempt to upgrade/uninstall vCenter that stuck due to this error.

Unable to edit LDAP identity source settings configured in vCenter.

In vCenter suppose if you have LDAP as identity source such as below screenshot.

ldap.jpg

However when we edit any setting such as changing the primary server URL and save then it fails with following error.

“The SSO server either failed to connect to or authenticate to the service at the specified URI”

You may see following error in vmware-sts-idmd.log

Appliance /var/log/vmware/sso/vmware-sts-idmd.log

Windows Appliance %ALLUSERSPROFILE%\VMWare\vCenterServer\logs\sso\vmware-sts-idmd.log


[2019-04-17T17:55:05.936-04:00 vsphere.local 14ec989e-7089-4c8d-aabf-b7d1b449f03c INFO ] [IdentityManager] Authentication succeeded for user [admin@domain] in tenant [vsphere.local] in [10] milliseconds with provider [domain.local] of type [com.vmware.identity.idm.server.provider.ldap.LdapWithAdMappingsProvider]

[2019-04-17T17:51:48.330-04:00 vsphere.local 8d1b773b-9441-4c1c-a77d-6c76c9011e2b WARN ] [ServerUtils] cannot bind connection: [ldap://dc.domain.local:3268,CN=admin,OU=Applicataion Users,DC=domain,DC=local]
[2019-04-17T17:51:48.330-04:00 vsphere.local 8d1b773b-9441-4c1c-a77d-6c76c9011e2b ERROR] [ServerUtils] cannot establish connection with uri: [ldap://dc.domain.local:3268]
[2019-04-17T17:51:48.330-04:00 vsphere.local 8d1b773b-9441-4c1c-a77d-6c76c9011e2b WARN ] [IdentityManager] Failed to probe provider connectivity [URI: ldap://dc.domain.local:3268]; tenantName [vsphere.local], userName [CN=admin,OU=Applicataion Users,DC=domain,DC=local]
[2019-04-17T17:51:48.330-04:00 vsphere.local 8d1b773b-9441-4c1c-a77d-6c76c9011e2b ERROR] [IdentityManager] Failed to set Ldap provider for tenant [vsphere.local]
[2019-04-17T17:51:48.330-04:00 vsphere.local 8d1b773b-9441-4c1c-a77d-6c76c9011e2b ERROR] [ServerUtils] Exception 'com.vmware.identity.idm.IDMLoginException: Failed to probe provider connectivity [URI: ldap://dc.domain.local:3268]; tenantName [vsphere.local], userName [CN=admin,OU=Applicataion Users,DC=domain,DC=local]'
com.vmware.identity.idm.IDMLoginException: Failed to probe provider connectivity [URI: ldap://dc.domain.local:3268]; tenantName [vsphere.local], userName [CN=admin,OU=Applicataion Users,DC=domain,DC=local]

As per the log, we can see Authentication works however one the server URL (typically a domain controller) isn’t reachable.

In this situation, you can’t modify setting due to unavailability of DC. The workaround is to either remove current identity LDAP source and then add it again OR create host file entry pointing to one of the working domain controller for DC that is failing as per log.

vCenter appliance 6.7 U1 affected because of high growth rate of ‘/dev/mapper/core_vg-core’

vCenter appliance 6.7 U1 shows 100% on ‘/dev/mapper/core_vg-core’. This affect vCenter services those can’t start.

root@vcsa1 [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.9G 0 4.9G 0% /dev
tmpfs 4.9G 712K 4.9G 1% /dev/shm
tmpfs 4.9G 688K 4.9G 1% /run
tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/sda3 11G 6.0G 4.1G 60% /
tmpfs 4.9G 1.4M 4.9G 1% /tmp
/dev/sda1 120M 31M 81M 28% /boot
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/seat_vg-seat 542G 389M 514G 1% /storage/seat
/dev/mapper/db_vg-db 9.8G 1.8G 7.5G 20% /storage/db
/dev/mapper/netdump_vg-netdump 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 9.8G 34M 9.2G 1% /storage/autodeploy
/dev/mapper/core_vg-core 25G 25G 0 100% /storage/core
/dev/mapper/archive_vg-archive 50G 28G 19G 60% /storage/archive
/dev/mapper/updatemgr_vg-updatemgr 99G 5.5G 88G 6% /storage/updatemgr
/dev/mapper/dblog_vg-dblog 15G 822M 14G 6% /storage/dblog
/dev/mapper/log_vg-log 9.8G 2.6G 6.7G 28% /storage/log

Following lines can be seen in logs

/var/log/vmware/messages

2019-04-02T13:19:58.021089-05:00 VCENTERSERVER pschealthd: Detected PSC system is not healthy - Wait for atleast one minute before failing
2019-04-02T13:20:01.792568-05:00 VCENTERSERVER CROND[58477]: (root) CMD ( /opt/vmware/vpostgres/current/scripts/pg_status_cron >/dev/null 2>&1)
2019-04-02T13:20:01.793977-05:00 VCENTERSERVER CROND[58478]: (root) CMD (. /etc/profile.d/VMware-visl-integration.sh; /usr/lib/applmgmt/backup_restore/scripts/SchedulerCron.py >>/var/log/vmware/applmgmt/backupSchedulerCron.log 2>&1)
2019-04-02T13:20:01.809506-05:00 VCENTERSERVER CROND[58481]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
2019-04-02T13:20:01.809886-05:00 VCENTERSERVER CROND[58482]: (root) CMD ( test -x /usr/sbin/cloudvm_ram_size_periodic && /usr/sbin/cloudvm_ram_size_periodic >/dev/null 2>&1)
2019-04-02T13:20:01.813858-05:00 VCENTERSERVER CROND[58483]: (root) CMD (/usr/sbin/logdiskcheck.sh >/dev/null 2>&1)

/var/log/vmware/sso/vmware-sts-idmd.log

[2019-04-01T18:18:28.839-05:00 IDM Startup INFO ] [IdmServer] IDM Server has started
[2019-04-01T18:18:30.017-05:00 INFO ] [NativeLibraryPreloader] jna.library.path: /opt/vmware/lib64:/usr/lib/vmware-vmdir/lib64:/usr/lib/vmware-vmafd/lib64:/opt/likewise/lib64:/usr/lib64
[2019-04-01T18:40:29.359-05:00 IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2019-04-01T18:40:29.376-05:00 IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
[2019-04-01T18:40:50.973-05:00 IDM Startup INFO ] [IdmServer] Starting IDM Server...
[2019-04-01T18:40:50.975-05:00 IDM Startup INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[], eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[SimpleMessage[message=IDM Server has started]], detailText=[null], corelationId=[IDM Startup], timestamp=[1554162050974]
[2019-04-01T18:40:50.975-05:00 IDM Startup INFO ] [IdmServer] IDM Server has started
[2019-04-01T18:40:52.062-05:00

/storage/log/vmware/sso/utils/vmware-stsd.err

Apr 02, 2019 1:23:55 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 541 ms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Apr 02, 2019 1:24:05 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 9793 ms
Service killed by signal 11

In the directory /storage/core, you may see lots of files core.jsvc creates after every minute.

root@vcsa1 [ ~ ]# cd /storage/core/
root@vcsa1 [ /storage/core ]# du -sh * | sort -nr
856M core.jsvc.6178
837M core.jsvc.15781
820M core.jsvc.10014
785M core.jsvc.1742
780M core.jsvc.7499
769M core.jsvc.13795
764M core.jsvc.16996
737M core.jsvc.11088
734M core.jsvc.14865
728M core.jsvc.21240

This happens because of vmware-stsd crashing with core.jsvc.xxx files and fills up /storage/core after upgrading to vCenter Server Appliance 6.7 update 1. Contact VMware support to know about this issue however as temporarily measure you can follow below steps.

Delete core.jsvc files those are created because of this issue.

root@vcsa1 [ /storage/core ]# rm core.jsvc.*

Stop and start vCenter services.

root@vcsa1 [ ~ ]# service-control --stop --all
root@vcsa1 [ ~ ]# service-control --start --all

Remove Active directory as identity source using vCenter web client.

\Home \ Administration \ Configuration under ‘Single sing-on’ \ Identity source.

Select AD domain and delete it.

Dis join vCenter from Active Directory domain.

\Home \ Administration \ Configuration under Deployment \ select Node then vCenter server under Nodes \ Mange then Active Directory under Advanced.

Click on leave.

If active directory is required for authentication then use Active directory as LDAP server as identity source.