Email notification for schedule task doesn’t go to relay server configured in vCenter.

Email notification can be configure for vCenter alerts as per documentation

However when you do any schedule activity such as snapshot / power on-off and give email address for this activity it may not go to relay server configured in the vCenter server as per above documentation.

As per the network traces you may see vCenter trying to contact email server as per MX record instead of going directly relay server configured as per vCenter.

To fix this issue we need to follow below steps.

  • Take snapshot of vCenter VM.
  • Take SSH session to vCenter VM, change to the /etc/mail folder:
cd /etc/mail
  • Make a backup copy of
  • Edit using vi, WinSCP, or any other preferable method and find these lines:
# "Smart" relay host (may be null)
  • After the “DS”, enter the FQDN of your SMTP server like this:
# "Smart" relay host (may be null)
  • Restart the sendmail service by running the below command:
systemctl restart sendmail.service

Now vCenter should send email to relay server configured in your environment.


Unable to open vCenter web console (Flash/HTML) after migrating from 6.5.X to 6.7.X

While opening Flash client (https://vcsa.domain.local/vsphere-client/), you may see error while opening Flash client.

An internal error has occurred-No state found from reature NativeBrowserFileTranster
Reloading the client is recommended so as to clear any problems left by this error.
click Yes to reload the vSphere web client.
Show error stack.

When we click on Yes then you see following stack.

Error Stack
ArgumentError No state found tor reature NativeBrowserFileTranster
at com vmware nexutil configuration: FeatureStateManagemsEnabled0
at conv vmware vmrc:
at conv vmware UtiLS/init()
at conv vmware ui ut":
at com vmware ui ut":
at com vmware nexutil proxies: ConfigurationServiceProxylgetConfiguration()
at conv vmware nexutil proxies ( ConfigurationServicePr0KylgetAProperties()
at conv vmware util: CommonApplnitialZernoadWebclientConfig()
at conv vmware ui util::
at conv vmware
at UlMediator/onBackendEndpointsSessionlnitialized()
at com vmware nexutil proxies: Baseproxy/notity()
at conv vmware nexutil
at com vmware nexutil proxies: : BaseProMonlnvocatimComplete()
at Operationlnvoker'resultResponseForRequest()
at Operationlnvoker/result()
at mx.rpc: :AsyncTokemhttp:/twww.adobe
my rpc events ResultEvenVhttp:/Äwmadobe_com.Q006.mex/mx/internal::callT*enResponders()
at mx.rpc: :AbstractOperatiomhttp:lhmwadobecomQOOömex/mVinternal::dispatchRpcEvent()
at mx.rpc: : Responder/resut()
at NetConnectimMessageResponderjresutHandIer()


HTML client (https://vcsa.domain.local/ui/) and VAMI page (https://vcsa.domain.local:5480) just load spinning wheel with blank screen and doesn’t load inventory.

This issue comes when vCenter migration from old VM to new VM and browser have old cache information. To isolate the issue try opening vCenter URL from any other workstation. To fix the issue clear browser cache OR reset it to default.

For example, for Google Chrome following steps can be carried out:-
Settings \ Expand Advanced \ section Reset and clean up then Restore settings to their original defaults.

vCenter backup via vCenter Appliance Management interface (VAMI) to SMB fails

Following error can be seen when you edit Backup schedule.

Error in method invocation module ‘util.Messages’ has no attribute ‘ScheduleLocationDoesNotExist’


Backup now (Immediate backup) fails with error. “SMB location is invalid”


Following lines can be seen in applmgmt.log (/var/log/vmware/applmgmt/)

2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Failed to mount the cifs share // at /storage/remote/backup/cifs/; Err: rc=32, stdOut:, stdErr: mount error(112): Host is down
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Couldn't mount the cifs share // at /storage/remote/backup/cifs/
2019-06-07T22:12:28.689 [15209]ERROR:vmware.appliance.vapi.impl:pint.Error('com.vmware.applmgmt.err_invalid_remote_loc', '%(0)s location is invalid.', **{'args': LocationType(string='SMB')})

This issue happens when SMB1 is disabled on File Server OR blocked in network. If you are using windows as file server then run following PowerShell to see current status of SMB version 1 & 2.

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol

Typical Output

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol
EnableSMB1Protocol EnableSMB2Protocol
------------------ ------------------
             False               True

Enabling SMB1 version may help to fix this issue.

Set-SmbServerConfiguration -EnableSMB1Protocol $true

Typical Output

Confirm, Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Server Configuration'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

Please note, if SMB1 is disabled deliberately due to security concern as it considered  weaker protocol compare to SMB2/3 then use alternate protocol (HTTP, FTP, NFS..ETC) to take vCenter backup via VAMI.

‘vMon-cli’ alternate to service-control command for vCenter services related task and troubleshooting.

Due to any issue if you are unable to run service-control command on vCenter then use vMon-cli as alternate.
This command is available in windows as well as appliance based vCenter.

Windows location.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -h


Appliance location.

/usr/lib/vmware-vmon/vmon-cli --h


Usage: vmon-cli

  -c, --vmonserver   vMon server connection path.
  -i, --start   Start service.
  -k, --stop    Stop Service.
  -r, --restart  Restart Service.
  -s, --status  Get Service status.
  -d, --dumpcore  Dump service process core.
  -U, --update  Update service state.
  -S  --starttype  Requires -U option.
  -R  --runasuser  Requires -U option.
  -l, --list               List registered services.
  --batchstart  Start all services in profile.
  --batchstop   Stop all services in profile.
  --ignorefail             Ignore batch operation failures.
  -h                       Print help information.


List all vCenter services.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -list


Status of VPXD service (service name can be changed)

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -s vpxd
C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --status vpxd
Name: vpxd
Starttype: AUTOMATIC
HealthState: HEALTHY


Stop VPXD service (service name can be changed), run one of the command.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -k vpxd
C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --stop vpxd
Completed Stop service request.


Start VPXD service (service name can be chagned), run one of the command.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -i vpxd
C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --start vpxd
Completed Start service request.


Stop all services

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --batchstop ALL
Successfully stopped all services.


Start all services.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --batchstart ALL
Successfully started all services.


Taking dump of VPXD service (works only for appliance)

root@vcsa1 [ ~ ]# /usr/lib/vmware-vmon/vmon-cli -d vpxd
Completed dump service livecore request. Created livecore file /var/core/livecore.vpxd-06-02-19-19:16:03.4378


Similarly you can update service startup type and service user credential if require for troubleshooting.


Unable to renew ESXi certificate just after making vCenter as subordinate CA

You may see following error when you attempt to renew ESXi certificate using vCenter web console (\ESXi\configure\certificate-renew). This error may also come when you connect ESXi host in vCenter that is either new OR disconnected previously.


On screen error stack

Error Stack
TypeError: Error #1009
	at com.vmware.vsphere.client.views.notification::OperationNotifyViewMediator/onSetContext()
	at com.vmware.vsphere.client.views.notification::OperationNotifyViewMediator/set _209484338contextObject()
	at com.vmware.vsphere.client.views.notification::OperationNotifyViewMediator/set contextObject()
	at BindingImpl/assign()
	at BindingImpl$/bindProperty()
	at com.vmware.flexutil.impl.binding::BindingUtil$/bindProperty()
	at com.vmware.flexutil::BindingSet/bindProperty()
	at com.vmware.frinje::ContextPropagationManager/bindChildToParentProperty()
	at com.vmware.frinje::ContextPropagationManager/createBindings()
	at com.vmware.frinje::ContextPropagationManager/bindToParent()
	at com.vmware.frinje::ContextPropagationManager/bindParentalMediatorChainFor()
	at com.vmware.frinje::ContextPropagationManager/addTarget()
	at com.vmware.frinje::ContextPropagationManager/addRemoveObject()
	at com.vmware.frinje::ContextPropagationManager/onInjectableObjectAddedRemoved()
	at com.vmware.frinje::ObjectRegistry/onObjectAdded()
	at com.vmware.flexutil::FunctionUtil$/invokeCallLater()
	at mx.core::UIComponent/callLaterDispatcher2()
	at mx.core::UIComponent/callLaterDispatcher()

In monitor tab, you may see following error:

A general system error occurred: Unable to get signed certificate for host: esxi_host name. Error: Start Time Error (70034)


You may see following lines in logs.


ERROR task-4065 -- certificateManager -- vim.CertificateManager.refreshCertificates: vmodl.fault.SystemError:
(vmodl.fault.SystemError) {
faultCause = (vmodl.MethodFault) null,
faultMessage = ;unset;,
reason = "Unable to get signed certificate forhost name '' ip '': Error: Start Time Error (70034)
"msg = ""}
Arg host:
(ManagedObjectReference) ['vim.HostSystem:7fcbc462-dcb8-45a3-b91a-e3524e1a048a:host-18']


2019-03-31T16:35:09.778456+00:00 info vmcad  t@140096531592960: VMCACheckAccessKrb: Authenticated user
2019-03-31T16:35:09.790024+00:00 info vmcad  t@140096531592960: Checking upn: cn=CAAdmins,cn=Builtin,dc=vsphere,dc=local against CA admin group:
2019-03-31T16:35:09.792511+00:00 info vmcad  t@140096531592960: Checking user's group: cn=DCAdmins,cn=Builtin,dc=vsphere,dc=local against CA admin group: cn=CAAdmins,cn=Builtin,dc=vsphere,dc=local
2019-03-31T16:35:09.793240+00:00 info vmcad  t@140096531592960: VMCASignedRequestPrivate: Invalid validity period requested
2019-03-31T16:35:09.793421+00:00 warning vmcad  t@140096531592960: error code: 0x00011192
2019-03-31T16:35:09.793908+00:00 warning vmcad  t@140096531592960: error code: 0x00000057
2019-03-31T16:35:09.794122+00:00 warning vmcad  t@140096531592960: error code: 0x00011192

This issue happens because vCenter VMware Certificate Authority predates VMware vSphere ESXi certificates by 24 hours to avoid time synchronization issues. You can wait for 24 hours after replacing the VMware Certificate Authority certificate with an enterprise subordinate certificate for ESXi OR attempting to add additional hosts to vCenter Server. If there is need to renew certificate for ESXi immediately then change the vpxd.certmgmt.certs.minutesBefore to 10 (default 1440 minutes means 24 hours) from vCenter advanced settings.

Select Administration > vCenter Server Settings to display the vCenter Server Settings dialog box.

In the settings list, select Advanced Settings, search for vpxd.certmgmt.certs.minutesBefore

Modify the value to 10



Analyze Enterprise Vault Dtraces using PowerShell-SQL tool.

This is for Enterprise Vault Engineers, however slight modification on this tool can help guys those are working on other technologies as well. The most difficult work for a support engineer is to analyze performance issue, where thousands of traces collected over time. In past, I developed a tool (combination of Power Shell & SQL script) that can upload Enterprise Vault dtrace to SQL server then assist on most common use cases such as finding out delay in specific function, searching a given keyword OR extracting entire thread.

You can either download executable file OR use PowerShell script on SQL 2016 server. Download link


  • Microsoft SQL 2016 installed on Windows 2012 R2.
  • This works with windows authentication so logged in user must have DB creator OR equivalent permission on SQL instance.
  • The tool can be executed on server where SQL server binary is installed. For example, you can run it on SQL1 and connect to SQL2. Both servers should have SQL installed.
  • Copy latest dtrace_analyser exe file to SQL server.
  • Upload & Analyse, both are different operation. For example, you can upload dtrace today and analyse it tomorrow.

Upload D-trace

Double click on dtrace_analyser.exe, a windows form and console (command prompt) will open. Windows form enable user to upload /analyze the data. Console shows the status of operation it executing in background. During upload/analyze, form will free and all options will be disable till execution completes.

Just place all the dtrace in a single directory then supply name of that directory.


Click on Upload Traces, copy dtrace folder location OR click on Browse to select the location.

If the Database (DtraceReview) and Table (DtraceContent) already exists in SQL then it will skip the creation of database/table but truncate existing set of upload dtrace for consistency and accuracy during analysis.

Upload speed depends on resources. Approximately 10-15 seconds will take to upload single 100 MB dtrace file when SQL server have 12 GB RAM & 8 CPU.

Analyze Traces

Click on Find Delay option, by default the location of output HTML file will be C:\temp\dtrace-YYYYMMDDMMSS.html, you can change the location by clicking on Browse option or just type the location in text bar.


This tool automatically review each thread and prepare HTML report. If the numbers of lines in thread is huge, form may go to Not Responding mode, leave it running (will fix in next version) however actual work can be seen from console window.


Once processing finish, HTML file will automatically open. Delay in seconds can be seen in last column. Any thread where delay is more than 2 second will only be visible in HTML output.


Please note, this tool isn’t intelligent to understand the lines written for d-trace logging. Few function are expected to have delay such as one below. Engineers should understand the limitation because some function completes when their sub function completes.


All process id, process name and thread id can be seen in HTML report.


Last section of HTML will show events captured across all d-trace files.


Use Search Keyword option to search specific function, exception OR line. By default output will be located in C:\temp\search_output_YYYYMMDDHHMMSS.txt. You can change location and file name by clicking on Browse OR manually type the location.


Use Extract thread option to download all lines of specific thread to text file. By default output will save in text file located in C:\temp\Dtrace_thread_ThreadID.txt. You can change location and file name by clicking on Browse OR manually type the location.


Figure out who is deleting files from windows operating system

File OR folder from windows operating system (client/OS) might miss due to many different reason. A user may logon to system interactively OR remotely then delete the file OR a malicious process may also delete the file. If you are unsure who is deleting files/folder then windows auditing is the best way to figure this out.

Follow this sequence to understand the concepts.

Enable windows auditing from Local Security Policy (run – secpol.msc). If you are doing against multiple servers then edit group policies from domain controller.


You can use following PowerShell to automate this step.

secedit /export /cfg c:\secpol.cfg
(gc C:\secpol.cfg).replace("AuditObjectAccess = 0", "AuditObjectAccess = 3") | Out-File C:\secpol.cfg
secedit /configure /db c:\windows\security\local.sdb /cfg c:\secpol.cfg /areas SECURITYPOLICY
rm -force c:\secpol.cfg -confirm:$false


Update group policy using following command.

gpupdate /force

Select folder that needs to be audited. In my example, I am enabling auditing for Delete action on c:\temp\temp folder


You can use below PowerShell

#Uncomment if foder you intending to be audited isn’t created so far.
#New-Item -type directory -path C:\temp\temp
$Folder= "c:\temp\temp"
$ACL = Get-Acl $Folder
$ar1 = New-object System.Security.AccessControl.FileSystemAuditRule ("EveryOne","Delete","3")
Set-Acl $Folder $ACL


Now if anyone (user/process) delete your file then event will be generated in your event viewer. For e.g. I am deleting File1.txt using windows explorer (right click \delete) second file using PowerShell.

RM -Force C:\Temp\TEMP\File2.txt -Confirm:$false


Open Event viewer and search Security log for Event ID 4656 with “File System” task category and with “Accesses: DELETE” string. “Subject: XXXX” will show you who has deleted a file.

Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Event ID: 4656
Task Category: File System
Level: Information
Keywords: Audit Success
User: N/A
Computer: server.domain.local
A handle to an object was requested.
Security ID: domain\user1
Account Name: user1
Account Domain: domain
Logon ID: 0x98B5C
Object Server: Security
Object Type: File
Object Name: C:\Temp\Temp\File2.txt
Handle ID: 0x774
Resource Attributes: -
Process Information:
Process ID: 0x4c4c
Process Name: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Access Request Information:
Transaction ID: {00000000-0000-0000-0000-000000000000}
Accesses: DELETE
Access Reasons: DELETE: Granted by D:(A;ID;FA;;;BA)
ReadAttributes: Granted by D:(A;ID;FA;;;BA)
Access Mask: 0x10080
Privileges Used for Access Check: -
Restricted SID Count: 0


OR you can below basic PowerShell to query system event viewer log.

Get-EventLog -LogName Security -InstanceId 4656


#new-item, #uncomment