Email notification for schedule task doesn’t go to relay server configured in vCenter.

Email notification can be configure for vCenter alerts as per documentation

However when you do any schedule activity such as snapshot / power on-off and give email address for this activity it may not go to relay server configured in the vCenter server as per above documentation.

As per the network traces you may see vCenter trying to contact email server as per MX record instead of going directly relay server configured as per vCenter.

To fix this issue we need to follow below steps.

  • Take snapshot of vCenter VM.
  • Take SSH session to vCenter VM, change to the /etc/mail folder:
cd /etc/mail
  • Make a backup copy of submit.cf
cp submit.cf submit.cf.orig
  • Edit submit.cf using vi, WinSCP, or any other preferable method and find these lines:
# "Smart" relay host (may be null)
DS
  • After the “DS”, enter the FQDN of your SMTP server like this:
# "Smart" relay host (may be null)
DS smtpserver.domain.edu
  • Restart the sendmail service by running the below command:
systemctl restart sendmail.service

Now vCenter should send email to relay server configured in your environment.

Advertisements

Unable to open vCenter web console (Flash/HTML) after migrating from 6.5.X to 6.7.X

While opening Flash client (https://vcsa.domain.local/vsphere-client/), you may see error while opening Flash client.

An internal error has occurred-No state found from reature NativeBrowserFileTranster
Reloading the client is recommended so as to clear any problems left by this error.
click Yes to reload the vSphere web client.
Show error stack.
YES No

When we click on Yes then you see following stack.

Error Stack
ArgumentError No state found tor reature NativeBrowserFileTranster
at com vmware nexutil configuration: FeatureStateManagemsEnabled0
at conv vmware vmrc:
at conv vmware UtiLS/init()
at conv vmware ui ut":
at com vmware ui ut":
at com vmware nexutil proxies: ConfigurationServiceProxylgetConfiguration()
at conv vmware nexutil proxies ( ConfigurationServicePr0KylgetAProperties()
at conv vmware util: CommonApplnitialZernoadWebclientConfig()
at conv vmware ui util::
at conv vmware
at UlMediator/onBackendEndpointsSessionlnitialized()
at com vmware nexutil proxies: Baseproxy/notity()
at conv vmware nexutil
at com vmware nexutil proxies: : BaseProMonlnvocatimComplete()
at Operationlnvoker'resultResponseForRequest()
at Operationlnvoker/result()
at mx.rpc: :AsyncTokemhttp:/twww.adobe
my rpc events ResultEvenVhttp:/Äwmadobe_com.Q006.mex/mx/internal::callT*enResponders()
at mx.rpc: :AbstractOperatiomhttp:lhmwadobecomQOOömex/mVinternal::dispatchRpcEvent()
at mx.rpc: : Responder/resut()
at NetConnectimMessageResponderjresutHandIer()

 

HTML client (https://vcsa.domain.local/ui/) and VAMI page (https://vcsa.domain.local:5480) just load spinning wheel with blank screen and doesn’t load inventory.

This issue comes when vCenter migration from old VM to new VM and browser have old cache information. To isolate the issue try opening vCenter URL from any other workstation. To fix the issue clear browser cache OR reset it to default.

For example, for Google Chrome following steps can be carried out:-
Settings \ Expand Advanced \ section Reset and clean up then Restore settings to their original defaults.

vCenter backup via vCenter Appliance Management interface (VAMI) to SMB fails

Following error can be seen when you edit Backup schedule.

Error in method invocation module ‘util.Messages’ has no attribute ‘ScheduleLocationDoesNotExist’

schedule.jpg

Backup now (Immediate backup) fails with error. “SMB location is invalid”

backupnow.jpg

Following lines can be seen in applmgmt.log (/var/log/vmware/applmgmt/)

2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Failed to mount the cifs share //ad.gsslabs.org/ at /storage/remote/backup/cifs/fs.labs.org/h3CYi0qm/g5dWY4YQ; Err: rc=32, stdOut:, stdErr: mount error(112): Host is down
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Couldn't mount the cifs share //ad.gsslabs.org/ at /storage/remote/backup/cifs/fs.labs.org/h3CYi0qm/g5dWY4YQ
2019-06-07T22:12:28.689 [15209]ERROR:vmware.appliance.vapi.impl:pint.Error('com.vmware.applmgmt.err_invalid_remote_loc', '%(0)s location is invalid.', **{'args': LocationType(string='SMB')})

This issue happens when SMB1 is disabled on File Server OR blocked in network. If you are using windows as file server then run following PowerShell to see current status of SMB version 1 & 2.

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol

Typical Output

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol
EnableSMB1Protocol EnableSMB2Protocol
------------------ ------------------
             False               True

Enabling SMB1 version may help to fix this issue.

Set-SmbServerConfiguration -EnableSMB1Protocol $true

Typical Output

Confirm, Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Server Configuration'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

Please note, if SMB1 is disabled deliberately due to security concern as it considered  weaker protocol compare to SMB2/3 then use alternate protocol (HTTP, FTP, NFS..ETC) to take vCenter backup via VAMI.

‘vMon-cli’ alternate to service-control command for vCenter services related task and troubleshooting.

Due to any issue if you are unable to run service-control command on vCenter then use vMon-cli as alternate.
This command is available in windows as well as appliance based vCenter.

Windows location.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -h

 

Appliance location.

/usr/lib/vmware-vmon/vmon-cli --h

 

Usage: vmon-cli

Options:
  -c, --vmonserver   vMon server connection path.
  -i, --start   Start service.
  -k, --stop    Stop Service.
  -r, --restart  Restart Service.
  -s, --status  Get Service status.
  -d, --dumpcore  Dump service process core.
  -U, --update  Update service state.
  -S  --starttype  Requires -U option.
  -R  --runasuser  Requires -U option.
  -l, --list               List registered services.
  --batchstart  Start all services in profile.
  --batchstop   Stop all services in profile.
  --ignorefail             Ignore batch operation failures.
  -h                       Print help information.

 

List all vCenter services.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -list
cm
content-library
eam
imagebuilder
mbcs
netdumper
perfcharts
rbd
rhttpproxy
sca
sps
vapi-endpoint
vmcam
vmonapi
vmsyslogcollector
vpxd
vpxd-svcs
vsan-health
vsm
vsphere-client
vsphere-ui

 

Status of VPXD service (service name can be changed)

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -s vpxd
C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --status vpxd
Name: vpxd
Starttype: AUTOMATIC
RunState: STARTED
HealthState: HEALTHY

 

Stop VPXD service (service name can be changed), run one of the command.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -k vpxd
C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --stop vpxd
Completed Stop service request.

 

Start VPXD service (service name can be chagned), run one of the command.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe -i vpxd
C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --start vpxd
Completed Start service request.

 

Stop all services

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --batchstop ALL
Successfully stopped all services.

 

Start all services.

C:\Program Files\VMware\vCenter Server\vmon>vmon-cli.exe --batchstart ALL
Successfully started all services.

 

Taking dump of VPXD service (works only for appliance)

root@vcsa1 [ ~ ]# /usr/lib/vmware-vmon/vmon-cli -d vpxd
Completed dump service livecore request. Created livecore file /var/core/livecore.vpxd-06-02-19-19:16:03.4378

 

Similarly you can update service startup type and service user credential if require for troubleshooting.

 

Unable to renew ESXi certificate just after making vCenter as subordinate CA

You may see following error when you attempt to renew ESXi certificate using vCenter web console (\ESXi\configure\certificate-renew). This error may also come when you connect ESXi host in vCenter that is either new OR disconnected previously.

esx_cert_renew.jpg

On screen error stack

Error Stack
---------------------
TypeError: Error #1009
	at com.vmware.vsphere.client.views.notification::OperationNotifyViewMediator/onSetContext()
	at com.vmware.vsphere.client.views.notification::OperationNotifyViewMediator/set _209484338contextObject()
	at com.vmware.vsphere.client.views.notification::OperationNotifyViewMediator/set contextObject()
	at BindingImpl/assign()
	at BindingImpl$/bindProperty()
	at com.vmware.flexutil.impl.binding::BindingUtil$/bindProperty()
	at com.vmware.flexutil::BindingSet/bindProperty()
	at com.vmware.frinje::ContextPropagationManager/bindChildToParentProperty()
	at com.vmware.frinje::ContextPropagationManager/createBindings()
	at com.vmware.frinje::ContextPropagationManager/bindToParent()
	at com.vmware.frinje::ContextPropagationManager/bindParentalMediatorChainFor()
	at com.vmware.frinje::ContextPropagationManager/addTarget()
	at com.vmware.frinje::ContextPropagationManager/addRemoveObject()
	at com.vmware.frinje::ContextPropagationManager/onInjectableObjectAddedRemoved()
	at flash.events::EventDispatcher/dispatchEvent()
	at com.vmware.frinje::ObjectRegistry/onObjectAdded()
	at flash.events::EventDispatcher/dispatchEvent()
	at com.vmware.flexutil.events::QueuingEventDispatcher/dispatchPendingEvents()
	at com.vmware.flexutil::FunctionUtil$/invokeCallLater()
	at mx.core::UIComponent/callLaterDispatcher2()
	at mx.core::UIComponent/callLaterDispatcher()

In monitor tab, you may see following error:

A general system error occurred: Unable to get signed certificate for host: esxi_host name. Error: Start Time Error (70034)

esx_cert_renew2.jpg

You may see following lines in logs.

/var/log/vmware/VPXD/VPXD.log

ERROR task-4065 -- certificateManager -- vim.CertificateManager.refreshCertificates: vmodl.fault.SystemError:
Result:
(vmodl.fault.SystemError) {
faultCause = (vmodl.MethodFault) null,
faultMessage = ;unset;,
reason = "Unable to get signed certificate forhost name 'esxi-2.ADdomain.org' ip '192.168.0.82': Error: Start Time Error (70034)
"msg = ""}
Args:
Arg host:
(ManagedObjectReference) ['vim.HostSystem:7fcbc462-dcb8-45a3-b91a-e3524e1a048a:host-18']

/var/log/vmware/vmcad/vmcad-syslog.log

2019-03-31T16:35:09.778456+00:00 info vmcad  t@140096531592960: VMCACheckAccessKrb: Authenticated user vcsa1.ADdomain.org@vsphere.local
2019-03-31T16:35:09.790024+00:00 info vmcad  t@140096531592960: Checking upn: cn=CAAdmins,cn=Builtin,dc=vsphere,dc=local against CA admin group: vcsa1.ADdomain.org@vsphere.local
2019-03-31T16:35:09.792511+00:00 info vmcad  t@140096531592960: Checking user's group: cn=DCAdmins,cn=Builtin,dc=vsphere,dc=local against CA admin group: cn=CAAdmins,cn=Builtin,dc=vsphere,dc=local
2019-03-31T16:35:09.793240+00:00 info vmcad  t@140096531592960: VMCASignedRequestPrivate: Invalid validity period requested
2019-03-31T16:35:09.793421+00:00 warning vmcad  t@140096531592960: error code: 0x00011192
2019-03-31T16:35:09.793908+00:00 warning vmcad  t@140096531592960: error code: 0x00000057
2019-03-31T16:35:09.794122+00:00 warning vmcad  t@140096531592960: error code: 0x00011192

This issue happens because vCenter VMware Certificate Authority predates VMware vSphere ESXi certificates by 24 hours to avoid time synchronization issues. You can wait for 24 hours after replacing the VMware Certificate Authority certificate with an enterprise subordinate certificate for ESXi OR attempting to add additional hosts to vCenter Server. If there is need to renew certificate for ESXi immediately then change the vpxd.certmgmt.certs.minutesBefore to 10 (default 1440 minutes means 24 hours) from vCenter advanced settings.

Select Administration > vCenter Server Settings to display the vCenter Server Settings dialog box.

In the settings list, select Advanced Settings, search for vpxd.certmgmt.certs.minutesBefore

Modify the value to 10

vCenter_ESX_Cert_setting.jpg

#1009

How to join VMware ESX / VCenter to Active Directory domain and manage using domain account.

Most of organization uses AD infrastructure for authentication and administrator the resources of organization. ESX servers and VSphere can also be joined to AD domain and administrator using domain account.

Here are the steps to do the same.

Confirm the appropriate DNS address & domain name is configured for ESX.  Login to ESX using VSphere client, select Configuration tab, select ‘DNS and routing’. If correct DNS IP is not configured then click on ‘Properties’ and save valid details.

1.jpg

Click on Authentication Services, select Properties to edit current settings, select directory service type as Active Directory. Type domain name then join to domain by giving domain admin credential.

2.jpg

Once ESX joined in the domain, you should be able to see computer account listed in ADUC (Active directory user’s & computer)

3.jpg

Now create a group called ‘VMWare Admin’ & a user ‘VAdmin’, this user will be the member of ‘VMware admin’ group.

4.jpg

In your VSphere client, select Permission tab, right click on empty space select ‘Add Permission’.

5.jpg

Click on ADD, from drop down menu select the domain, you should be able to see AD objects, select VMware admin group.

6.jpg

Select the role you wish to give to map with AD group ‘VMware admin’.

7.jpg

Once it added successfully, you should be able to login in ESX Vsphere client using domain credential. You can check box ‘use windows session credential’ if you wants to login using current windows login.

8.jpg

If you are managing ESX using VMWare VCenter then open VCenter page, go to ‘Administration’ page, select ‘Configuration’, In ‘Identity source’ option select ‘Active Directory (integrated windows authentication)’, Type appropriate domain name and click on OK.

9.jpg

Click on ‘Global Permission’ Tab, click on ‘+’ option then Add AD group ‘VMware admin’.

11.jpg

Select the role you wish to map with to ‘VMware admin’ AD group.

12.jpg

Now you should be able to login to VCenter using windows & AD authentication.

13.jpg

#active-directory, #vmware, #windows

How to create ISCSI shared disk in windows server for ESX VSphere environment.

This my first blog for ESX/VSphere where the requirement was to create ISCSI data store for HA & DRS (High Availability & Distributed Resource Scheduler).  We created storage from windows server instead of having a dedicated storage appliance/VM. The purpose of such configuration is only for learning & testing the HA/DRS activity.

This configuration is designed on windows 2012 R2 with ESX 6.0.

On windows server, install ‘File and ISCSI services’ and make sure ‘ISCSI Target server’ is selected.

1.jpg

Once the role is installed then select ‘File and storage services’ feature and then ‘ISCSI’.

Right click on empty space and select ‘New ISCSI Virtual Disk’ as shown in above example. In my screenshot one of the volume is created and I am creating a new one.

2.jpg

Now select the partition where you wish to place the Virtual Disk.

3.jpg

Give the appropriate Name and Description and confirm the path of SCSI target.

4.jpg

Give the appropriate size & select type of disk as mentioned below. In my example I have selected ‘Dynamically expending’ as I don’t need to worry about space.

5.jpg

You need to create target that will be the list of ESX servers participating in HA/DRS and access VHD using ISCSI protocol.

6.jpg

In my example, I have used DNS name of ESX servers as initiator but it can also be access using IQN, IP address OR MAC address.

16.jpg

In following screenshot, there are two ESX servers are added but you have even more.

8.jpg

Select type of authentication, I am leaving it blank to avoid confusion.

9.jpg

In next page, you can confirm the setting you have selected and the result of different section will be available for you.10.jpg

Now the shared ISCSI disk is ready, you can add this in ESX server using VSphere console. Select ESX server \ Manage \Storage \ Storage Adaptor \Target \ Add Target.

11.jpg

Rescan storage so all newly attache drives are visible.

12.jpg

Now you should be able to see the path of all available SCSI share disk.

13.jpg

SCSI disk will also be available as storage devices.

14.jpg

In your windows server, you would notice the target status as ‘connected’.

15.jpg

#disaster-recovery, #esx, #scsi, #storage, #vmware