Unhandled Exception while clicking on storage adapter using ESX UI.

This particular problem is seen with ESXi version 6.7 Update 2 (Build 13006603). Using host UI (https://HOSTNameORIP/ui ) when you click on Storage then Adapter then  following unhandled exception comes. This stop configuring storage related activity from ESXi.

Unhandled exception
-----------
Unfortunately, we hit an error that we weren't expecting.
The client may continue working, but at this point,
we recommend refreshing your browser and submitting a bug report.
Press the Esc key to hide this dialog and continue without refreshing

 

while clicking on Details options, following lines are seen.

Cause: Possibly unhandled rejection: {}
Version: 1.33.3
Build: 12923304
ESXi: 6.7.0
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Exception stack:
this.toAddress64@https://192.168.100.11/ui/scripts/main.js:422:206
updateAdapter@https://192.168.100.11/ui/scripts/main.js:365:26206
@https://192.168.100.11/ui/scripts/main.js:365:26542
k/<@https://192.168.100.11/ui/scripts/main.js:324:23176
$digest@https://192.168.100.11/ui/scripts/main.js:324:28780
$evalAsync/<@https://192.168.100.11/ui/scripts/main.js:324:30503
e@https://192.168.100.11/ui/scripts/main.js:323:10071
tg/h.defer/c<@https://192.168.100.11/ui/scripts/main.js:323:11522

 

VMkernel.log shows following lines.

2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)World: 11943: VC opID esxui-4004-bd50 maps to vmkernel opID 27da76af
2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)NVDManagement: 1461: No nvdimms found on the system

 

Clicking on Reload option log out ESXi web UI and you have to login to UI again.

 

This particular problem is fixed in ESXi build number 13981272 (ESXi 6.7 EP 10) OR later version. Refer KB  to know more about ESXi build version.
ESXi Patch download link
Select ESXi and version 6.7 then search to get latest patch.

The workaround is to storage setting using vCenter web administration (FLash/HTML).

Advertisements

VMware Appliance Monitoring Service (vmware-statsmonitor) doesn’t start.

After reboot vmware-statsmonitor service doesn’t start automatically. In some situation, this service even doesn’t start manually and fail with following error.

root@buildvcenter [ ~ ]# service-control --start vmware-statsmonitor
Operation not cancellable. Please wait for it to finish...
Performing start operation on service statsmonitor...
channel 3: open failed: administratively prohibited: open failed
Error executing start on service statsmonitor. Details {
    "detail": [
        {
            "args": [
                "statsmonitor"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'statsmonitor'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "problemId": null,
    "resolution": null,
    "componentKey": null
}

 

Following line can be seen in Vmon log (/var/log/vmware/vmon/vmon-syslog.log)

2019-08-08T09:26:42.809220-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:42.809574-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:45.810708-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:51.166333-07:00 warning vmon   Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /var/vmware/applmgmt/statsmonitor_health.xml. Error: [Errno 2] No such file or directory: '/var/vmware/applmgmt/statsmonitor_health.xml'
2019-08-08T09:26:51.166701-07:00 warning vmon
2019-08-08T09:26:51.194457-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:51.194832-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:54.195953-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:58.391456-07:00 notice vmon   Service start operation timed out.
2019-08-08T09:26:58.394656-07:00 notice vmon   Cancelling execution of pid 38052
2019-08-08T09:26:58.395009-07:00 warning vmon   Found empty StopSignal parameter in config file. Defaulting to SIGTERM

This issue happens because of startup delay and can be fix by following steps:-

  • Take snapshot of vCenter to be in safer side.
  • Take SSH to VCSA using root login.
  • Modify statsmonitor service config for vMon to set higher start up timeout:
sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
kill -HUP $(cat /var/run/vmon.pid)
  • Stop and start stats-monitor service explicitly.
/usr/lib/vmware-vmon/vmon-cli -k statsmonitor
/usr/lib/vmware-vmon/vmon-cli -i statsmonitor
  • Then restart vCenter to see if that service start automatically

Remote access for ESXi local user account ‘root’ has been locked for XXXX seconds after XXXX failed login attempts.

Due to consistent wrong password by application OR user, root a/c of ESXi may get locked out. This prevent any further login externally to ESXi host.

This happens most likely due to backup/monitoring application that has root user a/c configured the operation.

To solve this issue.

  • Take Console access (iDRAC/ILO/KVM..etc depending on hardware).
  • Press F2 to customize the system. Log in as root.
  • Use the Up/Down arrows to navigate to Troubleshooting Options > Enable ESXi Shell.
  • Press ALT+F1, login as root. Run the following commands to show number of failed attempts:
pam_tally2 --user root
  • Run the following command to unlock the root account:
pam_tally2 --user root --reset

 

Now you should be able to login to ESXi using root a/. You can review hostd log to find out from where failed login coming.

less /var/log/hostd.log |grep -i 'password'

2019-07-31T17:08:51.735Z info hostd[2099345] [Originator@6876 sub=Default 2019-07-31T20:51:07.055Z warning hostd[2205446] [Originator@6876 sub=Default opID=esxui-fca4-b52f] Rejected password for user root from 192.168.214.108
2019-07-31T20:51:11.056Z verbose hostd[2205444] [Originator@6876 sub=Solo.Vmomi] Arg password:
2019-07-31T20:51:39.634Z warning hostd[2099346] [Originator@6876 sub=Default opID=esxui-c6d2-b54a] Rejected password for user root from 192.168.214.108

 

Based on IP address check if any application configured with incorrect root password.

Please note in ESXi 6.5/6.7, invalid login may cause host to be unresponsive. Review KB  for proactive steps.

 

Unable to login to vCenter using Active directory user credential.

vCenter is joined to AD domain and identity source is configured as Integrated windows authentication but still unable to login to vCenter. You may see “access denied” in vSphere flash OR UI client.

You may see following lines in websso.log (var/log/vmware/sso/).

[2019-06-18T21:14:41.278Z tomcat-http--9 vsphere.local        2a51ab88-55aa-4194-9d63-ec5acbac4c27 INFO  auditlogger] {"user":"ADdomain\\ADUser","client":"10.200.201.223","timestamp":"06/18/2019 21:14:41 UTC","description":"User aduser@addomain.local 10.200.201.223 failed to log in with response code 401","eventSeverity":"INFO","type":"com.vmware.sso.LoginFailure"}
[2019-06-18T21:14:50.296Z tomcat-http--16 vsphere.local        e5a24519-8856-482f-a45a-9e4eb8d6eb8c ERROR com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [aduser@addomain.local] for tenant [vsphere.local] com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 851968][null][null]
at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.AuthenticateByPassword(LinuxIdmNativeAdapter.java:188) ~[vmware-identity-platform-7.0.0.jar:?]
at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.authenticate(ActiveDirectoryProvider.java:289) ~[vmware-identity-idm-server-7.0.0.jar:?]

When you create vCenter with IP address then you may see host name as photon-machine. However if the computer account in AD have duplicate/missing OR DNS record is missing then you see above error.

root@photon-machine [ /opt/likewise/bin ]# ./domainjoin-cli query
Name = photon-machine
Domain = ADDomain.local
Distinguished Name = CN=PHOTON-MACHINE,OU=Servers,DC=ADdomain,DC=local

 

If you have multiple computer a/c in AD with same name then follow below procedure.

# /opt/likewise/bin/domainjoin-cli leave

Reboot vCenter (using VAMI page).
Delete both computer a/c’s from Active directory.
Join vCenter into AD.

# /opt/likewise/bin/domainjoin-cli join addomain.local administrator@addomain.local

Reboot vCenter (using VAMI page).
Login using AD a/c.

 

If you then DNS record is missing for vCenter host name (as per above example it’s photon-machine) then create forward/reverse record in DNS server.

Email notification for schedule task doesn’t go to relay server configured in vCenter.

Email notification can be configure for vCenter alerts as per documentation

However when you do any schedule activity such as snapshot / power on-off and give email address for this activity it may not go to relay server configured in the vCenter server as per above documentation.

As per the network traces you may see vCenter trying to contact email server as per MX record instead of going directly relay server configured as per vCenter.

To fix this issue we need to follow below steps.

  • Take snapshot of vCenter VM.
  • Take SSH session to vCenter VM, change to the /etc/mail folder:
cd /etc/mail
  • Make a backup copy of submit.cf
cp submit.cf submit.cf.orig
  • Edit submit.cf using vi, WinSCP, or any other preferable method and find these lines:
# "Smart" relay host (may be null)
DS
  • After the “DS”, enter the FQDN of your SMTP server like this:
# "Smart" relay host (may be null)
DS smtpserver.domain.edu
  • Restart the sendmail service by running the below command:
systemctl restart sendmail.service

Now vCenter should send email to relay server configured in your environment.

Unable to open vCenter web console (Flash/HTML) after migrating from 6.5.X to 6.7.X

While opening Flash client (https://vcsa.domain.local/vsphere-client/), you may see error while opening Flash client.

An internal error has occurred-No state found from reature NativeBrowserFileTranster
Reloading the client is recommended so as to clear any problems left by this error.
click Yes to reload the vSphere web client.
Show error stack.
YES No

When we click on Yes then you see following stack.

Error Stack
ArgumentError No state found tor reature NativeBrowserFileTranster
at com vmware nexutil configuration: FeatureStateManagemsEnabled0
at conv vmware vmrc:
at conv vmware UtiLS/init()
at conv vmware ui ut":
at com vmware ui ut":
at com vmware nexutil proxies: ConfigurationServiceProxylgetConfiguration()
at conv vmware nexutil proxies ( ConfigurationServicePr0KylgetAProperties()
at conv vmware util: CommonApplnitialZernoadWebclientConfig()
at conv vmware ui util::
at conv vmware
at UlMediator/onBackendEndpointsSessionlnitialized()
at com vmware nexutil proxies: Baseproxy/notity()
at conv vmware nexutil
at com vmware nexutil proxies: : BaseProMonlnvocatimComplete()
at Operationlnvoker'resultResponseForRequest()
at Operationlnvoker/result()
at mx.rpc: :AsyncTokemhttp:/twww.adobe
my rpc events ResultEvenVhttp:/Äwmadobe_com.Q006.mex/mx/internal::callT*enResponders()
at mx.rpc: :AbstractOperatiomhttp:lhmwadobecomQOOömex/mVinternal::dispatchRpcEvent()
at mx.rpc: : Responder/resut()
at NetConnectimMessageResponderjresutHandIer()

 

HTML client (https://vcsa.domain.local/ui/) and VAMI page (https://vcsa.domain.local:5480) just load spinning wheel with blank screen and doesn’t load inventory.

This issue comes when vCenter migration from old VM to new VM and browser have old cache information. To isolate the issue try opening vCenter URL from any other workstation. To fix the issue clear browser cache OR reset it to default.

For example, for Google Chrome following steps can be carried out:-
Settings \ Expand Advanced \ section Reset and clean up then Restore settings to their original defaults.

vCenter backup via vCenter Appliance Management interface (VAMI) to SMB fails

Following error can be seen when you edit Backup schedule.

Error in method invocation module ‘util.Messages’ has no attribute ‘ScheduleLocationDoesNotExist’

schedule.jpg

Backup now (Immediate backup) fails with error. “SMB location is invalid”

backupnow.jpg

Following lines can be seen in applmgmt.log (/var/log/vmware/applmgmt/)

2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Failed to mount the cifs share //ad.gsslabs.org/ at /storage/remote/backup/cifs/fs.labs.org/h3CYi0qm/g5dWY4YQ; Err: rc=32, stdOut:, stdErr: mount error(112): Host is down
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
2019-06-07T22:12:28.673 [15209]ERROR:backupRestoreAPI:Couldn't mount the cifs share //ad.gsslabs.org/ at /storage/remote/backup/cifs/fs.labs.org/h3CYi0qm/g5dWY4YQ
2019-06-07T22:12:28.689 [15209]ERROR:vmware.appliance.vapi.impl:pint.Error('com.vmware.applmgmt.err_invalid_remote_loc', '%(0)s location is invalid.', **{'args': LocationType(string='SMB')})

This issue happens when SMB1 is disabled on File Server OR blocked in network. If you are using windows as file server then run following PowerShell to see current status of SMB version 1 & 2.

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol

Typical Output

Get-SmbServerConfiguration |select EnableSMB1Protocol, EnableSMB2Protocol
EnableSMB1Protocol EnableSMB2Protocol
------------------ ------------------
             False               True

Enabling SMB1 version may help to fix this issue.

Set-SmbServerConfiguration -EnableSMB1Protocol $true

Typical Output

Confirm, Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Server Configuration'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

Please note, if SMB1 is disabled deliberately due to security concern as it considered  weaker protocol compare to SMB2/3 then use alternate protocol (HTTP, FTP, NFS..ETC) to take vCenter backup via VAMI.