Unhandled Exception while clicking on storage adapter using ESX UI.

This particular problem is seen with ESXi version 6.7 Update 2 (Build 13006603). Using host UI (https://HOSTNameORIP/ui ) when you click on Storage then Adapter then  following unhandled exception comes. This stop configuring storage related activity from ESXi.

Unhandled exception
-----------
Unfortunately, we hit an error that we weren't expecting.
The client may continue working, but at this point,
we recommend refreshing your browser and submitting a bug report.
Press the Esc key to hide this dialog and continue without refreshing

 

while clicking on Details options, following lines are seen.

Cause: Possibly unhandled rejection: {}
Version: 1.33.3
Build: 12923304
ESXi: 6.7.0
Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Exception stack:
this.toAddress64@https://192.168.100.11/ui/scripts/main.js:422:206
updateAdapter@https://192.168.100.11/ui/scripts/main.js:365:26206
@https://192.168.100.11/ui/scripts/main.js:365:26542
k/<@https://192.168.100.11/ui/scripts/main.js:324:23176
$digest@https://192.168.100.11/ui/scripts/main.js:324:28780
$evalAsync/<@https://192.168.100.11/ui/scripts/main.js:324:30503
e@https://192.168.100.11/ui/scripts/main.js:323:10071
tg/h.defer/c<@https://192.168.100.11/ui/scripts/main.js:323:11522

 

VMkernel.log shows following lines.

2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)World: 11943: VC opID esxui-4004-bd50 maps to vmkernel opID 27da76af
2019-08-08T19:03:40.585Z cpu32:2099684 opID=27da76af)NVDManagement: 1461: No nvdimms found on the system

 

Clicking on Reload option log out ESXi web UI and you have to login to UI again.

 

This particular problem is fixed in ESXi build number 13981272 (ESXi 6.7 EP 10) OR later version. Refer KB  to know more about ESXi build version.
ESXi Patch download link
Select ESXi and version 6.7 then search to get latest patch.

The workaround is to storage setting using vCenter web administration (FLash/HTML).

Advertisements

VMware Appliance Monitoring Service (vmware-statsmonitor) doesn’t start.

After reboot vmware-statsmonitor service doesn’t start automatically. In some situation, this service even doesn’t start manually and fail with following error.

root@buildvcenter [ ~ ]# service-control --start vmware-statsmonitor
Operation not cancellable. Please wait for it to finish...
Performing start operation on service statsmonitor...
channel 3: open failed: administratively prohibited: open failed
Error executing start on service statsmonitor. Details {
    "detail": [
        {
            "args": [
                "statsmonitor"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'statsmonitor'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "problemId": null,
    "resolution": null,
    "componentKey": null
}

 

Following line can be seen in Vmon log (/var/log/vmware/vmon/vmon-syslog.log)

2019-08-08T09:26:42.809220-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:42.809574-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:45.810708-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:51.166333-07:00 warning vmon   Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /var/vmware/applmgmt/statsmonitor_health.xml. Error: [Errno 2] No such file or directory: '/var/vmware/applmgmt/statsmonitor_health.xml'
2019-08-08T09:26:51.166701-07:00 warning vmon
2019-08-08T09:26:51.194457-07:00 warning vmon   Service api healthcheck command returned unknown exit code 1
2019-08-08T09:26:51.194832-07:00 notice vmon   Re-check service health since it is still initializing.
2019-08-08T09:26:54.195953-07:00 notice vmon   Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
2019-08-08T09:26:58.391456-07:00 notice vmon   Service start operation timed out.
2019-08-08T09:26:58.394656-07:00 notice vmon   Cancelling execution of pid 38052
2019-08-08T09:26:58.395009-07:00 warning vmon   Found empty StopSignal parameter in config file. Defaulting to SIGTERM

This issue happens because of startup delay and can be fix by following steps:-

  • Take snapshot of vCenter to be in safer side.
  • Take SSH to VCSA using root login.
  • Modify statsmonitor service config for vMon to set higher start up timeout:
sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
kill -HUP $(cat /var/run/vmon.pid)
  • Stop and start stats-monitor service explicitly.
/usr/lib/vmware-vmon/vmon-cli -k statsmonitor
/usr/lib/vmware-vmon/vmon-cli -i statsmonitor
  • Then restart vCenter to see if that service start automatically

Remote access for ESXi local user account ‘root’ has been locked for XXXX seconds after XXXX failed login attempts.

Due to consistent wrong password by application OR user, root a/c of ESXi may get locked out. This prevent any further login externally to ESXi host.

This happens most likely due to backup/monitoring application that has root user a/c configured the operation.

To solve this issue.

  • Take Console access (iDRAC/ILO/KVM..etc depending on hardware).
  • Press F2 to customize the system. Log in as root.
  • Use the Up/Down arrows to navigate to Troubleshooting Options > Enable ESXi Shell.
  • Press ALT+F1, login as root. Run the following commands to show number of failed attempts:
pam_tally2 --user root
  • Run the following command to unlock the root account:
pam_tally2 --user root --reset

 

Now you should be able to login to ESXi using root a/. You can review hostd log to find out from where failed login coming.

less /var/log/hostd.log |grep -i 'password'

2019-07-31T17:08:51.735Z info hostd[2099345] [Originator@6876 sub=Default 2019-07-31T20:51:07.055Z warning hostd[2205446] [Originator@6876 sub=Default opID=esxui-fca4-b52f] Rejected password for user root from 192.168.214.108
2019-07-31T20:51:11.056Z verbose hostd[2205444] [Originator@6876 sub=Solo.Vmomi] Arg password:
2019-07-31T20:51:39.634Z warning hostd[2099346] [Originator@6876 sub=Default opID=esxui-c6d2-b54a] Rejected password for user root from 192.168.214.108

 

Based on IP address check if any application configured with incorrect root password.

Please note in ESXi 6.5/6.7, invalid login may cause host to be unresponsive. Review KB  for proactive steps.

 

How to restore accidentally deleted one note files.

Due to any unwanted circumstance such as incorrect migration, space issue.. etc, you may lose one note files. Fortunately Microsoft one note have automatic backup (out of your regular backup application) that keep certain numbers of files till the retention you have specified.

These settings can be configure from one note application itself.

Open one note, select File from menu list, select Options then select Save & Backup.

onenote_issue.jpg

By default the backup location is typically configured to user profile unless if not change by user/administrator.

e.g.

C:\Users\<UserAlias>\AppData\Local\Microsoft\OneNote\<OneNoteVersion>\Backup

backup_copy.jpg

 

From this location you should see different section and backup copies. Select latest copy then open in one note then right click select Move or Copy option OR you can merge with any existing one note section.

copymove.jpg

Life saver option who use one note

Unable to login to vCenter using Active directory user credential.

vCenter is joined to AD domain and identity source is configured as Integrated windows authentication but still unable to login to vCenter. You may see “access denied” in vSphere flash OR UI client.

You may see following lines in websso.log (var/log/vmware/sso/).

[2019-06-18T21:14:41.278Z tomcat-http--9 vsphere.local        2a51ab88-55aa-4194-9d63-ec5acbac4c27 INFO  auditlogger] {"user":"ADdomain\\ADUser","client":"10.200.201.223","timestamp":"06/18/2019 21:14:41 UTC","description":"User aduser@addomain.local 10.200.201.223 failed to log in with response code 401","eventSeverity":"INFO","type":"com.vmware.sso.LoginFailure"}
[2019-06-18T21:14:50.296Z tomcat-http--16 vsphere.local        e5a24519-8856-482f-a45a-9e4eb8d6eb8c ERROR com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [aduser@addomain.local] for tenant [vsphere.local] com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 851968][null][null]
at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.AuthenticateByPassword(LinuxIdmNativeAdapter.java:188) ~[vmware-identity-platform-7.0.0.jar:?]
at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.authenticate(ActiveDirectoryProvider.java:289) ~[vmware-identity-idm-server-7.0.0.jar:?]

When you create vCenter with IP address then you may see host name as photon-machine. However if the computer account in AD have duplicate/missing OR DNS record is missing then you see above error.

root@photon-machine [ /opt/likewise/bin ]# ./domainjoin-cli query
Name = photon-machine
Domain = ADDomain.local
Distinguished Name = CN=PHOTON-MACHINE,OU=Servers,DC=ADdomain,DC=local

 

If you have multiple computer a/c in AD with same name then follow below procedure.

# /opt/likewise/bin/domainjoin-cli leave

Reboot vCenter (using VAMI page).
Delete both computer a/c’s from Active directory.
Join vCenter into AD.

# /opt/likewise/bin/domainjoin-cli join addomain.local administrator@addomain.local

Reboot vCenter (using VAMI page).
Login using AD a/c.

 

If you then DNS record is missing for vCenter host name (as per above example it’s photon-machine) then create forward/reverse record in DNS server.

Email notification for schedule task doesn’t go to relay server configured in vCenter.

Email notification can be configure for vCenter alerts as per documentation

However when you do any schedule activity such as snapshot / power on-off and give email address for this activity it may not go to relay server configured in the vCenter server as per above documentation.

As per the network traces you may see vCenter trying to contact email server as per MX record instead of going directly relay server configured as per vCenter.

To fix this issue we need to follow below steps.

  • Take snapshot of vCenter VM.
  • Take SSH session to vCenter VM, change to the /etc/mail folder:
cd /etc/mail
  • Make a backup copy of submit.cf
cp submit.cf submit.cf.orig
  • Edit submit.cf using vi, WinSCP, or any other preferable method and find these lines:
# "Smart" relay host (may be null)
DS
  • After the “DS”, enter the FQDN of your SMTP server like this:
# "Smart" relay host (may be null)
DS smtpserver.domain.edu
  • Restart the sendmail service by running the below command:
systemctl restart sendmail.service

Now vCenter should send email to relay server configured in your environment.

Unable to open vCenter web console (Flash/HTML) after migrating from 6.5.X to 6.7.X

While opening Flash client (https://vcsa.domain.local/vsphere-client/), you may see error while opening Flash client.

An internal error has occurred-No state found from reature NativeBrowserFileTranster
Reloading the client is recommended so as to clear any problems left by this error.
click Yes to reload the vSphere web client.
Show error stack.
YES No

When we click on Yes then you see following stack.

Error Stack
ArgumentError No state found tor reature NativeBrowserFileTranster
at com vmware nexutil configuration: FeatureStateManagemsEnabled0
at conv vmware vmrc:
at conv vmware UtiLS/init()
at conv vmware ui ut":
at com vmware ui ut":
at com vmware nexutil proxies: ConfigurationServiceProxylgetConfiguration()
at conv vmware nexutil proxies ( ConfigurationServicePr0KylgetAProperties()
at conv vmware util: CommonApplnitialZernoadWebclientConfig()
at conv vmware ui util::
at conv vmware
at UlMediator/onBackendEndpointsSessionlnitialized()
at com vmware nexutil proxies: Baseproxy/notity()
at conv vmware nexutil
at com vmware nexutil proxies: : BaseProMonlnvocatimComplete()
at Operationlnvoker'resultResponseForRequest()
at Operationlnvoker/result()
at mx.rpc: :AsyncTokemhttp:/twww.adobe
my rpc events ResultEvenVhttp:/Äwmadobe_com.Q006.mex/mx/internal::callT*enResponders()
at mx.rpc: :AbstractOperatiomhttp:lhmwadobecomQOOömex/mVinternal::dispatchRpcEvent()
at mx.rpc: : Responder/resut()
at NetConnectimMessageResponderjresutHandIer()

 

HTML client (https://vcsa.domain.local/ui/) and VAMI page (https://vcsa.domain.local:5480) just load spinning wheel with blank screen and doesn’t load inventory.

This issue comes when vCenter migration from old VM to new VM and browser have old cache information. To isolate the issue try opening vCenter URL from any other workstation. To fix the issue clear browser cache OR reset it to default.

For example, for Google Chrome following steps can be carried out:-
Settings \ Expand Advanced \ section Reset and clean up then Restore settings to their original defaults.