vCenter appliance 6.7 U1 affected because of high growth rate of ‘/dev/mapper/core_vg-core’

vCenter appliance 6.7 U1 shows 100% on ‘/dev/mapper/core_vg-core’. This affect vCenter services those can’t start.

root@vcsa1 [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.9G 0 4.9G 0% /dev
tmpfs 4.9G 712K 4.9G 1% /dev/shm
tmpfs 4.9G 688K 4.9G 1% /run
tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/sda3 11G 6.0G 4.1G 60% /
tmpfs 4.9G 1.4M 4.9G 1% /tmp
/dev/sda1 120M 31M 81M 28% /boot
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/seat_vg-seat 542G 389M 514G 1% /storage/seat
/dev/mapper/db_vg-db 9.8G 1.8G 7.5G 20% /storage/db
/dev/mapper/netdump_vg-netdump 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 9.8G 34M 9.2G 1% /storage/autodeploy
/dev/mapper/core_vg-core 25G 25G 0 100% /storage/core
/dev/mapper/archive_vg-archive 50G 28G 19G 60% /storage/archive
/dev/mapper/updatemgr_vg-updatemgr 99G 5.5G 88G 6% /storage/updatemgr
/dev/mapper/dblog_vg-dblog 15G 822M 14G 6% /storage/dblog
/dev/mapper/log_vg-log 9.8G 2.6G 6.7G 28% /storage/log

Following lines can be seen in logs

/var/log/vmware/messages

2019-04-02T13:19:58.021089-05:00 VCENTERSERVER pschealthd: Detected PSC system is not healthy - Wait for atleast one minute before failing
2019-04-02T13:20:01.792568-05:00 VCENTERSERVER CROND[58477]: (root) CMD ( /opt/vmware/vpostgres/current/scripts/pg_status_cron >/dev/null 2>&1)
2019-04-02T13:20:01.793977-05:00 VCENTERSERVER CROND[58478]: (root) CMD (. /etc/profile.d/VMware-visl-integration.sh; /usr/lib/applmgmt/backup_restore/scripts/SchedulerCron.py >>/var/log/vmware/applmgmt/backupSchedulerCron.log 2>&1)
2019-04-02T13:20:01.809506-05:00 VCENTERSERVER CROND[58481]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
2019-04-02T13:20:01.809886-05:00 VCENTERSERVER CROND[58482]: (root) CMD ( test -x /usr/sbin/cloudvm_ram_size_periodic && /usr/sbin/cloudvm_ram_size_periodic >/dev/null 2>&1)
2019-04-02T13:20:01.813858-05:00 VCENTERSERVER CROND[58483]: (root) CMD (/usr/sbin/logdiskcheck.sh >/dev/null 2>&1)

/var/log/vmware/sso/vmware-sts-idmd.log

[2019-04-01T18:18:28.839-05:00 IDM Startup INFO ] [IdmServer] IDM Server has started
[2019-04-01T18:18:30.017-05:00 INFO ] [NativeLibraryPreloader] jna.library.path: /opt/vmware/lib64:/usr/lib/vmware-vmdir/lib64:/usr/lib/vmware-vmafd/lib64:/opt/likewise/lib64:/usr/lib64
[2019-04-01T18:40:29.359-05:00 IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2019-04-01T18:40:29.376-05:00 IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
[2019-04-01T18:40:50.973-05:00 IDM Startup INFO ] [IdmServer] Starting IDM Server...
[2019-04-01T18:40:50.975-05:00 IDM Startup INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[], eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[SimpleMessage[message=IDM Server has started]], detailText=[null], corelationId=[IDM Startup], timestamp=[1554162050974]
[2019-04-01T18:40:50.975-05:00 IDM Startup INFO ] [IdmServer] IDM Server has started
[2019-04-01T18:40:52.062-05:00

/storage/log/vmware/sso/utils/vmware-stsd.err

Apr 02, 2019 1:23:55 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 541 ms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Apr 02, 2019 1:24:05 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 9793 ms
Service killed by signal 11

In the directory /storage/core, you may see lots of files core.jsvc creates after every minute.

root@vcsa1 [ ~ ]# cd /storage/core/
root@vcsa1 [ /storage/core ]# du -sh * | sort -nr
856M core.jsvc.6178
837M core.jsvc.15781
820M core.jsvc.10014
785M core.jsvc.1742
780M core.jsvc.7499
769M core.jsvc.13795
764M core.jsvc.16996
737M core.jsvc.11088
734M core.jsvc.14865
728M core.jsvc.21240

This happens because of vmware-stsd crashing with core.jsvc.xxx files and fills up /storage/core after upgrading to vCenter Server Appliance 6.7 update 1. Contact VMware support to know about this issue however as temporarily measure you can follow below steps.

Delete core.jsvc files those are created because of this issue.

root@vcsa1 [ /storage/core ]# rm core.jsvc.*

Stop and start vCenter services.

root@vcsa1 [ ~ ]# service-control --stop --all
root@vcsa1 [ ~ ]# service-control --start --all

Remove Active directory as identity source using vCenter web client.

\Home \ Administration \ Configuration under ‘Single sing-on’ \ Identity source.

Select AD domain and delete it.

Dis join vCenter from Active Directory domain.

\Home \ Administration \ Configuration under Deployment \ select Node then vCenter server under Nodes \ Mange then Active Directory under Advanced.

Click on leave.

If active directory is required for authentication then use Active directory as LDAP server as identity source.

Advertisements