vCenter service ‘VPXD’ fails to start with error “No space left on device”’

When you start the service manually using following command it gives error.

Syntax

service-control --start vmware-vpxd

Error

Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False
2019-02-08T22:44:52.954Z   Service vpxd state STOPPED
Error executing start on service vpxd. Details {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Service-control failed. Error {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

Error in VPXD log


2019-02-08T22:44:53.608Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Starting VMware VirtualCenter 6.5.0 build-8024368
2019-02-08T22:44:53.608Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Log directory: /var/log/vmware/vpxd.
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=Main] Account name: root
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxUtil] [LoadMachineInstanceUuid] Local instance UUID: 6f0cbee2-56b1-43d3-b13a-348208627e07
2019-02-08T22:44:53.614Z error vpxd[7FB5996C4800] [Originator@6876 sub=Main] Init failed. Exception: N7Vmacore15SystemExceptionE(No space left on device)
--> [context]zKq7AVECAAAAADBxegAOdnB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAC8HisAtJgaAH6mGgG4w6d2cHhkAAEBxqcBCsinAeEoVQG9+1QBap9TAuAFAmxpYmMuc28uNgABdZdT[/context]
2019-02-08T22:44:53.615Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
2019-02-08T22:44:53.615Z info vpxd[7FB5996C4800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd.
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=CryptUtil] [static bool Vpx::Common::CryptUtil::UnmungePasswordToBuffer(VpxDecrypter*, const string&, char*, size_t)] invalid decrypter
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 00000:0
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType]: Database error: ODBC error: (00000) -
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Error getting configuration info from the database
2019-02-08T22:44:53.617Z warning vpxd[7FB5996C4800] [Originator@6876 sub=Main] Database not initialized. Nothing to unlock
2019-02-08T22:44:53.617Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

This issue may caused because of Inodes are full on vCenter appliance. More details on Inodes

To see current storage usage of iNodes, run following command.


root@vCenterServer [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 712704        0  100% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core

This command will give you iNode counts

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

sample output


    875 /usr/bin
   1417 /opt/vmware/lib/python2.7/test
   1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
   5362 /usr/share/man/man3
607550 /var/spool/mqueue

Go to path where you see most of the inodes, then get file count.

[/var/spool/mqueue ]# ls -ltrh | wc
607551 5467952 35237911

Delete inodes, take snapshot of vCenter VM for safer side.

find /var/spool/mqueue/ -type f -print0 | xargs  -0 rm -f

Verify space for inodes again using command ‘df -i

# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 105154   607550   15% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core

 

#linux, #vmware, #vpxd

SQL 2016 on Linux OS

Since SQL version 2016, customer can decide platform other than windows OS. SQL 2016 can be installed and configured on Linux OS and it’s completely supported. In my opinion the decision of allowing Linux platform is to compete with open source database engine such as MySQL, PostgreSQL and ORACLE those provide similar capabilities. The licensing & editions are still not clear but the platform provide most of features a DBA will look for.

Supported Linux Operating System

Red Hat Enterprise Linux 7.3 Workstation, Server, and Desktop
SUSE Enterprise Linux Server v12 SP2
Ubuntu 16.04LTS and 16.10
Docker Engine 1.8+ on Windows, Mac, or Linux
Operating system should minimum 3.25 GB of RAM and it’s tested up to 1 TB.

UN-Supported Features

Replication/mirroring
Poly-base, stretched DB
System extended stored procedures (XP_CMDShell, etc.)
Windows/AD authentication
User certificate SSL/TLS
Agent service / Brower / Reporting service
More Details

Installation demo of SQL on Ubuntu 16.04 Server

Configure prerequisite

Step 1 Download Ubuntu server ISO

Step 2 Create a new Linux OS in your virtual environment. You should have atleast 4 GB RAM &  20 GB disk space (Recommend  Test environment). Supply ‘User Name’ & ‘Password’ that will be used to login in terminal (command prompt of Linux), start the installation.

Step 3. Installation will continue without any user intervention.

Step 4. Once installed successfully, supply user name / password (Refer Step 2) to login in terminal.

Step 5. Get yourself familiarize with Linux Ubuntu commands.
Get the OS details.
IP address of machine
Hostname of machine
Check connectivity with internet using ‘ping’

lsb_release –a
ifconfig
hostname
ping help.ubuntu.com

If you wish to change IP address then refer

Step 6. Install the Open SSH client applications on Ubuntu server so  you will be able to take terminal access remotely using putty  and it will help to insert commands easily.

sudo apt-get install openssh-server openssh-client

 

(All above steps are shown in following slide show)

This slideshow requires JavaScript.

Installation & configuration of SQL Server

Step 1 Ensure your system is fully updated before installing SQL.

sudo apt-get update

Step 2. Install curl (curl is a tool to transfer data from OR to a server using supported protocol (http,imap, ldap…etc)

sudo su
sudo apt-get install curl

STEP 3 Add ‘SQL Server’ information to the repository on ‘/etc/apt/sources.list.d/’

curl https://packages.microsoft.com/config/ubuntu/16.04/mssql-server.list >/etc/apt/sources.list.d/mssql-server.list
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add

STEP 4 Install SQL server.

sudo apt-get install mssql-server

STEP 5 Run the configuration script to accept the license agreement and provide the System Administrator (SA) password.

sudo /opt/mssql/bin/mssql-conf setup

STEP 6 SQL service will be enable/start by default. If it does not then enable and start the service. Also verify if SQL is listening on port 1433

sudo systemctl enable mssql-server
sudo systemctl start mssql-server
sudo netstat -peant | grep ":1433"

STEP 7 The SQL server is ready, DBA now can access SQL server using ADO.NET client OR SQL Management Studio installed on Windows machine for Admin purpose.

(All above steps are shown in following slide show)

This slideshow requires JavaScript.

Installation of SQL Server command-line tools (SQLCMD)

Step1 Download SQLCMD tool and ODBC Drivers..

curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list>/etc/apt/sources.list.d/ms-prod.list
sudo apt-get update

Step 2 Install SQLCMD tool and ODBC Drivers.

sudo apt-get install mssql-tools

Accept license agreement.
Step 3 Create symlink for SQLCMD and

ln -sfn /opt/mssql-tools/bin/sqlcmd /usr/bin/sqlcmd

Step 4 Connect SQL server using SQLCMD then create database, table & inserts few rows.

sqlcmd -S 192.168.132.132 -U SA -P Password123#

Use [master]
create database LinuxDBTest
go
use LinuxDBTest
create table t1 (c1 int, c2 varchar(100))
go
insert into t1 values (1, 'pradeep')
go
select * from t1

Step 5 Verify DB, Tables & row using SSMS.

This slideshow requires JavaScript.

#linux, #sqlonlinux