Analyze Enterprise Vault Dtraces using PowerShell-SQL tool.

This is for Enterprise Vault Engineers, however slight modification on this tool can help guys those are working on other technologies as well. The most difficult work for a support engineer is to analyze performance issue, where thousands of traces collected over time. In past, I developed a tool (combination of Power Shell & SQL script) that can upload Enterprise Vault dtrace to SQL server then assist on most common use cases such as finding out delay in specific function, searching a given keyword OR extracting entire thread.

You can either download executable file OR use PowerShell script on SQL 2016 server. Download link

Prerequisite

  • Microsoft SQL 2016 installed on Windows 2012 R2.
  • This works with windows authentication so logged in user must have DB creator OR equivalent permission on SQL instance.
  • The tool can be executed on server where SQL server binary is installed. For example, you can run it on SQL1 and connect to SQL2. Both servers should have SQL installed.
  • Copy latest dtrace_analyser exe file to SQL server.
  • Upload & Analyse, both are different operation. For example, you can upload dtrace today and analyse it tomorrow.

Upload D-trace

Double click on dtrace_analyser.exe, a windows form and console (command prompt) will open. Windows form enable user to upload /analyze the data. Console shows the status of operation it executing in background. During upload/analyze, form will free and all options will be disable till execution completes.

Just place all the dtrace in a single directory then supply name of that directory.

First.jpg

Click on Upload Traces, copy dtrace folder location OR click on Browse to select the location.

If the Database (DtraceReview) and Table (DtraceContent) already exists in SQL then it will skip the creation of database/table but truncate existing set of upload dtrace for consistency and accuracy during analysis.

Upload speed depends on resources. Approximately 10-15 seconds will take to upload single 100 MB dtrace file when SQL server have 12 GB RAM & 8 CPU.

Analyze Traces

Click on Find Delay option, by default the location of output HTML file will be C:\temp\dtrace-YYYYMMDDMMSS.html, you can change the location by clicking on Browse option or just type the location in text bar.

second.jpg

This tool automatically review each thread and prepare HTML report. If the numbers of lines in thread is huge, form may go to Not Responding mode, leave it running (will fix in next version) however actual work can be seen from console window.

Third.jpg

Once processing finish, HTML file will automatically open. Delay in seconds can be seen in last column. Any thread where delay is more than 2 second will only be visible in HTML output.

Fourth.jpg

Please note, this tool isn’t intelligent to understand the lines written for d-trace logging. Few function are expected to have delay such as one below. Engineers should understand the limitation because some function completes when their sub function completes.

Fifth.jpg

All process id, process name and thread id can be seen in HTML report.

Six.jpg

Last section of HTML will show events captured across all d-trace files.

Seven.jpg

Use Search Keyword option to search specific function, exception OR line. By default output will be located in C:\temp\search_output_YYYYMMDDHHMMSS.txt. You can change location and file name by clicking on Browse OR manually type the location.

Eight.jpg

Use Extract thread option to download all lines of specific thread to text file. By default output will save in text file located in C:\temp\Dtrace_thread_ThreadID.txt. You can change location and file name by clicking on Browse OR manually type the location.

Ninth.jpg

Figure out who is deleting files from windows operating system

File OR folder from windows operating system (client/OS) might miss due to many different reason. A user may logon to system interactively OR remotely then delete the file OR a malicious process may also delete the file. If you are unsure who is deleting files/folder then windows auditing is the best way to figure this out.

Follow this sequence to understand the concepts.

Enable windows auditing from Local Security Policy (run – secpol.msc). If you are doing against multiple servers then edit group policies from domain controller.

SCpolicy1

You can use following PowerShell to automate this step.

secedit /export /cfg c:\secpol.cfg
(gc C:\secpol.cfg).replace("AuditObjectAccess = 0", "AuditObjectAccess = 3") | Out-File C:\secpol.cfg
secedit /configure /db c:\windows\security\local.sdb /cfg c:\secpol.cfg /areas SECURITYPOLICY
rm -force c:\secpol.cfg -confirm:$false

 

Update group policy using following command.

gpupdate /force

Select folder that needs to be audited. In my example, I am enabling auditing for Delete action on c:\temp\temp folder

SCpolicy2.jpg

You can use below PowerShell

#Uncomment if foder you intending to be audited isn’t created so far.
#New-Item -type directory -path C:\temp\temp
$Folder= "c:\temp\temp"
$ACL = Get-Acl $Folder
$ar1 = New-object System.Security.AccessControl.FileSystemAuditRule ("EveryOne","Delete","3")
$Acl.SetAuditRule($ar1)
Set-Acl $Folder $ACL

 

Now if anyone (user/process) delete your file then event will be generated in your event viewer. For e.g. I am deleting File1.txt using windows explorer (right click \delete) second file using PowerShell.

RM -Force C:\Temp\TEMP\File2.txt -Confirm:$false

 

Open Event viewer and search Security log for Event ID 4656 with “File System” task category and with “Accesses: DELETE” string. “Subject: XXXX” will show you who has deleted a file.

Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: MM/DD/YYYY HH:MM:SS
Event ID: 4656
Task Category: File System
Level: Information
Keywords: Audit Success
User: N/A
Computer: server.domain.local
Description:
A handle to an object was requested.
Subject:
Security ID: domain\user1
Account Name: user1
Account Domain: domain
Logon ID: 0x98B5C
Object:
Object Server: Security
Object Type: File
Object Name: C:\Temp\Temp\File2.txt
Handle ID: 0x774
Resource Attributes: -
Process Information:
Process ID: 0x4c4c
Process Name: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Access Request Information:
Transaction ID: {00000000-0000-0000-0000-000000000000}
Accesses: DELETE
ReadAttributes
Access Reasons: DELETE: Granted by D:(A;ID;FA;;;BA)
ReadAttributes: Granted by D:(A;ID;FA;;;BA)
Access Mask: 0x10080
Privileges Used for Access Check: -
Restricted SID Count: 0

 

OR you can below basic PowerShell to query system event viewer log.

Get-EventLog -LogName Security -InstanceId 4656

Power2-OUT.jpg

#new-item, #uncomment

Configuration of AWS S3 (Simple Storage Service) for application access.

Following are the steps can be useful if you have any application (service) in your on premise that need to access (download/upload) files to AWS S3 storage.

Sign in with root a/c credential to AWS Console

loginroot.jpg

Select IAM under ‘Security, identity and compliance’ container

IAM

Add a new user for API or Console access.

CreateUser.jpg

Give appropriate ‘User name’, Select Access Type.

Please note, selection of both access type isn’t recommended for production use due to accessibly issues. This demo require only ‘Programmatic acces. You can use same user account for delegation of AWS storage related stuff managed via AWS console.

UserProperties.jpg

Select ‘Attach existing policies directly’ then search for S3, attach ‘AmazonS3FullAccess’

Permission.jpg

Review the setting and then click on ‘Create user

Note down user name, access Key ID, Secret Access Key and Sign-in URL. You can additionally download CSV file for all these information.

UserDetails.jpg

Select ‘S3’ from ‘Storage’ section.

S3Page.jpg

 

Click on ‘Create Bucket’, give appropriate name and select ‘Region’, the bucket name should be unique in AWS infrastructure.  Then click on ‘Create’. I have skipped remaining criteria such as version, permission and website related stuff for this test. However if you need to have specific settings please refer.

CreateBucket.jpg

You can upload files manually using AWS Console.

Upload.jpg

 

Testing bucket access using on premise application.

Method 1-using CloudBerry

Install Cloudberry (freeware), Connect to AWS S3 bucket

CloudBerryTest.jpg

You can copy (/cut)-paste files from local machine to S3 OR vice versa.

CloudBerry-copypaste.jpg

 

Method 2 Using-Powershell

Install AWS tools Open PowerShell, use following commands to test bucket access.

Set Credential.

Set-AWSCredentials -AccessKey AKIAI3ZDRI4HGSD4NOGQ -SecretKey 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

Get-all buckets

Get-S3Bucket

Ps1.jpg

Show all contents of specified bucket.

Get-S3Object -BucketName appdatatest1 -MaxKey 100 |Format-Table

PS2.jpg

Refer  for more details on PowerShell commands AWS.

Retrieve Active Directory object properties using VBScript.

In many circumstance, you may need to isolate application and Active directory issues that may popup because of bad network OR configuration of environment. In one of last challenge to isolate application performance issues with network while retrieving AD object I used following VBScript rather than custom application.

Create a text file eg. RetrieveProxy.vbs in your desired location (mine is c:\tools). Paste following code into notepad.

This will retrieve ‘user1’ proxy address from Active directory.

StartTime=Now
REM wscript.echo Starttime
Set objUser=GetObject("LDAP://192.168.2.100/CN=user1,OU=Myusers,DC=domain,DC=local")
ProxyAddress=objUser.proxyAddresses
EndTime=Now
TimeTaken=DateDiff("s",StartTime,EndTime)
wscript.echo("Proxy Address: "&ProxyAddress", Bind took "&TimeTaken&" Seconds, From "&StartTime&" To "EndTime)

You can run either using command line.

commandout.jpg

Or by simply double clicking on “RetriveProxy.vbs”.

window1out.jpg

In above script,

  • You can modify the property name you wish to retrieve from AD. I have used “proxyAddress” you can have “whenCreated”, “whenChanged” OR any other available object properties.
  • You can change IP OR FQDN of Domain controller. It can be without IP/FQDN as well, Eg. “LDAP:// CN=user1,OU=Myusers,DC=domain,DC=local”, in that case it will connect nearest available domain controller.
  • Adding variable ‘StartTime’ OR ‘EndTime’ is completely optional, I have used it so I can see time taken during retrieval so that I can compare it other application doing similar stuff. If it take longer time then expected then definitely some of n/w resources are at fault somewhere.

Following is the modified version so it can run against all objects of an OU (Organizational unit).

StartTime=Now
REM wscript.echo Starttime
Set objUsers=GetObject("LDAP://192.168.2.100/OU=MyUsers,DC=domain,DC=local")
objUsers.Filter = Array("User")
Dim AllUsersProxy
For Each obj In objUsers
   AllUsersProxy=AllUsersProxy&obj.cn&"  "&obj.proxyAddresses&vbnewline
Next
EndTime=Now
TimeTaken=DateDiff("s",StartTime,EndTime)
wscript.echo("Proxy Addresses: "&vbnewline&AllUsersProxy&"Bind took "&TimeTaken&" Seconds, From " &StartTime&" To "&EndTime)

Please note, this script retrieve two properties and output will be new line for each objects.

commandout3

windowout2.jpg

 

Analyze IIS log using PowerShell

In my last blog, you learn how IIS (or similar) log can be uploaded to SQL DB. Once logs are uploaded you can analyze it using PowerShell and create fancy report as well. Let’s see how this can be done.

Following script gives number of request per IIS page with response code.

$Query1="
Use IISLogReview
     SELECT TOP 20 [sc-STATUS] Response, [cs-uri-stem] Access_Page,  count(*) Total_Request from IISLOG
     GROUP BY [sc-STATUS],  [cs-uri-stem]
     ORDER BY COUNT(*) DESC"
$SqlOut1=Invoke-Sqlcmd -ServerInstance SQL1 -Query $Query1 -QueryTimeout 65535
$PrintOut1=$SqlOut1|Select-Object Response,Access_Page,Total_Request|Format-Table -AutoSize
$PrintOut1

iis1.jpg

Following script gives number of request per user.

$Query2="
Use IISLogReview
    SELECT COUNT(*) Total_Request, [s-username] UserName  FROM IISLOG
    GROUP BY [s-username]
    ORDER BY Total_Request DESC"
$SqlOut2=Invoke-Sqlcmd -ServerInstance SQL1 -Query $Query2 -QueryTimeout 65535
$PrintOut2=$SqlOut2|Select-Object Total_Request,UserName |Format-Table -AutoSize
$PrintOut2

iis2.jpg

This is the complete tool/function however this time it will generate a HTML output.

Function AnalyseIIS
{
    PARAM
    (
    #Default SQL will be local host
    [string]$SQLServer=$env:computername,
    #Default HTML File will be c;\temp\yyyymmdd.html
    [string]$HTMLFile='C:\TEMP\'+(Get-Date).tostring("yyyyMMddhhmm")+'.html'
    )

    $Query1=
        "Use IISLogReview
        SELECT TOP 20 [sc-STATUS] Response, [cs-uri-stem] Access_Page,  count(*) Total_Request from IISLOG
        GROUP BY [sc-STATUS],  [cs-uri-stem]
        ORDER BY COUNT(*) DESC"

    $Query2=
        "Use IISLogReview
        SELECT COUNT(*) Total_Request, [s-username] UserName  FROM IISLOG
        GROUP BY [s-username]
        ORDER BY Total_Request DESC"

    $a = ""
    $a = $a + "BODY{background-color:peachpuff;font-family: Calibri; font-size: 12pt;}"
    $a = $a + "TABLE{border-width: 1px;border-style: solid;border-color: black;border-collapse: collapse;}"
    $a = $a + "TH{border-width: 1px;padding: 0px;border-style: solid;border-color: black;}"
    $a = $a + "TD{border-width: 1px;padding: 0px;border-style: solid;border-color: black;}"
    $a = $a + ""

    $SqlOut1=Invoke-Sqlcmd -ServerInstance $SQLServer -Query $Query1 -QueryTimeout 65535
    $PrintOut1=$SqlOut1|Select-Object Response,Access_Page,Total_Request| ConvertTo-HTML -PreContent '</pre>
<h2>Statistics Based on IIS Reponse, Page and Count</h2>
<pre>
'  -head $a |Out-String

    $SqlOut2=Invoke-Sqlcmd -ServerInstance $SQLServer -Query $Query2 -QueryTimeout 65535
    $PrintOut2=$SqlOut2|Select-Object Total_Request,UserName | ConvertTo-HTML -PreContent '</pre>
<h2>Statistics Based on UserName and Count</h2>
<pre>
'  -head $a |Out-String

    ConvertTo-Html -Title "IIS Log Analysis" -PostContent $PrintOut1,$PrintOut2 |Out-File $HTMLFile
    Invoke-Item $HTMLFile #open html after processing
}

AnalyseIIS
#OR Run with Parameter, Eg.
#AnalyseIIS -SQLServer SQL1 -HTMLFile C:\TEMP\IISReview.html 

iis3.jpg

You can have different select queries based on analysis you need and I believe my last two blogs can give you direction on how large set of text files can be uploaded to SQL then analyze it using PowerShell.

 

#default

Upload large logs files to SQL database

Most of us will agree if I say “SQL server is one of the powerful engine for query, reporting and statistic purpose”. As part of application troubleshooting, we come across many different set of logs those can be open in Notepad OR similar tools however it’s very different to take stats out it OR even open if it large. IIS log from web server is one of the nice example. Similarly you may have logs for different application such as D-trace for Enterprise vault.

Here in my blog, I am using IIS logs those can be uploaded to SQL server database using PowerShell script.

Create database (using PS script). If database already exists then it wouldn’t execute the command.

    $PrepareDB= "
    USE master
    DECLARE @DBname VARCHAR(100) ='IISLogReview'
    DECLARE @DBcreate varchar(max) ='CREATE DATABASE ' + @DBNAME
    IF  NOT EXISTS (SELECT name FROM sys.databases WHERE name = @DBname)
	    BEGIN
	    EXECUTE(@DBcreate)
	    END;
    GO
    "
    Invoke-Sqlcmd -ServerInstance SQL1 -Query $PrepareDB

Create table. If table already exists then it will skip but truncate existing rows so it hold only newest sets.

    $PrepareTable= "
    USE IISLogReview
    DROP TABLE IF EXISTS dbo.IISLOG
    CREATE TABLE dbo.IISLOG (
     [DATE] [DATE] NULL,
     [TIME] [TIME] NULL,
     [s-ip] [VARCHAR] (16) NULL,
     [cs-method] [VARCHAR] (8) NULL,
     [cs-uri-stem] [VARCHAR] (255) NULL,
     [cs-uri-query] [VARCHAR] (2048) NULL,
     [s-port] [VARCHAR] (4) NULL,
     [s-username] [VARCHAR] (16) NULL,
     [c-ip] [VARCHAR] (16) NULL,
     [cs(User-Agent)] [VARCHAR] (1024) NULL,
     [cs(Referer)] [VARCHAR] (4096) NULL,
     [sc-STATUS] [INT] NULL,
     [sc-substatus] [INT] NULL,
     [sc-win32-STATUS] [INT] NULL,
     [time-taken] [INT] NULL,
     INDEX cci CLUSTERED COLUMNSTORE
    )
    TRUNCATE TABLE [IISLogReview].dbo.IISLOG
    "
   Invoke-Sqlcmd -ServerInstance SQL1 –Query $PrepareTable

Once database & table is prepared then you can upload multiple IIS log files placed at given location using either ‘Bulk insert’ OR ‘BCP’.

Bulk Insert script

    $FolderPath='F:\IIS'
    $IISFiles=Get-ChildItem -Path $FolderPath -Name
    foreach ($File in $IISFiles)
        {
        $PATH=$FolderPath+'\'+$File
        $UploadQuery="
        BULK INSERT [IISLogReview].dbo.IISLOG
        FROM '$($PATH)'
        WITH (
            FIRSTROW = 2,
            FIELDTERMINATOR = ' ',
            ROWTERMINATOR = '\n'
              )"
        Invoke-Sqlcmd -ServerInstance SQL1 -Query $UploadQuery
        }

Please note, you may see “Bulk load data conversion” error those can ignored as IIS logs have few middle lines those are not well formatted and doesn’t need to be upload.

The upload activity can also be performed using BCP,  my favorite for import/export to/from SQL server.

    $FolderPath='F:\IIS'
    $IISFiles=Get-ChildItem -Path $FolderPath -Name
    foreach ($File in $IISFiles)
        {
        $PATH=$FolderPath+'\'+$File
        $PATH
        bcp [IISLogReview].dbo.IISLOG in $PATH -c -t" " -r\n -T -S SQL1
        }

Here is complete script, make sure to change SQL server name & IIS folder location at the end of the script.

Function UploadIIS
{
    PARAM
    (
    [Parameter(Mandatory=$true)]
    [string]$SQLServer,
    [Parameter(Mandatory=$true)]
    [string]$IISlogFolder
    )
    $PrepareDB= "
    USE master
    DECLARE @DBname VARCHAR(100) ='IISLogReview'
    DECLARE @DBcreate varchar(max) ='CREATE DATABASE ' + @DBNAME
    IF  NOT EXISTS (SELECT name FROM sys.databases WHERE name = @DBname)
	    BEGIN
	    EXECUTE(@DBcreate)
	    END;
    GO
    "
    $PrepareTable= "
    USE IISLogReview
    DROP TABLE IF EXISTS dbo.IISLOG
    CREATE TABLE dbo.IISLOG (
     [DATE] [DATE] NULL,
     [TIME] [TIME] NULL,
     [s-ip] [VARCHAR] (16) NULL,
     [cs-method] [VARCHAR] (8) NULL,
     [cs-uri-stem] [VARCHAR] (255) NULL,
     [cs-uri-query] [VARCHAR] (2048) NULL,
     [s-port] [VARCHAR] (4) NULL,
     [s-username] [VARCHAR] (16) NULL,
     [c-ip] [VARCHAR] (16) NULL,
     [cs(User-Agent)] [VARCHAR] (1024) NULL,
     [cs(Referer)] [VARCHAR] (4096) NULL,
     [sc-STATUS] [INT] NULL,
     [sc-substatus] [INT] NULL,
     [sc-win32-STATUS] [INT] NULL,
     [time-taken] [INT] NULL,
     INDEX cci CLUSTERED COLUMNSTORE
    )
    TRUNCATE TABLE [IISLogReview].dbo.IISLOG
    "
    $Queries = $PrepareDB,$PrepareTable
    foreach ($Query in $Queries)
        {
        Invoke-Sqlcmd -ServerInstance $SQLServer -Query $Query
        }
    $IISFiles=Get-ChildItem -Path $IISlogFolder -Name
    foreach ($File in $IISFiles)
        {
        $PATH=$IISlogFolder+'\'+$File
        bcp [IISLogReview].dbo.IISLOG in $PATH -c -t" " -r\n -T -S $SQLServer
        }
}

UploadIIS -SQLServer SQL1 -IISlogFolder F:\IISFolder

Although few of next blogs will be on analysis of IIS log uploaded on SQL but there is always a way to manually query the uploaded data using adhoc queries via SQL management studio.

 

SQL table export using PowerShell

Most of the PowerShell guys may need to extract data from SQL databases. There could be two or more ways to do the same task however as per my latest findings in one of the project I figured out including BCP in our script is the best way to achieve that. In my following two examples I am using AdventureWorksDB and a SQL query that write data into a text file.

Example 1 using Invoke-SQLCmd.

$SQLQueryy= "USE AdventureworksDW2016CTP3
SELECT  ProductKey, OrderDateKey, ShipDate FROM [dbo].[FactResellerSalesXL_CCI]
WHERE ProductKey =532"
$StartTime = (Get-Date)
Invoke-Sqlcmd -ServerInstance SQL2 -Query $SQLQueryy -QueryTimeout 65535 |Format-Table -HideTableHeaders -AutoSize |Out-File 'C:\temp\via_InvokeSQL.txt'  -Width 500
$Endtime=(Get-Date)
$TotalTime = "Total Elapsed Time : $(($Endtime-$StartTime).totalseconds) seconds"
$TotalTime

UsingSQLCmd.jpg

This script takes almost 14 seconds to extract the rows from SQL.

Example 2 Using BCP

$StartTime = (Get-Date)
bcp  "USE AdventureworksDW2016CTP3 SELECT  ProductKey, OrderDateKey, ShipDate FROM [dbo].[FactResellerSalesXL_CCI] WHERE ProductKey =532"  queryout 'C:\temp\via_BCP.txt' -T -c -S SQL2
$Endtime=(Get-Date)
$TotalTime = "Total Elapsed Time : $(($Endtime-$StartTime).totalseconds) seconds"
$TotalTime

UsingBCP.jpg

This query take less than 1 seconds with neat & clean output file.