Part 2 - Windows Scheduled task that auto-pulls Sitecore XMC Logs and sends to Splunk endpoint
In one of my earlier posts, I covered the Sitecore CLI PowerShell Script that downloads XMC logs from a Sitecore XMC environment. In this blog post, I cover the second part wherein I send the logs to a Splunk endpoint for ingestion. I also collate the logs in a local folder to facilitate filter and analysis of logs using SCLA - a capability not present in Sitecore XMC logs. As part of this approach, I also setup the scheduled task(s) auto-triggering the download and subsequent Splunk ingestion process. Although the ideal path "might" be to setup something in Azure that triggers and facilitates Splunk ingestion, the scheduled task used here is more of a self-contained approach to showcase the e2e flow - useful when you want to PoC the feasibility without waiting for access to Azure infrastructure in a long-winded, process-oriented environment. Furthermore, it is good to have multiple ways to accomplish the same end-result. This mindset not only improves confidence among stakeholders but also shows the capability of Sitecore CLI and PowerShell as such. So, without much ado, here are the steps involved:
How to create a scheduled task that downloads XMC logs in a periodic interval?
1. Windows Start menu > Task scheduler:
2. Task Scheduler library > Create Task:
3. Provide a meaningful name for the task:
Name: XMC Dev Env log download
Then, Click Change User button and provide admin credentials, in case if your windows login user doesn't have local admin permissions, since this would be used during task execution (prefix the relevant domain name, say devops, so that the user name is identified while you check name - devops\nadmnxsundarajan):
5. In the Triggers tab, Click New button to open the trigger window.
Since I want the scheduled task to run every 15 minutes without any end date/time, I make the following selections:
Under Settings, select One time
Under Advanced settings, select Repeat task every 15 minutes
for a duration of Indefinitely
Enabled - leave it checked
6. Next, in the actions tab, click New action.
In this case, I want the download logs PS script to be executed first. The name of the script is download-xmclogs.ps1 stored under c:\code in the machine. I also want the messages to go to log file and so, the log file name full path after *>
While, Start in specified the working directory for the PS Script
Program/script: C:\Windows\System32\cmd.exe
Add arguments (optional): /c powershell.exe "c:\code\download-xmclogs.ps1" *> "c:\code\download-xmclogs.log"
Start in (optional): c:\code
Download-XMCLogs.ps1
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #prerequisite: .net SDK - https://dotnet.microsoft.com/en-us/download | |
| #pass env id as param | |
| #https://github.com/svdoever/svdoever.github.io/blob/2afce8fa91dba2a08d1940c23910abd73ac97f7e/src/pages/XM_Cloud_build_and_deploy_like_a_pro.md?plain=1#L2 | |
| #https://thesitecorist.net/2022/12/19/sitecore-xm-cloud-logs/ | |
| #https://www.sergevandenoever.nl/XM_Cloud_build_and_deploy_like_a_pro/ | |
| [CmdletBinding()] | |
| Param | |
| ( | |
| [string] $orgClientId="sdssgs", | |
| [string] $orgClientSecret="dfsdgfsfs-sdfsfs", | |
| [string] $envId="sdfdsfsf", | |
| [string] $cmUrl="https://sdfsf-sdfsf-devd289.sitecorecloud.io/", | |
| [string] $outputPath=".", | |
| [bool] $firstLoad=$false, | |
| [bool] $downloadDeployLogs=$false, | |
| [string] $downloadFileName="" | |
| ) | |
| $watch = [System.Diagnostics.Stopwatch]::StartNew() | |
| $watch.Start() # Timer start | |
| $time = Get-Date -Format "yyyy-MM-dd HH:mm:ss" | |
| Write-Host("Start Date/Time - $time") | |
| function ConnecttoXMCEnvironment() | |
| { | |
| Register-PackageSource -Name Sitecore -Location https://nuget.sitecore.com/resources/v3/index.json -ProviderName NuGet #equivalent for VS nuget registration | |
| dotnet new tool-manifest --force | |
| dotnet tool install Sitecore.CLI --add-source https://sitecore.myget.org/F/sc-packages/api/v3/index.json | |
| dotnet tool restore | |
| Dotnet sitecore init | |
| dotnet sitecore plugin add -n Sitecore.DevEx.Extensibility.Xmcloud | |
| #add more plugins here | |
| #dotnet sitecore cloud login #- this pops the login page that is intrusive | |
| dotnet sitecore cloud login --client-credentials true --client-id $orgClientId --client-secret $orgClientSecret --allow-write | |
| dotnet sitecore connect -r xmcloud --cm $cmUrl --allow-write true --environment-name default | |
| } | |
| function DownloadDeploymentLogs() | |
| { | |
| $depPath=$outputPath + "\deploylogs" | |
| if (!(test-path -path $depPath)) {new-item -path $depPath -itemtype directory} | |
| if($depPath) { | |
| write-host "Downloading XMC deploy log list to "$depPath | |
| $depLogs=dotnet sitecore cloud deployment list --environment-id $envId --json | ConvertFrom-Json #Get deploy log list | |
| foreach($depLog in $depLogs) | |
| { | |
| #write-host $depObj.id | |
| if($depLog.id) {dotnet sitecore cloud deployment log --deployment-id $depLog.id --path $depPath} | |
| } | |
| } | |
| } | |
| function DownloadAllEnviromentLogs() | |
| { | |
| $envLogPath=$outputPath + "\envlogs" | |
| if (!(test-path -path $envLogPath)) {new-item -path $envLogPath -itemtype directory} | |
| write-host "Downloading XMC Env log list to "$envLogPath | |
| $envLogs=dotnet sitecore cloud environment log list -id $envId #Get env log list | |
| #write-host($envLogs) | |
| foreach($envLog in $envLogs) | |
| { | |
| if ($envLog.Trim() -Match "-") { | |
| $logFileName=$envLog.Trim().Split('-',2).Trim() | |
| write-host "Downloading log file named: "$logFileName | |
| #dotnet sitecore cloud environment log download --environment-id "sfsdfdsf" --log Log.lj4nr.20241011.041315.txt --path .\envlogs | |
| dotnet sitecore cloud environment log download --environment-id $envId --log $logFileName --path .\envlogs --timeout 600 | |
| } | |
| } | |
| } | |
| function DownloadLatestEnviromentLogs() | |
| { | |
| # dotnet sitecore cloud environment log list --latest --environment-id asdada | |
| $downloadPath="\envlogs\delta" | |
| if (!(test-path -path $downloadPath)) {new-item -path $downloadPath -itemtype directory} | |
| #CleanLogsFolder $downloadPath | |
| $envLogPath=$outputPath + $downloadPath | |
| write-host "Downloading XMC latest Env log list to "$envLogPath | |
| $envLogs=dotnet sitecore cloud environment log list --latest --environment-id $envId #Get env log list | |
| write-host "Latest logs:"$envLogs | |
| foreach($envLog in $envLogs) | |
| { | |
| if ($envLog.Trim() -Match "-") { | |
| $logFileName=$envLog.Trim().Split('-',2).Trim() | |
| write-host "Downloading log file named: "$logFileName | |
| dotnet sitecore cloud environment log download --environment-id $envId --log $logFileName --path $envLogPath --timeout 600 | |
| } | |
| } | |
| } | |
| function CleanLogsFolder($folderName) | |
| { | |
| #Get-ChildItem -Path (Join-Path $PSScriptRoot ".\deploylogs") | ForEach-Object { | |
| # $dataPath = $_.FullName | |
| # write-host($dataPath) | |
| # Get-ChildItem -Path $dataPath -Exclude ".gitkeep" -Recurse | Remove-Item -Force -Recurse -Verbose | |
| #} | |
| Get-ChildItem -Path (Join-Path $PSScriptRoot $folderName) | ForEach-Object { | |
| $deployPath = $_.FullName | |
| write-host($deployPath) | |
| Get-ChildItem -Path $deployPath -Exclude ".gitkeep" -Recurse | Remove-Item -Force -Recurse -Verbose | |
| } | |
| } | |
| #CleanLogsFolder #Optional call | |
| ConnecttoXMCEnvironment | |
| if ($downloadFileName) | |
| { | |
| dotnet sitecore cloud environment log download --environment-id $envId --log $downloadFileName --path .\envlogs --timeout 600 | |
| } | |
| else | |
| { | |
| if ($downloadDeployLogs) {DownloadDeploymentLogs} | |
| if ($firstLoad) { | |
| DownloadAllEnviromentLogs #first time download | |
| } | |
| else { | |
| DownloadLatestEnviromentLogs #incremental download | |
| } | |
| } | |
| $watch.Stop() # Stopping the timer | |
| Write-Host "Execution time - " $watch.Elapsed # Print script execution time | |
| Write-Host("Done") |
7. In the Conditions tab, leave everything as-is:
8. In the Settings tab, leave selections as-is:
9. The created task will look like this in the pane:
The important columns in the above pane are:
Next Run Time: When the task will be triggered based on the interval and start time specified in the trigger
Last Run Time: When was the task run last based on the specified frequency
Last Run Result: Task status 0x0 means successful run without any errors
The history tab is quite useful to check the execution status of the scheduled task:
One of the glitches with Sitecore XMC log files is, old log files (back in 2024) don't have a date in each line. So, you might need to get the date from the log file name and pass to Splunk if you are doing a one-time download of old files before sending to Splunk for ingestion. The recent log files do have date in each line. So, if your log integration is going to prod now, since each line has the date, no need to do any pre-processing before sending to Splunk for ingestion.
Old log file (15/Oct/2024):
New log file (24/Jan/2024):
PowerShell Script that uploads log files to Splunk endpoint
Prerequisite: Need a Splunk endpoint (http url and http header) as per this documentation that can be sent a successful curl request: https://docs.splunk.com/Documentation/Splunk/9.4.0/Data/UsetheHTTPEventCollector
E.g., curl https://hec.example.com:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}For instance, if you want to send an entire log file, this should work:
curl --data-binary "@C:\git\sitecore-splunk-logs\Scripts\envlogs\logilename.txt https://http-inputs-abc.splunkcloud.com/services/collector/raw -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0"
Actual PS Script that uploads log files:
The following PS script, gets the log files from the delta folder where latest log files are downloaded and present, then it uploads just those files to Splunk endpoint while subsequently moving over these files to the actual log files folder where all files are maintained. The cumulated log files folder can be useful in using SCLA to filter/analyse the log files locally.
########
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$uri = 'https://http-inputs-abc.splunkcloud.com/services/collector/raw'
$folderName="C:\code\envlogs\delta"
$destination="C:\code\envlogs"
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.add("Authorization", 'Splunk zczxczxcz-zxcz-zcxzxc-zcxcx-zxczxcvvc')
function MoveLogFiles($folderName)
{
#Get-ChildItem -Path (Join-Path $PSScriptRoot ".\deploylogs") | ForEach-Object {
# $dataPath = $_.FullName
# write-host($dataPath)
# Get-ChildItem -Path $dataPath -Exclude ".gitkeep" -Recurse | Remove-Item -Force -Recurse -Verbose
#}
Get-ChildItem -Path $folderName | ForEach-Object {
$deployPath = $_.FullName
write-host($deployPath)
Get-ChildItem -Path $deployPath | Move-Item -Force -Destination $destination
}
}
$watch = [System.Diagnostics.Stopwatch]::StartNew()
$watch.Start() # Timer start
$time = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
Write-Host("Start Date/Time - $time")
Get-ChildItem $folderName -Filter *.txt |
Foreach-Object {
if (!$_.Name.StartsWith('Rendering')) {
$fullName = $_.FullName
Write-host "Processing - "$fullName
$response = Invoke-RestMethod -Infile $fullName -Method 'POST' -Uri $uri -Headers $headers
Write-host "Processed -"$fullName "-" $response
}
}
MoveLogFiles $folderName
$watch.Stop() # Stopping the timer
Write-Host "Execution time - " $watch.Elapsed # Print script execution time
Write-Host("Done")
########
Similar to the earlier download PS script, this script will be part of another scheduled task.Note: Logically, this windows scheduled task should run after an interval of few minutes following the download task!
Remember to whitelist the computer ip address in Splunk cloud config to receive the logs.
The log entries in the log file created via the PowerShell script are always useful for debugging:
SCLA:
Splunk search results:
Comments
Post a Comment