Tactical Solution for integrating logs from XMCloud and other Sitecore SaaS offerings with Splunk - Log Download Process - Part 1
Currently, Sitecore DXP product SaaS suite doesn't support Common Audit Logging (CAL) as an e2e solution. So, as part of integrating the logs with an SIEM service like Splunk, you need to custom-stitch the integration by pulling all the logs across the suite in order to send those for observability to Splunk. So, in this post, I cover the first set of steps as part of the integration process.
Before proceeding further, here is some information from Sitecore:
========================================
These are different SaaS products available in your Sitecore Cloud:
- Sitecore Cloud Portal
- Sitecore XM Cloud
- Sitecore Search
- Sitecore Personalize
- Sitecore Connect
- Others
It's important to know that the Sitecore Cloud Portal provides the single entry point for other Cloud products.
Different products have different requirements for Audit Logs. For example:
- Identity logs (user logins, user management, etc) are tracked by Sitecore Cloud Portal.
- Application specific logs are tracked by specific applications.
Common Audit Logs
- The Sitecore Common Audit Logs is Sitecore' framework for integrating the product audit logs with external logging system (such as Splunk).
- To use the CAL today, you can create webhook (using the REST API). When an event is triggered from one of Sitecore' supported applications, it will trigger a notification with the event data to the webhook endpoint.
- The following products are supported today:
- Sitecore Personalize
- Sitecore CDP
- Sitecore Connect
- The following products are not supported but they will be delivered in the future (No ETA):
- Sitecore Cloud Portal
- Sitecore XM Cloud
- Sitecore Search
- Others
Because of this, you can stream application specific events (excluding Identity logs) from Sitecore Personalize to Splunk using this webhook method. However, you may face issues if your Splunk endpoint is configured to accept incoming requests from whitelisted IPs but CAL webhook doesn't provide static IPs for whitelisting. To address this issue, you can create a webhook using a custom APIM or Sitecore Connect to expose a static IP while forwarding the message from CAL to Splunk.
========================================
Now, my PoC:
Step-1 - PS Script that downloads XMCloud related logs in one-place:
Since XMC is the center piece, consolidating logs from XMC environment/deploy logs is one of the important aspects so, sharing here some of the important tips/notes.
Tip-1: Use organization Client ID and Client Secret for non-interactive XMCloud login:
You might want to use the download script as part of a scheduler so, generate organization (not environment) Client Id and secret in deploy.sitecorecloud.io:
Tip-2: Set longer timeout for log download:
One of the issues i faced with the above script was the following abort state during download since the environment log files were in mega bytes. I raised a Sitecore support ticket but i myself was soon to realize that there is a way to increase the timeout default from 100s to any number you desire using the timeout arg. When I did the same, to 600s, the log file download(s) was successful:
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
Execution was cancelled. This occurs when the task is killed or requests timeout.
Tip-3: Create a clean PS Script:
A clean script to delete the logs and restart is always handy while doing iterative development in the local environment.
##############
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Get-ChildItem -Path (Join-Path $PSScriptRoot ".\deploylogs") | ForEach-Object { | |
| $dataPath = $_.FullName | |
| write-host($dataPath) | |
| Get-ChildItem -Path $dataPath -Exclude ".gitkeep" -Recurse | Remove-Item -Force -Recurse -Verbose | |
| } | |
| Get-ChildItem -Path (Join-Path $PSScriptRoot ".\envlogs") | ForEach-Object { | |
| $deployPath = $_.FullName | |
| write-host($deployPath) | |
| Get-ChildItem -Path $deployPath -Exclude ".gitkeep" -Recurse | Remove-Item -Force -Recurse -Verbose | |
| } |
##############
Tip-4: Build separate functions for first time env logs download and incremental download:
#########first time download
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function DownloadAllEnviromentLogs() | |
| { | |
| $envLogPath=$outputPath + "\envlogs" | |
| write-host "Downloading XMC Env log list to "$envLogPath | |
| $envLogs=dotnet sitecore cloud environment log list -id $envId #Get env log list | |
| #write-host($envLogs) | |
| foreach($envLog in $envLogs) | |
| { | |
| if ($envLog.Trim() -Match "-") { | |
| $logFileName=$envLog.Trim().Split("-")[1].Trim() | |
| write-host "Downloading log file named: "$logFileName | |
| dotnet sitecore cloud environment log download --environment-id $envId --log $logFileName --path .\envlogs --timeout 600 | |
| } | |
| } | |
| } |
#########first time download
#########incremental download
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function DownloadLatestEnviromentLogs() | |
| { | |
| $envLogPath=$outputPath + "\envlogs" | |
| write-host "Downloading XMC latest Env log list to "$envLogPath | |
| $envLogs=dotnet sitecore cloud environment log list --latest --environment-id $envId #Get env log list | |
| #write-host($envLogs) | |
| foreach($envLog in $envLogs) | |
| { | |
| if ($envLog.Trim() -Match "-") { | |
| $logFileName=$envLog.Trim().Split("-")[1].Trim() | |
| write-host "Downloading log file named: "$logFileName | |
| dotnet sitecore cloud environment log download --environment-id $envId --log $logFileName --path $envLogPath --timeout 600 | |
| } | |
| } | |
| } |
#########incremental download
Consolidated Script at this point:
#########################
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #prerequisite: .net SDK - https://dotnet.microsoft.com/en-us/download | |
| #pass env id as param | |
| #https://github.com/svdoever/svdoever.github.io/blob/2afce8fa91dba2a08d1940c23910abd73ac97f7e/src/pages/XM_Cloud_build_and_deploy_like_a_pro.md?plain=1#L2 | |
| #https://thesitecorist.net/2022/12/19/sitecore-xm-cloud-logs/ | |
| #https://www.sergevandenoever.nl/XM_Cloud_build_and_deploy_like_a_pro/ | |
| [CmdletBinding()] | |
| Param | |
| ( | |
| [string] $orgClientId="fdsfsfsd", | |
| [string] $orgClientSecret="fdsdfs-sfdds", | |
| [string] $envId="sfdsfsd", | |
| [string] $cmUrl="https://xmc-sdfsfs-dsfsdsdf-prod.sitecorecloud.io/", | |
| [string] $outputPath="." | |
| ) | |
| $watch = [System.Diagnostics.Stopwatch]::StartNew() | |
| $watch.Start() # Timer start | |
| $time = Get-Date -Format "yyyy-MM-dd HH:mm:ss" | |
| Write-Host("Start Date/Time - $time") | |
| function ConnecttoXMCEnvironment() | |
| { | |
| dotnet new tool-manifest --force | |
| dotnet tool install Sitecore.CLI --add-source https://sitecore.myget.org/F/sc-packages/api/v3/index.json | |
| dotnet tool restore | |
| Dotnet sitecore init | |
| dotnet sitecore plugin add -n Sitecore.DevEx.Extensibility.Xmcloud | |
| #add more plugins here | |
| #dotnet sitecore cloud login #- this pops the login page that is intrusive | |
| dotnet sitecore cloud login --client-credentials true --client-id $orgClientId --client-secret $orgClientSecret --allow-write | |
| dotnet sitecore connect -r xmcloud --cm $cmUrl --allow-write true --environment-name default | |
| } | |
| function DownloadDeploymentLogs() | |
| { | |
| $depPath=$outputPath + "\deploylogs" | |
| write-host "Downloading XMC deploy log list to "$depPath | |
| $depLogs=dotnet sitecore cloud deployment list --environment-id $envId --json | ConvertFrom-Json #Get deploy log list | |
| foreach($depLog in $depLogs) | |
| { | |
| #write-host $depObj.id | |
| if($depLog.id) {dotnet sitecore cloud deployment log --deployment-id $depLog.id --path $depPath} | |
| } | |
| } | |
| function DownloadAllEnviromentLogs() | |
| { | |
| $envLogPath=$outputPath + "\envlogs" | |
| write-host "Downloading XMC Env log list to "$envLogPath | |
| $envLogs=dotnet sitecore cloud environment log list -id $envId #Get env log list | |
| #write-host($envLogs) | |
| foreach($envLog in $envLogs) | |
| { | |
| if ($envLog.Trim() -Match "-") { | |
| $logFileName=$envLog.Trim().Split("-")[1].Trim() | |
| write-host "Downloading log file named: "$logFileName | |
| dotnet sitecore cloud environment log download --environment-id $envId --log $logFileName --path .\envlogs --timeout 600 | |
| } | |
| } | |
| } | |
| function DownloadLatestEnviromentLogs() | |
| { | |
| $envLogPath=$outputPath + "\envlogs" | |
| write-host "Downloading XMC latest Env log list to "$envLogPath | |
| $envLogs=dotnet sitecore cloud environment log list --latest --environment-id $envId #Get env log list | |
| write-host("Latest logs:" + $envLogs) | |
| foreach($envLog in $envLogs) | |
| { | |
| if ($envLog.Trim() -Match "-") { | |
| $logFileName=$envLog.Trim().Split("-")[1].Trim() | |
| write-host "Downloading log file named: "$logFileName | |
| dotnet sitecore cloud environment log download --environment-id $envId --log $logFileName --path $envLogPath --timeout 600 | |
| } | |
| } | |
| } | |
| function CleanLogsFolder() | |
| { | |
| Get-ChildItem -Path (Join-Path $PSScriptRoot ".\deploylogs") | ForEach-Object { | |
| $dataPath = $_.FullName | |
| write-host($dataPath) | |
| Get-ChildItem -Path $dataPath -Exclude ".gitkeep" -Recurse | Remove-Item -Force -Recurse -Verbose | |
| } | |
| Get-ChildItem -Path (Join-Path $PSScriptRoot ".\envlogs") | ForEach-Object { | |
| $deployPath = $_.FullName | |
| write-host($deployPath) | |
| Get-ChildItem -Path $deployPath -Exclude ".gitkeep" -Recurse | Remove-Item -Force -Recurse -Verbose | |
| } | |
| } | |
| #CleanLogsFolder #Optional call | |
| ConnecttoXMCEnvironment | |
| DownloadDeploymentLogs | |
| DownloadAllEnviromentLogs #first time download | |
| DownloadLatestEnviromentLogs #incremental download | |
| $watch.Stop() # Stopping the timer | |
| Write-Host "Execution time - " $watch.Elapsed # Print script execution time | |
| Write-Host("Done") |
#########################
Running output at this point:
Just for reference, the XMC environment logs cover the events related to following entities:
Although raw information, note that rendering host logs, part of XMC environment logs, look like this:
You could refer this next part wherein I add further steps.
Although my purpose is to integrate Sitecore DXP logs, here are some blog posts generally discussing XMC logs:
https://thesitecorist.net/2022/12/19/sitecore-xm-cloud-logs/
https://www.sergevandenoever.nl/XM_Cloud_build_and_deploy_like_a_pro/
Comments
Post a Comment