Tactical Solution for integrating logs from XMCloud and other Sitecore SaaS offerings with Splunk - running note

Currently, Sitecore DXP product SaaS suite doesn't support Common Audit Logging (CAL) as an e2e solution. So, as part of integrating the logs with an SIEM service like Splunk, you need to custom-stitch the integration by pulling all the logs across the suite in order to send those for observability to Splunk. So, this post will be a running note where I will add information and analysis I make as part of the integration process. Note that since this is a WiP, the code and information will be tweaked for better and optimized platform integration.

Before proceeding further, here is some information from Sitecore:

========================================

These are different SaaS products available in your Sitecore Cloud: 

1. Sitecore Cloud Portal
2. Sitecore XM Cloud
3. Sitecore Search
4. Sitecore Personalize
5. Sitecore Connect
6. Others

It's important to know that the Sitecore Cloud Portal provides the single entry point for other Cloud products. 

Different products have different requirements for Audit Logs. For example:

1. Identity logs (user logins, user management, etc) are tracked by Sitecore Cloud Portal.
2. Application specific logs are tracked by specific applications. 

Common Audit Logs

• The Sitecore Common Audit Logs is Sitecore' framework for integrating the product audit logs with external logging system (such as Splunk).
• To use the CAL today, you can create webhook (using the REST API). When an event is triggered from one of Sitecore' supported applications, it will trigger a notification with the event data to the webhook endpoint. 
• The following products are supported today:
• Sitecore Personalize
• Sitecore CDP
• Sitecore Connect
• The following products are not supported but they will be delivered in the future (No ETA):
• Sitecore Cloud Portal
• Sitecore XM Cloud
• Sitecore Search
• Others

Because of this, you can stream application specific events (excluding Identity logs) from Sitecore Personalize to Splunk using this webhook method. However, you may face issues if your Splunk endpoint is configured to accept incoming requests from whitelisted IPs but CAL webhook doesn't provide static IPs for whitelisting. To address this issue, you can create a webhook using a custom APIM or Sitecore Connect to expose a static IP while forwarding the message from CAL to Splunk.

========================================

Step-1 - PS Script that downloads XMCloud related logs in one-place:

Since XMC is the center piece, consolidating logs from XMC environment/deploy logs is one of the important aspects so, sharing here some of the important tips/notes. 

Tip-1: Use organization Client ID and Client Secret for non-interactive XMCloud login:

You might want to use the download script as part of a scheduler so, generate organization (not environment) Client Id and secret in deploy.sitecorecloud.io:

Use the credentials in connect function for non-intrusive login:

Tip-2: Set longer timeout for log download:

One of the issues i faced with the above script was the following abort state during download since the environment log files were in mega bytes. I raised a Sitecore support ticket but i myself was soon to realize that there is a way to increase the timeout default from 100s to any number you desire using the timeout arg. When I did the same, to 600s, the log file download(s) was successful:

The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.

Execution was cancelled. This occurs when the task is killed or requests timeout.

Tip-3: Create a clean PS Script: 

A clean script to delete the logs and restart is always handy while doing iterative development in the local environment.

##############

##############

Tip-4: Build separate functions for first time env logs download and incremental download:

#########first time download

#########first time download

#########incremental download

#########incremental download

Consolidated Script at this point:

#########################

#########################

Running output at this point:

Just for reference, the XMC environment logs cover the events related to following entities:

Although raw information, note that rendering host logs, part of XMC environment logs, look like this:

Although my purpose is to integrate Sitecore DXP logs, here are some blog posts generally discussing XMC logs:

https://github.com/svdoever/svdoever.github.io/blob/2afce8fa91dba2a08d1940c23910abd73ac97f7e/src/pages/XM_Cloud_build_and_deploy_like_a_pro.md?plain=1#L2

https://thesitecorist.net/2022/12/19/sitecore-xm-cloud-logs/

https://www.sergevandenoever.nl/XM_Cloud_build_and_deploy_like_a_pro/

To be continued....