Part 3 - Azure Logic App that downloads XMC environment logs and sends to Splunk for ingestion

In my earlier blog post, I setup a windows task scheduler that runs in an interval to pull XMC logs then sends the logs to Splunk endpoint for ingestion. In this blog post, i have utilised the same PS scripts in an Azure logic app that does the same job. The PowerShell scripts definitely need some clean-up but this is more of a PoC to showcase that Azure logic app is indeed useful for this scenario of Sitecore XMC to Splunk ingestion of log files. I also realised that Azure logic apps are limitless. Moreover I love the affordable pay-as-you-go model that Azure offers - useful for people like me, who don't spend money on other gratifications but instead use it to experiment something without depending on the company/client. I know what you are thinking, but this is poor me, no bank balance, no dog, no life :D

1. Create Azure logic app from Azure marketplace.

2. Select Workflow Service Plan.

3. Give a suitable name for the logic app and select a region that is relevant to you then click Review+create button.

4. Then click Create button if validations are successful.

5. Once resource is created, go to resource and click workflow designer to create the workflow.

6. In the workflows pane, add a Stateful workflow and click Create button.

7. Click the refresh button to see the created workflow reflect in the bottom pane.

8. Click on the workflow name in the pane to subsequently open the much-needed designer.

9. Since a trigger is important to fire the logic app workflow, select the recurrence trigger among so many present in the list.

10.  Set a suitable duration for the trigger to fire.

11. Next add an action that should recur and pick Execute Powershell Code under Inline Code category.

12. Once the inline Powershell code action is added to the designer, you can see the PowerShell code window wherein you can write your code.

13. Since write-host function is every PS developer's friend, uncomment the write-host code line to see if it first of all gets logged to Application insights but yes, after saving the workflow

14. Duplicate the existing tab to open a separate tab for Application Insights. Go to the logic app home and then traverse to Application Insights menu in left side.

15. Now, you could use the run button (from the Overview left menu) in the adjacent tab to execute the logic app workflow to see the write-host message appear in the duplicated tab' log stream.

16. In a separate window/tab, Go to Kudu via Advanced tools.

17. Kudu > Debug console > CMD.

18. Now, within C:\home\site\wwwroot\ you can view the workflow artifacts including the inline PS script file.

19. Back in the designer, since logging is visible via log stream, you can now confidently modify the PowerShell inline code - connect to Sitecore XMC as first step, then pull XMC environment logs  before sending to Spunk as per earlier blog post. 

One of the vital changes I made in the download script was to register the nuget source for Sitecore via PS code since I did the same manually via Visual Studio nuget package manager in my local environment. I also ensured that the envlogs directory is created only when if it doesn't exist since locally, i created the envlogs and deploylogs directories manually. 

Register-PackageSource -Name Sitecore -Location https://nuget.sitecore.com/resources/v3/index.json -ProviderName NuGet #equivalent for Visual Studio Package Sources nuget registration

20. You can check that PS script logging and Application Insights integrate quite well via Log stream.

21 Finally, add another Inline Code action to upload the log files to Splunk endpoint.

One of the things to bear in mind is, the prerequisite in case of Download logs (second) step in the above logic app flow -  the logs necessary to continue with the delta download must be present and available in the Azure kudu file system before applying the delta logs on top of it. So, one way to accomplish is create another one-time logic app that -

1. Either will download all log files by sending a $true value to the $firstLoad param in Download-XMCLogs.ps1 script. Then, schedule the actual Download delta logic app (above) that downloads just the needful with $false value for the same PS script param. 

2. Or,  from the Sitecore XMC deploy environment, find the latest log file names and just download those again with a separate one-time logic app (there is a param to just download necessary log files). You could also manually upload just the necessary files to kudu file system. Then, allow the existing logic app to run at scheduled intervals to download the delta files thereafter.

Remember to whitelist the logic app ip address in Splunk cloud config to receive the logs.

Env Logs via kudu:

The Sitecore CLI artifacts in place in Azure file system: