If you are building out a Microsoft Sentinel environment, setting up the correct retention period is extremely important. By default, the retention is configured on the Azure Log Analytics workspace which is applicable to all data that resides in it.

The minimum retention is 30 days, while the maximum is 730 days. Choosing the correct retention period is something every organization has to decide on theirselves before they begin their Microsoft Sentinel journey and really depends on their own business and compliance requirements. While everybody security analyst would want to retain data forever, increasing the retention of a Log Analytic Workspace also increases the cost. If you are using Microsoft Sentinel, retention is free for up to 90 days, beyond that you will pay for every MB of data you retain.

The need for granular retention

While most organizations will be able to work with the global retention setting, some might have the need for granular retention. This could be because there are certain data sources they want to keep for a shorter time (to keep cost down) or for a longer time (for forensics). Let’s dive into some use cases.

Requiring a shorter retention period might sound odd at first, but I have seen a lot of customers run into it. Most organizations have setup their global retention to something in the range of 180-365 days. But some data sources that are being pulled in are in an extremely large volume and don’t provide a lot of value when retained for longer periods of times. Let’s take firewall logs as an example. Ingesting them into Microsoft Sentinel ensures you can match them with your custom Threat Intelligence indicators and are able to correlate the data to other sources during investigations. But firewall logs are usually pretty noisy and can be extremely costly to ingest and retain. In some scenarios, customers want to decrease the retention of just the firewall logs. This ensures all data is retained for 365 days, but firewall logs are removed after 90 days.

Another reason is to increase the retention for a particular table, a great example of this are audit logs. It happens multiple times a year where a customer asks me to check which administrator updated a particular setting a couple of months ago. This is a great reason to retain audit logs for a longer time as it improves the visibility and increases traceability.

Introducing table level retention

In order to solve these issues, Log Analytics supports table level retention. This allows you to setup a different retention period for specific tables. It’s important to note that this is applicable to all of the data within that table. Tables like ‘Syslog’ often contain data from multiple products. By implementing table level retention, you update the retention for all data within a table.

Table retention isn’t surfaced in the UI (which is unfortunate) and can only be checked or configured through the Azure Resource Manager which is configurable through ARM templates or the API.

Retrieving the current retention preriod

By using the following Powershell code, you are able to retrieve the current retention period of a table. If no table level retention is configured, the workspace level retention will be returned.

$bearerToken = "<Insert auth token>"
$SubscriptionID = "<Sub ID of Microsoft Sentinel>"
$ResourceGroup = "<>"
$WorkspaceName = "<>"
$tableName = "SigninLogs"


$data = Invoke-WebRequest -Uri "https://management.azure.com/subscriptions/$SubscriptionID/resourceGroups/$ResourceGroup/providers/Microsoft.OperationalInsights/workspaces/$WorkspaceName/Tables/$tableName`?api-version=2017-04-26-preview" `
-Method "GET" `
-Headers @{
  "Authorization"="Bearer $bearerToken"
} -ContentType "application/json"

$data.Content

Configuring retention

Configuring table retention can be done by using the API or an ARM template.

Using the Azure Management API

In order to update retention through the API, the following script can be used:

$bearerToken = "<Insert auth token>"
$SubscriptionID = "<Sub ID of Microsoft Sentinel>"
$ResourceGroup = "<>"
$WorkspaceName = "<>"
$tableName = "SigninLogs"

$body = @{
    "properties"=
        @{
            "retentionInDays"=$RetentionPeriod
        }
}

$JSON = ConvertTo-Json $body



$data = Invoke-WebRequest -Uri "https://management.azure.com/subscriptions/$SubscriptionID/resourceGroups/$ResourceGroup/providers/Microsoft.OperationalInsights/workspaces/$WorkspaceName/Tables/$tableName`?api-version=2017-04-26-preview" `
-Method "PUT" `
-Headers @{
  "Authorization"="Bearer $bearerToken"
} -ContentType "application/json" -Body $JSON

$data.Content

Using an ARM template

While using the Azure Management API is the easiest way in my opinion, updating table level retention is also possible by deploying an ARM template. This can be a great choise when you are dealing with a CI/CD environment.

An example of such an ARM template can be found on GitHub.

Closing off

Using tabel level retention is a great way to increase or decrease the retention of a particular log source, without the need to updating the general retention. This means it’s a great way to keep the cost of your Microsoft Sentinel environment down.

Interested to learn more about Microsoft Sentinel or Microsoft 365 Security in general? This and more is covered in our book which is available for purchase here.