Phase out Legacy Authentication - Gain insights

Blog series

This is part three of the six-part series on “Phase out Legacy Authentication”.

1. Preface
2. Enable Modern Authentication
3. Create prerequisites
4. Gain insights
5. The first 90%
6. The next 9%
7. Endgame

Recap

In the first part, Modern Authentication was enabled and in the second part, the prerequisites were created to create detailed reports and to disable Legacy Authentication for individual users.

Conditional Access policy logic

To better understand how it is possible to block individual users, let’s take a look at the Conditional Access Policies created and the login associated with them.

Phase 1 - Implicit Allow

In phase 1, we only want to block users who do not use Legacy Authentication (LA). This way, we prevent users from possibly using an app that does not support Modern Authentication (MA) during our journey.

Example

• Bob and Alice both use Exchange Online.
• Bob uses the native iOS mail app with Exchange ActiveSync Client (LA) and Microsoft Apps for business (MA).
• Alice uses the Outlook mobile app (MA) and Microsoft Apps for business (MA).

Since Alice only uses Modern Authentication, she can be included in the group “CAPolicy-Include-Block-Legacy-Authentication”. If she now tries to access Exchange with a legacy auth client, this attempt will be blocked by the conditional access policy “Temporary Policy: Block legacy authentication Rollout”.

Phase 2 - Explicit Allow

In Phase 2, Legacy Authentication (LA) is disabled for all users and only explicitly defined users can continue to use it.

Example

• Janine and Kevin both use Exchange Online
• Janine wants to set up a mail app with Exchange ActiveSync (LA) on her new Android smartphone.
• Kevin has an existing Android device and uses a mail app with Exchange ActiveSync (LA)

Kevin was added to the “CAPolicy-Exclude-Block-Legacy-Authentication” group and can therefore continue to use his mail app.
Janine is blocked from logging in and has to switch to a mail app that supports Modern Authentication.

Gain insights

Several tools are available to identify users for the first phase. Used together, these tools offer incredible flexibility and detailed insights into the use of Microsoft 365.

Workbooks

The Entra ID (Azure AD) Workbooks provided by Microsoft are a great place to start. Each workbook is designed for a specific analysis, so we will use some of the workbooks over the next while.

The workbooks are based on the Azure Monitor Workbooks service. Using this, data can be prepared in graphical form very easily and it is thus made easier for the user to extract information from the data volumes.

A big advantage of the workbooks is that they can be used interactively and data can be easily reduced using filters and drill-down options.

The source for this data is the Log Analytics Workspace created in the last chapter and the SignIn data collected there.

Navigation

Each workbook offers several filters that can be used to limit the displayed data. Always available is the time period, the other parameters vary depending on the selected workbook.

Sign-ins using Legacy Auth

The workbook “Sign-ins using Legacy Auth” provides insight into the use of legacy authentication and the apps and protocols affected by it.

Conditional Access Insights and Reporting

This workbook can be used to closely monitor the impact of a conditional access policy. For example, the data collected for the conditional access policy “Common Policy: Block legacy authentication” can be used to estimate what impact a shutdown will have on users.

This workbook also provides a more detailed insight into the distribution of the end devices used, the apps, sign-in risk and the location of the users, which is determined on the basis of the IP address.

In this case, filtering on the correct conditional access policy and time period is very important to get meaningful results.

Kusto queries

The workbooks just discussed are a great graphical and interactive tool and are, as mentioned, based on the recorded SignIn data in the Log Analytics Workspace. However, you can also conveniently query the actual raw data yourself using Kusto queries.

The existing workbooks offer a good start. Via the blue Log Analytics icon the underlying query opens and can be further modified.

Identify legacy protocols in use

The legacy protocols used are the starting point for deactivation. Depending on which service still uses legacy authentication, different countermeasures must be taken.

Exchange is one of the bigger troublemakers in this regard, as Mail is a very old service with many old protocols. Therefore, it is not surprising if a large part of the protocols are related to Exchange.

SigninLogs
// set TimeSpan to 7 days
| where TimeGenerated > ago(7d)
// Only query successful logins
| extend errorCode = toint(Status.errorCode)
| where errorCode == 0
// Check if ClientAppUsed is considered Legacy Auth - Empty is not considered Legacy Auth
| extend ClientAppUsed = iff(isempty(ClientAppUsed) == true, "Unknown", ClientAppUsed)  
| extend isLegacyAuth = case(ClientAppUsed contains "Browser", "No", ClientAppUsed contains "Mobile Apps and Desktop clients", "No", ClientAppUsed contains "Exchange ActiveSync", "Yes", ClientAppUsed contains "Unknown", "Unknown", "Yes") 
| where isLegacyAuth == "Yes"
| summarize Count=count()  by ClientAppUsed, AppDisplayName

Explanation

The query uses all data in the SigninLogs table created in the last 7 days as a data basis.

1
2
3

SigninLogs
// set TimeSpan to 7 days
| where TimeGenerated > ago(7d)

In the next step, only the logins that have successfuly logged into the system are analyzed. Whether a botfarm in Buxtehude could not log in successfuly is irrelevant to us in this context.

4
5
6

// Only query successful logins
| extend errorCode = toint(Status.errorCode)
| where errorCode == 0

Now the application that was used for the login is examined. If there is no information about it, the value for ClientAppUsed is set to “Unknown”.

7
8

// Check if ClientAppUsed is considered Legacy Auth - Empty is not considered Legacy Auth
| extend ClientAppUsed = iff(isempty(ClientAppUsed) == true, "Unknown", ClientAppUsed)  

Based on the client app, it is now defined if the signin is a legacy signin. Browsers, Microsoft mobile apps and desktop apps (>= 2013) are considered Modern Auth, unknown apps are marked as ‘Unknown’ and the rest is Legacy Authentication.

9

| extend isLegacyAuth = case(ClientAppUsed contains "Browser", "No", ClientAppUsed contains "Mobile Apps and Desktop clients", "No", ClientAppUsed contains "Exchange ActiveSync", "Yes", ClientAppUsed contains "Unknown", "Unknown", "Yes") 

Only LA SignIns are returned and aggregated based on the client app and the online service used.

10
11

| where isLegacyAuth == "Yes"
| summarize Count=count()  by ClientAppUsed, AppDisplayName

Example

Identify applications used

To better judge which applications are using these outdated protocols, the UserAgent can be evaluated.

SigninLogs
// set TimeSpan to 7 days
| where TimeGenerated > ago(7d)
// Only query successful logins
| extend errorCode = toint(Status.errorCode)
| where errorCode == 0
// Check if ClientAppUsed is considered Legacy Auth - Empty is not considered Legacy Auth
| extend ClientAppUsed = iff(isempty(ClientAppUsed) == true, "Unknown", ClientAppUsed)  
| extend isLegacyAuth = case(ClientAppUsed contains "Browser", "No", ClientAppUsed contains "Mobile Apps and Desktop clients", "No", ClientAppUsed contains "Exchange ActiveSync", "Yes", ClientAppUsed contains "Unknown", "Unknown", "Yes") 
| where isLegacyAuth == "Yes"
| summarize Count=count() by UserAgent, AppDisplayName
| sort by Count

Explanation

The first 10 lines of the query are identical to the first query.

The decisive factor is the grouping based on the UserAgent and the online service used in line 11.. Line 12 sorts the result in descending order by the number of logins.

11
12

| summarize Count=count() by UserAgent, AppDisplayName
| sort by Count

Example

Identify users without legacy authentication signins

So that in the next article Legacy Authentication can really be disabled for (hopefully 😉) 90% of all users, we need the information which users have not logged in using Legacy Authentication in the last 7 days.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

let distinctSigninLogs = SigninLogs
// set TimeSpan to 7 days
| where TimeGenerated > ago(7d)
// Only query successful logins
| extend Status = todynamic(Status)
| extend errorCode = toint(Status.errorCode)
| where errorCode == 0
// Check if ClientAppUsed is considered Legacy Auth - Empty is not considered Legacy Auth
| extend ClientAppUsed = iff(isempty(ClientAppUsed) == true, "Unknown", ClientAppUsed)  
| extend isLegacyAuth = case(ClientAppUsed contains "Browser", "No", ClientAppUsed contains "Mobile Apps and Desktop clients", "No", ClientAppUsed contains "Exchange ActiveSync", "Yes", ClientAppUsed contains "Unknown", "Unknown", "Yes");
let distinctAADNonInteractiveUserSignInLogs = AADNonInteractiveUserSignInLogs
// set TimeSpan to 7 days
| where TimeGenerated > ago(7d)
// Only query successful logins
| extend Status = todynamic(Status)
| extend errorCode = toint(Status.errorCode)
| where errorCode == 0
// Check if ClientAppUsed is considered Legacy Auth - Empty is not considered Legacy Auth
| extend ClientAppUsed = iff(isempty(ClientAppUsed) == true, "Unknown", ClientAppUsed)  
| extend isLegacyAuth = case(ClientAppUsed contains "Browser", "No", ClientAppUsed contains "Mobile Apps and Desktop clients", "No", ClientAppUsed contains "Exchange ActiveSync", "Yes", ClientAppUsed contains "Unknown", "Unknown", "Yes")
| distinct UserPrincipalName, isLegacyAuth;
let distinctAllSignInLogs = union distinctAADNonInteractiveUserSignInLogs, distinctSigninLogs
| distinct UserPrincipalName, isLegacyAuth;
let LegacyAuthSigninLogs = distinctAllSignInLogs
| where isLegacyAuth == "Yes"
| project UserPrincipalName;
distinctAllSignInLogs
| where UserPrincipalName !in (LegacyAuthSigninLogs)

Explanation

The first 10 lines are almost identical to the previous queries, but in line 1 the code is connected with the command let named distinctSigninLogs and can be used more easily in the subsequent code.

This makes the code leaner, since the same expression does not have to occur multiple times written out.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

let distinctSigninLogs = SigninLogs
// set TimeSpan to 7 days
| where TimeGenerated > ago(7d)
// Only query successful logins
| extend errorCode = toint(Status.errorCode)
| where errorCode == 0
// Check if ClientAppUsed is considered Legacy Auth - Empty is not considered Legacy Auth
| extend ClientAppUsed = iff(isempty(ClientAppUsed) == true, "Unknown", ClientAppUsed)  
| extend isLegacyAuth = case(ClientAppUsed contains "Browser", "No", ClientAppUsed contains "Mobile Apps and Desktop clients", "No", ClientAppUsed contains "Exchange ActiveSync", "Yes", ClientAppUsed contains "Unknown", "Unknown", "Yes")
| distinct UserPrincipalName, isLegacyAuth;

In line 10 the distinct operator is used to get each entry only once. So for each UserPrincipalName there is a maximum of three entries.

Lines 11 - 21 queries the same data from schema AADNonInteractiveUserSignInLogs to include non interactive sign-ins, such as service accounts, as well.

After that, in line 22 - 23 the result from distinctSigninLogs and distinctAADNonInteractiveUserSignInLogs is merged together in distinctAllSignInLogs.

22
23

let distinctAllSignInLogs = union distinctAADNonInteractiveUserSignInLogs, distinctSigninLogs
| distinct UserPrincipalName, isLegacyAuth;

In line 24-26 only those usernames are stored as LegacyAuthSigninLogs for which isLegacyAuth equals the value “Yes”.

24
25
26

let LegacyAuthSigninLogs = distinctAllSignInLogs
| where isLegacyAuth == "Yes"
| project UserPrincipalName;

All this was just preparation for the actual query. This query uses all unique SignIns from the “table” distinctSigninLogs and only returns user names that do not appear in the “table” LegacyAuthSigninLogs. Therefore only users who have not used legacy authentication.

27
28

distinctSigninLogs
| where UserPrincipalName !in (LegacyAuthSigninLogs)

What’s next

In the next article we will disable legacy authentication for the users we identified as not critical.