CA-03: Creating a Sign-In Risk-Based Policy

Overview

This guide walks you through creating a Conditional Access policy that responds to sign-in risk levels detected by Microsoft Entra ID Protection. This policy automatically escalates protection when Microsoft detects anomalous sign-in behavior such as impossible travel, anonymous IP addresses, or known malicious infrastructure.

Control ID: CA-03 Category: Conditional Access Severity: High License Required: Microsoft Entra ID P2 (included in Microsoft 365 E5 or standalone Entra ID P2)

Why This Matters

Microsoft analyzes billions of sign-ins daily and uses machine learning to detect suspicious authentication patterns. Sign-in risk detection identifies threats like:

• Impossible travel - Sign-in from two distant locations within impossible timeframes
• Anonymous IP addresses - Sign-ins from Tor exit nodes or anonymous VPNs
• Malware-linked IP addresses - Known botnet command and control infrastructure
• Unfamiliar sign-in properties - Unusual device, browser, or location combinations
• Password spray attacks - Multiple failed attempts across accounts

Risk-based policies automatically respond to these threats without affecting legitimate users during normal access.

Prerequisites

Required Roles

You need one of the following Entra ID roles:

• Conditional Access Administrator (recommended - least privilege)
• Security Administrator
• Global Administrator

Required Licenses

• Microsoft Entra ID P2 for each user in scope
• Included in: Microsoft 365 E5 or standalone Entra ID P2

Pre-Configuration Requirements

Before creating this policy:

1. Verify P2 licensing - Sign-in risk detection requires Entra ID P2
2. Emergency access accounts exist - Break-glass accounts must be excluded
3. Users have MFA registered - High-risk sign-ins will require MFA

Time Estimate

Step-by-Step Instructions

Step 1: Navigate to Conditional Access

1. Sign in to the Microsoft Entra admin center
2. In the left navigation, expand Protection
3. Select Conditional Access
4. Click Policies in the submenu

Step 2: Create a New Policy

1. Click + New policy at the top of the policies list
2. Enter a descriptive name: Block or MFA for Risky Sign-Ins

Step 3: Configure Users and Groups

1. Under Assignments, click Users
2. Select All users under Include
3. Under Exclude, click Users and groups
4. Add your emergency access accounts
5. Optionally add a pilot group exclusion for initial testing

Step 4: Configure Target Resources

1. Under Target resources, click Cloud apps
2. Select All cloud apps under Include

Step 5: Configure Sign-In Risk Condition

This is the key configuration for this policy:

1. Under Conditions, click Sign-in risk
2. Set Configure to Yes
3. Select the risk levels to target:

Recommended Configuration:

• High - Block access
• Medium - Require MFA

Note: You will need to create two policies (one for High blocking, one for Medium MFA) OR use a single policy with MFA for both and rely on Identity Protection defaults for blocking.

Step 6: Configure Access Controls (Option A - Single Policy)

For a simpler approach, require MFA for Medium and High risk:

1. Under Access controls, click Grant
2. Select Grant access
3. Check Require multifactor authentication
4. Click Select

Note: This approach requires MFA for risky sign-ins but does not block them. For blocking high-risk sign-ins, see Option B.

Step 6: Configure Access Controls (Option B - Block High Risk)

To block high-risk sign-ins outright, create two policies:

Policy 1: Block High-Risk Sign-Ins

1. Condition: Sign-in risk = High
2. Grant: Block access

Policy 2: Require MFA for Medium-Risk Sign-Ins

1. Condition: Sign-in risk = Medium
2. Grant: Require multifactor authentication

Step 7: Enable the Policy

Recommended approach - Start with Report-only:

1. Under Enable policy, select Report-only
2. Click Create
3. Monitor the Risky sign-ins report for 3-7 days
4. Review false positive rates and impact
5. Once validated, edit the policy and change to On

Understanding Sign-In Risk Levels

High Risk

Indicates a high probability the sign-in was not performed by the legitimate user.

Detection Examples:

• Sign-in from a known malicious IP address
• Impossible travel between distant locations
• Activity from Tor exit nodes with other suspicious indicators

Recommended Action: Block access or require password change + MFA

Medium Risk

Indicates a moderate probability of suspicious activity.

Detection Examples:

• Unfamiliar sign-in properties (new device/browser/location combination)
• Anonymous IP address usage
• Atypical travel patterns

Recommended Action: Require MFA to verify identity

Low Risk

Indicates minor anomalies that may warrant attention.

Detection Examples:

• Slightly unusual patterns
• Minor deviations from normal behavior

Recommended Action: Log and monitor (or require MFA in high-security environments)

Verification Checklist

After enabling the policy, verify successful implementation:

Immediate Checks

• Policy appears in the Conditional Access policies list with status "On" or "Report-only"
• Sign-in risk condition is properly configured (Medium, High selected)
• Emergency access accounts are listed in the exclusions
• Appropriate grant controls are configured

Identity Protection Dashboard

1. Navigate to Entra admin center > Protection > Identity Protection
2. Click Risky sign-ins to view detections
3. Verify that risky sign-ins show your policy as applied

Sign-in Log Validation

1. Navigate to Entra admin center > Monitoring > Sign-in logs
2. Click on any sign-in and check the Conditional Access tab
3. For risky sign-ins, verify your policy appears

Testing (Caution - Do Not Create Real Risks)

Note: Do not attempt to trigger sign-in risk artificially. Instead:

1. Review the "Risky sign-ins" report for recent detections
2. Validate that past risky sign-ins would have triggered the policy
3. Use the "What If" tool with a simulated high-risk scenario

Troubleshooting

No Risky Sign-Ins Detected

Symptom: The Risky sign-ins report is empty.

Solutions:

1. Verify Entra ID P2 licenses are assigned to users
2. Risk detection can take 5-10 minutes to process
3. Check the Risk detections report for raw detections
4. Ensure Conditional Access is not blocking sign-ins before risk can be evaluated

False Positives - Legitimate Users Blocked

Symptom: Legitimate users are being flagged as high risk.

Solutions:

1. Review the specific risk detections in Protection > Identity Protection > Risk detections
2. Check if users are using VPNs or privacy tools that trigger detections
3. Consider adding trusted locations to reduce false positives
4. Document the dismiss reason for each false positive

Users Cannot Self-Remediate

Symptom: Users cannot complete MFA or password reset when prompted.

Solutions:

1. Verify users have MFA methods registered
2. Ensure SSPR (Self-Service Password Reset) is enabled
3. Check if the user is blocked or disabled
4. An admin may need to manually dismiss the risk or reset credentials

Policy Not Applied to Risky Sign-Ins

Symptom: High-risk sign-ins are succeeding without MFA.

Solutions:

1. Verify the policy is set to "On" (not "Report-only")
2. Check if the user is excluded from the policy
3. Ensure the Sign-in risk condition is properly configured
4. Review policy evaluation order with the "What If" tool

Policy Configuration Summary

Single Policy Approach

Two-Policy Approach (Recommended)

Policy 1: Block High-Risk Sign-Ins

Policy 2: Require MFA for Medium-Risk Sign-Ins

Monitoring and Maintenance

Regular Reviews

1. Weekly: Review the Risky sign-ins report for patterns
2. Monthly: Analyze false positive rates and adjust if needed
3. Quarterly: Review sign-in risk policies with security team

Risk Dismissal

When investigating false positives:

1. Navigate to Protection > Identity Protection > Risky sign-ins
2. Select the sign-in and click Confirm safe or Dismiss risk
3. Document the reason for dismissal

Related Controls

• CA-04: Remediate High-Risk Users Automatically (user risk policy)
• CA-01: Require MFA for All Users (baseline policy)
• LOG-03: Stream Security Events to SIEM (for correlation)
• GOV-04: Automate Threat Response with SOAR (advanced automation)

Additional Resources

• Microsoft: Conditional Access - Sign-in risk
• Microsoft: Identity Protection overview
• Microsoft: Sign-in risk detections