The case of the... MapiExceptionShutoffQuotaExceeded
In this category I publish blog post explaining a problem and the solution but more important the analysis that lead to the root cause.
The name is a homage to “The Case of the Unexplained” from Mark Russinovich.
The error
Some Exchange Online users receive the following error message for one or more invited mailboxes when creating new meetings.
Delivery has failed to these recipients or groups:
John Doe (john.doe@bader.cloud)
There’s a problem with the recipient’s mailbox. Please try resending the message. If the problem continues, please contact your email admin.
Diagnostic information for administrators:
Generating server: VI1PR02MBnnnn.eurprd02.prod.outlook.com
john.doe@bader.cloud
Remote Server returned ‘554 5.2.0 STOREDRV.Deliver.Exception:QuotaExceededException.MapiExceptionShutoffQuotaExceeded; Failed to process message due to a permanent exception with message Move/Copy messages failed. […]’
The analysis
The MapiExceptionShutoffQuotaExceeded
cause specified in the error message suggests that the recipient mailbox is full and cannot accept new items. The cause will most likely not be found in the sender mailbox.
Analysis of the affected mailbox
The first consideration therefore goes directly to the mailbox size of the recipient’s mailbox. In this case, however, it is far from your limit of 100 GB.
Get-MailboxStatistics -Identity "$ExchangeGUID" | Select *TotalItemSize*
Hold configuration
In the affected environment, all mailboxes have been assigned a Microsoft retention policy. Depending on the configuration, this can be seen at the mailbox level or at the organization level.
Get-Mailbox -Identity "$ExchangeGUID" | Select LitigationHoldEnabled,InPlaceholds
The prefix mbx
identifies the hold type as Microsoft 365 Retention Policy.
If one or more policies are assigned to the entire Exchange organization, they can be displayed in the Exchange tenant’s organization configuration.
Get-OrganizationConfig | FL InPlaceHolds
Hold types
In this case, the mailbox, as well as the entire organization, is assigned a policy. However, these differ by the number after the colon.
Value | Description |
---|---|
1 | Deletion Policy - Data is deleted after a specified time |
2 | Hold Policy - Data is preserved for a specified time |
3 | Hold und Delete Policy - Data is preserved for a specified time and then deleted |
In this case, the personal mailbox is assigned a “deletion policy”. The entire organization has a deletion and a separate hold policy, thus also the mailbox.
Other affected mailboxes did not have a directly assigned policy, so this is ruled out as a possible cause.
Analysis of the Retention Policy
In the next step, I will therefore take a closer look at the Hold Retention Policy assigned to the organization. In addition to the already existing PSRemoting connection to Exchange Online, an additional connection to the Security & Compliance Center must be established for this purpose. The necessary cmdlets are included in the ExchangeOnlineManagement module.
The policy id must be specified without the prefix.
Connect-IPPSSession
Get-RetentionComplianceRule -Policy d4606a08bd3343d8afced3b2a697ba8c | Select Name, RetentionComplianceAction,RetentionDuration,RetentionDurationDisplayHint
In this case, the retention period is configured for 10 years. However, to prevent the mailbox from filling up, additional MRM policy rules have been created that move the items stored within the mailbox to the online archive after 3 years and move the ‘RecoverableItems’ to the online archive after 30 days.
Get-RetentionPolicyTag | ? RetentionAction -eq "MoveToArchive" | ft -AutoSize
This process is performed automatically in the background by the Managed Folder Assistant.
Check the archive
So far, everything looks fine with the mailbox. The next step is to check the archive. If there are problems here or if it is not enabled, the defined MRM policy rules cannot be executed.
Get-Mailbox -Identity "$ExchangeGUID" | Select ArchiveName,ArchiveStatus,ArchiveState
he online archive is active and so far no error is apparent.
RecoverableItems
Now I check the size of the single folders and in this case with a special attention to the Folder Scope RecoverableItems. Here not the “normal” folders are shown, but everything that has to do with holds, versioning, deleted items, etc..
$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Get-MailboxFolderStatistics -Identity "$ExchangeGUID" -IncludeAnalysis -FolderScope RecoverableItems | Format-Table Name,ItemsInFolder,FolderSize,*Subject*
Problem found!
The folder ‘DiscoveryHolds’ has a size of almost 100 GB and thus brings the mailbox to its limit. This is not apparent to the user, nor does she receive any information about the mailbox being too full.
Since you can also display the TopSubject
directly, you also get the information which element most likely causes the error. In this case it is a calendar entry, recognizable by the column TopSubjectClass
and the type IPM.Appointment
.
The root cause
Unfortunately, I could not find the cause of this behavior in this case. Even Microsoft Support, which was contacted, could not find any errors in the invitation sent.
It was a scheduled daily meeting that was configured as a Microsoft Teams event. Unfortunately, it remains unclear whether regular updates took place and the Managed Folder Assistant was unable to move the item to the online archive as a result.
In any case, the Managed Folder Assistant was not able to move the item from the Recoverable Items area of the mailbox to the online archive.
Normally, the folder should be cleaned up automatically by the MFA when the quota warning is reached. This cleanup process is done on a first-in-first-out basis.
This threshold is 90GB (soft) or 100GB (hard) for Exchange Online with an activated retention hold policy. Without a hold policy or a litigation hold, the value is 20 or 30 GB.
The solution
But the identified issue made it possible to find a solution to the problem.
- The Single Item Recovery feature must be disabled for the affected mailbox (recipient).
- Then delete the affected item completely
- A manual run of the Managed Folder Assistant should be started to clean up the mailbox
- And finally Single Item Recovery must be re-enabled
The Microsoft 365 Retention Policy is active all the time.
Disable Single Item Recovery
This step is necessary to allow the user to remove the faulty object completely.
$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Set-Mailbox -Identity "$ExchangeGUID" -SingleItemRecoveryEnabled $false
Delete the affected item
I did not automate this step via PowerShell because write access to the user’s mailbox was not wanted.
Therefore, the affected user, must delete the corresponding item completely.
- Delete from the calendar
- Delete from the Recycle Bin
- Delete from the Recoverable Items
This process can otherwise also be [automated] using New-ComplianceSearchAction
(https://learn.microsoft.com/en-us/exchange/security-and-compliance/recoverable-items-folder/clean-up-deleted-items?WT.mc_id=AZ-MVP-5004810).
Start Managed Folder Assistant (MFA)
To speed up the cleanup process in Exchange Online afterwards, the Managed Folder Assistant (MFA) can be started manually. It is important to specify the parameter HoldCleanup
.
$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Start-ManagedFolderAssistant -Identity "$ExchangeGUID" -HoldCleanup
Check
Whether the MFA has run can be checked using the Export-MailboxDiagnosticLogs
cmdlet. The returned data is in XML format and must therefore still be formatted accordingly.
$log = Export-MailboxDiagnosticLogs -Identity john.doe@bader.cloud -ExtendedProperties
$xml = [xml]($Log.MailboxLog)
$xml.Properties.MailboxTable.Property | ? {$_.Name -like "ELC*"}
Now the mailbox size can be checked again.
The folder DiscoveryHolds
should be greatly shrunk and the mailbox thus clean again.
$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Get-MailboxFolderStatistics -Identity "$ExchangeGUID" -IncludeAnalysis -FolderScope RecoverableItems | Format-Table Name,ItemsInFolder,FolderSize,*Subject*
Re-enable single item recovery
$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Set-Mailbox -Identity "$ExchangeGUID" -SingleItemRecoveryEnabled $true