Contents

The case of the... MapiExceptionShutoffQuotaExceeded

The case of the...

In this category I publish blog post explaining a problem and the solution but more important the analysis that lead to the root cause.

The name is a homage to “The Case of the Unexplained” from Mark Russinovich.

The error

Some Exchange Online users receive the following error message for one or more invited mailboxes when creating new meetings.

Warning

Delivery has failed to these recipients or groups:

John Doe (john.doe@bader.cloud)
There’s a problem with the recipient’s mailbox. Please try resending the message. If the problem continues, please contact your email admin.

Diagnostic information for administrators:

Generating server: VI1PR02MBnnnn.eurprd02.prod.outlook.com

john.doe@bader.cloud
Remote Server returned ‘554 5.2.0 STOREDRV.Deliver.Exception:QuotaExceededException.MapiExceptionShutoffQuotaExceeded; Failed to process message due to a permanent exception with message Move/Copy messages failed. […]’

The analysis

The MapiExceptionShutoffQuotaExceeded cause specified in the error message suggests that the recipient mailbox is full and cannot accept new items. The cause will most likely not be found in the sender mailbox.

Analysis of the affected mailbox

The first consideration therefore goes directly to the mailbox size of the recipient’s mailbox. In this case, however, it is far from your limit of 100 GB.

Get-MailboxStatistics -Identity "$ExchangeGUID" | Select *TotalItemSize*

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/TotalItemSize.png

Hold configuration

In the affected environment, all mailboxes have been assigned a Microsoft retention policy. Depending on the configuration, this can be seen at the mailbox level or at the organization level.

Get-Mailbox -Identity "$ExchangeGUID" | Select LitigationHoldEnabled,InPlaceholds

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/Holds.png

The prefix mbx identifies the hold type as Microsoft 365 Retention Policy.

If one or more policies are assigned to the entire Exchange organization, they can be displayed in the Exchange tenant’s organization configuration.

Get-OrganizationConfig | FL InPlaceHolds

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/OrganizationHolds.png

Hold types

In this case, the mailbox, as well as the entire organization, is assigned a policy. However, these differ by the number after the colon.

Value Description
1 Deletion Policy - Data is deleted after a specified time
2 Hold Policy - Data is preserved for a specified time
3 Hold und Delete Policy - Data is preserved for a specified time and then deleted

In this case, the personal mailbox is assigned a “deletion policy”. The entire organization has a deletion and a separate hold policy, thus also the mailbox.

Other affected mailboxes did not have a directly assigned policy, so this is ruled out as a possible cause.

Analysis of the Retention Policy

In the next step, I will therefore take a closer look at the Hold Retention Policy assigned to the organization. In addition to the already existing PSRemoting connection to Exchange Online, an additional connection to the Security & Compliance Center must be established for this purpose. The necessary cmdlets are included in the ExchangeOnlineManagement module.

The policy id must be specified without the prefix.

Connect-IPPSSession
Get-RetentionComplianceRule -Policy d4606a08bd3343d8afced3b2a697ba8c | Select Name, RetentionComplianceAction,RetentionDuration,RetentionDurationDisplayHint

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/RetentionPolicy.png

In this case, the retention period is configured for 10 years. However, to prevent the mailbox from filling up, additional MRM policy rules have been created that move the items stored within the mailbox to the online archive after 3 years and move the ‘RecoverableItems’ to the online archive after 30 days.

Get-RetentionPolicyTag | ? RetentionAction -eq "MoveToArchive" | ft -AutoSize

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/RetentionPolicyTag.png

This process is performed automatically in the background by the Managed Folder Assistant.

Check the archive

So far, everything looks fine with the mailbox. The next step is to check the archive. If there are problems here or if it is not enabled, the defined MRM policy rules cannot be executed.

Get-Mailbox -Identity "$ExchangeGUID" | Select ArchiveName,ArchiveStatus,ArchiveState

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/Archive.png

he online archive is active and so far no error is apparent.

RecoverableItems

Now I check the size of the single folders and in this case with a special attention to the Folder Scope RecoverableItems. Here not the “normal” folders are shown, but everything that has to do with holds, versioning, deleted items, etc..

$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Get-MailboxFolderStatistics -Identity "$ExchangeGUID" -IncludeAnalysis -FolderScope RecoverableItems | Format-Table Name,ItemsInFolder,FolderSize,*Subject*

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/SizeBefore.png

Problem found!

The folder ‘DiscoveryHolds’ has a size of almost 100 GB and thus brings the mailbox to its limit. This is not apparent to the user, nor does she receive any information about the mailbox being too full.

Since you can also display the TopSubject directly, you also get the information which element most likely causes the error. In this case it is a calendar entry, recognizable by the column TopSubjectClass and the type IPM.Appointment.

The root cause

Unfortunately, I could not find the cause of this behavior in this case. Even Microsoft Support, which was contacted, could not find any errors in the invitation sent.

It was a scheduled daily meeting that was configured as a Microsoft Teams event. Unfortunately, it remains unclear whether regular updates took place and the Managed Folder Assistant was unable to move the item to the online archive as a result.

In any case, the Managed Folder Assistant was not able to move the item from the Recoverable Items area of the mailbox to the online archive.
Normally, the folder should be cleaned up automatically by the MFA when the quota warning is reached. This cleanup process is done on a first-in-first-out basis.

This threshold is 90GB (soft) or 100GB (hard) for Exchange Online with an activated retention hold policy. Without a hold policy or a litigation hold, the value is 20 or 30 GB.

The solution

But the identified issue made it possible to find a solution to the problem.

  • The Single Item Recovery feature must be disabled for the affected mailbox (recipient).
  • Then delete the affected item completely
  • A manual run of the Managed Folder Assistant should be started to clean up the mailbox
  • And finally Single Item Recovery must be re-enabled
Info
The mailbox hold status is not changed with this solution.
The Microsoft 365 Retention Policy is active all the time.

Disable Single Item Recovery

This step is necessary to allow the user to remove the faulty object completely.

$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Set-Mailbox -Identity "$ExchangeGUID" -SingleItemRecoveryEnabled $false

Delete the affected item

I did not automate this step via PowerShell because write access to the user’s mailbox was not wanted.

Therefore, the affected user, must delete the corresponding item completely.

  • Delete from the calendar
  • Delete from the Recycle Bin
  • Delete from the Recoverable Items

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/DeleteRecoverableItems-en.png

This process can otherwise also be [automated] using New-ComplianceSearchAction(https://learn.microsoft.com/en-us/exchange/security-and-compliance/recoverable-items-folder/clean-up-deleted-items?WT.mc_id=AZ-MVP-5004810).

Start Managed Folder Assistant (MFA)

To speed up the cleanup process in Exchange Online afterwards, the Managed Folder Assistant (MFA) can be started manually. It is important to specify the parameter HoldCleanup.

$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Start-ManagedFolderAssistant -Identity "$ExchangeGUID" -HoldCleanup

Check

Whether the MFA has run can be checked using the Export-MailboxDiagnosticLogs cmdlet. The returned data is in XML format and must therefore still be formatted accordingly.

$log = Export-MailboxDiagnosticLogs -Identity john.doe@bader.cloud -ExtendedProperties
$xml = [xml]($Log.MailboxLog)
$xml.Properties.MailboxTable.Property | ? {$_.Name -like "ELC*"}

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/MailboxDiagnosticLogs.png

Now the mailbox size can be checked again.
The folder DiscoveryHolds should be greatly shrunk and the mailbox thus clean again.

$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Get-MailboxFolderStatistics -Identity "$ExchangeGUID" -IncludeAnalysis -FolderScope RecoverableItems | Format-Table Name,ItemsInFolder,FolderSize,*Subject*

/en/the-case-of-mapiexceptionshutoffquotaexceeded/images/SizeAfter.png

Wichtig
Finally, do not forget to enable the Single Item Recovery feature again.

Re-enable single item recovery

$ExchangeGUID = Get-Mailbox john.doe@bader.cloud | Select-Object -ExpandProperty ExchangeGUID
Set-Mailbox -Identity "$ExchangeGUID" -SingleItemRecoveryEnabled $true