# Advanced topic: reports processing reference

This page contains a technical reference for how DMARCwise processes DMARC reports, especially when it comes to normalization and correction of invalid reports. It may in some cases help explain discrepancies you may see in the dashboard compared to the original reports.

Once received as an attachment to an email message, a DMARC report is parsed and passed through a validator.

The validator is typically **lax** and forgives many issues that would be unacceptable according to the DMARC specification. Some fields in the XML reports are however **validated more strictly**, otherwise the report would be very hard to use in practice. This includes for example the policy fields, which must contain a valid policy name.

Although blatantly invalid reports are rare, when one such report is received it is not imported and is retained for manual review.

## Corrections

To prevent reports from being discarded for minor reasons, we apply **corrections**, which fall into three categories:
- **Normalization**: a field is cleaned up with the aim of improving compatibility or making the data more usable. Examples include lowercasing values, removing invalid punctuation, accepting typos. These are high-confidence harmless corrections.
- **Recovered**: a field contains an **invalid value** that violates the DMARC specification, but the intent is clear enough that we can replace it with a valid value. Our judgement is that replacing or inferring the intended value is more useful than discarding the report.
- **Discarded**: an individual unusable or invalid item is removed so the rest of the report can still be processed. An example is discarding a DKIM auth result which contains no data. These items are usually the result of bugs in the software that generated the report.

These corrections are based on bad reports we've seen in the past and are updated when new invalid reports come through.

When you [browse a report](/docs/dmarc/reports) in DMARCwise, you already see the normalized/corrected data. When a correction was applied, a badge is shown and clicking on it lets you see how the data was modified.

![DMARC report corrections badge in report page]($docs/images/dmarc-report-corrections.png)

You can also always see the original unprocessed report in **XML format**.

Our implementation of the parsing, correction and validation of DMARC reports is thoroughly tested with automated tests.

## List of corrections

The tables below list the **corrections we currently apply** to new reports we receive, grouped by category. Each row references the original DMARC XML element (using the path from the report's `<feedback>` root) and explains what happens to it. Note that these rules change over time, so you may see different corrections in older reports.

### Normalization

| XML element | Correction |
| --- | --- |
| `report_metadata/data_range` | The misspelled `data_range` element is renamed to the correct `date_range`. |
| `policy_published/p` | Surrounding semicolons are trimmed, e.g. `reject;` becomes `reject`. |
| `policy_published/p` | An empty value is treated as missing (as if the `p` tag was absent). |
| `policy_published/sp` | Surrounding semicolons are trimmed. |
| `policy_published/sp` | An empty value is treated as missing. |
| `policy_published/pct` | A trailing percent sign is trimmed, e.g. `100%` becomes `100`. |
| `policy_published/pct` | An empty value is treated as missing. |
| `policy_published/aspf` | An empty value is treated as missing. |
| `policy_published/adkim` | An empty value is treated as missing. |
| `record/identities` | The misspelled `identities` element is renamed to the correct `identifiers`. |
| `record/row/source_ip` | The source IP address is canonicalized to a single stable form (e.g. compressed IPv6, lowercased hex, leading zeros removed), so that the same address always appears identically. |
| `record/auth_results/spf/result` | Lowercased. |
| `record/auth_results/dkim/domain` | Lowercased. |
| `record/auth_results/dkim/selector` | Lowercased. |
| `record/auth_results/dkim/result` | Lowercased. |

### Recovered

| XML element | Correction |
| --- | --- |
| `policy_published/p` | The invalid value `unknown` is treated as missing (as if the `p` tag was absent). |
| `policy_published/sp` | The invalid value `unknown` is treated as missing. |
| `policy_published/aspf` | The invalid value `unknown` is treated as missing. |
| `policy_published/adkim` | The invalid value `unknown` is treated as missing. |
| `record/identifiers/header_from` | When empty or missing, it is filled with the value of `envelope_from`. |
| `record/row/policy_evaluated/dkim` | A non-pass DKIM result (`none`, `neutral`, `temperror`, `permerror`) is replaced with `fail`. The alignment result can only be `pass` or `fail`, and alignment cannot exist without a DKIM `pass`, so any recognized non-pass result is treated as an alignment failure. |
| `record/row/policy_evaluated/spf` | A non-pass SPF result (`none`, `neutral`, `softfail`, `temperror`, `permerror`) is replaced with `fail`. The alignment result can only be `pass` or `fail`, and alignment cannot exist without an SPF `pass`, so any recognized non-pass result is treated as an alignment failure. |
| `record/auth_results/spf/result` | A non-standard SPF result is mapped to its likely intended [RFC 8601](https://datatracker.ietf.org/doc/html/rfc8601) value: `hardfail` becomes `fail`, `unknown` becomes `temperror`, and `error` becomes `permerror`. |
| `record/auth_results/dkim/result` | A non-standard DKIM result is mapped to its likely intended [RFC 8601](https://datatracker.ietf.org/doc/html/rfc8601) value: `unknown` and `tmpfail` become `temperror`, and `error` becomes `permerror`. |

### Discarded

| XML element | Correction |
| --- | --- |
| `record` | When the report contains a single record row whose `count` is `0`, the record is discarded but the report is kept. These records are typically completely empty and would fail validation. |
| `record/auth_results/spf` | Discarded when its `result` is empty. |
| `record/auth_results/spf` | Discarded when its `domain` is empty. |
| `record/auth_results/dkim` | Discarded when its `result` is `none`: a result of `none` [means](https://datatracker.ietf.org/doc/html/rfc8601#section-2.7.1) the message was not signed, so it makes little sense to present it as a DKIM signature. |
| `record/auth_results/dkim` | Discarded when its `result` is empty. |
| `record/auth_results/dkim` | Discarded when its `domain` is empty. |