An AI meeting summary can sound fluent while omitting a decision, condition, objection, risk or action. It may also add an unsupported explanation or owner. A useful audit therefore compares the summary with a controlled source and measures both missing required information and material additions.
Define the summary’s purpose
State whether the output is a quick internal brief, formal meeting note, action record, customer update or decision record. The required content changes with the purpose.
Set the audit rules before reading the generated summary. Decide which source items are mandatory, what counts as a material assertion and which omissions automatically fail the review.
Secure and verify the source
Use the original audio, a checked transcript and any approved agenda, slides, chat messages or decision log. Correct names, dates, numbers, amounts, technical terms and negations before treating the transcript as a reliable source.
Mark inaudible or uncertain passages. An auditor should not assume that an unclear source supports a definite summary statement.
Create a required-information register
Before checking the summary, list the source items that should appear:
- confirmed decisions;
- actions, owners and deadlines;
- conditions and dependencies;
- risks, objections and dissent;
- open questions;
- evidence limitations;
- material figures, dates and reference numbers;
- follow-up or escalation routes.
Assign every required item a stable identifier and severity. This prevents the reviewer from deciding after the fact that an omitted item was unimportant.
Build a source-to-summary matrix
For every required source item, record:
- source ID and timestamp or document reference;
- source wording or controlled paraphrase;
- required summary category;
- whether it is present, partly present, missing or incorrect;
- summary passage reference;
- severity and reviewer note;
- correction required.
The matrix creates traceability and makes the audit repeatable.
Calculate omission rate
Omission rate = missing required source items ÷ total required source items.
Report the raw numbers with the rate. For example, if 3 of 40 required items are missing, the omission rate is 7.5%.
Also report partial and incorrect items separately. Do not count a decision as present when the summary removes the condition that limits it.
Measure category-specific omissions
Calculate separate rates or raw counts for decisions, actions, risks, conditions, dissent and evidence limitations. An overall rate can hide a serious pattern, such as consistently dropping objections while retaining routine discussion.
High-risk categories may have a zero-tolerance rule even when the overall summary passes.
Audit decisions carefully
Check that every confirmed decision is included and that proposals are not presented as approved outcomes. Preserve:
- decision owner or approving body;
- date and status;
- conditions and dependencies;
- scope and exclusions;
- remaining approvals or evidence.
A summary that states “approved” instead of “recommended subject to budget approval” has not merely shortened the wording; it has changed the decision.
Audit actions and ownership
For each action, check task, owner, deadline, dependency and status. Do not invent an owner or date where none was agreed. Mark unresolved ownership clearly and assign a follow-up action if needed.
Separate external commitments, internal tasks and suggestions. A speaker volunteering to explore an option is not necessarily accepting delivery responsibility.
Audit risk, conditions and dissent
Fluent summaries often compress qualifications. Check whether they retain safety concerns, legal or budget conditions, minority views, objections and unresolved evidence gaps.
Do not treat silence as agreement. Where the source records conflicting views, the summary should preserve the disagreement or explain the decision process used to resolve it.
Audit evidence boundaries
Check that reported accounts remain distinct from verified facts and that interpretations remain distinct from observations. A summary should not imply certainty beyond the source.
Preserve labels such as provisional, estimated, reported, disputed, unverified and outside scope where they affect how the reader should use the information.
Calculate unsupported-addition rate
List each material assertion in the summary and trace it to the source.
Unsupported-addition rate = unsupported material assertions ÷ total material assertions in the summary.
Report the raw numerator and denominator. Unsupported additions may include invented decisions, actions, owners, reasons, evidence or conclusions.
Define the assertion unit before auditing. A stable unit can be a sentence-level claim, action row, decision row or other material proposition.
Measure incorrect transformation
Some failures are neither omissions nor additions. The source item is present but its meaning changes. Track these as incorrect transformations, including:
- proposal converted into decision;
- qualified statement converted into certainty;
- deadline or amount changed;
- dissent removed from a consensus statement;
- speaker or owner changed;
- cause asserted where only correlation was discussed.
Report these errors separately so a low omission rate does not create false confidence.
Apply severity without hiding raw counts
Use a predefined severity scale:
- Low: minor context loss with no operational effect.
- Medium: creates confusion, extra work or weakens traceability.
- High: changes a decision, owner, deadline, obligation, safety issue, material figure or evidence boundary.
A weighted score can help prioritise correction, but always publish the underlying counts by category and severity. Weighting should not allow several high-risk omissions to disappear inside a strong aggregate score.
Use independent review
Have two reviewers independently audit a representative subset. Record agreement, disputed classifications and adjudication. Blind reviewers to the system version where practical.
Low reviewer agreement may indicate an unclear source, vague required-information rules or an unsuitable assertion unit.
Reconcile the entire audit
A final audit table should show:
- total required source items;
- present, partial, missing and incorrect items;
- omission rate;
- category and severity counts;
- total material summary assertions;
- unsupported assertions and rate;
- incorrect transformations;
- reviewer disagreements;
- corrections and approval status.
Sample ongoing use responsibly
For routine governance, review a risk-based sample that includes different teams, meeting types, lengths, speakers and operating conditions. Include complaints, incidents and unusual outputs rather than selecting only convenient meetings.
Increase sampling after a model, application, prompt, template or workflow change. Rerun the same controlled samples where appropriate.
Correct and approve the final record
Update the summary, preserve the audit trail and obtain approval from the responsible person when the note supports a formal decision or action process. Mark drafts and superseded versions clearly.
Restrict access to audio and transcripts containing personal or confidential information. Delete unnecessary duplicates according to the approved retention period.
The NERALVO Halo AI Voice Recorder can support capture and DOWAY-generated summaries, templates and structured outputs. Those outputs should remain drafts until omissions, unsupported additions and changed meanings have been checked against the source.
Final checklist
- Is the summary purpose defined?
- Was a required-information register created first?
- Can every required item be traced through the matrix?
- Are omission rates reported with raw counts?
- Are decisions, actions, conditions, dissent and evidence limits checked?
- Are unsupported additions and incorrect transformations measured?
- Are severity and reviewer disagreement visible?
- Was the corrected summary approved and version-controlled?
Ready to capture meetings properly?
View the NERALVO Halo AI voice recorder with 64GB local storage, meeting capture, compatible phone-call recording workflows and one year of DOWAY Max included.
View NERALVO Halo