How to use this document

Ten questions, drawn from four of Adeo’s oversight domains. For each question:

  • A short explanation of what the question is testing and why it matters
  • What a good answer looks like from the internal team or the MSP
  • What a not-good answer looks like
  • The next action, where the not-good answer appears

The document is not a scoring framework. It is a conversation aid. The value comes from asking the questions, in order, at a board or board sub-committee meeting, and noticing which answers come back clean and which come back vague.

If three or more answers come back vague, the organisation has a governance gap. That gap often points to an absence of independent measurement rather than an MSP at fault.

Domain I — Service delivery

1. What does the last twelve months of MSP ticket data tell us?

What it tests. Whether the MSP produces service-quality reporting that tells the organisation something it did not already know.

Good answer. “We receive a monthly service-report summarising ticket volume, category, resolution time, and recurrence. We can hand the board the last twelve months of data on request within one business day.”

Not-good answer. “We get reports but I’d have to ask the MSP.”

Next action. Request the raw ticket data for the last twelve months as a CSV. The export itself is a signal — resistance, delay, or missing fields reveal more than the data would.

2. Which recurring issues have we had more than twice this year?

What it tests. Whether the MSP is resolving the underlying cause or servicing the symptom.

Good answer. A list of three or four issue categories, each with a root-cause note and a plan to prevent recurrence.

Not-good answer. “I don’t know, the MSP handles that.”

Next action. Ask the MSP to group the last twelve months of tickets by root-cause code (not symptom). If the grouping is thin, the MSP is closing tickets without analysing them.

3. When did the board last hear from someone independent about the MSP’s performance?

What it tests. Whether the organisation has any voice other than the MSP’s own in the conversation about the MSP.

Good answer. “Our external adviser completed an annual independent review last quarter, and their findings are on file with the board minutes.”

Not-good answer. “We haven’t had an external review.” Or: “The MSP presents their own report at our quarterly meeting.”

Next action. Commission a lightweight, fixed-fee independent review — a Quick-Scan or equivalent — to establish the baseline.

Domain II — Commercial discipline

4. Are we paying for M365 (or equivalent) licences we do not use?

What it tests. Whether anyone in the organisation has visibility into licence allocation.

Good answer. “Our last licence-utilisation review was in March. We identified 7 seats assigned to dormant users and 4 tier mis-alignments; we remediated all of them, for an annualised saving of about A$6,000.”

Not-good answer. “The MSP manages our licences.”

Next action. Request an export of active users, assigned licences, and last sign-in date from Entra (or equivalent). Reconcile against current payroll. The work takes an hour.

5. Do our last three MSP invoices reconcile to our contract?

What it tests. Whether any internal party has reconciled recurring invoices against contracted scope in the last quarter.

Good answer. “Yes, we reconcile monthly and the last six months were clean.” Or: “Yes, we found two items last quarter that didn’t map to the contract and had them corrected.”

Not-good answer. “We pay the invoice the MSP sends us.”

Next action. Finance team reconciles the last three invoices against the contracted services and rates. Note any recurring charges for services that no longer run, or any out-of-scope charges that weren’t pre-approved in writing.

6. When did we last benchmark what we pay for what we get?

What it tests. Whether the commercial terms are periodically tested against the market.

Good answer. “Our last market benchmark was a year ago, conducted by an independent adviser. We are within market range for the in-scope services.”

Not-good answer. “We’ve been with this MSP for years; the rates have stayed the same.”

Next action. Have an independent party benchmark the current fee structure against Australian SME MSP market pricing. The market has moved; the contract has usually not.

Domain III — Roadmap and governance

7. Where is the written IT roadmap, and what does it say?

What it tests. Whether there is a formal register of committed technical work, and whether it is current.

Good answer. “Our current roadmap covers the next twelve months, contains 19 items, was last reviewed at our quarterly board IT meeting, and can be produced on request.”

Not-good answer. “The MSP mentions it in our QBR.”

Next action. Request a written roadmap, in a format that can be tabled at the next board meeting, within thirty days.

8. What percentage of last year’s roadmap items were closed on time?

What it tests. Whether the roadmap is a commitment or a suggestion.

Good answer. “Of 19 items on last year’s roadmap, 13 closed on time, 4 slipped with documented reasons and new dates, and 2 were deferred with a board-level rationale.”

Not-good answer. “I’d need to ask.”

Next action. Independent review of the roadmap’s completion rate. If the closure rate is below fifty percent without documented reasons for slippage, the roadmap is decorative.

Domain IV — Security and compliance

9. Could we produce evidence tomorrow of the core controls our cyber insurer will ask for at renewal?

What it tests. Whether evidence for MFA coverage, backup restore-test, privileged access, endpoint coverage, and patch compliance is on file or needs to be collected on demand.

Good answer. “Yes, the evidence pack is maintained by our external adviser. The pack was refreshed last month and is stored in the board’s secure document area.”

Not-good answer. “We could ask the MSP.”

Next action. Commission a Cyber Insurance Readiness review eight to twelve weeks before the next renewal. Not at the point the questionnaire arrives.

10. If a threat actor encrypted our primary office email system tonight, what happens in the next seventy-two hours?

What it tests. Whether there is a documented incident-response plan, who the owner is, and whether it has been exercised.

Good answer. “We have a documented plan owned by our operations director. It was last exercised at a tabletop run-through in February. The recovery path from immutable backup is documented and tested. Notification obligations under the Privacy Act are documented in the same plan.”

Not-good answer. “We’d call the MSP.”

Next action. Document the incident-response plan, assign an owner, exercise it annually. The exercise does not need to be a full drill; a tabletop run-through with the board chair and the MSP account manager in the room is a meaningful first step.

Scoring the answers

Count the clean answers.

  • 9–10 clean answers. The governance is in order. A lightweight annual independent review keeps the posture honest.
  • 6–8 clean answers. There are identifiable gaps. A Quick-Scan or equivalent closes them without disrupting the MSP relationship.
  • 3–5 clean answers. The gap is structural. A Baseline Audit is the right level of engagement.
  • 0–2 clean answers. The board is exposed at both commercial and insurance levels. At this level an independent review is the logical next step.

Appendix — where each question fits into a standard framework

For boards that want to see how the ten questions map to the frameworks an independent reviewer would use:

Question ACSC E8 NIST CSF 2.0 Insurer Control Set
1. Ticket data — ID.RA —
2. Recurring issues — ID.IM —
3. Independent voice — GV.OV —
4. Licence utilisation — ID.AM, PR.DS —
5. Invoice reconciliation — GV.SC —
6. Market benchmark — GV.SC —
7. Written roadmap — GV.SC, GV.OV —
8. Roadmap completion — GV.OV —
9. Insurance control evidence All eight PR, DE insurer control attestation
10. Incident response — RS, RC IR attestation

About this document

Adeo is an independent, vendor-neutral IT advisory for Australian SMEs. We audit, score, and oversee what a managed service provider delivers. We do not resell products, we do not take commissions, and our only source of revenue is the fee our clients pay us.

For a thirty-minute conversation on how the answers from your organisation map to an engagement, write to contact@adeo.au. Replies within one business day.

adeo.au · contact@adeo.au · Adelaide, South Australia · ABN 50 698 595 523