How to use this document

Ten questions, drawn from four of the five domains Adeo measures. For each question:

  • A short explanation of what the question is testing and why it matters
  • What a good answer looks like from the internal team or the MSP
  • What a not-good answer looks like
  • The next action to take when the answer is not good

The document is a conversation aid. The value comes from asking the questions, in order, at a board or board sub-committee meeting, and noticing which answers come back clean and which come back vague. The sample answers, and the figures in them, are illustrative; no figure in this paper is an Adeo fee or a real client result.

If three or more answers come back vague, the organisation has a governance gap. That gap is as likely to reflect an absence of independent measurement as a fault in the MSP.

Domain I — Service delivery

1. What does the last twelve months of MSP ticket data tell us?

What it tests. Whether the MSP produces service-quality reporting that tells the organisation something it did not already know.

Good answer. “We receive a monthly service report summarising ticket volume, category, resolution time, and recurrence. We can hand the board the last twelve months of data on request within one business day.”

Not-good answer. “We get reports but I’d have to ask the MSP.”

Next action. Request the raw ticket data for the last twelve months as a CSV. The export itself is a signal — resistance, delay, or missing fields reveal more than the data would.

2. Which recurring issues have we had more than twice this year?

What it tests. Whether the MSP is resolving the underlying cause or servicing the symptom.

Good answer. A list of three or four issue categories, each with a root-cause note and a plan to prevent recurrence.

Not-good answer. “I don’t know, the MSP handles that.”

Next action. Ask the MSP to group the last twelve months of tickets by root-cause code rather than by symptom. If the grouping is thin, the MSP is closing tickets without analysing them.

3. When did the board last hear from someone independent about the MSP’s performance?

What it tests. Whether the organisation has any voice other than the MSP’s own in the conversation about the MSP.

Good answer. “Our external adviser completed an annual independent review last quarter, and their findings are on file with the board minutes.”

Not-good answer. “We haven’t had an external review.” Or: “The MSP presents their own report at our quarterly meeting.”

Next action. Commission a lightweight, fixed-fee independent review — a Quick-Scan or equivalent — to establish the baseline.

Domain II — Commercial discipline

4. Are we paying for Microsoft 365 (or equivalent) licences we do not use?

What it tests. Whether anyone in the organisation has visibility into licence allocation.

Good answer. “Our last licence-utilisation review was in March. We identified seven seats assigned to dormant users and four tier misalignments; we remediated all of them, for an annualised saving of about A$6,000.”

Not-good answer. “The MSP manages our licences.”

Next action. Request an export of active users, assigned licences, and last sign-in date from Entra (or equivalent). Reconcile against current payroll. The work takes an hour.

5. Do our last three MSP invoices reconcile to our contract?

What it tests. Whether any internal party has reconciled recurring invoices against contracted scope in the last quarter.

Good answer. “Yes, we reconcile monthly and the last six months were clean.” Or: “Yes, we found two items last quarter that didn’t map to the contract and had them corrected.”

Not-good answer. “We pay the invoice the MSP sends us.”

Next action. Finance team reconciles the last three invoices against the contracted services and rates. Note any recurring charges for services that no longer run, or any out-of-scope charges that weren’t pre-approved in writing.

6. When did we last benchmark what we pay for what we get?

What it tests. Whether the commercial terms are periodically tested against the market.

Good answer. “Our last market benchmark was a year ago, conducted by an independent adviser. We are within market range for the in-scope services.”

Not-good answer. “We’ve been with this MSP for years; the rates have stayed the same.”

Next action. Have an independent party benchmark the current fee structure against Australian SME MSP market pricing. Rates in the broader market change over time; a contract left unreviewed remains on its original terms.

Domain III — Roadmap and governance

7. Where is the written IT roadmap, and what does it say?

What it tests. Whether there is a formal register of committed technical work, and whether it is current.

Good answer. “Our current roadmap covers the next twelve months, contains 19 items, was last reviewed at our quarterly board IT meeting, and can be produced on request.”

Not-good answer. “The MSP mentions it in our QBR.”

Next action. Request a written roadmap, in a format that can be tabled at the next board meeting, within thirty days.

8. What percentage of last year’s roadmap items were closed on time?

What it tests. Whether the roadmap is a commitment or a suggestion.

Good answer. “Of 19 items on last year’s roadmap, 13 closed on time, 4 slipped with documented reasons and new dates, and 2 were deferred with a board-level rationale.”

Not-good answer. “I’d need to ask.”

Next action. Independent review of the roadmap’s completion rate. If the closure rate is below fifty per cent without documented reasons for slippage, the roadmap is decorative.

Domain IV — Security and compliance

9. Could we produce evidence tomorrow of the core controls our cyber insurer will ask for at renewal?

What it tests. Whether evidence for MFA coverage, backup restore-test, privileged access, endpoint coverage, and patch compliance is on file or needs to be collected on demand.

Good answer. “Yes, the evidence pack is maintained by our external adviser. The pack was refreshed last month and is stored in the board’s secure document area.”

Not-good answer. “We could ask the MSP.”

Next action. Commission a Cyber-Insurance Readiness review eight to twelve weeks before the next renewal, with time in hand before the questionnaire arrives.

10. If a threat actor encrypted our primary office email system tonight, what happens in the next seventy-two hours?

What it tests. Whether there is a documented incident-response plan, who the owner is, and whether it has been exercised.

Good answer. “We have a documented plan owned by our operations director. It was last exercised at a tabletop run-through in February. The recovery path from immutable backup is documented and tested. Notification obligations under the Privacy Act are documented in the same plan.”

Not-good answer. “We’d call the MSP.”

Next action. Document the incident-response plan, assign an owner, exercise it annually. The exercise does not need to be a full drill; a tabletop run-through with the board chair and the MSP account manager in the room is a meaningful first step.

Reading the answers

Count the clean answers.

  • 9–10 clean answers. The governance is in order. A lightweight annual independent review keeps the posture honest.
  • 6–8 clean answers. There are identifiable gaps. A Quick-Scan or equivalent closes them without disrupting the MSP relationship.
  • 3–5 clean answers. The gap is structural. A Baseline Audit is the right level of engagement.
  • 0–2 clean answers. The board is exposed on both the commercial and the insurance fronts; an independent review is the logical next step.

Appendix — where each question fits into a standard framework

For boards that want to see how the ten questions map to the frameworks an independent reviewer would use:

Question ACSC Essential Eight NIST CSF 2.0 Insurer Control Set
1. Ticket data — ID.RA —
2. Recurring issues — ID.IM —
3. Independent voice — GV.OV —
4. Licence utilisation — ID.AM, PR.DS —
5. Invoice reconciliation — GV.SC —
6. Market benchmark — GV.SC —
7. Written roadmap — GV.SC, GV.OV —
8. Roadmap completion — GV.OV —
9. Insurance control evidence All eight PR, DE Insurer control attestation
10. Incident response — RS, RC IR attestation

About this document

Adeo is an independent, vendor-neutral IT advisory for Australian SMEs. We audit, score, and oversee what a managed service provider delivers. We do not resell products, we do not take commissions, and our only source of revenue is the fee our clients pay us.

To talk through how the answers from your organisation map to an engagement, write to contact@adeo.au.

Adeo Partners Pty Ltd · ACN 698 595 523 · ABN 50 698 595 523 · adeo.au · contact@adeo.au · Adelaide, South Australia