How to tell if your MSP is delivering

Why this matters

Many SME relationships with an MSP are long-standing — often the result of a decision made several roles ago, never reviewed since, and embedded in how the organisation operates. The monthly invoice arrives. The tickets get closed. The board assumes it is being looked after.

The question is not whether the MSP is competent. Most are competent at what they were engaged to do. The question is whether what the MSP is contracted to do is still the right set of work, and whether the delivery is being measured by anyone who is not the party being measured.

Five questions help the board self-diagnose the relationship in an afternoon. If four of them can be answered cleanly, the MSP is likely delivering and the relationship can be continued with minor tuning. If fewer than three can be answered cleanly, there is material value in an independent review.

1. Is the SLA the right SLA?

Most MSP contracts contain a service level agreement specifying response times, resolution times, and uptime targets. The first question is not whether the SLA is being met; it is whether the SLA measures anything the organisation actually needs.

Good test: if the SLA is entirely about ticket response and resolution times, it does not measure service quality in any business-facing way. A ticket closed in forty minutes that did not resolve the problem is a metric win for the MSP and a measurable failure for the organisation.

Better SLA elements (rarer in SMB contracts, common in enterprise):

Problem-recurrence tracking (was the same issue closed three times this quarter?)
End-user satisfaction on closure (not on first-touch; on closure)
Time to resolution of specific classes of incident (not all incidents averaged together)
Outcome measurement for enhancement requests, not just reactive tickets

Ask the MSP for the last twelve months of ticket data exported to CSV. The export alone is a useful signal — the time it takes, and the resistance encountered, tell you something before you have read a row.

2. What does the ticket data actually say?

If the ticket-data export arrives, five patterns are worth looking for:

Recurrence clusters — the same category of issue appearing multiple times across the year. A healthy environment resolves the underlying cause once; recurrence that is never traced to a root cause is a cost the organisation keeps paying.
Severity distribution — if everything is priority 3, the prioritisation mechanism is inert. If everything is priority 1, it is being gamed for SLA.
Same-user concentration — if twenty percent of tickets come from ten percent of users, the MSP has a user-training opportunity, not a technical opportunity.
Time-of-closure clustering — tickets closing precisely at the SLA window is a signal of SLA management, not of resolution rhythm.
Root-cause coding — good MSPs code the root cause; poor MSPs code only the symptom. If every ticket root cause is “software error,” the data is not being analysed.

Reading ticket data well is a skill. Reading it badly is still useful; a partner, CFO, or IT-aware board member will see more than they expect in an afternoon.

3. Is the roadmap progressing?

Every competent MSP relationship has a roadmap — a register of improvements, migrations, and enhancements agreed between the organisation and the MSP. Usually it lives in the quarterly business review deck, is referenced verbally, and is not tracked rigorously.

The question: in the last twelve months, how many roadmap items have been closed, how many are in progress, and how many have slipped past their original date without a written re-commitment?

A reasonable picture might be roughly two-thirds of items closed on time, the rest in progress or pushed with a documented reason. A weaker one is a quarter closed, half in progress, and a quarter quietly no longer referenced.

The warning sign is not the slippage itself — projects slip for good reasons — it is the absence of a written register. If the roadmap lives only in the account manager’s head, there is no audit trail, and the organisation cannot tell the difference between a good quarter and a bad one.

4. Are the commercial terms being honoured?

Three commercial signals worth investigating annually:

Licence utilisation. Is the organisation paying for Microsoft 365 or equivalent licences it is not using? Are users assigned to tiers they do not need? Are there seats for people who have left? Exports for M365 licence allocation, Entra sign-in activity, and group membership reveal this in a morning.
Invoice-to-contract match. Are the last three invoices reconciling against the contract? Are there line items that appeared without a corresponding change order? Are there out-of-scope hours that were not pre-approved in writing?
Vendor markup. If the MSP resells software, hardware, or third-party services, is the markup visible to the organisation? Is it within the range the organisation agreed to? Is it within the range the market considers reasonable?

The presence of commercial drift is not, on its own, evidence of wrongdoing. MSPs operate on thin margins, and licence management is genuinely hard. The presence of drift without measurement is the governance failure.

5. Is the cyber-insurance control set being substantiated?

The question the insurer is asking at renewal is being asked, indirectly, of the MSP — because the MSP is operating most of the controls. But the insurer is asking the insured, not the MSP, and the insured bears the premium.

Five sub-questions:

Has the MSP provided a current export of MFA coverage?
Has the MSP provided a current export of conditional-access policy state?
Has the MSP provided a dated log of a successful restore test in the last ninety days?
Has the MSP provided a current patch-compliance export?
Has the MSP provided a current EDR coverage export?

If any of the answers is “I would need to ask,” the organisation is not in a position to respond to an underwriter in the window the underwriter expects. That is the gap most worth closing before an underwriter asks.

What to do with the answers

If the organisation can answer four or five of the five cleanly, the MSP relationship is in good shape and the review could plausibly be annual rather than monthly. A quarterly independent scorecard (Adeo Pulse or equivalent) is light-touch confirmation, not a structural intervention.

If the organisation can answer three cleanly, the relationship is functional and there are identifiable gaps worth addressing. A Quick-Scan is typically the right level of engagement — the more targeted entry point, scoped in the written proposal.

If the organisation can answer fewer than three cleanly, an independent review is the logical next step. A Baseline Audit is the right level of engagement — a focused multi-week engagement, wider in scope, producing a board-ready written assessment and a remediation roadmap.

In any case, the first step is the organisation asking the questions directly. The answers produced are evidence, either way.

A board-level framing

The question the board should ask its own leadership is not “are we safe?” — the MSP will always answer yes, and the answer will be truthful to the limit of the MSP’s visibility. The question the board should ask is:

“Could we produce, on short notice, an independent document that explains what our MSP delivered this year and what we paid for it?”

If the answer is yes, the governance is in order. If the answer is no, the governance is not in order, and that is the specific gap an independent advisor exists to close.

What this article does not cover

The specific mechanics of dropping, changing, or retendering an MSP. That is a different conversation and a different document.
Any recommendation of specific MSPs, vendors, or products. Adeo takes no commissions and names no preferred suppliers publicly.
Incident-response playbooks. That is a specialised field and Adeo will refer cleanly to a specialist when an incident is in flight.

Adeo is an independent, vendor-neutral IT advisory for Australian SMEs. We audit, score, and oversee what a managed service provider delivers. We do not resell, and we take no commissions. Contact: contact@adeo.au · adeo.au.