More hospitals now run phishing simulations. Most of them use the open rate as the measure of success: 12% last time, 8% this time, reported as progress.
There is a structural problem in that design. Making a lower open rate the goal puts pressure on the organisation not to get caught — which means the person who does click stays quiet. In a real attack, silence is precisely what you cannot afford. A click is a problem on one endpoint; thirty unreported minutes is a problem for the whole hospital.
The objective needs restating. Not a lower open rate, but a higher report rate — and an organisation where people who report are visibly better off for it.
Disclaimer: General information. Before running simulations, confirm your own position on employment-law considerations, whether staff must be told in advance, and the terms of any contract with a delivery vendor.
Restating the Objective
| Metric | What it tells you | How to target it |
|---|---|---|
| Report rate | Whether people who notice actually act | Raise it. The primary metric |
| Time to first report | Whether the reporting route works | Shorten it |
| Open / click rate | General awareness level | Reference only. Not a target |
| Credential entry rate | The behaviour that causes harm | Lower it, without blame |
| Reports reaching IT | Whether IT actually receives them | Approach 100% |
Making report rate primary changes the design. Is there an easy route? How do you respond to reporters? What do you do when reports exceed what IT can triage? Those become the questions.
Watch time to first report too. A high report rate is meaningless if the first report arrives the next day. "How many minutes until the first report?" is your best estimate of when a real response would begin — see First Response to Ransomware.
Designing Out the Blame
Simulations fail in how the results are handled. The moment individually identifiable results are shared, the next report rate falls.
- Do not send individual results to line managers. Keep statistics at department level at most
- Do not use them in performance appraisal, state this in writing, and say so in advance
- Make the landing page educational — "here is how this one could be spotted", not "you failed"
- Reply to people who report — even an auto-acknowledgement, with thanks
- Share results as organisational tendencies — "mails imitating billing correspondence had the highest open rate", not "the administration department was worst"
Point 5 is the operational trick. Naming a department pressures its head, and that pressure lands on staff. Make the scenario the subject, not the people.
Whether to pre-announce the exercise at all is a judgement. A complete surprise measures real awareness but weighs on staff. Announcing that a simulation will happen this year while withholding the date is the workable middle.
Building Scenarios in a Hospital Context
| Scenario type | Example | Intended audience |
|---|---|---|
| Posing as a business system | Password expiry, account unlock | All staff |
| Posing as an internal request | A submission request from administration, HR paperwork | All staff |
| Posing as a supplier | Invoices, quotations, deliveries | Administration, billing, procurement |
| Posing as a conference or course | Registration, abstract submission | Doctors, researchers |
| Posing as a patient or family | Enquiries, complaints, requests for records | Patient liaison, medical records |
| Posing as a public body | Notices, survey requests | Management functions |
Raise difficulty in stages. Starting with a hard scenario produces a high open rate and a chilled organisation. Year one should leave obvious tells and confirm that the reporting route works; refine later.
Two cautions: do not use the names of real suppliers or staff — it damages trust in those parties — and avoid extreme urgency, such as clinical emergencies, which disrupts care even in a drill.
The Follow-Up Is the Exercise
| Timing | Action |
|---|---|
| Immediately | Educational landing page; acknowledgement to reporters |
| Same / next day | Check that reports are arriving and the route is not blocked |
| Within a week | Share results with all staff, scenario-first, showing the tells |
| Within two weeks | Consolidate issues: routes, communication, materials |
| Before next time | Fix the issues and brief departments individually |
Show the tells concretely. "Be careful with suspicious mail" changes nothing. Show the actual screen: the sender domain, the odd salutation, the mismatch between displayed link and destination, the small errors in the Japanese.
And treat reporting as the behaviour to celebrate: "N people reported; the first came M minutes after delivery. Against a real attack we would have started M minutes in." That framing is what makes reporting normal.
See Designing Security Training and Drills and A BCP for Cyberattacks.
Preparing the Receiving End
Raising the report rate raises the load on IT. Start without preparing for that and reports go unanswered — after which nobody reports again.
- One reporting route, which every member of staff can state — mail, phone, or a button. Multiple routes create hesitation
- A defined out-of-hours route, or an explicit statement that reports are handled in working hours
- One-click reporting where the mail client supports it; it materially raises the rate
- Triage criteria — simulation, genuine suspicious mail, or harmless
- Prepared reply templates
"Too many reports" is a good state. Raising the bar to reduce false positives loses the real ones. Solve it by making triage efficient. Keep the records alongside your log management and incident plan records.
Conclusion
- The objective is a higher report rate, not a lower open rate
- Measure time to first report — it approximates when a real response would start
- Design out blame: no individual results to managers, no use in appraisal, stated in advance
- Share results with the scenario as the subject, never the department
- Build scenarios in your own operational context and escalate difficulty; never use real supplier names
- The follow-up fortnight is the exercise. Show the tells, and celebrate reporting
- Prepare the receiving end first: one route, an out-of-hours rule, ready replies
Most of the effort sits in the reporting route and the follow-up, not the send. Pottech advises on governance and operating rules from the standpoint of building and running medical information systems — contact us. To assess a supplier's own training regime, see What Is an ISMS (ISO/IEC 27001)?.
References and Sources
- Guidelines for the Safe Management of Medical Information Systems | MHLW
- Information-technology Promotion Agency (IPA)
- National center of Incident readiness and Strategy for Cybersecurity (NISC)
- FY2026 Medical Fee Revision | MHLW
Note: how simulations are run, whether staff must be informed, and how records are handled should be decided in light of your own employment policies and applicable law. Guideline and reimbursement requirements are subject to revision.