AI in Social Care: The Questions Every Provider Should Be Asking

Why Now?

If you run a domiciliary care service in the UK or Ireland, AI has probably already arrived in your organisation, whether you decided on it or not. It sits inside the scheduling tool that proposes who visits whom, powers the note-taking app a carer downloaded to save time, and has likely been switched on quietly in software updates you accepted.

The technology has matured to the point where it changes what “good” looks like. Digital social care records alone, the foundation AI now sits on top of, already save an estimated 30 million administrative hours a year, giving back at least 20 minutes per care worker per shift.

The problem is that the conversation about AI is happening in two separate languages.

Technology partners talk about efficiency and competitive advantage. Regulators talk about risk, ethics, and the duty not to harm vulnerable people.

Neither, on its own, helps a Registered Manager or Operations Director decide what to adopt, what to defer, and what to avoid. This blog post is the bridge by drawing on a our new white paper, The Questions Nobody is Asking About AI in Social Care, written by Group Chief Technology Officer Seán Morris, and turns its findings into the practical questions you should put to any technology partner before you sign.

The Stethoscope Moment:

When the stethoscope was invented in 1816, doctors distrusted it as impersonal and unreliable. Within a generation, practising medicine without one became unthinkable.

It never replaced the physician’s judgement. It extended it. AI in social care sits at a similar point.

AI will not replace the expertise of your carers, coordinators, and managers, but used well it gives them insight and capacity they could not otherwise have. The question is not whether AI arrives in home care, it’s how you govern it when it does. And the clock is already running.

The Regulatory Clock is Ticking:

If you operate across both the UK and Ireland, you face overlapping frameworks that all point at the same expectation: show me your documentation.

Does the EU AI Act Apply to UK and Irish Care Providers:

The EU AI Act is the world’s first comprehensive AI regulation, and Irish providers are directly subject to it.

UK providers are not automatically exempt: if your technology partner processes data through EU infrastructure, you can find yourself within scope, which is simply how modern cloud services work.

The Act sorts AI into risk tiers, and for social care the high-risk category is the one that matters.

Scheduling and resource-allocation systems that decide which carer is assigned to which service user are likely to fall here, as are predictive tools that flag someone as at elevated risk or as a safeguarding concern. High-risk systems must meet requirements covering risk management, data governance, technical documentation, transparency, human oversight, and accuracy.

The compliance deadline is 2 August 2026, and most providers have not yet checked whether their tools fall within scope.

The obligations do not sit with your technology partner alone. As the organisation using a high-risk system in your service, you are a “deployer” under the Act, with your own duties around human oversight, monitoring, and record-keeping, separate from the technology partner’s duties as the system’s provider.

What Does CQC, the ICO, and the MHRA Expect?

The Care Quality Commission’s (CQC) Single Assessment Framework does not yet contain AI-specific criteria, but it places heavy weight on governance and on evidence that policies work in practice.

Inspectors may never say “AI” out loud, but “can you demonstrate that this system is governed properly?” applies to everything you use, including AI running quietly inside your scheduling or documentation tools.

The Information Commissioner’s Office (ICO), the UK’s data protection regulator, is specific about meaningful human oversight. Its guidance on automated decision-making makes clear that human involvement must be active rather than tokenistic, with genuine discretion to overturn an automated decision rather than rubber-stamp it.

If your coordinators cannot realistically override a scheduling recommendation, you do not have human oversight. You have a compliance fig leaf.

The MHRA has also set up a national commission on AI in healthcare, and NHS England runs a supplier registry for ambient voice technology in care settings.

Where Does Ireland Stand?

Irish providers face the EU AI Act layered on top of the Health Information Quality Authority’s (HIQA) standards for records management, medication management, and governance.

Dedicated AI guidance from HIQA remains limited, so you are interpreting existing rules against technology they were never written for. That is uncomfortable, but it is also an opportunity.

Providers who get ahead of the guidance will be in a far stronger position when it lands.

If you also operate in Australia, the same direction of travel applies through the Aged Care Quality Standards, NDIS Practice Standards, and the TGA’s medical device rules for software, all covered in the full white paper.

Five Risks Worth Understanding Before You Buy:

1. The Line Between Assistance and Decision-Making:

This is the single most important design choice in any care-related AI system. AI can help a carer document an observation more completely, but it cannot diagnose a condition.

It can recommend that a carer is well-matched to a service user, but it cannot decide that a visit is unnecessary. Where a vendor draws that line determines whether their product is treated as a regulated medical device or an administrative tool.

Technology partners who blur it, through ambitious marketing or designs that skip proper human review gates, create regulatory and clinical risk for everyone who deploys them.

2. Hallucination is a Service User Safety Issue:

A fabricated vital sign that was never recorded. A medication that was never administered. A behavioural change that never happened.

In care documentation, an AI “hallucination” is not a minor glitch. It is a service user safety incident.

A 2025 peer-reviewed study in npj Digital Medicine, the largest manual evaluation of its kind, analysed nearly 13,000 clinician-annotated sentences.

Even in a controlled research setting, a carefully optimised GPT-4 pipeline with iterative prompt refinement, the model produced a 1.47% hallucination rate and a 3.45% omission rate, and 44% of those hallucinations were rated “major”, meaning they could change a patient’s diagnosis or management if left uncorrected.

In an everyday, unoptimised commercial deployment, the rates will be higher.

The conclusion is simple: human review before records are finalised is not optional.

Any technology partner whose tool touches care documentation should be able to describe their safeguards in technical detail. If they cannot, that should end the conversation.

3. Where is Your Data Actually Processed?

Social care data is among the most sensitive personal data that exists, and cloud-based AI services can route it through several regions without you ever knowing.

A provider in Manchester whose care data is processed through a US-hosted AI model has created a cross-border transfer that needs specific safeguards under UK GDPR, and an Irish provider whose data crosses outside the EU faces similar obligations.

“Our servers are in the UK” is not the same as “no AI model in our stack processes your data outside the UK”. Your partner should demonstrate this at an infrastructure level, not just promise it in a contract.

4. Bias and the Invisible Rationing Problem:

AI learns from historical data, and historical data reflects historical inequities.

A scheduling algorithm trained on past patterns can quietly carry them forward, and a risk model can underestimate risk for groups that were historically underserved.

A system that allocates fewer resources to certain groups based on learned patterns is rationing care by algorithm.

Nobody voted for that, nobody consented to it, and in most cases nobody even knows it is happening. Responsible AI requires active bias testing before deployment, ongoing monitoring afterwards, and a documented process to fix it when bias is found.

5. The Update Problem:

Foundation AI models are updated regularly by their developers, and a single update can change the behaviour of every feature built on top of it.

A change to how clinical language is interpreted, or how a risk score is calculated, affects care delivery, and deserves the same rigour as a change to your care protocols.

Your provider should notify you in advance of model changes, test that the system still behaves correctly before an update goes live, and let you roll back if quality drops.

Where the Sector is Heading: The Four Shifts:

The white paper sets out four shifts that will shape the next few years, and for home care providers two of them land hard.

Document intelligence moves AI from recording observations to understanding them, surfacing the declining fluid intake across several visits, or the near-miss cluster at a particular time of day, that are currently buried in notes nobody has time to read.

Intelligent scheduling moves from building rosters faster to building better ones, balancing carer skills, visit windows, travel time, continuity, and availability. The next generation should hand the coordinator an optimised draft to review rather than asking them to build from scratch, and eventually forecast demand before gaps appear.

Regulatory intelligence turns compliance into a proactive loop, where a regulatory change triggers a policy update, then a training requirement, all tracked and evidenced for inspection across CQC, HIQA, and NDIS.

The data estate is the quiet differentiator. With 80% of CQC-registered providers now using digital social care records, up from 41% in December 2021, the sector has built the foundation AI needs. The organisations that benefit most will be those investing in data quality and governance now.

Why This Matters So Much in Home Care:

The pressures are not theoretical.

Adult social care in England employs 1.6 million people yet carries 111,000 vacant posts, a vacancy rate three times that of the wider economy, with annual turnover equivalent to around 335,000 workers leaving every year.

Skills for Care estimates the sector will need 470,000 additional posts by 2040, overseas recruitment has fallen sharply, and in September 2025 NHS beds occupied by patients medically fit for discharge cost an estimated £220 million in a single month.

These are not problems you can solve by working harder. They call for working smarter, which is exactly why AI is arriving and exactly why governing it well matters.

Get it right and you free your team for the human work only they can do. Get it wrong and you have automated risk into the most vulnerable part of your service.

The Seven Questions to Ask Before You Sign:

If you take one thing from this post, make it this checklist. Put these seven questions to any AI enabled technology provider, for any product. Vague or evasive answers are a red flag.

1. Where is my data actually processed? Not stored, processed. By which AI models, in which regions? Can you show it at an infrastructure level, not just in a contract?
2. What safeguards exist against hallucination? What stops fabricated content entering a care record? What are the confidence thresholds, and where is human review mandatory before content is saved?
3. Is this classified as a medical device? Has it been assessed under DCB0129 (UK), EU MDR (Ireland), or TGA (Australia)? If not, what design constraints keep it outside that classification, and how are they audited?
4. What is your EU AI Act position? For high-risk systems such as scheduling and risk prediction, what technical documentation and conformity assessments do you provide? The deadline is 2 August 2026.
5. How will I know when the model changes? What is your notification policy, how is regression testing done, and can you roll back if quality drops?
6. How is bias tested and monitored? What methodology tests outputs across protected characteristics, and what is the documented remediation process?
7. Can our staff actually challenge this? What explainability is built in? Can a coordinator understand why a schedule was proposed, or a manager why a risk flag was raised? The ICO is clear that oversight that cannot be exercised in practice is not oversight.

The Standards We Should Set:

AI in social care is not a question of whether, but of how.

The providers who navigate this well will treat AI adoption as a governance decision, not a technology decision.

They will ask hard questions of their partners, invest in data quality as a strategic priority, and make sure their boards understand the obligations that come with AI. Just as the stethoscope extended the physician’s judgement rather than replacing it, AI should extend the expertise of your team. The difference comes down to governance.

Frequently Asked Questions:

Does the EU AI Act Apply

Irish providers are directly subject to the EU AI Act. UK providers can also fall within scope if their technology partner processes data through EU infrastructure, which is common with cloud-based AI services. The compliance deadline for high-risk systems is 2 August 2026.

Is AI Note-Taking Safe to Use for Care Records?

It can be, with the right safeguards. Peer-reviewed research has found that even carefully optimised clinical AI pipelines produce hallucinations and omissions. Any tool that touches care documentation should ground outputs to source data, apply confidence thresholds, and require human review before a record is finalised.

It can be, with the right safeguards. Peer-reviewed research has found that even carefully optimised clinical AI pipelines produce hallucinations and omissions. Any tool that touches care documentation should ground outputs to source data, apply confidence thresholds, and require human review before a record is finalised.

What Does the ICO Mean by Meaningful Human Oversight

The ICO expects genuine discretion to overturn an automated decision, not a review process that rubber-stamps the algorithm. If your staff cannot realistically override a scheduling recommendation or challenge a risk flag, you do not have meaningful human oversight.

Read the Full White Paper:

For the complete regulatory breakdown across the UK, Ireland, and Australia, the evidence behind each risk, and the full partner-assessment checklist, download the white paper, The Questions Nobody is Asking About AI in Social Care.

And if you want to see how OneTouch can support your service, from care management and scheduling to compliance, and talk through what responsible AI adoption could look like for your organisation, book a demo with our team.