. Introduction
Welcome back to Impact of AI: Explored, the podcast where we explore how artificial intelligence is changing technology, organisations and everyday life. In this episode, James O’Regan and Gerjon Kunst sit down with Lee Crossley, a consultant CTO, co-founder and engineer with two decades of experience across education, policing, healthcare, energy and digital transformation.
Our topic is local AI: models that run on devices or infrastructure we own rather than exclusively in a public cloud. We discuss what “local” really means, why it matters, how models should be routed between local and cloud environments, and what happens when AI becomes invisible—quietly embedded in our phones, laptops, offices and other devices.
The conversation begins with a deliberately provocative question: if every AI provider wants us to consume more tokens, could the smarter move be to use fewer—or none at all?
2. Meet the Guest
Lee Crossley is Co-Founder of Overshow and Director at San Digital, a UK-based consultancy that delivers fixed-price strategy and engineering services for scaleups. He is a multi-skilled director and engineer with a Computer Science honours degree from Lancaster University and around twenty years of experience in consulting and product engineering.[web:1]
His career includes roles as Chief Engineer at BJSS, CTO and co-founder of Presciense (an award-winning IoT company connecting the smart grid to the smart home), and technical consulting work for NHS Digital, NHS (NICE), Cambridge Assessment, Payzone and others.[web:1] Lee has led development offices in Manchester and Taipei, engaged with major energy providers and telco operators across Europe, and passed the UK government’s “Secure by Design” principles, independently verified by Gemserv.[web:1]
He has won eight hackathons in the UK and across Europe, maintains a significant open-source following and is a regular speaker at technical and industry conferences worldwide.[web:1] His work spans firmware, cloud services, mobile applications, IoT and technology strategy—experience that informs his pragmatic view of AI: less interest in benchmark hype, more focus on whether a system is appropriately designed, measurable, secure and useful.
Lee is also working on a product based on the idea of technology inception: anticipating where local models will be in the future and designing a product that can run on a standard laptop while remaining interchangeable as models improve. His approach is deliberately model-agnostic. The model is a component that can be replaced, rather than the product itself.
3. Setting the Stage
Local AI matters because the current AI economy is largely built around consumption. Cloud providers sell inference, usually measured in tokens, while users are expected to choose models, manage prompts, understand context windows and decide how much compute a task deserves.
But tokens are only one part of the picture. A cloud request may also involve latency, data transfer, privacy exposure, availability, vendor dependency and the operational cost of reviewing the result. For many tasks, particularly small and sensitive ones, sending data to a remote service may be unnecessary.
That does not mean the cloud is going away. Instead, we see a future built around right-sized and hybrid AI:
- Small, private or latency-sensitive tasks can run locally.
- Large research workloads can be sent to cloud models.
- A routing layer can choose the right model automatically.
- Users should not need to understand tokens, model names or prompt engineering just to get useful work done.
In this blogpost, we explore the practical and strategic questions behind that future—and the governance challenges that come with it.
4. Episode Highlights
Local AI is broader than large language models
One of the first important clarifications is that local AI does not necessarily mean running a huge chatbot on a high-end workstation. A local model could run on a phone, laptop, small server or even an embedded device. Lee gives the example of an IoT device using a focused model for something like burn detection in a smart appliance.
The key question is not “Can this device run the biggest model?” but “What is the smallest, most suitable model for this task?”
The future is routing, not model selection
The discussion then moves towards semantic or intelligent routing. A system could inspect a request and decide where it belongs. A question containing personally identifiable information might be handled locally, while a long-running research task could use a powerful cloud model.
This would remove one of today’s biggest frustrations: users having to decide which model to select, how many tokens to spend and which instructions to provide. The intelligence would move from the chat box into the surrounding system.
As Lee puts it, the future is not necessarily a prompt and a chat box. AI should understand the user’s context and surface useful assistance at the right moment, rather than asking people to become prompt engineers.
5. Deep Dive: Groundedness in AI—Why Local Context Matters
The most important insight in this episode is that local AI is not only about saving money. It is about control, context and trust.
Privacy and latency
Local speech-to-text is a strong example. Lee describes working on a product where speech recognition runs on the device. This can provide excellent accuracy without sending audio to a cloud service. It also avoids network latency, data-transfer costs and concerns about how recordings or voiceprints are stored.
The same principle applies to smaller tasks such as categorisation, extracting key facts from a meeting transcript or identifying sensitive information. If the job can be completed locally, the organisation may not need to expose the underlying data to a third-party provider.
Right-size the model
A recurring theme is avoiding over-specification. A simple text-editing task should not automatically be sent to the most advanced frontier model. Using a larger model may increase cost and latency without improving the outcome.
Right-sizing means matching the model and execution environment to the work:
- A compact local model for transcription or classification.
- A specialised model for search enhancement.
- A cloud model for large-scale research or complex reasoning.
- A hybrid workflow when sensitive data and heavy computation appear together.
This is similar to good architecture in any other area: do not deploy an unnecessarily expensive system when a smaller, more focused component solves the problem.
Invisible intelligence
Lee argues that prompts should eventually become less important as an interface. A blank chat box is an “anything machine,” which can be difficult for people to use effectively. A more useful system understands the current context and provides assistance inside the workflow.
Imagine a meeting where a local model transcribes the discussion, identifies decisions, highlights contradictions and surfaces relevant information—all in real time. The model does not need to be presented as a separate chatbot. It can simply become part of the environment in which people work.
This is where local AI becomes especially interesting. Running transcription, context extraction and personalisation locally could make the experience faster, more private and more responsive.
Control comes with responsibility
Local processing also creates new assurance requirements. Organisations may need to know:
- Which model is installed on each device.
- Whether data is encrypted locally.
- How models are updated and patched.
- What happens if a device is lost or a vendor disappears.
- Which applications, agents or MCP servers can access the model.
- What was called, when it was called and whether the action was permitted.
Traditional compliance controls may not answer all of these questions. A managed local-AI platform could eventually provide central deployment, hardware-aware model selection, security policies, updates, auditing and data protection.
For a Microsoft-oriented enterprise, it is easy to imagine this becoming part of endpoint management: deploy a model through a management platform, select a model based on available RAM and hardware acceleration, enforce policies and store sensitive information in a protected local data vault.
The goal is to make local AI simple for the user without making it uncontrolled for the organisation.
6. Real-Life Stories & Examples
Local speech-to-text
Lee’s practical example is local speech-to-text. Running the model on the device can deliver near-perfect accuracy for the required use case while avoiding the latency and privacy issues associated with uploading audio. This is particularly relevant for meetings, interviews, healthcare, legal work and other environments where voice data is sensitive.
A specialist model for search
Lee also describes building a product that used a local model to enhance search. Instead of bundling an unnecessarily large multimodal model, he stripped out capabilities that the product did not need and reduced the model footprint by roughly one-third.
The lesson is clear: a local model should be designed around a task. The objective is not to reproduce a cloud chatbot; it is to embed the right intelligence into a useful product.
Model routing based on sensitivity
Gerjon raises the idea of a routing layer that checks a request and sends sensitive content to a local model while forwarding less sensitive or more demanding work to the cloud. Lee points to emerging examples of this approach, including systems that detect personally identifiable information and route it locally.
However, detection alone is not enough. Organisations still need to ask what happens to the data on the device, whether it is encrypted and how the routing decision is audited.
The AI vampire problem
The conversation takes a humorous but serious turn when the hosts discuss developers who stay up late to monitor agents or keep coding tasks running. James describes people leaving social occasions to check on an agent, while Lee compares token consumption to a casino-style feedback loop: a good result encourages another run, and then another.
This can create the illusion of productivity while generating more review work, defects and rollbacks. Local AI does not automatically solve that problem. Whether the model runs locally or in the cloud, teams still need sensible workloads, clear boundaries, human review and the ability to roll back changes.
Starting with Ollama
For people who want to experiment, Lee suggests starting with a tool such as Ollama. A mid-range laptop with around 16 GB of memory may be suitable for a compact multimodal model, depending on the exact hardware and workload.
The advice is not to expect a local model to be a direct replacement for ChatGPT or another cloud assistant. Instead, start with a specific task, test whether it works with a smaller model and only then decide whether local deployment is worthwhile.
Enterprise deployment
For organisations with thousands of endpoints, token savings cannot be the only success metric. Lee recommends starting with a real business workflow and measuring the baseline before introducing AI. Then measure the result afterwards.
If a new AI workflow is introduced but the old process remains active, the organisation may not save anything. In some cases, employees become faster at producing output but create more work for reviewers. The cost of verification can be higher than the token cost itself.
That is why enterprise local AI needs central policies, default-deny access, audit history, controlled MCP integrations and clear measures of business outcome.
7. Key Takeaways
- Local AI means running an appropriate model on hardware you own or control; it does not necessarily mean running a giant language model.
- The best model is the one that fits the task, device, privacy requirement and performance target.
- Local speech-to-text and small classification tasks are strong candidates because they can reduce latency and data exposure.
- Cloud AI will remain important for large, complex or long-running workloads.
- Hybrid routing can automatically send sensitive tasks locally and demanding tasks to the cloud.
- Users should not have to manage tokens, model selection and prompt files to get useful assistance.
- Local AI introduces new requirements for deployment, patching, encryption, auditing and access control.
- In enterprise environments, measure the full workflow—not just token consumption.
- Faster generation does not automatically equal higher productivity; review, testing and rollback still matter.
- The likely future is invisible intelligence embedded in devices and workflows, with users rarely needing to know which model handled a task.
8. Closing Thoughts
This conversation left us with a slightly different view of the local-versus-cloud debate. It is not a simple choice between two camps, and it is not just an argument about cost. The more useful question is: what is the right model, in the right place, for the right task, with the right controls?
We expect the next phase of AI to be less about choosing the latest model and more about the layer around the models. Routing, context, consent, security and orchestration will determine whether AI feels helpful or frustrating. The models may become interchangeable commodities, while the real intelligence sits in the system that decides when and how to use them.
Lee’s prediction is that the next twelve months will bring many more open-weight models, significant contributions from China and continued progress in local deployment—but not local AGI in everyone’s pocket. Instead, control will gradually move closer to people and their devices, often without users even noticing that a local model is running.
Thanks for joining us for this episode of Impact of AI: Explored. We would love to hear how you are using local models today. Are you experimenting on a laptop, deploying models in an enterprise environment or building hybrid workflows? Join the conversation and let us know what you think the future of local AI should look like.
