# items

added_at: 2025-11-30 18:12:57
thumb_url: https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent.png
url:
https://www.windowslatest.com/2025/11/30/microsoft-says-ai-agents-are-risky-but-its-moving-ahead-with-the-plan-on-windows-11/

title:
Microsoft admits AI agents can hallucinate and fall for attacks, but they’re still coming to Windows 11

content_html:
<p>For the past few weeks, Microsoft has been associating <a href="https://www.windowslatest.com/2025/11/14/windows-11-agentic-os-ai-upgrade-faces-backlash-microsoft-responds-by-closing-replies/" rel="noopener noreferrer" target="_blank">AI agents with the future of Windows</a>. But the company’s own documentation openly admits that such agents can hallucinate, act unpredictably, and even fall for attacks that didn’t exist a year ago. Yet, the fourth-largest organization is still pushing ahead with agentic features in Windows 11.</p>
<p>If Microsoft believes these agents are risky enough to need separate accounts, isolated sessions, and tamper-evident audit logs, why is Windows 11 becoming the test bed for them? And why now, at a time when users are already exhausted by the AI-fication of the OS?</p>
<h2>Microsoft’s big bet on agentic computing is already locked in</h2>
<p>In mid-October 2025, Microsoft said that they are “making every Windows 11 PC an AI PC.” The company unveiled a wave of AI integrations meant to let you “talk” to your computer, show it what’s on your screen, and then have it act on your behalf.</p>
<p>Microsoft essentially wants you to replace keystrokes and mouse clicks with natural language, and we got to see a preview of this plan with <a href="https://www.windowslatest.com/2025/02/06/microsofts-copilot-voice-ai-is-expanding-beyond-english-to-take-on-chatgpt-gemini/" rel="noopener noreferrer" target="_blank">Copilot Voice</a>, <a href="https://www.windowslatest.com/2025/08/07/windows-11s-built-in-copilot-vision-that-can-see-your-screen-now-works-for-free-everywhere-hands-on/" rel="noopener noreferrer" target="_blank">Copilot Vision</a>, and the agentic part, Copilot Actions.</p>
<p>The latest moves make the Windows 11 taskbar the nerve centre of this AI-fication. Windows 11’s Search box is being replaced (optional, for now) with a new “<a href="https://www.windowslatest.com/2025/10/19/microsoft-cant-fix-windows-11-search-so-its-handing-it-to-ask-copilot-on-the-taskbar/" rel="noopener noreferrer" target="_blank">Ask Copilot</a>” interface that lets you summon AI agents or Copilot with a single click or type. From there, agents can run tasks in the background, and you can monitor their progress directly from the taskbar, as if they were regular apps.</p>
<figure><img alt="Invoking agent from Ask Copilot in Taskbar" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 2216px) 100vw, 2216px" src="https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent.png" srcset="https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent.png 2216w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-300x169.png 300w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-1024x576.png 1024w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-768x432.png 768w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-1536x864.png 1536w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-2048x1152.png 2048w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-696x391.png 696w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-1068x601.png 1068w, https://www.windowslatest.com/wp-content/uploads/2025/11/Ask-Copilot-on-taskbar-add-agent-747x420.png 747w"/><figcaption>Invoking agent from Ask Copilot in Taskbar. Credit: Microsoft</figcaption></figure>
<p>Even if today the agentic functionality is limited and opt-in, the architecture and roadmap clear the air around the fact that agentic computing is the next core paradigm for Windows.</p>
<h2>Microsoft openly says AI agents can misbehave, but still wants them inside your files and apps</h2>
<p>On the bright side, Microsoft doesn’t pretend this is safe or foolproof. The company’s official <a href="https://support.microsoft.com/en-au/windows/experimental-agentic-features-a25ede8a-e4c2-4841-85a8-44839191dfb3#wl" rel="noopener" target="_blank">documentation</a> warns that these AI agents <em>“face functional limitations in terms of how they behave and occasionally may hallucinate and produce unexpected outputs.”</em></p>
<h3>Agents are vulnerable to Cross Prompt Injection (XPIA), malicious prompts, and malware</h3>
<p>One of the biggest risks that Microsoft talks of is Cross Prompt Injection (XPIA). It describes a situation where an AI agent gets tricked by malicious content embedded in UI elements, documents, or apps. Such content could potentially override the agent’s original instructions and force it to perform harmful actions like copying sensitive files or leaking data.</p>
<p>Security researchers have already <a href="https://arxiv.org/abs/2504.11281" rel="noopener noreferrer" target="_blank">flagged GUI-based agents</a> as vulnerable to these kinds of indirect attacks, the reason being the high privileges given to such AI Agents.</p>
<p>While we appreciate Microsoft being open about this, there is a certain distrust that pops up, considering all the <a href="https://www.windowslatest.com/2025/11/20/microsoft-says-copilot-codes-faster-than-you-drink-coffee-devs-say-fix-windows-11-first-like-slow-file-explorer/" rel="noopener noreferrer" target="_blank">hatred that Copilot is garnering</a> these days. And if you think <a href="https://www.windowslatest.com/2024/05/21/microsoft-details-windows-11-recall-ai-privacy-security-it-records-screen/" rel="noopener noreferrer" target="_blank">Recall was a privacy nightmare</a>, AI agents are a whole different ballpark.</p>
<p><img alt="Recall in Windows 11 24H2" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 500px) 100vw, 500px" src="https://www.windowslatest.com/wp-content/uploads/2025/09/Recall-in-Windows-11-24H2.jpg" srcset="https://www.windowslatest.com/wp-content/uploads/2025/09/Recall-in-Windows-11-24H2.jpg 786w, https://www.windowslatest.com/wp-content/uploads/2025/09/Recall-in-Windows-11-24H2-255x300.jpg 255w, https://www.windowslatest.com/wp-content/uploads/2025/09/Recall-in-Windows-11-24H2-768x905.jpg 768w, https://www.windowslatest.com/wp-content/uploads/2025/09/Recall-in-Windows-11-24H2-696x820.jpg 696w, https://www.windowslatest.com/wp-content/uploads/2025/09/Recall-in-Windows-11-24H2-357x420.jpg 357w"/></p>
<p>Microsoft insists that agents run under separate accounts, with limited permissions, controlled folder access, and tamper-evident logs. But it still grants these agents read and write access to some of our most personal locations in the PC, specifically Documents, Downloads, Desktop, Videos, Pictures, and Music, which Microsoft calls <a href="https://learn.microsoft.com/en-us/windows/win32/shell/known-folders" rel="noopener noreferrer" target="_blank">known folders</a>.</p>
<p>“…malicious content embedded in UI elements or documents can override agent instructions, leading to unintended actions like data exfiltration or malware installation,” Microsoft warned in a <a href="https://support.microsoft.com/en-us/windows/experimental-agentic-features-a25ede8a-e4c2-4841-85a8-44839191dfb3" rel="noopener" target="_blank">support document</a> published earlier this month. “We recommend you read through this information and understand the security implications of enabling an agent on your computer.”</p>
<p>So, given the risks, if Microsoft wants agents to interact with apps and files like a real person, how exactly does it stop the whole system from collapsing under its own weight?</p>
<h2>The entire thing depends on a new feature called Agent Workspace</h2>
<p>Agent Workspace is the backbone of Microsoft’s vision for an Agentic OS. Everything the company has promised, including the AI that uses apps for you, edits files, moves documents around, and completes multi-step tasks without bothering you, only works because Windows 11 can now create dedicated sessions for these agents to operate in.</p>
<p>It is unlike a virtual machine or <a href="https://www.windowslatest.com/2025/10/27/how-i-installed-microsoft-store-in-windows-sandbox-with-powershell-script/" rel="noopener noreferrer" target="_blank">Windows Sandbox</a>. Agent Workspace is a parallel Windows environment, complete with its own account, its own desktop, its own process tree, and its own permission boundary.</p>
<p>Giving a separate workspace for AI agents is Microsoft’s first attempt at giving them a “place to exist” inside Windows, without letting it sit directly inside the user’s session.</p>
<p>Each agent gets a separate standard account on your PC, and Windows treats this account like a controlled, limited user who can do only the things you explicitly allow. Such restrictions are Microsoft’s response to the same problems they warned about.</p>
<p><strong>How AI agents work inside Windows 11</strong></p>
<p>Inside this workspace, the Agent interacts with applications the same way we do. It can click UI buttons, type into text fields. Scroll through windows, drag files, and do tasks that involve multiple steps. The AI handles the reasoning behind these steps.</p>
<figure><img alt="Copilot operator" loading="lazy" referrerpolicy="no-referrer" src="https://www.windowslatest.com/wp-content/uploads/2025/11/Copilot-operator.gif"/><figcaption>Copilot Actions using Agent Workspace on Windows 11</figcaption></figure>
<p>Copilot Actions already uses this model. Instead of asking a cloud model to generate text, the agent literally performs the steps in software installed on your PC. That’s why Microsoft needs to give it separate Windows sessions.</p>
<p>If an agent misinterprets a prompt or if XPIA is triggered inside a document, the damage will be, technically, contained within a boundary where Windows can supervise and log every action.</p>
<p>Agent Workspace is responsible for deciding what to show to agents. As I mentioned, agents only get access to the six “known folders”. Everything else in the user profile is off-limits, that is, unless you give it access.</p>
<p>This should also stop agents from crawling into system directories, credential stores, or app data folders where unintended reads or writes would cause chaos for app developers. Microsoft also uses Access Control Lists to prevent the agent account from going beyond the permissions of the user who enabled it.</p>
<p>To enable any of this feature, you need to turn on the <strong>Experimental Agentic Features</strong>, which is off by default.</p>
<p><img alt="Experimental agentic features in Windows 11" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 1089px) 100vw, 1089px" src="https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11.jpg" srcset="https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11.jpg 1089w, https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11-300x209.jpg 300w, https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11-1024x715.jpg 1024w, https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11-768x536.jpg 768w, https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11-696x486.jpg 696w, https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11-1068x745.jpg 1068w, https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11-602x420.jpg 602w, https://www.windowslatest.com/wp-content/uploads/2025/11/Experimental-agentic-features-in-Windows-11-100x70.jpg 100w"/></p>
<figure><img alt="Windows 11 Agent Workspace" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 1075px) 100vw, 1075px" src="https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace.jpg" srcset="https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace.jpg 1075w, https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace-300x181.jpg 300w, https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace-1024x617.jpg 1024w, https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace-768x463.jpg 768w, https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace-696x420.jpg 696w, https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace-1068x644.jpg 1068w, https://www.windowslatest.com/wp-content/uploads/2025/11/Windows-11-Agent-Workspace-697x420.jpg 697w"/><figcaption>Image Courtesy: WindowsLatest.com</figcaption></figure>
<p>Microsoft says, <em>“This feature has no AI capabilities on its own, it is a security feature for agents like Copilot Actions. Enabling this toggle allows the creation of a separate agent account and workspace on the device, providing a contained space to keep agent activity separate from the user.” </em></p>
<p><strong>MCP protocol controls what agents can touch</strong></p>
<p>Microsoft is positioning the Model Context Protocol (MCP) as the standardized bridge between agents and applications. That’s how the agent communicates with tools on the system.</p>
<p>MCP allows the agent to discover tools, call functions, read file metadata, and interact with services through a predictable JSON-RPC layer. This prevents any direct access and gives Windows a central enforcement point where authentication, permission to use tools, capability declarations, and logging happen. If it isn’t for the MCP, an agent would be blind. The workspace keeps it within safe limits.</p>
<h2>Why Microsoft believes the risk with AI Agents is worth it?</h2>
<p>From Microsoft’s point of view, stepping back from AI isn’t an option anymore. The company wants people to use AI naturally in Windows to the point that the OS becomes a <em>“canvas for AI”.</em></p>
<p>Apple is hard at work with Apple Intelligence, especially since the plan to use a <a href="https://www.theverge.com/news/814654/apple-intelligence-google-gemini-ai-siri" rel="noopener noreferrer" target="_blank">custom version of Gemini</a>, which brings us to Google already planning to enter the <a href="https://www.business-standard.com/technology/tech-news/google-aluminium-os-android-pc-platform-windows-macos-rival-125112600784_1.html" rel="noopener noreferrer" target="_blank">PC market with Aluminium OS</a>.</p>
<p><a href="https://www.windowslatest.com/2025/11/27/affordable-macbook-could-make-windows-11-devices-cheaper-and-analysts-agree/" rel="noopener noreferrer" target="_blank">Apple’s upcoming budget MacBook</a>, with a full version of Apple Intelligence, will be more appealing to many, just because of the company’s desirability factor. So, if Windows isn’t already prepared, there is a real risk that the platform starts to look boring, all while being hated for the existing issues in Windows 11, like the <a href="https://www.windowslatest.com/2025/11/28/tested-windows-11s-faster-file-explorer-preloaded-is-still-slower-than-windows-10-and-uses-additional-ram/" rel="noopener noreferrer" target="_blank">slow File Explorer</a>.</p>
<p>Large corporations pushing users to try new stuff that eventually gives them millions in ROI isn’t something new, but should you trust Microsoft?</p>
<p>Windows 11 does not have a great reputation to begin with. People already complain about how bloated it feels.</p>
<figure><img alt="Community Notes on X point to the Copilot mistake and recommends the right way to change text size" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 526px) 100vw, 526px" src="https://www.windowslatest.com/wp-content/uploads/2025/11/ac8e405d-26f6-479e-9742-56315508d498.jpeg" srcset="https://www.windowslatest.com/wp-content/uploads/2025/11/ac8e405d-26f6-479e-9742-56315508d498.jpeg 526w, https://www.windowslatest.com/wp-content/uploads/2025/11/ac8e405d-26f6-479e-9742-56315508d498-253x300.jpeg 253w, https://www.windowslatest.com/wp-content/uploads/2025/11/ac8e405d-26f6-479e-9742-56315508d498-355x420.jpeg 355w"/><figcaption>Community Notes on X point to the Copilot mistake and recommends the right way to change text size</figcaption></figure>
<p>Microsoft’s Recall feature has become the textbook example of how not to launch an AI product on a desktop OS. Security researchers, privacy advocates, and regular users all raised the alarm over the idea of constant screenshots of your activity being stored on disk.</p>
<p>The backlash was loud enough that <a href="https://www.windowslatest.com/2024/06/15/windows-11-24h2-kb5038575-removes-microsoft-recall-ai/" rel="noopener noreferrer" target="_blank">Microsoft delayed the feature</a>, reworked it to be opt-in, and still cannot fully shake the “privacy nightmare” label. Even now, privacy-focused apps like Signal, Brave, and AdGuard ship with measures that <a href="https://www.theverge.com/news/713676/brave-adguard-windows-recall-block-microsoft" rel="noopener noreferrer" target="_blank">block Recall out of the box</a>.</p>
<p>All of this context makes people nervous about Windows becoming an agentic OS. If Recall struggled to respect boundaries, what happens when agents can also click, type, and move files around for you?</p>
<h2>Microsoft is building a risky future and hoping users follow</h2>
<p>Microsoft has made its choice to rebuild Windows 11 around AI agents that can do work on your behalf. The company is brave enough to admit the risks, yet confident enough to keep moving forward.</p>
<p>Honestly, on paper, the architecture looks smart. Separate accounts for agents, isolated workspaces, limited folder access, strict logging, and a protocol layer that lets Windows stand between agents and tools. In practice, this will live or die on execution. One serious exploit could undo a lot of the trust Microsoft is trying to rebuild after Recall. At least, the Experimental Agentic features are optional for now.</p>
<p>The uncomfortable truth is that an agentic OS is probably inevitable, and I’m not just talking about Windows. Every major platform vendor is pushing towards a future where AI does more than chat with you.</p>
<p>What is not inevitable is trust. Microsoft will have to earn that, especially from users who already feel like Windows 11 is working against them. If the company wants people to accept AI agents that live inside their personal folders, they will need to start by making everything completely optional, and then giving valid use cases.</p>

content_text:
For the past few weeks, Microsoft has been associating

AI agents with the future of Windows

. But the company’s own documentation openly admits that such agents can hallucinate, act unpredictably, and even fall for attacks that didn’t exist a year ago. Yet, the fourth-largest organization is still pushing ahead with agentic features in Windows 11.

If Microsoft believes these agents are risky enough to need separate accounts, isolated sessions, and tamper-evident audit logs, why is Windows 11 becoming the test bed for them? And why now, at a time when users are already exhausted by the AI-fication of the OS?

Microsoft’s big bet on agentic computing is already locked in

In mid-October 2025, Microsoft said that they are “making every Windows 11 PC an AI PC.” The company unveiled a wave of AI integrations meant to let you “talk” to your computer, show it what’s on your screen, and then have it act on your behalf.

Microsoft essentially wants you to replace keystrokes and mouse clicks with natural language, and we got to see a preview of this plan with

Copilot Voice

,

Copilot Vision

, and the agentic part, Copilot Actions.

The latest moves make the Windows 11 taskbar the nerve centre of this AI-fication. Windows 11’s Search box is being replaced (optional, for now) with a new “

Ask Copilot

” interface that lets you summon AI agents or Copilot with a single click or type. From there, agents can run tasks in the background, and you can monitor their progress directly from the taskbar, as if they were regular apps.

Invoking agent from Ask Copilot in Taskbar. Credit: Microsoft

Even if today the agentic functionality is limited and opt-in, the architecture and roadmap clear the air around the fact that agentic computing is the next core paradigm for Windows.

Microsoft openly says AI agents can misbehave, but still wants them inside your files and apps

On the bright side, Microsoft doesn’t pretend this is safe or foolproof. The company’s official

documentation

warns that these AI agents

“face functional limitations in terms of how they behave and occasionally may hallucinate and produce unexpected outputs.”

Agents are vulnerable to Cross Prompt Injection (XPIA), malicious prompts, and malware

One of the biggest risks that Microsoft talks of is Cross Prompt Injection (XPIA). It describes a situation where an AI agent gets tricked by malicious content embedded in UI elements, documents, or apps. Such content could potentially override the agent’s original instructions and force it to perform harmful actions like copying sensitive files or leaking data.

Security researchers have already

flagged GUI-based agents

as vulnerable to these kinds of indirect attacks, the reason being the high privileges given to such AI Agents.

While we appreciate Microsoft being open about this, there is a certain distrust that pops up, considering all the

hatred that Copilot is garnering

these days. And if you think

Recall was a privacy nightmare

, AI agents are a whole different ballpark.

Microsoft insists that agents run under separate accounts, with limited permissions, controlled folder access, and tamper-evident logs. But it still grants these agents read and write access to some of our most personal locations in the PC, specifically Documents, Downloads, Desktop, Videos, Pictures, and Music, which Microsoft calls

known folders

.

“…malicious content embedded in UI elements or documents can override agent instructions, leading to unintended actions like data exfiltration or malware installation,” Microsoft warned in a

support document

published earlier this month. “We recommend you read through this information and understand the security implications of enabling an agent on your computer.”

So, given the risks, if Microsoft wants agents to interact with apps and files like a real person, how exactly does it stop the whole system from collapsing under its own weight?

The entire thing depends on a new feature called Agent Workspace

Agent Workspace is the backbone of Microsoft’s vision for an Agentic OS. Everything the company has promised, including the AI that uses apps for you, edits files, moves documents around, and completes multi-step tasks without bothering you, only works because Windows 11 can now create dedicated sessions for these agents to operate in.

It is unlike a virtual machine or

Windows Sandbox

. Agent Workspace is a parallel Windows environment, complete with its own account, its own desktop, its own process tree, and its own permission boundary.

Giving a separate workspace for AI agents is Microsoft’s first attempt at giving them a “place to exist” inside Windows, without letting it sit directly inside the user’s session.

Each agent gets a separate standard account on your PC, and Windows treats this account like a controlled, limited user who can do only the things you explicitly allow. Such restrictions are Microsoft’s response to the same problems they warned about.

How AI agents work inside Windows 11

Inside this workspace, the Agent interacts with applications the same way we do. It can click UI buttons, type into text fields. Scroll through windows, drag files, and do tasks that involve multiple steps. The AI handles the reasoning behind these steps.

Copilot Actions using Agent Workspace on Windows 11

Copilot Actions already uses this model. Instead of asking a cloud model to generate text, the agent literally performs the steps in software installed on your PC. That’s why Microsoft needs to give it separate Windows sessions.

If an agent misinterprets a prompt or if XPIA is triggered inside a document, the damage will be, technically, contained within a boundary where Windows can supervise and log every action.

Agent Workspace is responsible for deciding what to show to agents. As I mentioned, agents only get access to the six “known folders”. Everything else in the user profile is off-limits, that is, unless you give it access.

This should also stop agents from crawling into system directories, credential stores, or app data folders where unintended reads or writes would cause chaos for app developers. Microsoft also uses Access Control Lists to prevent the agent account from going beyond the permissions of the user who enabled it.

To enable any of this feature, you need to turn on the

Experimental Agentic Features

, which is off by default.

Image Courtesy: WindowsLatest.com

Microsoft says,

“This feature has no AI capabilities on its own, it is a security feature for agents like Copilot Actions. Enabling this toggle allows the creation of a separate agent account and workspace on the device, providing a contained space to keep agent activity separate from the user.”

MCP protocol controls what agents can touch

Microsoft is positioning the Model Context Protocol (MCP) as the standardized bridge between agents and applications. That’s how the agent communicates with tools on the system.

MCP allows the agent to discover tools, call functions, read file metadata, and interact with services through a predictable JSON-RPC layer. This prevents any direct access and gives Windows a central enforcement point where authentication, permission to use tools, capability declarations, and logging happen. If it isn’t for the MCP, an agent would be blind. The workspace keeps it within safe limits.

Why Microsoft believes the risk with AI Agents is worth it?

From Microsoft’s point of view, stepping back from AI isn’t an option anymore. The company wants people to use AI naturally in Windows to the point that the OS becomes a

“canvas for AI”.

Apple is hard at work with Apple Intelligence, especially since the plan to use a

custom version of Gemini

, which brings us to Google already planning to enter the

PC market with Aluminium OS

.

Apple’s upcoming budget MacBook

, with a full version of Apple Intelligence, will be more appealing to many, just because of the company’s desirability factor. So, if Windows isn’t already prepared, there is a real risk that the platform starts to look boring, all while being hated for the existing issues in Windows 11, like the

slow File Explorer

.

Large corporations pushing users to try new stuff that eventually gives them millions in ROI isn’t something new, but should you trust Microsoft?

Windows 11 does not have a great reputation to begin with. People already complain about how bloated it feels.

Community Notes on X point to the Copilot mistake and recommends the right way to change text size

Microsoft’s Recall feature has become the textbook example of how not to launch an AI product on a desktop OS. Security researchers, privacy advocates, and regular users all raised the alarm over the idea of constant screenshots of your activity being stored on disk.

The backlash was loud enough that

Microsoft delayed the feature

, reworked it to be opt-in, and still cannot fully shake the “privacy nightmare” label. Even now, privacy-focused apps like Signal, Brave, and AdGuard ship with measures that

block Recall out of the box

.

All of this context makes people nervous about Windows becoming an agentic OS. If Recall struggled to respect boundaries, what happens when agents can also click, type, and move files around for you?

Microsoft is building a risky future and hoping users follow

Microsoft has made its choice to rebuild Windows 11 around AI agents that can do work on your behalf. The company is brave enough to admit the risks, yet confident enough to keep moving forward.

Honestly, on paper, the architecture looks smart. Separate accounts for agents, isolated workspaces, limited folder access, strict logging, and a protocol layer that lets Windows stand between agents and tools. In practice, this will live or die on execution. One serious exploit could undo a lot of the trust Microsoft is trying to rebuild after Recall. At least, the Experimental Agentic features are optional for now.

The uncomfortable truth is that an agentic OS is probably inevitable, and I’m not just talking about Windows. Every major platform vendor is pushing towards a future where AI does more than chat with you.

What is not inevitable is trust. Microsoft will have to earn that, especially from users who already feel like Windows 11 is working against them. If the company wants people to accept AI agents that live inside their personal folders, they will need to start by making everything completely optional, and then giving valid use cases.


url: https://www.theverge.com/tech/831379/microsoft-developer-ai-usage-stats-notepad
title: How Microsoft’s developers are using AI | The Verge
added_at: 2025-11-30 20:15:35
content_html:
<p>Microsoft is pitching a future where AI controls everything on your PC and agents go and do work for you in the background. But before the company gets there, it has to build the tools to make these systems work and convince its own developers that AI is actually capable of achieving these big promises.</p><p>Microsoft CEO<strong> Satya Nadella</strong> <a href="https://www.theverge.com/news/658584/up-to-30-percent-of-some-microsoft-code-is-now-written-by-ai" rel="noopener noreferrer" target="_blank">revealed earlier this year</a> that up to 30 percent of the code of “some of our projects” is written by AI, and I’ve been eager to learn exactly how Microsoft’s developers are using the technology ever since. I’ve been speaking to sources and company execs to get a better idea of how AI is being used by Microsoft developers. Some employees have told me they’re skeptical that AI agents will be able to fully replace the work of humans, leaving developers to fix the mistakes of automated agents.</p><p>When I ask the company for more specifics, though, Microsoft touts its early success in deploying AI internally.</p><p>“We want to really look at where there’s developer toil, where we have inefficiencies,” says <strong>Amanda Silver</strong>, a CVP in Microsoft’s CoreAI team who leads product for the company’s Apps &amp; Agents platform, in an interview with <em>Notepad</em>. “Part of what we’re looking at is both how we can apply [AI] and where we can apply it.”</p><p>There are over 100,000 code repositories inside Microsoft, from brand new projects to legacy codebases that are more than 20 years old and still up and running. “We have pretty much every programming language, architecture, and lifecycle stage that you can imagine, and this really reflects a lot of our customers,” says Silver.</p>

content_text:
Microsoft is pitching a future where AI controls everything on your PC and agents go and do work for you in the background. But before the company gets there, it has to build the tools to make these systems work and convince its own developers that AI is actually capable of achieving these big promises.

Microsoft CEO

Satya Nadella

revealed earlier this year

that up to 30 percent of the code of “some of our projects” is written by AI, and I’ve been eager to learn exactly how Microsoft’s developers are using the technology ever since. I’ve been speaking to sources and company execs to get a better idea of how AI is being used by Microsoft developers. Some employees have told me they’re skeptical that AI agents will be able to fully replace the work of humans, leaving developers to fix the mistakes of automated agents.

When I ask the company for more specifics, though, Microsoft touts its early success in deploying AI internally.

“We want to really look at where there’s developer toil, where we have inefficiencies,” says

Amanda Silver

, a CVP in Microsoft’s CoreAI team who leads product for the company’s Apps & Agents platform, in an interview with

Notepad

. “Part of what we’re looking at is both how we can apply [AI] and where we can apply it.”

There are over 100,000 code repositories inside Microsoft, from brand new projects to legacy codebases that are more than 20 years old and still up and running. “We have pretty much every programming language, architecture, and lifecycle stage that you can imagine, and this really reflects a lot of our customers,” says Silver.

thumb_url:
https://platform.theverge.com/wp-content/uploads/sites/2/2025/02/STK095_MICROSOFT_2_CVirginia_B.jpg?quality=90&strip=all&crop=0%2C10.732984293194%2C100%2C78.534031413613&w=1200


added_at: 2025-12-09 22:37:05
thumb_url: https://miro.medium.com/v2/resize:fit:1200/1*1KDv-77ZwlKWoh6397MC7w.png
url:
https://medium.com/@jonataneduard/building-a-real-time-on-device-speech-to-text-in-swiftui-with-whisper-core-ml-ios-17-b1d468e44f4d

title:
Building a Real-Time On-Device Speech-to-Text in SwiftUI with Whisper + Core ML (iOS 17+) | by Jonatan Ortiz | Medium

content_html:
<h1>Building a Real-Time On-Device Speech-to-Text in SwiftUI with Whisper + Core ML (iOS 17+)</h1><figure>Press enter or click to view image in full size</figure><p>In this guide we’ll build a SwiftUI feature that transcribes speech <strong>fully on-device</strong> using <a href="https://github.com/ggml-org/whisper.cpp" rel="noopener ugc nofollow" target="_blank">Whisper (ggml)</a> with a <strong>Core ML encoder</strong> for ANE acceleration. We will: prepare a clean Python environment, download models, generate the Core ML encoder, add everything to Xcode, wire up a low-latency audio pipeline with AVAudioEngine, and decode streaming text in SwiftUI.</p><p><strong>What you’ll get</strong></p><ul><li>A reusable SpeechToTextViewModel that:</li><li>Captures mic audio and normalizes to <strong>16 kHz mono Float32</strong>.</li><li>Uses a <strong>sliding window</strong> to trigger streaming transcription.</li><li>Runs Whisper (ggml decoder) + Core ML encoder, publishing text to SwiftUI.</li><li>A minimal SpeechToTextView for demo + a didactic in-app guide view.</li></ul><h2>Why on-device Whisper with Core ML?</h2><ul><li><strong>Privacy</strong>: No audio leaves the device.</li><li><strong>Latency</strong>: Low end-to-end latency when you keep the model local and the encoder on ANE.</li><li><strong>Offline</strong>: Works without network access.</li><li><strong>Portability</strong>: Ship a consistent UX across devices.</li></ul><h2>What we are building</h2><p>We will implement:</p><ul><li>SpeechToTextViewModel — handles audio capture, normalization to 16 kHz mono Float32, frame windowing, inference triggers, and Whisper decoding.</li><li>SpeechToTextView — small SwiftUI screen that shows live transcription and a Start/Stop button.</li></ul><h2>Requirements</h2><ul><li>Xcode 15+ and iOS 17+ (physical device recommended; Simulator audio behavior differs).</li><li>Apple Silicon Mac (recommended) for Core ML model generation and testing.</li><li>Enough free storage to include the model files in your app target.</li></ul><p><strong>Xcode target settings</strong></p><ul><li>Add to <strong>Info.plist</strong>:</li><li>NSMicrophoneUsageDescription — reason string for microphone access.</li></ul><h2>Step 0 — Create a clean Python environment (macOS)</h2><blockquote><p><em>We’ll use Python </em><strong><em>3.11</em></strong><em> for best compatibility with coremltools and the conversion scripts.</em></p></blockquote><pre><br/>python3.11 -m venv whisper311<br/>source whisper311/bin/activate<br/><br/><br/>python -m pip install --upgrade pip<br/><br/><br/>pip install coremltools==8.3.0 openai-whisper ane_transformers<br/><br/><br/><br/></pre><blockquote><p><em>If you prefer Conda, create conda create -n whisper311 python=3.11 -y then conda activate whisper311.</em></p></blockquote><h2>Step 1 — Get Whisper.cpp and the ggml model</h2><p>We will use <strong>whisper.cpp</strong> utilities to download the ggml model and generate a Core ML encoder.</p><pre><br/>git clone https://github.com/ggml-org/whisper.cpp<br/>cd whisper.cpp<br/><br/><br/>./models/download-ggml-model.sh tiny.en</pre><p>This will place a .bin file under whisper.cpp/models, e.g.:</p><ul><li>models/ggml-tiny.en.bin (English only)</li><li>or models/ggml-tiny.bin (multilingual)</li></ul><blockquote><p><strong><em>Tip</em></strong><em>: If you plan to speak Portuguese in your demo, use the </em><strong><em>multilingual</em></strong><em> model (tiny, base, …). English-only (*.en) is optimized for English input.</em></p></blockquote><h2>Step 2 — Generate the Core ML encoder (ANE acceleration)</h2><p>The decoder remains ggml; the <strong>encoder</strong> can run on <strong>Core ML / ANE</strong> for a solid speed-up.</p><pre><br/><br/>./models/generate-coreml-model.sh tiny.en<br/><br/></pre><p>Expected output (example):</p><pre>done converting<br/>models/coreml-encoder-tiny.en.mlmodelc<br/>models/ggml-tiny.en-encoder.mlmodelc</pre><p><strong>You need both files in Xcode</strong>:</p><ul><li>ggml-tiny.en.bin (or ggml-tiny.bin)</li><li>ggml-tiny.en-encoder.mlmodelc (or ggml-tiny-encoder.mlmodelc)</li></ul><blockquote><p><strong><em>Note</em></strong><em>: You do </em><strong><em>not</em></strong><em> need to build the whisper.cpp CLI (-DWHISPER_COREML=1) for iOS. We are only using its scripts to produce the Core ML encoder.</em></p></blockquote><h2>Step 3 — Add models to Xcode</h2><p>Create a folder (e.g. Resources/models/) and drag:</p><ul><li>ggml-tiny.en.bin (or ggml-tiny.bin)</li><li>ggml-tiny.en-encoder.mlmodelc (or ggml-tiny-encoder.mlmodelc)</li></ul><p>Check:</p><ul><li><strong>Copy items if needed</strong> ✅</li><li><strong>Add to targets</strong> ✅ (your app target)</li><li>Do <strong>not</strong> mark as “Remove on Install.”</li></ul><h2>Step 4 — Add SwiftWhisper via SPM</h2><p>In Xcode:</p><ul><li><strong>File &gt; Add Packages…</strong></li><li>Search for <strong>SwiftWhisper</strong> (the Whisper wrapper for Swift).</li><li>Add the package to your app target.</li></ul><blockquote><p><em>If Xcode requests GitHub authentication and you use 2FA, set up a Personal Access Token (PAT) and choose HTTPS. Once added, you should not need to re-enter credentials.</em></p></blockquote><h2>Step 5 — Implement the Swift side</h2><h2>SpeechToTextViewModel.swift</h2><ul><li>Captures mic audio with AVAudioEngine.</li><li>Normalizes to 16 kHz mono Float32 (downmix + resample if necessary).</li><li>Accumulates frames and triggers inference with a sliding window.</li><li>Uses SwiftWhisper to transcribe chunks asynchronously.</li><li>Publishes text to SwiftUI.</li></ul><pre>import SwiftUI<br/>import SwiftWhisper<br/>import AVFoundation<br/><br/><br/><br/><br/><br/><br/><br/>final class SpeechToTextViewModel: ObservableObject {<br/><br/>    <br/>    @Published var transcription: String = ""<br/>    @Published var isRecording: Bool = false<br/><br/>    <br/>    private var whisper: Whisper?<br/><br/>    <br/>    private let audioEngine = AVAudioEngine()<br/>    private var inputNode: AVAudioInputNode?<br/><br/>    <br/>    private let targetSampleRate: Double = 16_000<br/>    private let windowSeconds: Double = 3.0<br/>    private let hopSeconds: Double = 1.0<br/>    private let minFirstTriggerSeconds: Double = 2.0<br/>    private let minDecibelsFullScaleToTranscribe: Double = -45.0<br/><br/>    private var accumulatedFrames: [Float] = []<br/>    private var isRunningInference: Bool = false<br/>    private var lastInferenceTime: TimeInterval = 0<br/>    private var hasTriggeredOnce: Bool = false<br/><br/>    <br/><br/>    init() {<br/>        let binaryCandidates = ["ggml-tiny.en", "ggml-tiny", "tiny.en", "tiny"]<br/>        let encoderCandidates = ["ggml-tiny.en-encoder", "ggml-tiny-encoder"]<br/>        let dirs = ["", "models", "Model", "Sample/Model"]<br/><br/>        guard<br/>            let binURL = Self.locateResource(<br/>                basenames: binaryCandidates,<br/>                ext: "bin",<br/>                subdirectories: dirs<br/>            ),<br/>            let _ = Self.locateResource(<br/>                basenames: encoderCandidates,<br/>                ext: "mlmodelc",<br/>                subdirectories: dirs<br/>            )<br/>        else {<br/>            DispatchQueue.main.async {<br/>                self.transcription = "Whisper model files were not found in the app bundle."<br/>            }<br/>            return<br/>        }<br/><br/>        whisper = Whisper(fromFileURL: binURL)<br/>    }<br/><br/>    <br/><br/>    func requestPermissions() {<br/>        if #available(iOS 17.0, *) {<br/>            AVAudioApplication.requestRecordPermission { _ in }<br/>        } else {<br/>            AVAudioSession.sharedInstance().requestRecordPermission { _ in }<br/>        }<br/>    }<br/><br/>    <br/><br/>    func toggleRecording() {<br/>        isRecording ? stopRecording() : startRecording()<br/>    }<br/><br/>    <br/><br/>    @discardableResult<br/>    private func configureAudioSession() -&gt; Bool {<br/>        let session = AVAudioSession.sharedInstance()<br/><br/>        do {<br/>            try session.setCategory(<br/>                .playAndRecord,<br/>                mode: .voiceChat,<br/>                options: [<br/>                    .duckOthers,<br/>                    .defaultToSpeaker,<br/>                    .allowBluetooth<br/>                ]<br/>            )<br/>        } catch {  }<br/><br/>        do {<br/>            try session.setPreferredSampleRate(targetSampleRate)<br/>        } catch {  }<br/><br/>        do {<br/>            try session.setPreferredIOBufferDuration(0.01)<br/>        } catch {  }<br/><br/>        do {<br/>            try session.setActive(<br/>                true,<br/>                options: .notifyOthersOnDeactivation<br/>            )<br/>        } catch {<br/>            return false<br/>        }<br/><br/>        do {<br/>            try session.setPreferredInputNumberOfChannels(1)<br/>        } catch {  }<br/><br/>        return true<br/>    }<br/><br/>    <br/><br/>    func startRecording() {<br/>        requestPermissions()<br/>        guard configureAudioSession() else { return }<br/><br/>        let input = audioEngine.inputNode<br/>        inputNode = input<br/><br/>        let inputFormat = input.outputFormat(forBus: 0)<br/>        let sr = inputFormat.sampleRate<br/>        let ch = Int(inputFormat.channelCount)<br/><br/>        let tapBufferSize: AVAudioFrameCount = 8_192<br/>        let windowFrames = Int(windowSeconds * targetSampleRate)<br/><br/>        <br/>        accumulatedFrames.removeAll(keepingCapacity: true)<br/>        isRunningInference = false<br/>        lastInferenceTime = 0<br/>        hasTriggeredOnce = false<br/><br/>        let alreadyTarget =<br/>            abs(sr - targetSampleRate) &lt; 0.5 &amp;&amp;<br/>            ch == 1 &amp;&amp;<br/>            inputFormat.commonFormat == .pcmFormatFloat32<br/><br/>        if alreadyTarget {<br/>            <br/>            input.installTap(<br/>                onBus: 0,<br/>                bufferSize: tapBufferSize,<br/>                format: inputFormat<br/>            ) { [weak self] buffer, _ in<br/>                guard let self = self else { return }<br/>                let count = Int(buffer.frameLength)<br/>                guard count &gt; 0, let ptr = buffer.floatChannelData?[0] else { return }<br/><br/>                self.accumulatedFrames.append(<br/>                    contentsOf: UnsafeBufferPointer(start: ptr, count: count)<br/>                )<br/>                if self.accumulatedFrames.count &gt; windowFrames * 2 {<br/>                    let drop = self.accumulatedFrames.count - windowFrames * 2<br/>                    self.accumulatedFrames.removeFirst(drop)<br/>                }<br/>                self.maybeTriggerInference(<br/>                    windowFrameCount: windowFrames,<br/>                    contextTag: "fast-path"<br/>                )<br/>            }<br/>        } else {<br/>            <br/>            guard<br/>                let monoFormat = AVAudioFormat(<br/>                    commonFormat: .pcmFormatFloat32,<br/>                    sampleRate: sr,<br/>                    channels: 1,<br/>                    interleaved: false<br/>                ),<br/>                let outFormat = AVAudioFormat(<br/>                    commonFormat: .pcmFormatFloat32,<br/>                    sampleRate: targetSampleRate,<br/>                    channels: 1,<br/>                    interleaved: false<br/>                ),<br/>                let downmix = AVAudioConverter(from: inputFormat, to: monoFormat),<br/>                let resample = AVAudioConverter(from: monoFormat, to: outFormat)<br/>            else { return }<br/><br/>            let ratio = targetSampleRate / sr<br/><br/>            input.installTap(<br/>                onBus: 0,<br/>                bufferSize: tapBufferSize,<br/>                format: inputFormat<br/>            ) { [weak self] inputBuffer, _ in<br/>                guard let self = self else { return }<br/>                let inFrames = Int(inputBuffer.frameLength)<br/>                guard inFrames &gt; 0 else { return }<br/><br/>                <br/>                guard<br/>                    let monoPCM = AVAudioPCMBuffer(<br/>                        pcmFormat: monoFormat,<br/>                        frameCapacity: AVAudioFrameCount(inFrames)<br/>                    )<br/>                else { return }<br/><br/>                var downErr: NSError?<br/>                var fed = false<br/>                _ = downmix.convert(<br/>                    to: monoPCM,<br/>                    error: &amp;downErr<br/>                ) { _, outStatus in<br/>                    if fed || inputBuffer.frameLength == 0 {<br/>                        outStatus.pointee = .noDataNow<br/>                        return nil<br/>                    }<br/>                    fed = true<br/>                    outStatus.pointee = .haveData<br/>                    return inputBuffer<br/>                }<br/>                if downErr != nil { return }<br/>                let monoFrames = Int(monoPCM.frameLength)<br/>                guard monoFrames &gt; 0 else { return }<br/><br/>                <br/>                let estOut = Int(ceil(Double(monoFrames) * ratio))<br/>                let capacity = max(monoFrames, estOut)<br/><br/>                guard<br/>                    let outPCM = AVAudioPCMBuffer(<br/>                        pcmFormat: outFormat,<br/>                        frameCapacity: AVAudioFrameCount(capacity)<br/>                    )<br/>                else { return }<br/><br/>                var resErr: NSError?<br/>                var given = false<br/>                _ = resample.convert(<br/>                    to: outPCM,<br/>                    error: &amp;resErr<br/>                ) { _, outStatus in<br/>                    if given || monoPCM.frameLength == 0 {<br/>                        outStatus.pointee = .noDataNow<br/>                        return nil<br/>                    }<br/>                    given = true<br/>                    outStatus.pointee = .haveData<br/>                    return monoPCM<br/>                }<br/>                if resErr != nil { return }<br/>                let produced = Int(outPCM.frameLength)<br/>                guard produced &gt; 0, let ptr = outPCM.floatChannelData?[0] else { return }<br/><br/>                self.accumulatedFrames.append(<br/>                    contentsOf: UnsafeBufferPointer(start: ptr, count: produced)<br/>                )<br/>                if self.accumulatedFrames.count &gt; windowFrames * 2 {<br/>                    let drop = self.accumulatedFrames.count - windowFrames * 2<br/>                    self.accumulatedFrames.removeFirst(drop)<br/>                }<br/>                self.maybeTriggerInference(<br/>                    windowFrameCount: windowFrames,<br/>                    contextTag: "convert-path"<br/>                )<br/>            }<br/>        }<br/><br/>        do {<br/>            try audioEngine.start()<br/>            isRecording = true<br/>        } catch {<br/>            <br/>        }<br/>    }<br/><br/>    func stopRecording() {<br/>        audioEngine.stop()<br/>        inputNode?.removeTap(onBus: 0)<br/>        isRecording = false<br/>        accumulatedFrames.removeAll(keepingCapacity: false)<br/>        isRunningInference = false<br/>        hasTriggeredOnce = false<br/>        try? AVAudioSession.sharedInstance().setActive(<br/>            false,<br/>            options: .notifyOthersOnDeactivation<br/>        )<br/>    }<br/><br/>    <br/><br/>    private func maybeTriggerInference(<br/>        windowFrameCount: Int,<br/>        contextTag: String<br/>    ) {<br/>        let now = Date.timeIntervalSinceReferenceDate<br/>        let minFirst = Int(minFirstTriggerSeconds * targetSampleRate)<br/><br/>        let hasFull = accumulatedFrames.count &gt;= windowFrameCount<br/>        let enoughForFirst = accumulatedFrames.count &gt;= minFirst || hasTriggeredOnce<br/>        let debounced = (now - lastInferenceTime) &gt;= hopSeconds<br/>        let canTrigger = !isRunningInference &amp;&amp; enoughForFirst &amp;&amp; debounced<br/>        guard canTrigger else { return }<br/><br/>        let desired = hasFull ? windowFrameCount : max(accumulatedFrames.count, minFirst)<br/>        let start = max(0, accumulatedFrames.count - desired)<br/>        let frames = Array(accumulatedFrames[start..&lt;accumulatedFrames.count])<br/><br/>        <br/>        let (_, dbFS) = Self.audioLevelMetrics(for: frames)<br/>        guard dbFS &gt;= minDecibelsFullScaleToTranscribe else { return }<br/><br/>        lastInferenceTime = now<br/>        isRunningInference = true<br/><br/>        let localWhisper = whisper<br/>        let copyFrames = frames<br/><br/>        Task.detached { [weak self, copyFrames, localWhisper] in<br/>            guard let self = self else { return }<br/>            guard let w = localWhisper else {<br/>                await MainActor.run { self.isRunningInference = false }<br/>                return<br/>            }<br/><br/>            do {<br/>                let segments = try await w.transcribe(audioFrames: copyFrames)<br/>                let text = segments<br/>                    .map(\.text)<br/>                    .joined(separator: " ")<br/>                    .trimmingCharacters(in: .whitespacesAndNewlines)<br/><br/>                await MainActor.run {<br/>                    if !text.isEmpty {<br/>                        self.transcription += self.transcription.isEmpty ? text : " \(text)"<br/>                        self.hasTriggeredOnce = true<br/>                    }<br/>                    self.isRunningInference = false<br/>                }<br/>            } catch {<br/>                await MainActor.run {<br/>                    self.isRunningInference = false<br/>                }<br/>            }<br/>        }<br/>    }<br/><br/>    <br/><br/>    private static func audioLevelMetrics(for frames: [Float]) -&gt; (Double, Double) {<br/>        guard !frames.isEmpty else { return (0.0, -120.0) }<br/><br/>        var sumSquares: Double = 0<br/>        let step = max(1, frames.count / 8_000)<br/>        var count = 0<br/>        var i = 0<br/><br/>        while i &lt; frames.count {<br/>            let s = Double(frames[i])<br/>            sumSquares += s * s<br/>            count += 1<br/>            i += step<br/>        }<br/><br/>        guard count &gt; 0 else { return (0.0, -120.0) }<br/>        let rms = sqrt(sumSquares / Double(count))<br/>        let dbFS = 20.0 * log10(max(rms, 1e-8))<br/>        return (rms, dbFS)<br/>    }<br/><br/>    private static func locateResource(<br/>        basenames: [String],<br/>        ext: String,<br/>        subdirectories: [String]<br/>    ) -&gt; URL? {<br/>        let bundle = Bundle.main<br/><br/>        for base in basenames {<br/>            for dir in subdirectories {<br/>                if dir.isEmpty, let url = bundle.url(forResource: base, withExtension: ext) {<br/>                    return url<br/>                }<br/>                if let url = bundle.url(forResource: base, withExtension: ext, subdirectory: dir) {<br/>                    return url<br/>                }<br/>            }<br/>        }<br/><br/>        if let root = bundle.resourceURL {<br/>            for base in basenames {<br/>                for dir in subdirectories {<br/>                    let candidate = dir.isEmpty<br/>                    ? root.appendingPathComponent("\(base).\(ext)")<br/>                    : root.appendingPathComponent(dir).appendingPathComponent("\(base).\(ext)")<br/>                    if FileManager.default.fileExists(atPath: candidate.path) {<br/>                        return candidate<br/>                    }<br/>                }<br/>            }<br/>        }<br/>        return nil<br/>    }<br/>}</pre><h2>SpeechToTextView.swift</h2><pre>import SwiftUI<br/><br/>struct SpeechToTextView: View {<br/>    @StateObject private var viewModel = SpeechToTextViewModel()<br/><br/>    var body: some View {<br/>        VStack(alignment: .leading, spacing: 16) {<br/>            Title("Speech-to-Text with Core ML")<br/>            BodyText(<br/>                "This example uses Whisper via Core ML to transcribe your speech in real time."<br/>            )<br/><br/>            DividerLine()<br/><br/>            Subtitle("Live Transcription")<br/>            ScrollView {<br/>                Text(viewModel.transcription)<br/>                    .font(.system(size: 16))<br/>                    .padding()<br/>                    .frame(<br/>                        maxWidth: .infinity,<br/>                        alignment: .leading<br/>                    )<br/>                    .background(Color.backgroundSurface)<br/>                    .cornerRadius(10)<br/>            }<br/><br/>            Button(action: {<br/>                viewModel.toggleRecording()<br/>            }) {<br/>                Text(<br/>                    viewModel.isRecording<br/>                    ? "Stop Recording"<br/>                    : "Start Recording"<br/>                )<br/>                .fontWeight(.semibold)<br/>                .frame(maxWidth: .infinity)<br/>                .padding()<br/>                .background(Color.accentColor)<br/>                .foregroundColor(.white)<br/>                .cornerRadius(10)<br/>            }<br/>            .accessibilityLabel(<br/>                viewModel.isRecording<br/>                ? "Stop Recording"<br/>                : "Start Recording"<br/>            )<br/>        }<br/>        .padding(20)<br/>        .onAppear { viewModel.requestPermissions() }<br/>    }<br/>}</pre><h2>Performance tuning</h2><ul><li><strong>Model choice</strong>: tiny is small but less accurate; base is a good middle ground. If your input is Portuguese, use <strong>multilingual</strong> (tiny, base) rather than English-only (*.en).</li><li><strong>Window length</strong>: Increase windowSeconds for more context (accuracy) at the cost of latency.</li><li><strong>Hop size</strong>: Increase hopSeconds to reduce compute frequency on constrained devices.</li><li><strong>Silence gate</strong>: Adjust minDecibelsFullScaleToTranscribe (e.g., from −45 dBFS to −55 dBFS) if you need more sensitivity.</li><li><strong>ANE warm-up</strong>: First run may be slower due to Core ML compilation on device.</li></ul>

content_text:
Building a Real-Time On-Device Speech-to-Text in SwiftUI with Whisper + Core ML (iOS 17+)

Press enter or click to view image in full size

In this guide we’ll build a SwiftUI feature that transcribes speech

fully on-device

using

Whisper (ggml)

with a

Core ML encoder

for ANE acceleration. We will: prepare a clean Python environment, download models, generate the Core ML encoder, add everything to Xcode, wire up a low-latency audio pipeline with AVAudioEngine, and decode streaming text in SwiftUI.

What you’ll get

A reusable SpeechToTextViewModel that:

Captures mic audio and normalizes to

16 kHz mono Float32

.

Uses a

sliding window

to trigger streaming transcription.

Runs Whisper (ggml decoder) + Core ML encoder, publishing text to SwiftUI.

A minimal SpeechToTextView for demo + a didactic in-app guide view.

Why on-device Whisper with Core ML?

Privacy

: No audio leaves the device.

Latency

: Low end-to-end latency when you keep the model local and the encoder on ANE.

Offline

: Works without network access.

Portability

: Ship a consistent UX across devices.

What we are building

We will implement:

SpeechToTextViewModel — handles audio capture, normalization to 16 kHz mono Float32, frame windowing, inference triggers, and Whisper decoding.

SpeechToTextView — small SwiftUI screen that shows live transcription and a Start/Stop button.

Requirements

Xcode 15+ and iOS 17+ (physical device recommended; Simulator audio behavior differs).

Apple Silicon Mac (recommended) for Core ML model generation and testing.

Enough free storage to include the model files in your app target.

Xcode target settings

Add to

Info.plist

:

NSMicrophoneUsageDescription — reason string for microphone access.

Step 0 — Create a clean Python environment (macOS)

We’ll use Python

3.11

for best compatibility with coremltools and the conversion scripts.

python3.11 -m venv whisper311

source whisper311/bin/activate

python -m pip install --upgrade pip

pip install coremltools==8.3.0 openai-whisper ane_transformers

If you prefer Conda, create conda create -n whisper311 python=3.11 -y then conda activate whisper311.

Step 1 — Get Whisper.cpp and the ggml model

We will use

whisper.cpp

utilities to download the ggml model and generate a Core ML encoder.

git clone https://github.com/ggml-org/whisper.cpp

cd whisper.cpp

./models/download-ggml-model.sh tiny.en

This will place a .bin file under whisper.cpp/models, e.g.:

models/ggml-tiny.en.bin (English only)

or models/ggml-tiny.bin (multilingual)

Tip

: If you plan to speak Portuguese in your demo, use the

multilingual

model (tiny, base, …). English-only (*.en) is optimized for English input.

Step 2 — Generate the Core ML encoder (ANE acceleration)

The decoder remains ggml; the

encoder

can run on

Core ML / ANE

for a solid speed-up.

./models/generate-coreml-model.sh tiny.en

Expected output (example):

done converting

models/coreml-encoder-tiny.en.mlmodelc

models/ggml-tiny.en-encoder.mlmodelc

You need both files in Xcode

:

ggml-tiny.en.bin (or ggml-tiny.bin)

ggml-tiny.en-encoder.mlmodelc (or ggml-tiny-encoder.mlmodelc)

Note

: You do

not

need to build the whisper.cpp CLI (-DWHISPER_COREML=1) for iOS. We are only using its scripts to produce the Core ML encoder.

Step 3 — Add models to Xcode

Create a folder (e.g. Resources/models/) and drag:

ggml-tiny.en.bin (or ggml-tiny.bin)

ggml-tiny.en-encoder.mlmodelc (or ggml-tiny-encoder.mlmodelc)

Check:

Copy items if needed

✅

Add to targets

✅ (your app target)

Do

not

mark as “Remove on Install.”

Step 4 — Add SwiftWhisper via SPM

In Xcode:

File > Add Packages…

Search for

SwiftWhisper

(the Whisper wrapper for Swift).

Add the package to your app target.

If Xcode requests GitHub authentication and you use 2FA, set up a Personal Access Token (PAT) and choose HTTPS. Once added, you should not need to re-enter credentials.

Step 5 — Implement the Swift side

SpeechToTextViewModel.swift

Captures mic audio with AVAudioEngine.

Normalizes to 16 kHz mono Float32 (downmix + resample if necessary).

Accumulates frames and triggers inference with a sliding window.

Uses SwiftWhisper to transcribe chunks asynchronously.

Publishes text to SwiftUI.

import SwiftUI

import SwiftWhisper

import AVFoundation

final class SpeechToTextViewModel: ObservableObject {

@Published var transcription: String = ""

@Published var isRecording: Bool = false

private var whisper: Whisper?

private let audioEngine = AVAudioEngine()

private var inputNode: AVAudioInputNode?

private let targetSampleRate: Double = 16_000

private let windowSeconds: Double = 3.0

private let hopSeconds: Double = 1.0

private let minFirstTriggerSeconds: Double = 2.0

private let minDecibelsFullScaleToTranscribe: Double = -45.0

private var accumulatedFrames: [Float] = []

private var isRunningInference: Bool = false

private var lastInferenceTime: TimeInterval = 0

private var hasTriggeredOnce: Bool = false

init() {

let binaryCandidates = ["ggml-tiny.en", "ggml-tiny", "tiny.en", "tiny"]

let encoderCandidates = ["ggml-tiny.en-encoder", "ggml-tiny-encoder"]

let dirs = ["", "models", "Model", "Sample/Model"]

guard

let binURL = Self.locateResource(

basenames: binaryCandidates,

ext: "bin",

subdirectories: dirs

),

let _ = Self.locateResource(

basenames: encoderCandidates,

ext: "mlmodelc",

subdirectories: dirs

)

else {

DispatchQueue.main.async {

self.transcription = "Whisper model files were not found in the app bundle."

}

return

}

whisper = Whisper(fromFileURL: binURL)

}

func requestPermissions() {

if #available(iOS 17.0, *) {

AVAudioApplication.requestRecordPermission { _ in }

} else {

AVAudioSession.sharedInstance().requestRecordPermission { _ in }

}

}

func toggleRecording() {

isRecording ? stopRecording() : startRecording()

}

@discardableResult

private func configureAudioSession() -> Bool {

let session = AVAudioSession.sharedInstance()

do {

try session.setCategory(

.playAndRecord,

mode: .voiceChat,

options: [

.duckOthers,

.defaultToSpeaker,

.allowBluetooth

]

)

} catch {  }

do {

try session.setPreferredSampleRate(targetSampleRate)

} catch {  }

do {

try session.setPreferredIOBufferDuration(0.01)

} catch {  }

do {

try session.setActive(

true,

options: .notifyOthersOnDeactivation

)

} catch {

return false

}

do {

try session.setPreferredInputNumberOfChannels(1)

} catch {  }

return true

}

func startRecording() {

requestPermissions()

guard configureAudioSession() else { return }

let input = audioEngine.inputNode

inputNode = input

let inputFormat = input.outputFormat(forBus: 0)

let sr = inputFormat.sampleRate

let ch = Int(inputFormat.channelCount)

let tapBufferSize: AVAudioFrameCount = 8_192

let windowFrames = Int(windowSeconds * targetSampleRate)

accumulatedFrames.removeAll(keepingCapacity: true)

isRunningInference = false

lastInferenceTime = 0

hasTriggeredOnce = false

let alreadyTarget =

abs(sr - targetSampleRate) < 0.5 &&

ch == 1 &&

inputFormat.commonFormat == .pcmFormatFloat32

if alreadyTarget {

input.installTap(

onBus: 0,

bufferSize: tapBufferSize,

format: inputFormat

) { [weak self] buffer, _ in

guard let self = self else { return }

let count = Int(buffer.frameLength)

guard count > 0, let ptr = buffer.floatChannelData?[0] else { return }

self.accumulatedFrames.append(

contentsOf: UnsafeBufferPointer(start: ptr, count: count)

)

if self.accumulatedFrames.count > windowFrames * 2 {

let drop = self.accumulatedFrames.count - windowFrames * 2

self.accumulatedFrames.removeFirst(drop)

}

self.maybeTriggerInference(

windowFrameCount: windowFrames,

contextTag: "fast-path"

)

}

} else {

guard

let monoFormat = AVAudioFormat(

commonFormat: .pcmFormatFloat32,

sampleRate: sr,

channels: 1,

interleaved: false

),

let outFormat = AVAudioFormat(

commonFormat: .pcmFormatFloat32,

sampleRate: targetSampleRate,

channels: 1,

interleaved: false

),

let downmix = AVAudioConverter(from: inputFormat, to: monoFormat),

let resample = AVAudioConverter(from: monoFormat, to: outFormat)

else { return }

let ratio = targetSampleRate / sr

input.installTap(

onBus: 0,

bufferSize: tapBufferSize,

format: inputFormat

) { [weak self] inputBuffer, _ in

guard let self = self else { return }

let inFrames = Int(inputBuffer.frameLength)

guard inFrames > 0 else { return }

guard

let monoPCM = AVAudioPCMBuffer(

pcmFormat: monoFormat,

frameCapacity: AVAudioFrameCount(inFrames)

)

else { return }

var downErr: NSError?

var fed = false

_ = downmix.convert(

to: monoPCM,

error: &downErr

) { _, outStatus in

if fed || inputBuffer.frameLength == 0 {

outStatus.pointee = .noDataNow

return nil

}

fed = true

outStatus.pointee = .haveData

return inputBuffer

}

if downErr != nil { return }

let monoFrames = Int(monoPCM.frameLength)

guard monoFrames > 0 else { return }

let estOut = Int(ceil(Double(monoFrames) * ratio))

let capacity = max(monoFrames, estOut)

guard

let outPCM = AVAudioPCMBuffer(

pcmFormat: outFormat,

frameCapacity: AVAudioFrameCount(capacity)

)

else { return }

var resErr: NSError?

var given = false

_ = resample.convert(

to: outPCM,

error: &resErr

) { _, outStatus in

if given || monoPCM.frameLength == 0 {

outStatus.pointee = .noDataNow

return nil

}

given = true

outStatus.pointee = .haveData

return monoPCM

}

if resErr != nil { return }

let produced = Int(outPCM.frameLength)

guard produced > 0, let ptr = outPCM.floatChannelData?[0] else { return }

self.accumulatedFrames.append(

contentsOf: UnsafeBufferPointer(start: ptr, count: produced)

)

if self.accumulatedFrames.count > windowFrames * 2 {

let drop = self.accumulatedFrames.count - windowFrames * 2

self.accumulatedFrames.removeFirst(drop)

}

self.maybeTriggerInference(

windowFrameCount: windowFrames,

contextTag: "convert-path"

)

}

}

do {

try audioEngine.start()

isRecording = true

} catch {

}

}

func stopRecording() {

audioEngine.stop()

inputNode?.removeTap(onBus: 0)

isRecording = false

accumulatedFrames.removeAll(keepingCapacity: false)

isRunningInference = false

hasTriggeredOnce = false

try? AVAudioSession.sharedInstance().setActive(

false,

options: .notifyOthersOnDeactivation

)

}

private func maybeTriggerInference(

windowFrameCount: Int,

contextTag: String

) {

let now = Date.timeIntervalSinceReferenceDate

let minFirst = Int(minFirstTriggerSeconds * targetSampleRate)

let hasFull = accumulatedFrames.count >= windowFrameCount

let enoughForFirst = accumulatedFrames.count >= minFirst || hasTriggeredOnce

let debounced = (now - lastInferenceTime) >= hopSeconds

let canTrigger = !isRunningInference && enoughForFirst && debounced

guard canTrigger else { return }

let desired = hasFull ? windowFrameCount : max(accumulatedFrames.count, minFirst)

let start = max(0, accumulatedFrames.count - desired)

let frames = Array(accumulatedFrames[start..<accumulatedFrames.count])

let (_, dbFS) = Self.audioLevelMetrics(for: frames)

guard dbFS >= minDecibelsFullScaleToTranscribe else { return }

lastInferenceTime = now

isRunningInference = true

let localWhisper = whisper

let copyFrames = frames

Task.detached { [weak self, copyFrames, localWhisper] in

guard let self = self else { return }

guard let w = localWhisper else {

await MainActor.run { self.isRunningInference = false }

return

}

do {

let segments = try await w.transcribe(audioFrames: copyFrames)

let text = segments

.map(\.text)

.joined(separator: " ")

.trimmingCharacters(in: .whitespacesAndNewlines)

await MainActor.run {

if !text.isEmpty {

self.transcription += self.transcription.isEmpty ? text : " \(text)"

self.hasTriggeredOnce = true

}

self.isRunningInference = false

}

} catch {

await MainActor.run {

self.isRunningInference = false

}

}

}

}

private static func audioLevelMetrics(for frames: [Float]) -> (Double, Double) {

guard !frames.isEmpty else { return (0.0, -120.0) }

var sumSquares: Double = 0

let step = max(1, frames.count / 8_000)

var count = 0

var i = 0

while i < frames.count {

let s = Double(frames[i])

sumSquares += s * s

count += 1

i += step

}

guard count > 0 else { return (0.0, -120.0) }

let rms = sqrt(sumSquares / Double(count))

let dbFS = 20.0 * log10(max(rms, 1e-8))

return (rms, dbFS)

}

private static func locateResource(

basenames: [String],

ext: String,

subdirectories: [String]

) -> URL? {

let bundle = Bundle.main

for base in basenames {

for dir in subdirectories {

if dir.isEmpty, let url = bundle.url(forResource: base, withExtension: ext) {

return url

}

if let url = bundle.url(forResource: base, withExtension: ext, subdirectory: dir) {

return url

}

}

}

if let root = bundle.resourceURL {

for base in basenames {

for dir in subdirectories {

let candidate = dir.isEmpty

? root.appendingPathComponent("\(base).\(ext)")

: root.appendingPathComponent(dir).appendingPathComponent("\(base).\(ext)")

if FileManager.default.fileExists(atPath: candidate.path) {

return candidate

}

}

}

}

return nil

}

}

SpeechToTextView.swift

import SwiftUI

struct SpeechToTextView: View {

@StateObject private var viewModel = SpeechToTextViewModel()

var body: some View {

VStack(alignment: .leading, spacing: 16) {

Title("Speech-to-Text with Core ML")

BodyText(

"This example uses Whisper via Core ML to transcribe your speech in real time."

)

DividerLine()

Subtitle("Live Transcription")

ScrollView {

Text(viewModel.transcription)

.font(.system(size: 16))

.padding()

.frame(

maxWidth: .infinity,

alignment: .leading

)

.background(Color.backgroundSurface)

.cornerRadius(10)

}

Button(action: {

viewModel.toggleRecording()

}) {

Text(

viewModel.isRecording

? "Stop Recording"

: "Start Recording"

)

.fontWeight(.semibold)

.frame(maxWidth: .infinity)

.padding()

.background(Color.accentColor)

.foregroundColor(.white)

.cornerRadius(10)

}

.accessibilityLabel(

viewModel.isRecording

? "Stop Recording"

: "Start Recording"

)

}

.padding(20)

.onAppear { viewModel.requestPermissions() }

}

}

Performance tuning

Model choice

: tiny is small but less accurate; base is a good middle ground. If your input is Portuguese, use

multilingual

(tiny, base) rather than English-only (*.en).

Window length

: Increase windowSeconds for more context (accuracy) at the cost of latency.

Hop size

: Increase hopSeconds to reduce compute frequency on constrained devices.

Silence gate

: Adjust minDecibelsFullScaleToTranscribe (e.g., from −45 dBFS to −55 dBFS) if you need more sensitivity.

ANE warm-up

: First run may be slower due to Core ML compilation on device.


url: https://medium.com/@craig_32726/agile-is-out-architecture-is-back-7586910ab810
title: Agile is Out, Architecture is Back | by Craig Adam | Medium
added_at: 2025-12-22 13:20:50
thumb_url: https://miro.medium.com/v2/da:true/resize:fit:1200/0*Bn_BbUoA8OKCG9Zm
content_html:
<figure>Press enter or click to view image in full size<figcaption>Photo by <a href="https://unsplash.com/@enginakyurt?utm_source=medium&amp;utm_medium=referral" rel="noopener ugc nofollow" target="_blank">engin akyurt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral" rel="noopener ugc nofollow" target="_blank">Unsplash</a></figcaption></figure><h2>The pendulum swings</h2><p>Software development has always been defined by its extremes. In the early days, we planned everything. Specs were sacred. Architecture diagrams came before a single line of code. And every change felt like steering a cargo ship — slow, bureaucratic, and heavily documented.</p><p>Then came Agile, and the pendulum swung hard in the other direction. We embraced speed, iteration, and imperfection. “Working software over comprehensive documentation” became the battle cry of a new generation. Shipping fast was more important than getting it right the first time. And to be fair, that shift unlocked enormous productivity. It changed the culture of software for good.</p><p>Now, we’re entering a new era — one driven by AI tools that can generate code from a sentence. Tools like GitHub Copilot and Claude Code are reshaping what it means to be a developer. It’s not just about writing code anymore — it’s about designing the environment in which code gets written.</p><p>And that pendulum? It’s swinging back again.</p><p>Not all the way to waterfall, but toward a more intentional practice. One where design and documentation matter again — not for the intern or the next engineer — but for the machine that will generate the next 10,000 lines of code. We’re moving past the era of the “vibe coder” — those who prompt and ship without understanding the implications — and into a world where thoughtful, human-led architecture becomes the cornerstone of high-quality software.</p><p>I once worked under a CTO who was genuinely stunned by how little upfront thinking went into our software delivery. With a team of over a hundred developers, he asked me to solve a deceptively simple problem: get people to spend more time thinking before they started typing. At the time, it may have felt like swimming against the Agile tide. But looking at where the industry is headed now — with AI generating the bulk of the code and developers shifting into roles of system design and oversight — I think he’d be relieved. The pendulum is finally swinging back toward intentionality.</p><p>Because if we let machines vibe without boundaries, we’ll bury ourselves in tech debt. If we design the <em>systems</em> they work within, we can scale faster — and better — than we ever did before.</p><h2>Vibe coding and prompt engineering</h2><blockquote><p>“Just vibe it.”</p></blockquote><p>That phrase — popularized by Andrej Karpathy — is shorthand for a new generation of software development where AI tools do the heavy lifting. Need a React component? Prompt it. API integration? Prompt it. CRUD with pagination, error handling, and a loading state? One good prompt can get you 80% of the way there.</p><p>This is vibe coding: a blend of natural language prompting, AI-assisted scaffolding, and rapid iteration. It feels magical. And for some developers, especially newer ones, it’s the only way they’ve ever coded.</p><p>The appeal is obvious. Vibe coding removes friction. It skips boilerplate. It amplifies your velocity. You can build a prototype in an afternoon that would’ve taken a team days just a few years ago.</p><p>But here’s the problem: code written at the speed of thought tends to age like milk, not wine.</p><p>Vibe coding encourages <strong>shallow</strong> understanding. It prioritizes what looks right now over what will hold up six months from now. Architecture decisions get made implicitly by the model. Patterns get embedded without review. And before long, you’re swimming in generated complexity that no one quite understands — not even the person who shipped it.</p><p>Used responsibly, vibe coding is a superpower. Used recklessly, it’s a one-way ticket to tech debt hell.</p><p>The solution isn’t to slow down — it’s to change who’s holding the steering wheel. We don’t need more vibe coders. We need people thinking about the system the AI is working within. We need architects who can harness the power of vibe coding without being owned by it.</p><h2>Designing frameworks instead of functions</h2><p>The work of writing individual functions is being automated. That’s not speculation — it’s happening in real time. AI can generate a well-formed TypeScript resolver, a GraphQL schema, or a Flutter widget in seconds. The result? The tactical layer of development is becoming commoditized.</p><p>But the strategic layer — that’s still very much a human game.</p><p>Modern software developers are no longer just builders; they’re becoming architects. Not in the enterprise title inflation sense, but in the literal sense: their job is to design the structures within which software is created. That means curating libraries, enforcing boundaries, and defining patterns that allow AI-generated code to integrate cleanly and scale sustainably.</p><p>This is where the future-proof dev lives — not in the loop that renders a component, but in the decisions about what that component should be, how it interacts with the rest of the system, and why it exists at all.</p><p>The shift is subtle but powerful:</p><ul><li>Instead of asking “What’s the best way to implement this endpoint?”, we’re asking “What’s the cleanest contract for this part of the system?”</li><li>Instead of “How do I fix this bug?”, it’s “How do I prevent this class of bug entirely through structure?”</li><li>Instead of spending hours refactoring files, we’re building constraints that prevent bad patterns from showing up in the first place.</li></ul><p>In this new paradigm, the most valuable developers aren’t the ones who write the most code — they’re the ones who write the best systems for code to live in. The frameworks, scaffolds, patterns, and guardrails that allow AI to operate effectively without creating a mess.</p><p>The job isn’t to out-code the machine. It’s to out-think it.</p><figure>Press enter or click to view image in full size<figcaption>Photo by <a href="https://unsplash.com/@apellaes?utm_source=medium&amp;utm_medium=referral" rel="noopener ugc nofollow" target="_blank">Alexandre Pellaes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral" rel="noopener ugc nofollow" target="_blank">Unsplash</a></figcaption></figure><h2>A new audience</h2><p>In the old world, we wrote clean code and thorough documentation for the next developer. You wanted your future teammate — or your future self — to be able to pick things up without tearing their hair out. Good comments, clear structure, naming that made sense.</p><p>Now, the next “developer” is an AI.</p><p>This shift changes everything. Instead of writing for a junior dev who’s ramping up, we’re writing for a model that can generate 500 lines of code in a few seconds — but only if we give it a solid foundation. AI is a pattern-matching machine. It doesn’t reason like a human. It doesn’t ask for clarification. It does exactly what your structure, naming, and examples teach it to do — whether or not that’s what you meant.</p><p>That means our systems need to be legible to the machine. We need:</p><ul><li><strong>Predictable patterns</strong> — so the AI can see what “right” looks like.</li><li><strong>Tight constraints</strong> — so it doesn’t wander into dangerous territory.</li><li><strong>Curated examples</strong> — because what you show the AI becomes what it repeats.</li><li><strong>Clean abstractions</strong> — not just for humans, but for token-hungry models trying to reason across files.</li></ul><p>Think of it this way: we’re not just writing code anymore, we’re designing the <em>training data</em> for our future co-pilot. Every good function, every well-labeled type, every carefully enforced boundary is a breadcrumb the model will follow when it fills in the gaps.</p><p>This isn’t just good engineering hygiene — it’s now core to velocity.</p><p>So yes, we still care about naming and consistency and interfaces. But not just because it helps humans onboard. Now it helps our machine teammates build the next 100 features without creating chaos.</p><p></p><h2>Get Craig Adam’s stories in your inbox</h2><p>Join Medium for free to get updates from this writer.</p><p>That’s the real job of tomorrow’s dev: don’t just make it work — make it obvious, reproducible, and extensible by machines.</p><h2>A new manifesto</h2><p>For two decades, the Agile Manifesto has shaped how teams build software. Its principles rescued us from bloated specs and 18-month waterfall timelines. We stopped writing Word docs and started shipping MVPs. It was a massive improvement.</p><p>But we’ve overcorrected.</p><p>Agile taught us to value “working software over comprehensive documentation.” Fair enough — until working software became synonymous with “just get it done.” In the age of vibe coding and AI-assisted prompts, that principle starts to break down.</p><p>Because the software may be working… but it’s not always understood.</p><p>Now, with AI generating more and more of the codebase, the pendulum swings again. We’re rediscovering the value of the old things: documentation, specifications, guardrails. But not because humans need them — because our AI coworkers do.</p><p>The core values of Agile aren’t invalid, but some of them need reinterpretation. In the new world:</p><ul><li>We might value <strong>comprehensive structure over working software</strong> — because software that works today but collapses tomorrow is a liability.</li><li>We might value <strong>curated systems over individuals and interactions</strong> — because the individuals are increasingly machines.</li><li>We might value <strong>responding to context over responding to change</strong> — because stability and repeatability are what enable rapid iteration, not chaos.</li></ul><p>This isn’t a return to bureaucracy. It’s the emergence of a new agility: one built on thoughtful constraints, strong architectural bones, and machine-optimizable scaffolds.</p><p>Think of it this way: in 2005, the bottleneck was speed. In 2025, the bottleneck is direction.</p><p>Agile got us moving. Now we need a map.</p><figure>Press enter or click to view image in full size<figcaption>Photo by <a href="https://unsplash.com/@aldebarans?utm_source=medium&amp;utm_medium=referral" rel="noopener ugc nofollow" target="_blank">Aldebaran S</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral" rel="noopener ugc nofollow" target="_blank">Unsplash</a></figcaption></figure><h2>The next generation</h2><p>If you’re a senior developer or tech lead today, your role is already shifting — even if your title hasn’t caught up.</p><p>You’re no longer just writing features. You’re defining the environment in which those features get created. That means owning the architecture, enforcing patterns, and designing systems that guide not just human teammates, but increasingly, AI collaborators.</p><p>To lead effectively in this new landscape, here’s where to focus:</p><p><strong>1. Think in Systems, Not Snippets</strong></p><p>Being good at writing code isn’t enough anymore. You need to shape systems:</p><ul><li>Where does this boundary live?</li><li>Which decisions need to be made once and encoded?</li><li>What abstractions will reduce churn over time?</li></ul><p>Codebases now grow faster than ever. Weak structure leads to rapid collapse.</p><p><strong>2. Build Guardrails, Not Just Features</strong></p><p>Set patterns that both humans and machines can follow safely. Use types, linters, test suites, and schemas not just to enforce correctness — but to <em>communicate intent</em>.</p><p>Where you can’t enforce behaviour through automation, enforce it through clear and repeatable structures. Frameworks, not just libraries.</p><p><strong>3. Curate Examples Like a Librarian</strong></p><p>AI tools rely on pattern recognition. Good examples lead to good outputs. Bad ones multiply confusion.</p><p>Your codebase is now a learning environment. Clean it up! Remove contradictory styles. Document intent when it matters. Treat it like you’re preparing training data — because in a way, you are.</p><p><strong>4. Own the Review Layer</strong></p><p>AI can generate functional code. What it can’t do — yet — is make thoughtful architectural tradeoffs.</p><p>That’s your job. Review for coherence, not just correctness. Look for patterns that create long-term complexity. Be a curator of quality, not just a gatekeeper for bugs.</p><p><strong>5. Don’t Be the Genius Developer</strong></p><p>The smartest person in the room isn’t the one solving the most problems — it’s the one preventing problems from happening in the first place.</p><p>Leadership in this context means building systems that scale beyond your individual effort. It means encoding judgment into the system, not keeping it locked in your head.</p><blockquote><p>The next generation of developers won’t be measured by how fast they can ship code — but by how well they shape the systems that allow others to ship safely, sustainably, and intelligently.</p></blockquote><h2>TL;DR</h2><p>The future of software development won’t be defined by who types the fastest. That race is already over — and the machines won.</p><p>But we’re not out of a job. We’re just playing a different one now.</p><p>The most valuable developers of the next decade won’t be vibe coders chasing short-term velocity. They’ll be designers of long-term systems — architects who can harness AI’s raw power while maintaining clarity, cohesion, and direction.</p><p>The pendulum is swinging again. From over-documented specs to “just ship it” chaos, and now toward a new center: intentional systems design, optimized not for humans alone, but for AI collaborators too. We’re not going back to the old way. We’re moving forward to something more sophisticated — more powerful — because it lets us scale without surrendering control.</p><p>In this new world, speed still matters. But direction matters more. Structure matters more. Principles matter more.</p><p>So yes — ship fast. But ship smart. Build systems that last. And write for the next generation of coders: not interns, not coworkers, but intelligent machines trying their best to follow your lead.</p><p>The future of software is not just written. It’s designed.</p>

content_text:
Press enter or click to view image in full size

Photo by

engin akyurt

on

Unsplash

The pendulum swings

Software development has always been defined by its extremes. In the early days, we planned everything. Specs were sacred. Architecture diagrams came before a single line of code. And every change felt like steering a cargo ship — slow, bureaucratic, and heavily documented.

Then came Agile, and the pendulum swung hard in the other direction. We embraced speed, iteration, and imperfection. “Working software over comprehensive documentation” became the battle cry of a new generation. Shipping fast was more important than getting it right the first time. And to be fair, that shift unlocked enormous productivity. It changed the culture of software for good.

Now, we’re entering a new era — one driven by AI tools that can generate code from a sentence. Tools like GitHub Copilot and Claude Code are reshaping what it means to be a developer. It’s not just about writing code anymore — it’s about designing the environment in which code gets written.

And that pendulum? It’s swinging back again.

Not all the way to waterfall, but toward a more intentional practice. One where design and documentation matter again — not for the intern or the next engineer — but for the machine that will generate the next 10,000 lines of code. We’re moving past the era of the “vibe coder” — those who prompt and ship without understanding the implications — and into a world where thoughtful, human-led architecture becomes the cornerstone of high-quality software.

I once worked under a CTO who was genuinely stunned by how little upfront thinking went into our software delivery. With a team of over a hundred developers, he asked me to solve a deceptively simple problem: get people to spend more time thinking before they started typing. At the time, it may have felt like swimming against the Agile tide. But looking at where the industry is headed now — with AI generating the bulk of the code and developers shifting into roles of system design and oversight — I think he’d be relieved. The pendulum is finally swinging back toward intentionality.

Because if we let machines vibe without boundaries, we’ll bury ourselves in tech debt. If we design the

systems

they work within, we can scale faster — and better — than we ever did before.

Vibe coding and prompt engineering

“Just vibe it.”

That phrase — popularized by Andrej Karpathy — is shorthand for a new generation of software development where AI tools do the heavy lifting. Need a React component? Prompt it. API integration? Prompt it. CRUD with pagination, error handling, and a loading state? One good prompt can get you 80% of the way there.

This is vibe coding: a blend of natural language prompting, AI-assisted scaffolding, and rapid iteration. It feels magical. And for some developers, especially newer ones, it’s the only way they’ve ever coded.

The appeal is obvious. Vibe coding removes friction. It skips boilerplate. It amplifies your velocity. You can build a prototype in an afternoon that would’ve taken a team days just a few years ago.

But here’s the problem: code written at the speed of thought tends to age like milk, not wine.

Vibe coding encourages

shallow

understanding. It prioritizes what looks right now over what will hold up six months from now. Architecture decisions get made implicitly by the model. Patterns get embedded without review. And before long, you’re swimming in generated complexity that no one quite understands — not even the person who shipped it.

Used responsibly, vibe coding is a superpower. Used recklessly, it’s a one-way ticket to tech debt hell.

The solution isn’t to slow down — it’s to change who’s holding the steering wheel. We don’t need more vibe coders. We need people thinking about the system the AI is working within. We need architects who can harness the power of vibe coding without being owned by it.

Designing frameworks instead of functions

The work of writing individual functions is being automated. That’s not speculation — it’s happening in real time. AI can generate a well-formed TypeScript resolver, a GraphQL schema, or a Flutter widget in seconds. The result? The tactical layer of development is becoming commoditized.

But the strategic layer — that’s still very much a human game.

Modern software developers are no longer just builders; they’re becoming architects. Not in the enterprise title inflation sense, but in the literal sense: their job is to design the structures within which software is created. That means curating libraries, enforcing boundaries, and defining patterns that allow AI-generated code to integrate cleanly and scale sustainably.

This is where the future-proof dev lives — not in the loop that renders a component, but in the decisions about what that component should be, how it interacts with the rest of the system, and why it exists at all.

The shift is subtle but powerful:

Instead of asking “What’s the best way to implement this endpoint?”, we’re asking “What’s the cleanest contract for this part of the system?”

Instead of “How do I fix this bug?”, it’s “How do I prevent this class of bug entirely through structure?”

Instead of spending hours refactoring files, we’re building constraints that prevent bad patterns from showing up in the first place.

In this new paradigm, the most valuable developers aren’t the ones who write the most code — they’re the ones who write the best systems for code to live in. The frameworks, scaffolds, patterns, and guardrails that allow AI to operate effectively without creating a mess.

The job isn’t to out-code the machine. It’s to out-think it.

Press enter or click to view image in full size

Photo by

Alexandre Pellaes

on

Unsplash

A new audience

In the old world, we wrote clean code and thorough documentation for the next developer. You wanted your future teammate — or your future self — to be able to pick things up without tearing their hair out. Good comments, clear structure, naming that made sense.

Now, the next “developer” is an AI.

This shift changes everything. Instead of writing for a junior dev who’s ramping up, we’re writing for a model that can generate 500 lines of code in a few seconds — but only if we give it a solid foundation. AI is a pattern-matching machine. It doesn’t reason like a human. It doesn’t ask for clarification. It does exactly what your structure, naming, and examples teach it to do — whether or not that’s what you meant.

That means our systems need to be legible to the machine. We need:

Predictable patterns

— so the AI can see what “right” looks like.

Tight constraints

— so it doesn’t wander into dangerous territory.

Curated examples

— because what you show the AI becomes what it repeats.

Clean abstractions

— not just for humans, but for token-hungry models trying to reason across files.

Think of it this way: we’re not just writing code anymore, we’re designing the

training data

for our future co-pilot. Every good function, every well-labeled type, every carefully enforced boundary is a breadcrumb the model will follow when it fills in the gaps.

This isn’t just good engineering hygiene — it’s now core to velocity.

So yes, we still care about naming and consistency and interfaces. But not just because it helps humans onboard. Now it helps our machine teammates build the next 100 features without creating chaos.

Get Craig Adam’s stories in your inbox

Join Medium for free to get updates from this writer.

That’s the real job of tomorrow’s dev: don’t just make it work — make it obvious, reproducible, and extensible by machines.

A new manifesto

For two decades, the Agile Manifesto has shaped how teams build software. Its principles rescued us from bloated specs and 18-month waterfall timelines. We stopped writing Word docs and started shipping MVPs. It was a massive improvement.

But we’ve overcorrected.

Agile taught us to value “working software over comprehensive documentation.” Fair enough — until working software became synonymous with “just get it done.” In the age of vibe coding and AI-assisted prompts, that principle starts to break down.

Because the software may be working… but it’s not always understood.

Now, with AI generating more and more of the codebase, the pendulum swings again. We’re rediscovering the value of the old things: documentation, specifications, guardrails. But not because humans need them — because our AI coworkers do.

The core values of Agile aren’t invalid, but some of them need reinterpretation. In the new world:

We might value

comprehensive structure over working software

— because software that works today but collapses tomorrow is a liability.

We might value

curated systems over individuals and interactions

— because the individuals are increasingly machines.

We might value

responding to context over responding to change

— because stability and repeatability are what enable rapid iteration, not chaos.

This isn’t a return to bureaucracy. It’s the emergence of a new agility: one built on thoughtful constraints, strong architectural bones, and machine-optimizable scaffolds.

Think of it this way: in 2005, the bottleneck was speed. In 2025, the bottleneck is direction.

Agile got us moving. Now we need a map.

Press enter or click to view image in full size

Photo by

Aldebaran S

on

Unsplash

The next generation

If you’re a senior developer or tech lead today, your role is already shifting — even if your title hasn’t caught up.

You’re no longer just writing features. You’re defining the environment in which those features get created. That means owning the architecture, enforcing patterns, and designing systems that guide not just human teammates, but increasingly, AI collaborators.

To lead effectively in this new landscape, here’s where to focus:

1. Think in Systems, Not Snippets

Being good at writing code isn’t enough anymore. You need to shape systems:

Where does this boundary live?

Which decisions need to be made once and encoded?

What abstractions will reduce churn over time?

Codebases now grow faster than ever. Weak structure leads to rapid collapse.

2. Build Guardrails, Not Just Features

Set patterns that both humans and machines can follow safely. Use types, linters, test suites, and schemas not just to enforce correctness — but to

communicate intent

.

Where you can’t enforce behaviour through automation, enforce it through clear and repeatable structures. Frameworks, not just libraries.

3. Curate Examples Like a Librarian

AI tools rely on pattern recognition. Good examples lead to good outputs. Bad ones multiply confusion.

Your codebase is now a learning environment. Clean it up! Remove contradictory styles. Document intent when it matters. Treat it like you’re preparing training data — because in a way, you are.

4. Own the Review Layer

AI can generate functional code. What it can’t do — yet — is make thoughtful architectural tradeoffs.

That’s your job. Review for coherence, not just correctness. Look for patterns that create long-term complexity. Be a curator of quality, not just a gatekeeper for bugs.

5. Don’t Be the Genius Developer

The smartest person in the room isn’t the one solving the most problems — it’s the one preventing problems from happening in the first place.

Leadership in this context means building systems that scale beyond your individual effort. It means encoding judgment into the system, not keeping it locked in your head.

The next generation of developers won’t be measured by how fast they can ship code — but by how well they shape the systems that allow others to ship safely, sustainably, and intelligently.

TL;DR

The future of software development won’t be defined by who types the fastest. That race is already over — and the machines won.

But we’re not out of a job. We’re just playing a different one now.

The most valuable developers of the next decade won’t be vibe coders chasing short-term velocity. They’ll be designers of long-term systems — architects who can harness AI’s raw power while maintaining clarity, cohesion, and direction.

The pendulum is swinging again. From over-documented specs to “just ship it” chaos, and now toward a new center: intentional systems design, optimized not for humans alone, but for AI collaborators too. We’re not going back to the old way. We’re moving forward to something more sophisticated — more powerful — because it lets us scale without surrendering control.

In this new world, speed still matters. But direction matters more. Structure matters more. Principles matter more.

So yes — ship fast. But ship smart. Build systems that last. And write for the next generation of coders: not interns, not coworkers, but intelligent machines trying their best to follow your lead.

The future of software is not just written. It’s designed.


