For the past few years, much of the enterprise AI conversation has been framed around productivity.
How much time can a Copilot save? How quickly can an employee find information, prepare a document, analyse data, write code, or respond to a customer?
These are reasonable questions. They have shaped the first meaningful wave of generative AI adoption inside organisations.
But I’d argue they are becoming insufficient.
The more important development is that AI systems are beginning to move beyond helping an individual complete a task and towards participating across workflows, applications, and business processes. That matters more than the steady improvement in what any single assistant can do.
That shift takes us from Copilots towards what could reasonably be described as digital workforces.
The distinction matters.
A Copilot primarily raises questions about individual productivity, adoption and appropriate use. A coordinated ecosystem of agents raises larger questions about authority, identity, access, architecture, accountability and control.
The next phase of enterprise AI is therefore not simply a technology deployment challenge.
It is an organisational one.
Productivity Was the Starting Point
The productivity case for AI is well established.
Organisations have introduced generative AI into knowledge work to help employees research information, summarise meetings, create content, analyse data, and reduce repetitive effort.
The next transition changes the unit of analysis.
Rather than asking how AI can help a person perform an individual activity, organisations are beginning to consider how agents can execute sequences of activities, interact with systems, and coordinate an outcome.
Microsoft’s 2025 Work Trend Index provides one indication of the direction. Its research described a progression from AI assistants, through agents completing tasks under human direction, towards environments in which people set direction while agents run larger portions of workflows and business processes.
Importantly, this is unlikely to be a neat or universal progression. Microsoft itself notes that the journey is not strictly linear.
There will probably be no single moment when an organisation can declare that it has moved from Copilots to a digital workforce.
An employee may use a Copilot to prepare a briefing while another part of the organisation uses an agent for research, another uses traditional workflow automation, and a more mature business function coordinates several agents across an operational process.
The important distinction is not the label attached to the technology.
It is how much authority the system has, how many organisational boundaries it can cross, and what happens when it acts.
The Shift Is from Assistance to Coordination
A productivity assistant is relatively easy to conceptualise.
There is a person. There is a task. There is an AI capability that helps the person complete it.
The accountability chain remains reasonably visible.
Coordinated agents make the picture more complicated.
Imagine an agent that identifies a supply issue. Another evaluates alternatives. A third examines relevant contractual information. Another initiates an approval workflow. Enterprise systems provide information throughout the process, while people become involved when defined conditions or exceptions occur.
Whatever technology sits underneath that scenario, the management problem has changed.
The unit being designed is no longer simply a tool.
It is a system of work.
The significance of this shift is sometimes reduced to autonomy. I think coordination is just as important.
The more systems, data sources, decisions and agents involved in pursuing an outcome, the more dependent that outcome becomes on the architecture and governance connecting them.
An individual Copilot can improve a task.
A coordinated agent ecosystem can change the process.
And when the process changes, the organisation around it has to change as well.
A Digital Workforce Needs an Operating Model
This is where terminology can become misleading.
Calling agents a “digital workforce” may suggest that organisations simply need to accommodate a new category of worker.
But agents are not employees.
Treating them as digital equivalents of people risks obscuring the more important governance problem.
Employees operate within a dense organisational structure. There are reporting lines, delegations, policies, approval authorities, professional standards, and consequences for inappropriate actions.
Those mechanisms have developed over decades.
Digital agents do not inherit that environment automatically.
Their boundaries must be designed.
An organisation considering greater agent autonomy therefore needs answers to some practical questions:
- Who owns each agent?
- What objective has it been authorised to pursue?
- Which identities and credentials does it use?
- Which applications can it access?
- What information can it retrieve?
- What actions can it initiate?
- What decisions can it make without approval?
- What requires human authorisation?
- How are exceptions handled?
- How are actions logged and monitored?
- Who can suspend the agent?
- What happens when several agents contribute to an undesirable outcome?
These are not model-performance questions.
They are operating model questions.
In my experience, they are also far easier to answer before an agent reaches production than after it.
Established AI governance frameworks increasingly approach the challenge in similar organisational terms. Risk management, accountability, lifecycle governance, and continual improvement cannot simply be added after an AI capability has been deployed. They need to form part of the environment in which that capability operates.
This becomes increasingly important as organisations scale.
Scale magnifies whatever operating model sits underneath it.
If ownership is clear, access is appropriately bounded and controls are effective, scale can create significant value.
If accountability is fragmented, permissions are excessive or processes are poorly understood, coordination can magnify those weaknesses as well.
Architecture Becomes Part of Governance
Enterprise architecture is sometimes treated as a technical discipline separate from governance.
Digital workforces make that separation increasingly difficult to sustain.
Consider identity.
An agent acting across business systems needs a mechanism through which its authority is established. That immediately raises questions about authentication, authorisation, privilege, segregation of duties and lifecycle management.
Consider data.
An agent may require information from multiple systems to complete an objective. That introduces questions about classification, confidentiality, lineage and whether access that is appropriate for an employee in one context should automatically translate into machine access in another.
Consider integration.
Agents may invoke APIs, applications, workflows, or other agents. Each connection potentially creates another pathway through which information and authority can travel.
Architecture choices therefore shape the control environment as much as they shape technical performance.
This becomes particularly important with agentic systems because an AI capability connected to tools and enterprise applications is no longer simply generating a response.
It may be capable of acting.
That means security, identity, data governance and enterprise architecture increasingly converge around the same question:
What is this agent authorised to do?
Architecture diagrams will therefore need to tell executives more than where systems connect.
They must increasingly help answer:
Where does authority travel, where are decisions made, and where can the organisation intervene?
That is architecture as governance.
Human Oversight Has to Be Designed
One response to increasing AI autonomy is to insist that there will always be a human in the loop.
The intention is sensible.
The phrase is less useful than it first appears.
If every action requires human approval, much of the value of autonomous execution disappears.
If people are asked to approve such a large number of machine-generated decisions that approval becomes routine, oversight may become ceremonial rather than meaningful.
The better question is not whether a human appears somewhere in the workflow.
It is where human judgement is necessary, and what that person needs to make an informed decision.
Different processes will require different answers.
A low-risk internal administrative activity may justify broad automation within a tightly bounded environment.
An action involving significant financial authority, sensitive information, safety or a regulated decision may require explicit human approval.
Some situations may require an agent to stop when policy, confidence, or operational thresholds are breached.
Others may benefit from retrospective assurance rather than intervention on every transaction.
The control model should follow the consequences. I’d suggest starting there, with the consequences of an action, rather than with the technology performing it.
This principle appears consistently across international and Australian responsible AI guidance. Accountability, transparency, safety, and appropriate human oversight remain central even as the technology itself becomes more capable.
For leaders, the implication is important.
Human oversight is an operating model design task, not an abstract principle attached to an AI strategy.
The New Capability Gap
There is an interesting contradiction developing in enterprise AI.
The technology is becoming easier to access while the organisational capabilities required to use it responsibly are becoming more demanding.
Microsoft’s 2025 research illustrates the intensity of executive interest. It found that 81 per cent of surveyed leaders expected agents to be moderately or extensively integrated into their company’s AI strategy within the next 12 to 18 months. Nearly half (45 per cent) said expanding team capacity with digital labour was a top priority over the same period, second only to upskilling existing employees (47 per cent). The survey covered 31,000 workers across 31 markets, including Australia and New Zealand.
Those figures should be interpreted for what they are: vendor research indicating executive sentiment rather than an independent market forecast.
But they point to an important question.
Is organisational capability developing at the same pace as technical capability?
An organisation may be technically capable of building agents while remaining operationally unprepared to govern them.
It may have AI expertise but unclear ownership of processes.
It may have sophisticated models but fragmented identity controls.
It may have automation capability without mature exception management.
It may have extensive data but insufficient understanding of how that data influences automated decisions.
It may have an AI policy while lacking a practical operating model for assigning accountability when an agent acts.
This is the capability gap that I believe deserves more executive attention.
The ability to deploy AI and the ability to govern AI are related, but they are different capabilities, and they rarely mature at the same pace.
Governance Before Scale
This leads to a broader principle.
The appropriate level of governance should follow the authority and consequences of the system.
Not every agent requires the same controls.
A tightly bounded research agent retrieving approved public information is materially different from an agent capable of updating customer records, committing expenditure or initiating activity across production systems.
Governance that ignores this distinction can fail in two directions.
One is over-control.
Low-risk use cases become unnecessarily cumbersome, reducing legitimate experimentation and creating incentives for people to work around established processes.
The other is under-control.
Higher-authority agents are treated like productivity assistants even though the consequences of their actions are materially different.
A more mature approach is to classify agents and agent-enabled processes according to characteristics such as:
- autonomy
- data sensitivity
- privileges
- business criticality
- reversibility of actions
- financial or operational impact
- regulatory significance
- requirement for human judgement
- potential impact when multiple agents interact
Controls can then follow the risk and authority profile. In practice, I’d suggest starting with a small number of these characteristics and refining over time, rather than waiting for a perfect taxonomy.
This is consistent with established approaches to AI risk management. It is also much closer to how enterprises already think about identity, cybersecurity, information management, and operational risk.
Digital workforce governance does not require organisations to abandon those disciplines. It requires them to connect those disciplines around a new form of participant in enterprise processes.
From AI Readiness to Organisational Readiness
Perhaps the biggest implication is that the definition of AI readiness needs to change.
During the first phase of generative AI adoption, readiness often revolved around access to models, data quality, security, acceptable-use policies, training and experimentation.
Those foundations remain important.
But readiness for digital workforces goes further.
It asks whether the organisation understands its processes well enough to delegate parts of them.
Whether decision rights are sufficiently clear to encode.
Whether identity and access architecture can constrain machine actors appropriately.
Whether operations teams can observe agent activity.
Whether people know when to intervene.
Whether leaders can establish accountability across a process in which human and machine participants both contribute.
And whether the organisation can stop, investigate and recover when something goes wrong.
This is where the broader development of AI governance becomes important.
International standards increasingly treat AI through organisational management and risk management systems rather than evaluating individual applications in isolation.
Australian guidance is moving in a similar direction. The Guidance for AI Adoption, published by the National AI Centre within the Department of Industry, Science and Resources in October 2025, sets out six essential practices, from deciding who is accountable through to maintaining human control. Policy and guidance for government from the Digital Transformation Agency, including material addressing agentic AI, provides a further signal.
Principles remain necessary.
But principles eventually must become operating mechanisms.
Who is accountable for each outcome?
What authority has been delegated?
How is access constrained?
What is monitored?
When does a person intervene?
How does the organisation respond when the system behaves in a way that was not expected?
These questions go beyond responsible AI. They are indicators of organisational maturity.
The organisations that deploy the largest number of agents will not necessarily be the most advanced.
The stronger signal will be whether they can increase autonomy without losing accountability.
Executive Implications
The transition from Copilots to digital workforces should prompt leadership teams to examine some uncomfortable questions.
What assumptions are being challenged?
Is the organisation still treating AI primarily as an individual productivity initiative?
If agents begin participating in processes rather than merely assisting employees, productivity metrics alone will not describe success.
Leaders will need to consider process outcomes, reliability, control, resilience, and the quality of human-machine coordination.
How does accountability change?
Who owns an outcome when several agents, applications, and people have contributed to it?
Execution may become distributed.
Accountability cannot.
Ownership needs to remain identifiable from the business outcome through to the process and the systems acting within it.
How must governance evolve?
Policies are necessary, but they are not sufficient.
Governance must be reflected in architecture, permissions, decision rights, monitoring, escalation rules, and operational controls.
The test is what the organisation has actually authorised an agent to do, and whether that authority is visible and enforced.
What capability gaps exist?
Before expanding agent deployment, leaders should examine whether the organisation can:
- inventory agents and assign accountable owners;
- classify them according to authority and risk;
- control identities, permissions and data access;
- observe agent activity across workflows;
- maintain appropriate audit records;
- detect exceptions and unintended behaviour;
- introduce meaningful human intervention points;
- suspend access when necessary; and
- recover safely when automated activity fails.
An organisation unable to answer those questions has not necessarily discovered an obstacle to innovation.
It has discovered a governance requirement.
What leadership actions are required?
Digital workforces cannot become the exclusive responsibility of the CIO, CISO, HR function, or an AI centre of excellence.
Technology leaders need to establish secure and observable foundations.
Risk and governance leaders need to determine appropriate control expectations.
Business leaders need to own the outcomes being delegated.
HR and organisational leaders need to consider how roles, skills, and operating models change.
Boards and executives ultimately need confidence that increasing machine autonomy does not create decreasing organisational accountability.
Leadership Checkpoint
Before your organisation expands the authority of its agents, I’d suggest the leadership team can answer these questions with confidence:
1. Can we name an accountable business owner for every agent operating in our environment today, and for the outcome it contributes to?
2. Have we classified our agents by authority and consequence, so that controls are proportionate rather than uniform?
3. If an agent acted outside its intended boundaries tomorrow, could we detect it, suspend it, reconstruct what happened and recover safely?
4. Where have we deliberately designed human judgement into our most important processes, and do the people in those roles have what they need to exercise it meaningfully?
5. Are identity, data and integration architecture decisions being reviewed as governance decisions, or only as technical design?
6. Are we measuring success by process outcomes, reliability and control, rather than individual productivity alone?
Final Thought: The Real Transition
The progression from Copilots to digital workforces will produce impressive demonstrations. Demonstrations are the easy test.
The real test comes when an agent is given access to sensitive information, authority to act across systems, responsibility for part of an important workflow, or the ability to coordinate with other agents.
At that point, the conversation changes.
The question is no longer:
What can AI do?
It becomes:
What should AI be authorised to do, under whose accountability, within which boundaries, and with what capacity for the organisation to observe and intervene?
That is a much harder question.
It is also the question leadership teams need to answer before scale makes the decision for them.
Digital workforces will require more than better AI. They will require clearer organisational architecture, mature governance, deliberate human oversight, and stronger operational discipline.
The opportunity is significant.
But sustainable advantage will come from building the organisational capability to coordinate agents responsibly, rather than from deploying more of them than everyone else.
That is the shift from AI adoption to AI operating maturity, and it is where leadership accountability begins.
Let’s build wisely.
References
5. Organisation for Economic Co-operation and Development. OECD AI Principles. Updated May 2024.
8. Australian Government Digital Transformation Agency. Artificial Intelligence in Government.
9. OWASP Foundation. OWASP GenAI Security Project.
10. MITRE. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. https://atlas.mitre.org/
11. Microsoft. 2025 Work Trend Index Annual Report: The Year the Frontier Firm Is Born. 23 April 2025.
