CSIRO's Insights on Next-Gen AI Agents from M365 Copilot Trial

The CSIRO (Commonwealth Scientific and Industrial Research Organisation) is taking a closer look at the role of next-generation AI agents in workplaces after its trial with M365 Copilot. While CSIRO acknowledges that AI agents can significantly impact organizations, it remains skeptical about whether the Microsoft tool is the suitable embodiment of this future.

Participating in a six-month government pilot program, CSIRO has shared its findings on M365 Copilot, following similar reports from the Treasury. The insights published on arXiv align with broader outcomes observed during the government’s technological trial: while some utility was noted, the product did not entirely match its bold claims, leaving certain users feeling underwhelmed.

According to CSIRO, the researchers stated, “AI copilots are marketed as transformative technologies, but their real-world value depends on several socio-technical factors, including integration with existing workflows, user trust, and the extent to which they align with professional demands.”

In their analysis, CSIRO utilized both quantitative and qualitative methods, with a significant focus on in-depth interviews conducted with 27 trial participants within the agency.

Findings indicate that while M365 Copilot made measurable enhancements in specific tasks such as meeting summarization, drafting emails, and retrieving basic information, the tool struggled in areas that require domain-specific insights, creative problem-solving, and complex decision-making. The researchers explained that users experienced what they termed as a productivity paradox, where the time saved through automation was often counterbalanced by the need to extensively validate and modify AI-generated results.

As organizations reflect on the return on investment associated with AI copilots, it is vital to consider whether these tools truly bolster productivity or merely redirect cognitive challenges elsewhere.”

CSIRO’s unique environment, focused on scientific research rather than typical corporate or administrative settings, posed specific challenges during their use of M365 Copilot.

The researchers highlighted, “The integration of AI solutions in scientific research presents distinct challenges and opportunities.”

On a more positive note, users recognized that M365 Copilot was particularly effective for summarization, drafting documents, organizing meeting-related tasks, and troubleshooting technical issues. They appreciated its ability to condense larger documents or web content into actionable points, assist in generating initial drafts, and streamline workflows through concise meeting summaries and action items.

Despite M365 Copilot not being the ideal solution, both CSIRO and some users emerged from the trial with a sense of optimism regarding the potential of AI agents or assistants. They believe that the next generation of AI systems is on the horizon.

The researchers pointed out, “The rise of AGI artificial general intelligence and AI agents means that the current generation of copilots, including M365 Copilot, will soon be eclipsed by more advanced, autonomous AI assistants.”

They emphasized that, unlike M365 Copilot—which primarily enhances the Microsoft ecosystem—AI agents developed by various firms are advancing toward greater autonomy in decision-making capabilities. This shift prompts organizations to rethink strategies not just about whether to adopt AI copilots, but about how to effectively incorporate AI agents in a manner that aligns with governance, workforce dynamics, and ethical considerations.

As multiple forms of AI systems—capable of understanding text, images, and voice—develop at a rapid pace, organizations need to gear up for a future where AI agents are more intricately integrated into everyday operations alongside human employees than what current copilots permit.

AI, Copilot, Research