Alan Turing, a renowned British mathematician, deciphered the Enigma code used by Germany during World War II and devised the ‘Turing Test.’ The test was designed to evaluate if a machine could mimic human-like intelligence. Recent endeavours by an Australian government Senate committee aimed to use the infamous test to judge whether a generative artificial intelligence large language model (GenAI LLM) could equal or surpass the quality of work delivered by human staff.
In cooperation with a consultancy group from Amazon, the committee embarked on a five-week experimental trial. Public submissions addressing a parliamentary investigation were summarised by both a selected LLM and human personnel. These resulting summaries were put through a blind test and examined by five business representatives. The involvement of GenAI was undisclosed to the evaluators.
After the process, it was revealed to the judges about the real nature of the trial and they were queried on the reasons for their rating for each summary. It was at this point, they were informed about the automated summarisation. However, three out of five mentioned that they had speculation of a GenAI test.
Contrary to expectations, the outcome indicated that the GenAI summaries rated lower on all measures when juxtaposed with human summaries by 47% to 81%, thus failing to pass the Turing Test.
The evaluators pointed out that the automated summaries often lacked emphasis, nuance and context. They contained inaccuracies, overlooked relevant information and introduced unrelated commentary. Thus, they concluded that GenAI turned out to be counterproductive, giving rise to additional work due to the necessity of verifying facts and referring back to the original public submissions.
Detailed committee discussions and the comprehensive report can be accessed on a public platform at the given url. Wall Street seems to harbour increasing doubts about the substantial return on investment from GenAI. For investors, the projected ‘revolutionary’ technology remains costly relative to its tangible business impact and has yet to offer any groundbreaking applications for public use.
Microsoft has seen a 75% rise in its capital expenditure from the previous year, with almost all its second-quarter earnings – around $22 billion – being ploughed into investments in cloud and GenAI. Alphabet, the parent company of Google downplays its GenAI investments but acknowledges a significant increase in its capital expenditure this year compared to the previous one. Amazon has spent $30 billion on capital expenditure this current year compared to $48 billion in 2023 and has been comparably reticent about it.
Sam Altman, the CEO of ChatGPT’s OpenAI, has been lobbying the US government to participate with investors to fund a national GenAI infrastructure project. The initiative, which is projected to come at a hefty price of “tens of billions of dollars,” encompasses data centres, power generation, and national grid enhancements.
A contentious report, GenAI: Too much spend, too little benefit?, was released by Goldman Sachs in June (https://www.goldmansachs.com/images/migrated/insights/pages/gs-research/gen-ai–too-much-spend,-too-little-benefit-/TOM_AI%202.0_ForRedaction.pdf). In the report, a variety of analysts deliberated the potential economic profits from GenAI over the next ten years. Goldman Sachs concluded that there remains an opportunity for returns for investors, either due to the future success of GenAI or the gradual slowdown of investments.
Without justifying its substantial investment, GenAI has still been able to intrigue and captivate. New search engines such as Perplexity.ai have made significant progress over Google, despite the operational cost being six to ten times higher. Tools like GitHub Copilot aid routine software development, but generate incorrect code causing frustration. The hyperrealistic pictures created by tools like Flux 1 from Black Forest Labs have considerable promise, particularly for online retail, enabling virtual ‘try-before-you-buy’ for clothes and accessories but their practical application remains unvalidated.
Supporters of GenAI are optimistic, arguing that the technology is still in its nascent stage. They suggest that combining GenAI with autonomous actions could be promising. Such AI agents can plan, execute, and modify tasks based on their past experiences. For instance, an automated holiday planner could plan holidays tailored to the user’s preferences and experiences of previous holiday-goers.
The San Francisco-based start-up Altera has taken an innovative step in artificial intelligence by deploying 1,000 autonomous GenAI agents within the environment of the widely-known game, Minecraft. These agents, functioning independently, have cultivated their own society, complete with a cultural structure, thriving economy, religious practices, and system of governance. This unique, virtual society is showcased in a YouTube synopsis, at this address: https://www.youtube.com/watch?v=2tbaCn0Kl90.
Within this simulation, notable community developments have taken place. A marketplace has emerged, serving as a platform for the trade of goods. Unexpectedly, the figure of highest wealth in this society is the community priest, who has amassed riches through the process of religious conversion. The society has also engaged in the drafting, revising, and democratic acceptance of their own constitution. An additional testament to their societal cohesion is their collective response to citizens going astray, with the illumination of torches to guide them back home.
These actions have been autonomously performed by the AI community without the presence of a preexisting script. These artificial beings exercise independence in their planning, coordination, and actions.
While GenAI does not currently meet the standards set by Turing’s test, this benchmark is inherently focused on drawing comparisons to human intelligence. This could arguably be a somewhat narrow assessment. We are now experiencing a new kind of intelligence that we’re yet to fully comprehend; one that doesn’t easily submit to mathematical analysis or prediction. When such intelligence is allowed to control its own actions entirely, it raises a set of philosophical, ethical, and practical dilemmas.