What should artificial intelligence be allowed to do?
The EU is attempting to regulate AI systems through legislation. However, this proves difficult. The development is moving too quickly, there are too many unanswered questions, and too much can go wrong. It is far from simple to define a legal framework for AI systems. The EU is only now learning.
When discussing the risks posed by so-called artificial intelligence (AI), you may recall the “Chaos GPT” developed in April 2023. This ChatGPT-based application’s goal is to exterminate humanity. However, an AI does not have to be malicious in order to cause harm. The software will be used to support an increasing number of decision-making processes. Machine learning systems may soon be able to decide who gets a job or a loan, make medical diagnoses, or control drones, vehicles, or entire power grids autonomously. But on what basis do they do this, and what rules should the systems follow?
The European Union (EU) is currently attempting to provide answers to these questions. They have been working on artificial intelligence regulation, known as the AI Act, for more than two years. The EU Parliament has already voted on a draft of the rules, and the Commission, Council, and Parliament are now negotiating the final version in a so-called trilogue, which could be adopted before the end of this year.
The task appeared to be relatively simple at the start of the legislative process. A so-called risk class model presented itself as a reasonable solution: the draft divides artificial intelligence systems (the definition of which is still up for debate) based on their intended use and importance. The risk class model imposes little regulation on less risky use cases, whereas riskier purposes, such as discrimination, must meet certain conditions such as transparency and explainability. The draft prohibits applications that are too risky, such as the annihilation of humanity, as well as so-called social scoring and social credit systems, which allow states to analyze their citizens’ behavior and document the consequences.
The world is changing faster than you can make laws
Since the negotiations began, a lot has changed. For example, there was no mention at the time of generative models, which are also the foundation of chatbots like ChatGPT. These so-called foundation models were less well known at the time and, more importantly, had far less public clout. However, the risk class model is hampered by these models, some of which are also referred to as »general purpose AI. There is no unique use case for them that could be classified using the risk class model.
After all, the technology can be used to run a non-threatening chatbot for cooking recipes as well as a chatbot that evaluates application documents and may discriminate against female applicants. Not only that, but generative AI lowers the so-called “barrier to entry” for malicious actors, according to computer scientist Anka Reuel, who studies the effects and risks of AI systems at Stanford University. “Currently, you still need very specific knowledge about hacking and coding to find and exploit vulnerabilities, but that is about to change,” says one expert in cyber security.
Anyone who feeds ChatGPT code and inquires about errors or vulnerabilities will receive responses today. They don’t always make sense, and the system frequently refuses to provide information due to built-in filters, but that’s only the start. As a result, some experts advocated for those Foundation Models to be classified as high-risk AI, while others warned that overly strict rules would stifle competition and slow down those who want to responsibly develop such systems.
The current AI Act compromise requires providers of generative AI models to make transparent where their data comes from, how much energy data processing consumes, which measures are intended to be used to identify and minimize risks, and what the limitations of their models are. Furthermore, AI-generated content should be labeled as such. However, according to a Stanford University study, the providers of these models have only met a few of the required points thus far.
“Who checks whether the rules are being followed?” So there is still a lot to be done before the regulation goes into effect. And some providers, such as the CEO of OpenAI, the company behind ChatGPT, have already threatened to leave Europe if the AI Act is implemented in this manner. Researchers, on the other hand, point out that critical details are still completely unknown, which has a significant impact on the effectiveness of regulation.
According to Ulrike Luxburg, professor of machine learning theory at the University of Tübingen, “the right topics, such as a possible bias in the data or the risk of discriminatory decisions, are addressed there.” The question is, however, how this should be implemented in practice. Among other things, the draft so far requires companies to confirm that appropriate measures have been initiated. “But who controls that?” Luxburg wonders. “Who would have the authority to say that something is not cutting-edge?” The draft still mentions “conformity assessment,” which means that companies must document that their applications meet the relevant criteria to avoid risks.
The requirement in several places in the AI Act for transparency and explainability of decisions made by machine learning systems is also difficult to implement. This is necessary, for example, to protect citizens from the potentially serious consequences of unjustified or incorrect decisions and to allow them to contest them. Technically, however, this is questionable, according to Luxburg: “In deep networks, for example, there are no methods that can be used to explain in a meaningful way how they come to their decisions; it helps to understand what criteria were applied to a decision, but nobody has a chance to understand them because of the many millions of parameters.” Machine learning certainly has explainability methods. These procedures do not explain the legislatively intended sense of justification. According to Luxburg, in computer science, there is a distinction between interpretable and explainable. Interpretable means that there is a decision rule that can be understood using human logic, such as anyone earning less than sum X does not qualify for a loan greater than sum Y. This is precisely what does not work with deep neural networks, i.e., with multi-layered systems on which generative AI models are also based.
Inscrutable decisions with arbitrary justifications
There, you work with explainability, which is especially useful for processes that are incomprehensible using human logic. “Many heuristics are created for this,” Luxburg says, implying that it has been tested to some extent. For example, how does the system decide if you increase a person’s income slightly? Will she be able to get a loan then? Because deep neural networks consider all possible factors when searching for patterns in large amounts of data and humans are unable to understand which path the decision takes, this is the only way to estimate which factors influence the decision and in what form. But is that a legitimate explanation? “You try to explain that afterward,” Luxburg says, “but you don’t understand the decision.”
This is not helpful, particularly in cases where an institution and an affected citizen have different interests: “The interest of the bank would be an explanation with which it does not make itself vulnerable.” For example, the explanation that the person concerned is below a certain income limit and therefore does not get a loan—and not because she is a woman, for example. But the latter could influence the decision of a deep neural network, except that it is not transparent. If the bank both trains the system and is responsible for explaining it, then it cannot be checked whether the explanation given is correct. Furthermore, such subsequently generated explanations are highly arbitrary. Numerous explainability algorithms produce varying results; simply select the algorithm that produces the desired result. “So the bank can give any explanation,” Luxburg explains. That, of course, is not the goal of regulation. “Then you can also omit the requirement for explainability.” If, on the other hand, the AI Act requires a human-readable explanation, deep neural networks should not be used for risky applications. Only models that correspond to humanely comprehensible logic, such as a decision tree, would be called into question.
The human factor
However, Anjali Mazumder, head of AI and Justice and Human Rights at the Alan Turing Institute in London, warns that another focus is missing, namely the focus on the human being at the other end of machine learning systems. While the AI Act includes numerous measures and rules for the development of systems, how they are used is equally important. The most obvious example is the doomsday AI ChaosGPT, which did not decide to wipe out humanity on its own; it was programmed to do so by someone with evil intentions or, at the very least, an odd sense of humor.
Even with desirable goals and tasks for machine learning systems, the question of the role of humans arises. “The way we interact with these systems as humans is becoming increasingly important,” Mazumder says. After all, it is frequently argued that, in the end, it is the human decision that matters, not the machines. However, at the end of the process, a human must be able to make that decision. “In areas like aviation or healthcare, we’ve been discussing automation for a long time, and the people there have undergone long and detailed training before they make decisions.” They understand both the consequences of a wide range of decisions under certain conditions and the weaknesses of AI systems.
Because the potential disruptive factors in AI decisions are increasing, human control at the end of the decision chain is also critical. Long-known issues, such as racial bias in training data, which led to discriminatory decisions, have reached a new level, according to Mazumder. The nature of generative AI itself threatens to exacerbate the problem further: “Not only do we have this problem on a different scale, but we now also run the risk of these systems becoming potentially racist or generating sexist data and images themselves.« This can create a vicious circle. However, given recent advances in generative AI, such as ChatGPT, the trend is for AI systems to be used for decisions by professional groups with little understanding of how the systems work. The hope is that thanks to intuitive language systems, people will be able to perform tasks for which they have no formal training. But will they question the machine’s decisions in the way that is required when the much-touted “human in the loop” is the last line of defense between a machine’s potentially incorrect decision and the people affected? And who is then held accountable for errors?
Who is responsible?
Mazumder contends that responsibility in generative models remains ambiguous due to the numerous levels involved, including building the model, using the data, storing it, and building another system on top. This obscures the issue of accountability. Despite the clarification of responsibility for machine decisions, the question of whether the person has the necessary specialist knowledge to assess and question the results, as well as the necessary information for the various development steps and actors involved remains.
Unfortunately, the EU’s AI Act raises this issue as well. A lack of reliable knowledge makes regulation significantly more difficult; new technology has not yet been thoroughly researched enough to provide an overview and realistically assess the long-term consequences. Anka Reuel, a computer scientist at Stanford University, is currently working on a framework that will address all aspects of the subject. When she shows her document, it is obvious how broad the subject is and how many questions generative AI and its legal regulation raise. Her visualization, which includes keywords and connections, cannot be viewed on a computer screen. She zooms in and out repeatedly to explain which dimensions must be considered. “It depends on how good the information we get from a system is,” she says, “but it also depends on how faulty the systems themselves are.” Privacy and Data Governance: What data goes into the model, and how secure is the data used? How secure are they against external attacks? “
The AI comes into the physical world
Reuel, along with US cognitive scientist and well-known AI critic Gary Marcus, warned in the »Economist« of potential risks posed by AI systems, including existential threats to humanity. But she doesn’t want to be labeled as a doomsday prophet, as some AI critics are in the highly polarized debate. They are currently engaged in a toxic debate with other experts who warn of current problems such as racist bias in systems, which include many big names from tech companies and AI research.
According to Reuel, this debate, whether about existential threats to humanity or solving short-term problems, is not productive. After all, the current issues may become existential. “First and foremost, we would have to debate what it means for a technology to be existentially risky.” In the public debate, this is frequently equated with an AI that develops consciousness, will, and bad intentions.
People are sufficiently quiet for this. A developer had already created “Auto-GPT” as an open-source application in April 2023, a system based on ChatGPT that can update itself and pursue a given goal. It wasn’t long before someone else used the freely available AI system to create the ChaosGPT apocalypse automaton. The system began by looking up the most dangerous and powerful weapon in history on Google and ended up learning about psychology and manipulation. Of course, due to a lack of interfaces to the real world, the system was unable to carry out its plans to wipe out humanity.
People who interact with AI, on the other hand, may act as their extended arm into reality, potentially endangering humanity’s existence or setting in motion other dangerous developments, according to Reuel: “The dangerous thing is that people tend to consider an AI’s decision to be correct, even if they are aware that they are interacting with an AI and that this AI is flawed.”
Even that may not be necessary soon. Increasingly, the systems can directly influence the physical world, such as an AI controlling a robot or drone. “We’re giving the models more and more interfaces to interact in the real world,” Reuel says, “and that’s what concerns me.”
