examining the potential challenges | 10.1163/22131035-14020001
21
the human brain functions and results in decisions.113 What matters most
is whether the decision can be justified with reasons grounded in fact and
law. Some black-box ai systems can produce results but are unable to offer
reasons or provide clear, understandable explanations for the results that were
reached.114 Humans may not give reasons or clear reasons for their decisions, but
they possess the ability to do so. Since the legal system demands transparency
in decision-making, decision-makers can be challenged by affected parties
through appeals and may be compelled to justify their decisions. As a result,
humans can reflect on their decision-making processes and provide reasons.115
With ai black box models, they lack the ability to provide the rationale for
their decisions even if the law imposes this duty. This is a deeper problem than
just refusing to give reasons. This suggests that a full ability to provide reasons
flowing from ai decision-making can be very difficult.
Empirical research has shown that ai systems are unable to accurately
perform the complex legal reasoning required in decision-making and as a
consequence, they cannot provide legally meaningful explanations for their
outcome of a case.116 Kolkman et al. asserts that ai lacks the ability to adapt its
reasoning to the evolving nature of the legal system and lacks the flexibility to
apply exceptions to general rules in novel cases. This is because ai algorithms
tend to learn the most general rules (patterns, correlations) that appear in its
training data, the algorithm cannot exercise discretion, the autonomy to decide
what should be done for each individual case.117 ai algorithms is problematic
because it does not follow rational evaluation or human-like reasoning. As a
result, ai algorithms cannot handle complex matters that impact on individual
lives.118
Explainable ai systems have been developed to provide post hoc explanations
of the black-box model’s outputs. These explanations capture simplifications
of a black-box system’s algorithm decision-making and do not contain all its
technical details. It identifies the most important features that influenced a
113
114
115
116
117
118
John Zerilli and others, ‘Transparency in Algorithmic and Human Decision-making: Is
there a Double Standard?’ (2019) 32(4) Philosophy and Technology 661–683.
R v McCann [2019] uksc 34; Adamantia Rachovitsa and Niclas Johann, ‘The Human
Rights Implications of the Use of ai in the Digital Welfare State: Lessons learned from
the Dutch SyRI Case’ (2022) 22 Human Rights Law Review 1–15; Hassija et al, (n 10) at 46.
Uwe Peters, ‘Explainable ai Lacks Regulative Reasons: Why ai and Human DecisionMaking are not Equally Opaque’ (2023) 3(3) ai and Ethics 968–969.
Daan Kolkman and others, ‘Justitia ex machina: The impact of an ai system on legal
decision-making and discretionary authority’ (2024) 11(2) Big Data & Society 3.
Ibid. Johan Egbert Hans Korteling and others, ‘Human- Versus Artificial Intelligence’
(2021) 4 Front Artif Intell 1–10.
Uwe, (n 115) at 971.
International Human Rights Law Review (2025) 1–31