When Algorithms Make Decisions, Who Remains Accountable? | RVS CAS MBA

Artificial Intelligence

A recruitment team receives a ranked list of applicants. A lender sees a risk score. A retailer is advised to reduce the price of a slow-moving product. In each situation, an algorithm makes a recommendation that can quickly become an action. The screen looks precise; the consequences extend beyond it. Someone gains an opportunity, loses access or carries a cost. The managerial question is therefore larger than whether the model works: who must answer for what happens next?

Artificial intelligence can improve consistency, identify patterns and help people work through complex information. Yet greater analytical capacity does not remove the need for judgment. It redistributes judgment across decisions about objectives, data, thresholds, workflows and exceptions. Accountability begins by making those decisions visible, including the ones made long before an individual result appears.

The hidden judgment inside a prediction

Imagine a hypothetical lending company using a model to predict repayment difficulty. Its output is a probability, not a complete answer about whether a loan should be offered. Managers still choose the acceptable level of risk, the evidence required for review and the treatment of unusual applicants. They also decide how commercial goals should be balanced against fair access and customer welfare.

A highly accurate model can support a poorly designed process. If staff are rewarded only for approving cases quickly, they may stop examining borderline recommendations. If the system is evaluated only on average performance, recurring errors affecting a smaller customer group may escape attention. Technical quality matters, but the surrounding management choices determine how that quality translates into practice.

This is a problem of construct validity: does the measurable target represent the outcome the organisation actually values? Predicting an observable label accurately is different from establishing that the label deserves to guide a decision. A model can become more reliable at reproducing an inadequate definition of success.

Responsibility travels across the organisation

The data team influences what the model can learn. A procurement team influences which vendor is selected. Business leaders define objectives, while operational managers decide how recommendations enter daily work. Senior management allocates resources for testing, training and oversight. Accountability cannot sensibly be assigned only to the employee who clicks the final button.

The NIST AI Risk Management Framework 1.0 provides a useful foundation. Its core functions are Govern, Map, Measure and Manage. It emphasises documented responsibilities, leadership ownership of risk and continuing evaluation throughout a system’s life. This is a voluntary risk-management framework, rather than a universal legal rule; the managerial value lies in making ownership explicit (NIST, 2023).

For a business, that can mean naming an accountable owner for the overall application and identifying who controls each critical stage. The owner need not write the code. The owner must ensure that the right expertise, evidence and authority are available. Shared participation becomes useful only when responsibilities are clear enough to prevent important work falling between departments.

Human oversight must have substance

Adding a person to a workflow does not automatically make it accountable. Consider a reviewer handling hundreds of recommendations with little explanation, limited time and no permission to challenge the result. Formally, a human is involved. Practically, the reviewer may be functioning as a signature rather than a safeguard.

Meaningful oversight requires information, competence and authority. A reviewer needs to understand what the system is estimating, what it does not know and which circumstances require escalation. There must also be a workable route to pause a decision. An override mechanism that employees fear using is unlikely to offer much protection.

Oversight should match the consequences. A recommendation about arranging a product catalogue may need different controls from a recommendation affecting employment or essential services. The practical question is how much independent scrutiny a decision deserves, given its possible harm, reversibility and uncertainty. Uniform approval procedures can waste effort on minor issues while leaving serious ones insufficiently examined.

Explanations should enable action

An explanation is useful when it helps someone decide what to do. A technical team may need diagnostic measures and model documentation. A frontline manager may need to know which evidence is missing. An affected individual may need a clear reason for the outcome and a way to correct inaccurate information.

These needs are connected, but they are not identical. A list of influential variables may describe a model without explaining the organisation’s decision. A customer should not have to understand statistical architecture to discover that an old address or an incomplete record shaped an outcome.

A practical explanation therefore connects the output to the decision rule and the available remedy. What information was used? What important limitation applies? Who can reconsider the case? Where disclosure is constrained by privacy or security, the organisation still needs a meaningful review process. Accountability concerns the ability to question and correct an outcome, as well as the ability to describe it.

When a defensible proxy creates an indefensible result

Obermeyer and colleagues examined a healthcare algorithm that predicted expenditure as a proxy for need. At the same risk score, Black patients were sicker than White patients. Unequal expenditure on care meant that predicting cost accurately could still misrepresent health needs. The important failure was embedded in the target, rather than explained simply by the presence of race as an input (Obermeyer et al., 2019).

This finding does not establish that every predictive application contains the same disparity. It demonstrates a mechanism managers should investigate: a convenient business measure can encode an existing inequality. In recruitment, for example, prior performance ratings might contain both useful information and the consequences of unequal opportunities. Removing an explicit demographic variable would not, by itself, resolve that concern.

Managers can ask whether errors differ across relevant groups, whether new applicants resemble the population used for testing and whether performance changes after deployment. They can also examine what employees do with recommendations. A system that performs well in a test environment may be used differently when teams face commercial pressure.

A further distinction is between prediction and intervention. A model may predict who usually succeeds without identifying who would benefit from training, credit or additional support. Allocating an opportunity changes the conditions being predicted. Managers who confuse these questions can systematically reward an existing advantage while believing they are discovering potential.

Accountability includes the ability to stop

Organisations often prepare carefully for launch and less carefully for withdrawal. Yet an application may become unsuitable when customer behaviour changes, data quality deteriorates or the business expands into a different market. Approval should be understood as conditional on continuing fitness for purpose.

Before deployment, managers should define warning signs, escalation routes and the authority to suspend use. They should consider how work would continue if the system were unavailable. A fallback process may seem unremarkable during a successful demonstration, but it becomes central when an important service depends on the technology.

The same reasoning applies to vendors. Purchasing an external tool can transfer technical tasks without removing the organisation’s responsibility for its own use of that tool. Contracts, documentation and independent evaluation should support informed oversight. A promise of proprietary sophistication cannot substitute for knowing whether an application is appropriate.

A classroom question with professional consequences

Students can explore accountability by mapping a hypothetical automated decision from problem definition to appeal. Assign roles to a business head, analyst, operational reviewer and affected customer. Ask each group what evidence it needs, what it can change and what it must explain. The gaps in the map often reveal the most important management issues.

Faculty can enrich this exercise by bringing perspectives from analytics, human resources, finance and ethics. The aim is to develop a habit of inquiry: whose objective does the system optimise, who bears its mistakes and who can secure a correction? These questions connect technical learning with organisational responsibility.

AI becomes a stronger managerial resource when its limits can be discussed and its use can be challenged. The algorithm may calculate, classify or recommend. People and institutions still choose the purposes it serves, the authority it receives and the remedies available when it fails. Those choices are where accountability must remain.

References

1\. NIST (2023). Artificial Intelligence Risk Management Framework 1.0, Section 5: AI RMF Core. Voluntary framework; cited version 1.0.

https\://airc.nist.gov/airmf-resources/airmf/5-sec-core/

2\. Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366, 447–453. doi:10.1126/science.aax2342. University of California institutional repository.

https\://escholarship.org/uc/item/6h92v832

Sources verified online: 20 October 2026\.