Interpretability vs. Explainability

The black box problem.

AI and ML models developed through deep learning are often described as black boxes. They produce outputs — classifications, predictions, decisions — that are opaque to observers and frequently difficult even for the engineers who built them to reverse engineer or explain.

That opacity was an acceptable trade-off when AI stayed in research settings. It is not acceptable when AI makes medical diagnoses, approves or rejects loan applications, or flags individuals in security systems. In those contexts, errors must be explained. Biases must be identified and removed. Decisions must be auditable.

Two frameworks exist to address this. They are related, they overlap in places, and they are regularly conflated. They are not the same thing.

Understanding how the model thinks.

Interpretability refers to the degree to which a human observer can understand why a model produced a specific output. A fully interpretable model allows its inputs to be mapped to its outputs through reasoning that is legible without additional tools.

Interpretable models tend to be structurally simpler — linear regressions, decision trees, rule-based classifiers — where the decision-making mechanism is transparent by design. The model does not need to be interrogated after the fact. Its logic is directly readable.

There are two sub-divisions worth distinguishing:

Global interpretability

The model as a whole is understandable. An observer can inspect the full decision logic and understand how the system behaves across all inputs, not just a specific case.

Local interpretability

Individual decisions within a model can be traced and understood. The model may not be globally transparent, but specific inference outputs can be followed back to their causes.

Accounting for what the model did.

Explainability focuses on post hoc analysis. Rather than requiring that the model's inner workings be legible, it asks: can we produce a coherent account of why a specific output was generated, even if the underlying mechanism remains opaque?

This makes explainability more accessible in the context of complex models. A deep neural network trained on millions of parameters is unlikely to be fully interpretable. But tools like SHAP and LIME can highlight which features most influenced a given prediction — providing an explanation without requiring full transparency of the model itself.

Explainability is typically local in scope. It addresses individual outputs rather than the model as a system. It does not require the model to be simpler. It adds an interrogation layer on top of whatever model is deployed.

Same goal. Different approach.

Both involve trade-offs.

The interpretability trade-off

Interpretable models are transparent by design — but transparency often comes at the cost of performance. Simpler model architectures may lack the capacity to capture the complexity required for high-accuracy outcomes on large, messy datasets. Choosing interpretability can mean accepting a ceiling on predictive power.

The explainability trade-off

Post hoc explanations approximate the model's behaviour rather than revealing it directly. They can oversimplify or misrepresent what actually occurred inside a complex model. Generating explanations also adds computational overhead — a relevant constraint in real-time or edge deployments where resources are limited.

What regulators are asking for.

Frameworks such as GDPR, the EU AI Act, and sector-specific financial and healthcare regulations are increasingly explicit about the obligation to explain automated decisions. When AI is used to process personal data or make consequential decisions, individuals may have the right to understand the basis of those decisions and to contest them.

Explainability is often cited as the mechanism for meeting this obligation — because it can be applied to complex models without requiring a simpler, potentially less accurate replacement. But regulators are becoming more discerning. An explanation that approximates rather than reflects the underlying decision process may satisfy the letter of a requirement while failing its intent.

Interpretable models offer a more robust answer to that scrutiny. When the model's logic is the explanation — rather than an approximation of it — there is less room for the account to be challenged as misleading.

Developers and organisations must weigh the complexity of the model against the level of transparency the regulatory and operational context demands. Explainability is not a substitute for interpretability where genuine transparency is required. It is a practical alternative where full interpretability is not achievable.

Logic as the foundation for both.

Most AI systems force a choice: interpretable but limited, or capable but opaque.

Logic-Based Networks are built differently. Rather than learning statistical weights, they learn logical relationships — patterns expressed through propositional logic and binarised representations of input data. Because the learned behaviour is expressed as logic, it can be inspected, traced, and explained without requiring a post hoc approximation layer.

That means LBNs are both interpretable and explainable in a way that most high-performance models are not. The model's decisions are not buried inside a statistical cloud. They are grounded in conditions and rules that can be read by an engineer, communicated to an auditor, and queried by a compliance team.

Interpretability by architecture

Because LBNs express their learning through logical structures, the model's behaviour is legible rather than approximated. Global and local interpretability are natural properties of the architecture — not features that need to be added after training.

Explainability without proxies

When an LBN produces a classification, the path to that output can be traced through the logic it learned. The explanation is not a post hoc reconstruction — it is a direct account of the decision pathway, which means it is accurate rather than approximate.

Performance without compromise

LBNs are not simpler models. They learn from data and perform competitively on benchmarks, including against gradient-boosted and deep learning approaches. The trade-off between transparency and accuracy is reduced — not eliminated, but substantially narrowed.

Build interpretable AI from your own data.

Literal Labs is building a tool that will let you train Logic-Based Networks on your own data — producing models that are accurate, interpretable, and explainable without requiring specialist AI expertise.

Fill in the form to receive an alert when we launch the training tool. If you'd prefer to discuss how you can utilise our AI pipeline now, please contact us.