--- layout: 2026 title: "Explainable AI: Inherently Interpretable Models | Literal Labs" excerpt: "Most XAI tools explain black boxes after the fact. Logic-Based Networks are interpretable by design — no SHAP, no LIME, no approximation gap." permalink: /explainable-ai/index-2.html cssheadinc: jsfootinc: lastupdated: 2026-05-26 sitemap: exclude: 'yes' ---

Explainable AI

Regulators demand it. Engineers need it. And your models have to deliver it. Explainable AI (XAI) refers to methods and techniques that make AI model decisions understandable to humans. There are two broad approaches: post-hoc explanation, which approximates why a black-box model reached a decision after the fact, and inherent interpretability, where the model's structure is itself human-readable. Logic-Based Networks belong to the second category — every decision is traceable to a logical rule, with no explanation gap.

Scales icon Explainability fundamentals

What Is Explainable AI?

Explainable AI (XAI) is the field concerned with making AI systems' decisions understandable — to engineers, to operators, to regulators, and to the people affected by those decisions.

The demand for XAI arises directly from the opacity of modern machine learning. Deep neural networks can achieve remarkable predictive accuracy, but their internal structure — millions of floating-point weights distributed across hundreds of layers — does not map to anything a human can inspect and understand. You can observe what a neural network predicts; you cannot directly observe why.

This matters wherever AI decisions carry consequences. A medical diagnosis model that produces a prediction without explanation leaves clinicians unable to verify or contextualise the output. A credit risk model that denies a loan application without traceable reasoning cannot be audited for bias or challenged under data protection law. An industrial safety system that flags an anomaly without identifying the source gives operators nowhere to look first.

XAI attempts to solve this problem. The question is how.

Code icon Two approaches

Two Approaches to XAI

Post-Hoc Explanation

Post-hoc explanation tools — SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), attention visualisation, integrated gradients, and others — work by interrogating a trained model after the fact to produce human-readable explanations of its predictions.

SHAP, for example, estimates each input feature's contribution to a prediction by computing a weighted average of marginal contributions across all possible feature subsets. This produces an attribution score for each feature — a number indicating how much that feature pushed the prediction up or down.

These tools are valuable. For many deployed neural network systems, they are the only practical path to any form of explanation. But they have structural limitations that cannot be engineered away:

  • They are approximations. Post-hoc explanation tools do not recover the actual reasoning of the model. They construct a simplified model of the model's behaviour. The explanation may be accurate on average whilst being wrong for specific predictions.
  • They can be inconsistent. Different explanation methods applied to the same model and same prediction can produce different feature attributions. There is no ground truth to arbitrate between them.
  • They add computational overhead. SHAP computation scales with the number of features and the complexity of the model. For real-time inference on constrained hardware, this overhead can be prohibitive.
  • They may not satisfy regulatory requirements. Regulators increasingly want to know not just what the model tends to do, but why it made this specific decision. An approximation that is correct 95 per cent of the time may not be sufficient when the remaining 5 per cent represents the decisions under audit.

Inherent Interpretability

An inherently interpretable model is one whose structure can be directly inspected and understood. The explanation is not computed after inference — it is the inference.

Decision trees are the classic example: every path from root to leaf is a sequence of conditions that can be read as a plain logical rule. The model's reasoning for any prediction is the path it took. There is no approximation.

Logic-Based Networks extend this principle to a class of models powerful enough for real-world classification tasks on structured data. LBNs learn propositional clauses — rules of the form "if condition A and condition B and not condition C, then vote for class X." Every inference is the evaluation of those clauses. Every prediction is traceable to the specific clauses that fired. The explanation is exact, not approximate.

Lightning icon Competitive advantage

Why Inherent Interpretability Wins

The debate between post-hoc explanation and inherent interpretability is sometimes framed as a trade-off: interpretable models sacrifice accuracy for transparency, so post-hoc tools allow you to have both. This framing is increasingly contested — and for edge AI applications, it misrepresents the actual trade-offs.

No approximation gap

With an inherently interpretable model, the explanation you give is the reasoning the model used. With post-hoc tools, there is always a gap between the explanation and the actual computation. That gap is small in typical cases and potentially significant in outlier cases — which are often precisely the cases that matter most.

Auditability is complete

In regulated contexts, being able to explain a specific decision — not the model's general behaviour — is often what the regulation requires. Inherently interpretable models support this without qualification. Post-hoc explanations support it approximately.

No additional infrastructure

Post-hoc explanation tools add code, computation, and maintenance burden. An inherently interpretable model requires none of this; the explainability is already there.

Compliance confidence is higher

For EU AI Act compliance, GDPR's right to explanation, and sector-specific requirements in healthcare, finance, and safety systems, inherently interpretable models offer a cleaner compliance position. See AI Act compliance for edge AI for more detail.

File icon Logic-based networks

How Logic-Based Networks Achieve Inherent Interpretability

LBNs learn propositional clauses through a reinforcement-based training process. The trained model is a set of these clauses — logical rules that can be inspected, read aloud, and debated by engineers and domain experts.

For an anomaly detection model trained on vibration sensor data, a clause might read: "if vibration amplitude in the 200–400Hz band is high AND operating temperature is elevated AND the running variance of amplitude has increased over the past 10 seconds: vote anomaly." This is not a simplification or summary; it is the actual learned rule. It can be verified against domain knowledge, challenged if wrong, and used to guide diagnostic action.

This is what explanation means in practice. Not a bar chart of SHAP values that requires a data scientist to interpret, but a logical condition that an engineer can read, evaluate, and act on.

The full technical account of LBN architecture and training is on the Logic-Based AI page.

Globe icon Edge deployment

XAI for Edge AI

Most XAI discussion focuses on large models deployed in cloud or server environments. The edge AI context introduces additional constraints that make post-hoc explanation particularly difficult and inherent interpretability particularly valuable.

Post-hoc explanation tools are computationally expensive. Running SHAP on an embedded device with a 32MHz processor and 256KB of RAM is not practical for real-time inference. The explanation tooling that works well in a cloud environment cannot be ported to constrained hardware without significant additional engineering — and often cannot be ported at all.

LBNs on the edge are inherently explainable without any additional computation. The clause evaluation that produces the prediction simultaneously constitutes the explanation. There is nothing extra to compute and no additional memory required.

This matters for deployments where explainability is required at the point of inference — safety systems that must indicate which condition triggered an alert, medical devices that must surface the features driving a classification, industrial monitors that must guide operator response. Explore the explainable edge AI page for a detailed treatment of this intersection.

Scales icon Compliance

The Regulatory Landscape

Regulatory pressure on AI explainability has moved rapidly. Several frameworks are now directly relevant to deployed AI systems:

EU AI Act

High-risk AI systems — including those used in healthcare, critical infrastructure, employment, education, and several other categories — must meet transparency and explainability requirements. Systems must provide sufficient information for users to understand the system's outputs and for operators to detect and correct errors. Inherently interpretable models have a significantly cleaner compliance position than black-box models with post-hoc explanation overlays.

GDPR

The right to explanation under GDPR — specifically Article 22 and Recital 71 — requires that individuals subject to automated decision-making be able to obtain a meaningful explanation of the decision. "Meaningful" is not yet precisely defined in case law, but post-hoc approximations are a weaker position than inherent interpretability.

Sector-specific requirements

Medical device regulation (MDR, FDA guidance), financial services regulation (SR 11-7 on model risk management, PRA/FCA expectations), and industrial safety standards (IEC 61508 and related) each impose requirements on model transparency and traceability that are more easily satisfied by inherently interpretable models.

See AI Act compliance for edge AI for a more detailed treatment of the regulatory landscape.

Explore more

Explore the Explainable AI Cluster

Scales icon Questions
What is explainable AI (XAI)?

Explainable AI refers to methods and techniques that make the outputs of AI systems understandable to humans. This includes both post-hoc tools that approximate the reasoning of black-box models (such as SHAP and LIME) and inherently interpretable models whose structure is directly human-readable. The goal is to enable trust, auditability, error correction, and regulatory compliance.

What is the difference between explainability and interpretability?

Interpretability refers to the degree to which a model's internal workings can be directly understood. Interpretable models — such as decision trees and Logic-Based Networks — produce decisions that can be traced to readable rules. Explainability is broader: it includes post-hoc tools that explain a model's outputs even when the model itself is opaque. An inherently interpretable model is always explainable; an explainable model is not always interpretable. The interpretability vs explainability page covers this distinction in detail.

Do Logic-Based Networks need SHAP or LIME?

No. SHAP, LIME, and equivalent tools are designed to explain models whose internal reasoning is not directly accessible — primarily neural networks. Logic-Based Networks are inherently interpretable: every prediction is traceable to the specific logical clauses that fired. No post-hoc approximation is required, and no explanation gap exists.

Next steps

Build Inherently Explainable Edge AI

ModelMill is the platform for training Logic-Based Networks on your sensor data and deploying explainable edge AI on standard microcontroller hardware. The explainability is built in — no SHAP, no LIME, no additional tooling required.

Request early access to ModelMill