Small Models, Big Results: How Orchestrated LLM Pipelines Beat Frontier AI at FMEA Generation
Aug 2026 · 5 min read
Elector beats frontline LLM models in FMEA generation by 30 % - read the details.
Mateus Gonzalez Spencer, Calibras
The Problem
Failure Mode and Effects Analysis (FMEA) is one of manufacturing’s most important quality tools – a structured process where engineers systematically document what could go wrong with each feature of a part, how severe the consequences would be, and how to prevent them. Every hole, thread, and surface finish on a component gets its own row in the FMEA table.
The problem is volume. A single complex part generates hundreds of FMEA rows, and engineers juggle multiple parts at once. Under that pressure, FMEAs become a documentation checkbox rather than genuine risk analysis. Engineers reuse phrasing, copy entries between parts, and fall back on generic labels – “assembly issues”, “part has no functionality” – that are true but say nothing specific about the feature. The result is consistent and fast, but shallow.
The natural next step – handing a drawing to a frontier AI assistant – hits three walls: drawings are sensitive IP that cannot leave company premises, per-call costs are unsustainable at volume, and compliance requires traceable outputs with explicit reasoning. No frontier model provides any of this.
Elector’s Approach
Rather than a single monolithic prompt, we decompose the problem into specialised stages connected by structured data flow:
- Feature extraction and enrichment. Engineering drawings are processed to identify measurable features. These sparse geometric payloads are enriched by a vision-language model that produces contextual descriptions of the part, the drawing view, and each specific feature – anchoring downstream reasoning in actual geometry and specific features.
- Orchestrated column generation. Each FMEA column is generated by a dedicated prompt that receives precisely the context it requires. The pipeline runs sequentially: failure mode → potential effect → severity score. Every stage produces both a concise table entry and a detailed reasoning trace.
- Structured outputs with audit trails. Every entry carries an explicit chain of reasoning – why this failure mode applies, how the effect follows, and which rubric band justified the severity score. Engineers can inspect, validate, and contest each step.
The whole pipeline runs on small, locally-deployable models (4B – 9B), requiring no external API calls and zero data egress, all contained inside the Elector system – either on-premise or on EU cloud.
How We Measure Quality
We use a reference-free rubric. For each column we define three quality dimensions covering correctness, scope, and format, each scored 1 – 3 by an LLM judge that receives the drawing image, the generated output, and the rubric criteria—scoring against the rubric rather than any reference string. This makes it ideal for production monitoring: a set of generated rows is automatically scored, flagged if below threshold, and surfaced for review – no human annotation bottleneck, no reliance on stale reference data ensuring quality.
Results
When scored against a reference-free rubric, the pipeline outperforms both the template-generated human reference data and nine frontier AI models:
| Configuration | C2 | C3 | C4 |
|---|---|---|---|
| Human reference data | 5.44 | 6.17 | 3.54 |
| Claude Opus 4.7 | 7.55 | 7.25 | 8.60 |
| GLM 5.1 | 7.10 | 7.10 | 8.05 |
| Kimi K2.6 | 7.25 | 6.55 | 7.55 |
| DeepSeek Expert | 7.00 | 6.70 | 7.15 |
| Gemini 3.1 Pro | 6.45 | 8.05 | 7.15 |
| Qwen 3.7-max | 6.40 | 7.40 | 7.25 |
| Minimax 2.7 MAX | 6.55 | 6.70 | 7.50 |
| Mistral Medium 3.5 | 6.55 | 7.85 | 6.10 |
| ChatGPT GPT-5.5 | 6.80 | 7.50 | 6.60 |
| Frontier mean | 6.85 | 7.23 | 7.33 |
| Pipeline | 8.49 | 8.37 | 8.05 |
Scores out of 9.0 per column.
The pipeline wins on every column. The reference data scores at floor on severity (3.54) – human annotators assigned scores without documenting reasoning, while the pipeline always provides a trace. The frontier gap tells a deeper story: the narrow spread across nine diverse models confirms this is a task-architecture problem, not a model-selection problem.
The deficit cascades. Without VLM-enriched feature context, frontier models produce imprecise failure mode labels, and that imprecision propagates through effects and into severity. On severity specifically, frontier models actually produce better-sounding justifications than the pipeline (Context Awareness favours them), yet consistently miscalibrate the final score because they lack the rubric-literal reasoning and band-comparison scaffolding the pipeline provides. The result: plausible prose, wrong numbers. Across 438 raw severity scores, frontier models average 5.36 versus the pipeline’s 8.17 – a 34.4% shortfall.
What Scale Buys You (and What It Doesn’t)
A key practical finding: quality on Effect and Severity plateaus from 4B parameters upward under the structured pipeline. Models from 4B to 120B achieve near-identical scores on these two columns. A 4B model on consumer hardware delivers production quality on two of three columns, at zero API cost and full data privacy. Model scale is not the bottleneck. Pipeline architecture is.
Why This Matters
Three consequences follow directly:
- Data privacy. Engineering drawings remain on-premise. No external API calls are required. Every inference happens locally.
- Cost. A small local model eliminates per-call inference costs that are unsustainable at production volume. The difference between a frontier API and a local 4B model is the difference between a recurring operational cost and a one-time hardware investment.
- Traceability. Every FMEA entry carries an explicit reasoning trace – failure mode rationale, effect chain, severity band comparison. This is the explainability gap at the core of both template systems and one-shot LLM generation.
Conclusion
The path from manual FMEA to generative FMEA is not a text-generation problem. It is an architecture problem. A structured pipeline enables small local models to outperform the largest frontier systems used in one-shot mode. Structure, not scale, is what produces quality.
The frontier models’ uniform underperformance confirms that increasing model capability is not the path forward. The bottleneck is not language understanding – it is the absence of structured decomposition: visual grounding, sequential reasoning, and rubric-anchored calibration.
For manufacturers, the best FMEA quality does not require sending sensitive data to external APIs or trusting a black-box process. It requires a well-designed pipeline running on local hardware.
