Human in the Loop Doesn’t Limit AI Scalability — It Enables It

Figure 1 : The folder view illustrates that only 4% of the captured information has been reviewed. It was considered sufficient for the user to trust the other 96%

And, as the cherry on top 🍒, HITL doesn’t just verify critical results—it actively improves detection models. Every human validation becomes a valuable learning signal, helping the system to better understand unusual contexts, and continuously boost accuracy. In other words, the more users interact with HITL, the smarter the AI gets—a virtuous cycle of trust, quality, and ongoing evolution.

HITL accelerates model improvement as data grows

More documents mean more context, more edge cases, and ultimately more opportunities for humans to correct detections and refine prompts or models.

On large datasets, HITL becomes a continuous learning engine for both supervised machine‑learning models and computer‑vision models.

Improving Machine‑Learning (ML) Model

In the era of LLMs, are traditional ML models for keyword or pattern extraction still relevant?

At AgileDD, the answer is yes.

ML‑powered capture agents offer several advantages:

·       They are inexpensive to operate.

·       They can be trained quickly on a specific document family.

·       They respond directly to user feedback.

When a user marks a captured value as correct or wrong, that feedback becomes a direct training signal.

The model improves immediately, without requiring massive datasets or complex fine‑tuning pipelines.

Improving CNN‑Based Computer Vision Models

The same principle applies to agents that detect or classify figures inside documents.

These models can begin performing well with as few as 20 examples per category, and they improve rapidly with HITL validation.

One AgileDD customer uses such an agent to extract geological descriptions from mudlogs.

When the capture is incorrect, the cause is often simple: the layout of the log differs slightly from the training examples.

With HITL, the workflow becomes:

·       The user corrects a few examples on the first pages.

·       The model is retrained — a process now much faster thanks to AgileDD’s migration to Kubernetes.

·       The document is re‑processed.

The result:

The model learns this new layout and becomes fully reliable in that context. HITL transforms a one‑off correction into lasting model improvement.

What about the capture agents powered by a LLM ?

AgileDD allows users to create capture agents powered by LLMs, whether hosted via an external API or deployed on‑premises. We do not train or fine‑tune LLMs directly — but that doesn’t mean these agents are outside human control.

Just like any other extraction:

·       LLM‑generated values can be validated

·       marked as incorrect

·       edited

·       deleted

·       or replaced with manual captures

And HITL feedback doesn’t stop there.

Validation and refutation can be used to train an additional model that becomes part of the agent — effectively creating a dual‑engine agent where a lightweight supervised model helps the LLM better understand the context in which certain information should or should not appear.

This hybrid approach combines the flexibility of generative AI with the precision and governance of supervised learning.

Cross Validation: Understanding Whether Your Agent Truly Learned

When you train an agent—whether in AgileDD or any other platform—the first question is always the same: Did the model actually learn from the data I provided?

To answer this, AgileDD applies cross validation during training and computes several key performance indicators:

• F1 Score
• Precision
• Recall
• AUC (Area Under the Curve)

These metrics are calculated five times, each on 20% of the training dataset, providing a robust view of model performance. From this process, several insights emerge immediately:

• The higher the F1, Precision, and Recall, the better the model performs. These metrics reflect the model’s ability to correctly identify relevant information while minimizing errors.
• An AUC close to 1 indicates excellent separability between true positives and false positives. In practical terms, it means the model’s confidence scores are meaningful and can be used to filter or prioritize results.
• If F1 scores remain consistent across all folds, the model generalizes well across the entire training context. This consistency shows that the model isn’t overfitting to a subset of the data.
• If F1 scores vary significantly between folds, the model needs more training data to cover all contexts reliably. In other words, the model has learned some situations well but struggles with others.

Cross validation gives users a transparent, data driven understanding of how their agent behaves—before it ever processes a single production document. It’s an essential step for building trust and ensuring that HITL feedback will compound into meaningful improvements over time.