Workflow type
PII detection across rapid reports, file notes, forms, and other claim-document text in a roughly 70M-document historical claims corpus.
Deep dive
Reviewable PII detection for regulated claim workflows.
A concise look at the WCB claim-document pipeline: prompt-routed LLM extraction, deterministic validation, post-processing, release gates, and workflow-level evaluation.
Built for a roughly 70M-document historical claims corpus, ~5K new documents/day, noisy PDF-extracted claim text, configurable local or Databricks execution, and traceable outputs staff can evaluate before downstream use.
System boundary
Outputs stay bounded to 14 PII/entity classes and structured results staff can review.
Platform layer
One YAML-configured workflow supports local and Databricks execution with MLflow, run artifacts, release gates, and audit logging.
Delivery flow
Detection only counts when the evidence is clean.
The pipeline moves from extracted claim text to routed prompts, hybrid validation, scoring, and traceable structured output.
01
Text intake
Load PDF-extracted claim text from approved tabular sources spanning the historical corpus and ~5K new documents/day.
02
Prompt routing
Route documents by type and text shape before inference.
03
LLM extraction
Run LLM inference through approved local or platform runtimes for bounded PII/entity classes.
04
Hybrid validation
Combine model output with regex signals, dictionary matches, structural lookups, and span repair before scoring.
05
Audit + evaluation
Score against labeled data, write run artifacts, and prepare traceable structured output for downstream use.
System map
The parts a reviewer can point at.
The same pipeline as a structure instead of a sequence.
Runtime
Detection
Evaluation and governance
Output
Select a component to see its role.
Operating principles
Design principles for reliable detection.
Scope
Detect bounded entity types across noisy documents.
The pipeline focuses on names, dates, claim numbers, contact details, and regulated identifiers rather than open-ended generation.
Grounding
Pair LLM output with deterministic evidence.
Regex matches, structural lookups, dictionaries, and post-processing help remove weak spans and support cleaner evidence.
Runtime
Keep one config contract across local and Databricks runs.
A single YAML-driven runtime path keeps local development and platform execution aligned.
Governance
Make every run inspectable.
Run summaries, event logs, and MLflow metrics make it easier to trace why a result was accepted, rejected, or tuned.
Evaluation rubric
How the pipeline is measured.
Accuracy
Do precision, recall, and F1 hold up against labeled claim-document data?
Evidence quality
Are weak detections removed, overlaps resolved, and spans repaired before writeback?
Runtime consistency
Does the same config behave reliably across local development and Databricks execution?
Traceability
Can each run be explained through summaries, events, and MLflow metrics?
Failure boundaries
Reasons to stop and review.
01
Outputs that cannot be traced back to model or rule evidence.
02
Entity spans that overlap, drift, or stay too weak after post-processing.
03
Runtime differences between local and Databricks execution that change evaluation behavior.
04
Writebacks that are not ready for safe downstream use.
For confidential work, the right public evidence is disciplined: pipeline shape, evaluation logic, auditability, and delivery judgment without exposing sensitive documents.