Real Estate Tax Fraud Detection: How Local Government Agencies Use Fine-Tuned Llama Models

The Before: The Fraud Problem

Midsize county assessor offices process between 15,000 and 45,000 property tax assessments annually. Historically, fraud detection relied entirely on manual audits—typically triggered by citizen complaints or random sampling. One Midwest county discovered that their manual approach caught approximately 2-3% of fraudulent filings, with an average investigation taking 40-60 hours per case.

Common fraud patterns included:

Property value underreporting (homeowners claiming false improvements to reduce assessed value)

Commercial property misclassification (reporting commercial buildings as agricultural to avoid higher tax rates)

Exemption abuse (claiming agricultural exemption on properties used for residential or commercial purposes)

Transfer price manipulation (reporting artificially low sale prices to reduce assessed value)

Document forgery (submitting false renovation receipts or structural reports)

One county's assessment office had a backlog of 890 cases awaiting investigation. The team consisted of 12 assessors and auditors. At their current pace, clearing the backlog would take approximately 7-8 months of full-time work, during which new suspicious cases continued accumulating.

Financial impact: One county audit found that undetected fraud in just 3 years resulted in approximately $1.2 million in lost tax revenue across 247 fraudulent assessments.

The Solution: Fine-Tuned Llama Implementation

A county in the Great Lakes region partnered with a regional technology firm to deploy a fine-tuned Llama 2 70B model (later upgraded to Llama 3) for automated fraud detection screening. The implementation used:

Infrastructure:

Local deployment on county servers (privacy-critical requirement—property data cannot leave government systems)

4x NVIDIA A100 GPUs for inference

Custom vector database (Pinecone) storing 35,000 historical assessment cases

REST API connecting to existing property assessment management system (Accela GovPilot)

Model & Training:

Base: Meta's Llama 2 70B model

Fine-tuning dataset: 8,500 manually-labeled historical cases (3,200 confirmed fraud cases, 5,300 legitimate cases)

Training hardware: 2x A100 GPUs over 18 hours

Framework: Hugging Face's `peft` library with LoRA (Low-Rank Adaptation) for efficient fine-tuning

Training cost: ~$400 (compute only, not including data prep labor)

Additional Tools:

Document OCR: Tesseract 5.0 for scanning submitted permits and receipts

Named Entity Recognition: spaCy for extracting property addresses, owner names, and financial figures

Workflow automation: n8n connecting model outputs to assessor dashboard

Validation layer: Human-in-the-loop review interface built on React

Step-by-Step Implementation Workflow

Phase 1: Data Preparation (Weeks 1-4)

Step 1: Historical Case Audit

The county began by auditing 10,000 historical assessments from the past 5 years. Trained auditors reviewed cases marked as "fraud confirmed" by courts or settlement, and randomly sampled legitimate cases. Result: 8,500 cases were labeled with 47 distinct fraud indicators. Time investment: 240 hours across 3 auditors.

Step 2: Feature Extraction Pipeline

Technical team created a structured extraction process:

Property address standardization

Sale price comparison against county records

Claimed improvements vs. photographic evidence (drone imagery from county GIS)

Property classification history (tracking reclassifications)

Owner information consistency (cross-referencing deed records)

Timeline analysis (suspiciously short time between purchase and assessment reduction claim)

Each case was converted into a 2,000-character text summary including property details, claimed improvements, and supporting documentation status.

Step 3: Dataset Splitting

Training: 6,800 cases (80%)

Validation: 850 cases (10%)

Testing: 850 cases (10%)

Stratified by fraud type to ensure balanced representation

Phase 2: Model Fine-Tuning (Weeks 5-7)

Step 4: LoRA Configuration

The team configured parameter-efficient fine-tuning:

LoRA rank: 64

LoRA alpha: 128

Target modules: q_proj, v_proj (attention layers)

Batch size: 16 (per GPU)

Learning rate: 5e-4

Epochs: 3

Max sequence length: 2,048 tokens

Rationale: LoRA approach reduced fine-tuning time from 72 hours to 18 hours while maintaining 99.2% of full fine-tuning performance.

Step 5: Training Execution

Using Hugging Face Transformers library:

Training loss decreased from 0.89 to 0.12 over 3 epochs

Validation accuracy improved from 71% to 94.3%

Model size: 35GB (manageable for local deployment)

Inference speed: 2.3 seconds per assessment on single A100

Step 6: Baseline Testing

Model evaluated against held-out test set:

Precision: 91.7% (false positives = 8.3%)

Recall: 89.2% (false negatives = 10.8%)

F1 Score: 0.904

Compared against previous rule-based system (precision 65%, recall 34%)

Phase 3: Integration & Deployment (Weeks 8-10)

Step 7: API Development

Technical team wrapped fine-tuned model in REST API with:

Input validation (property data structure conformance)

Output standardization (fraud risk score 0-100, confidence threshold, flagged fraud indicators)

Rate limiting (500 assessments/day per deployment requirements)

Logging and audit trail (all model decisions recorded for compliance)

Step 8: Dashboard Development

Created web interface for assessor staff:

Real-time fraud risk scores displayed during assessment entry

Highlighted risk factors (e.g., "98% confidence: claimed $45K renovation but no permit records")

Case prioritization queue (automatically sorting by risk)

One-click investigation flagging

Historical case reference retrieval

Step 9: Soft Launch

Introduced system to 2 assessors and 1 supervisor for 2 weeks:

System ran in "advisory" mode (no automated flags)

Team provided feedback on false positive scenarios

Identified 12 edge cases requiring prompt refinement

Retrained model with 47 new labeled examples addressing identified gaps

Phase 4: Full Production Deployment (Weeks 11-16)

Step 10: Staged Rollout

Week 11: All new assessments screened by model (historical backlog excluded)

Week 12: Assessors begin investigating top-50 flagged cases

Week 13: Expand to top-150 flagged cases

Week 14-16: Full operation with continuous performance monitoring

Step 11: Human-in-Loop Integration

Established protocol:

Model flags cases with risk score > 75 for auditor review

Auditor reviews flagged case (average 15 minutes vs. 45 minutes for cold investigation)

Auditor confirms, overrides, or escalates

All decisions feed back into model improvement dataset

Monthly retraining on accumulated feedback (every 200 new cases)

Step 12: Monitoring & Optimization

Set up automated monitoring dashboard tracking:

Model precision/recall trends

False positive rate by category

Investigation conversion rate (% of flagged cases resulting in confirmed fraud)

Auditor workload distribution

Model inference latency

Results: Concrete Numbers

Investigation Efficiency:

Average investigation time: 45 minutes → 18 minutes (-60%)

Cases processed per auditor per week: 8 → 24 (+200%)

Backlog clearance: 890 cases cleared in 9 weeks vs. projected 30 weeks

Fraud Detection:

Detection rate: 2-3% → 31.2% of submitted assessments flagged for review

Confirmed fraud in flagged cases: 18.7% actual fraud rate (vs. 2.1% in pre-model era)

New fraud cases identified in Year 1: 847 confirmed (vs. average 58 in previous years)

Financial Impact:

Recovered tax revenue Year 1: $3.8 million (from caught fraud + corrected assessments)

Implementation cost: $185,000 (hardware, software licenses, labor)

ROI Year 1: 2,050% ($3.8M/$185K)

Annual ongoing cost: $52,000 (maintenance, retraining, infrastructure)

Operational Metrics:

Staff training time: 12 hours per assessor (one-time)

System uptime: 99.7% over 12 months

False positive rate (cases auditor investigated but no fraud): 11.3% (acceptable threshold was <20%)

User satisfaction (staff survey): 8.4/10

False Negative Analysis:

In post-implementation audits, 42 fraud cases were later discovered that model had missed (risk score <75). Analysis revealed:

18 involved obscure property classifications not well-represented in training data

14 were complex multi-property schemes requiring cross-property analysis

10 involved collusion between property owners and assessor staff (outside model scope)

Improvement: Added cross-property analysis module, reducing false negatives by 35% in Year 2.

What Made It Work

1. Quality Training Data

Success depended entirely on accurate historical labels. The county invested 240 hours upfront ensuring cases were properly labeled by experienced auditors. Garbage data = garbage model. The 8,500 carefully-labeled cases proved invaluable.

2. Local Deployment

Property data is sensitive government information. Deploying locally on county servers (rather than cloud APIs) eliminated privacy concerns, ensuring staff and leadership would actually use the system.

3. Human-in-Loop Design

The model was positioned as an "investigator assistant," not replacement. Auditors made final decisions. This preserved trust and allowed continuous learning from expert feedback. Monthly retraining on accumulated corrections was critical.

4. Specific Problem Framing

The model wasn't asked to "detect all fraud." It was optimized for the top 5 fraud patterns accounting for 87% of historical cases. Narrow, focused optimization beats broad generalization.

5. Integration with Existing Workflow

System plugged into Accela GovPilot (software assessors already used). No major workflow redesign. Adoption required minimal training.

6. Clear Business Metrics

Leadership understood ROI in concrete terms: $3.8M recovered in Year 1 vs. $185K investment. This justified expanded funding.

Common Mistakes to Avoid

Mistake 1: Insufficient Training Data

Many agencies started with 500-1,000 labeled examples. The county learned that <2,000 labeled examples resulted in 67% accuracy. Jumping to 8,500 labeled cases improved accuracy to 94%+.

Lessons: Invest in labeling. Hire retired auditors part-time to label historical cases.

Mistake 2: Cloud-Based Deployment

Initial proposal was to use OpenAI API or cloud-based inference. Legal review rejected this—property data cannot leave government systems. This requirement drove decision toward local deployment, which actually reduced API costs long-term ($3K/year local vs. projected $180K/year in API calls for 500K annual assessments).

Mistake 3: Over-Reliance on Model

Early enthusiasm led to suggestion of "auto-flagging" cases for reduction without human review. Auditors pushed back. This would have created liability and legal challenges. The human-in-loop approach is legally defensible and maintains accountability.

Mistake 4: Ignoring Model Drift

Initial proposal was to train model once and run for 3 years. In reality, fraud patterns evolve. Monthly retraining on new cases was essential. Without this, accuracy degraded from 94% to 78% by month 9.

Mistake 5: Inadequate Change Management

One neighboring county attempted similar deployment but failed to involve assessor staff in system design. Staff perceived it as surveillance. Adoption was <30%. The successful county involved staff from week 1, giving them veto power over alert thresholds and investigation prioritization.

How To Replicate This Approach

For a County of 200,000-500,000:

Phase 1: Preparation (6 weeks, $15,000)

Hire consultant or train staff on AI/ML fundamentals (20 hours)

Audit and label 4,000-6,000 historical assessment cases (160 hours)

Procure hardware: 2x NVIDIA RTX 4090 ($3,000) or 1x A100 ($15,000)

Set up infrastructure team and AI lead ($80K/year ongoing, but work is 20% on this project)

Phase 2: Development (8 weeks, $25,000)

Fine-tune Llama model using labeled dataset ($400 compute)

Develop API wrapper and REST endpoints ($3,000 contractor)

Build assessor dashboard ($8,000 contractor)

Create training materials ($2,000)

Phase 3: Deployment (4 weeks, $10,000)

Soft launch with pilot team

Gather feedback and refine

Full rollout and monitoring setup

Total Year 1 Investment: $50,000-75,000

Ongoing Annual Cost: $40,000-55,000 (infrastructure, retraining, monitoring)

Key Success Factors:

Dedicate 1 person (0.5 FTE) as "AI coordinator" for first year

Budget 200 hours for data labeling—this is non-negotiable

Involve assessor staff in design from week 1

Plan for monthly model retraining (4 hours work)

Set realistic expectations: expect 15-25% fraud detection improvement Year 1, 40-55% by Year 2-3

Realistic Expectations

What This Approach Will Do:

Increase fraud detection rate from 2-5% to 20-35% in Year 1

Reduce investigation time per case by 50-65%

Generate ROI of 500-2,000% depending on fraud prevalence in your jurisdiction

Provide auditors with structured investigation priorities

Create defensible audit trail for legal challenges

What This Approach Will NOT Do:

Eliminate need for human auditors (they become more efficient, not redundant)

Catch 100% of fraud (estimated ceiling is 60-75% with current technology)

Replace property inspectors or appraisers

Work without quality historical training data

Operate without ongoing maintenance and retraining

Realistic Timeline:

Months 1-3: Planning, data prep, model training

Months 4-5: Soft launch, feedback, refinement

Month 6+: Full production (continuously improving)

Year 1: 30-50% improvement in detection rate

Year 2-3: 50-75% improvement as model matures

Who This Works For

Best Fit:

County or municipal assessor offices (population 150K+)

Jurisdictions processing 10,000+ assessments annually

Regions with known fraud patterns (suburban/exurban areas highest risk)

Agencies with existing digital assessment records

Organizations with IT infrastructure to support local deployment

Strong Indicators of Success:

Current fraud detection rate <10% (room for improvement)

Investigator backlog >300 cases

Leadership commitment to technology investment

Available budget for initial implementation ($50K-100K)

Staff willing to learn new tools (age <55 avg. strongest predictor)

Poor Fit:

Small counties (<50,000 population) with <3,000 annual assessments—ROI insufficient

Fully paper-based assessment systems (requires digitization first)

Organizations mandating cloud-only infrastructure (privacy concerns)

Jurisdictions unable to commit 1 staff person (0.5 FTE) to project

Agencies with very clean historical fraud records (<1% fraud rate—low improvement potential)

Conclusion

This approach has proven effective in 7 county implementations across the Midwest and Southeast U.S. (as of 2024). The key is treating this as an operational efficiency project with clear ROI, not an experimental AI pilot. The fine-tuned Llama model works because it's trained on your specific data, deployed locally on your infrastructure, and integrated into your existing workflow. Start with quality data, involve your users, expect continuous improvement, and track metrics obsessively.