How We Fixed a Fintech AI Assistant That Ignored Its Knowledge Base

Project Overview

A German B2B fintech company partnered with Perfsys to improve the reliability of its AWS-based AI assistant through AI model optimization. The client is a team of fewer than 10 people that runs a well-known platform of verified customer reviews for financial advisors, banks and insurers. The platform helps consumers make informed financial decisions.

The client's vision was to build a reliable, knowledge-based AI assistant that answers complex user queries from verified data and keeps context during long interactions.

Client Background and Existing AWS Setup

The client had already implemented a serverless AWS architecture consisting of:

• Amazon Bedrock for AI inference
• Amazon S3 as a knowledge base repository
• AWS Lambda and API Gateway for orchestration
• A web UI for the frontend interface

Each AI agent represented a unique financial advisor persona sharing access to a centralized knowledge base stored in S3.

Despite this advanced setup, the agents were inconsistent, prone to hallucination and often ignored the knowledge base, which compromised reliability. The client engaged us to measure, diagnose and systematically improve agent performance.

The Challenge: Hallucinations and Ignored Knowledge Base Data

While the infrastructure was functional, the core challenge lay in AI quality and consistency:

• Agents forgot their personalities or initial instructions during extended conversations
• Context retention dropped significantly after 3 to 4 exchanges
• Agents produced hallucinated or incorrect answers, sometimes ignoring KB data
• No automated evaluation existed to track answer accuracy or reference validity

The client's main goal was clear:

"Ensure the AI agent provides accurate, reference-backed answers from the knowledge base, with measurable and repeatable quality metrics."

Our Approach: Evaluation First, Then Model and Prompt Tuning

Perfsys designed a three-phase strategy: automated evaluation first, then model experimentation, then prompt-level AI model optimization.

Building an LLM Evaluation Pipeline with DeepEval

We began by developing a custom Evaluation Pipeline based on the DeepEval framework. This pipeline allowed automatic testing of hundreds of AI interactions to measure:

• KB reference accuracy (how well answers use the knowledge base)
• Response consistency
• Invocation time (latency)

The evaluation pipeline enabled:

• Running 500+ automated test cases across multiple sessions
• Establishing quantitative baselines for each tested model
• Reproducing real user interaction patterns

This became the foundation for comparing every candidate model. If you are planning a similar build, our guide to building an AI agent MVP on Amazon Bedrock covers the architecture side.

Comparing Claude 3 Haiku, Claude 3.7 Sonnet and Amazon Nova Pro

We tested three models on Amazon Bedrock against the same evaluation set:

The testing revealed that Claude 3 Haiku, the client's initial choice, failed to reference the KB correctly in 80% of cases.

While Claude 3.7 Sonnet had better accuracy, Amazon Nova Pro offered optimal performance-to-cost ratio and superior consistency within Bedrock's ecosystem.

Post-Migration Issue Resolution

After moving to Amazon Nova Pro, we found and fixed four system-level issues:

Results: 95% Knowledge Base Accuracy in Six Weeks

Within six weeks, Perfsys successfully delivered a measurable improvement in AI performance and consistency through targeted AI model optimization.

Key Quantitative Outcomes

• 100% of critical issues resolved (language, fallback, KB consistency and hallucination handling)
• Knowledge base accuracy improved from 80% to 95%, ensuring nearly all answers are KB-based
• Evaluation automation reduced manual QA time by 70%, validating 500+ test cases per iteration

Impact Summary

• Valid answer consistency and reliability of KB usage significantly improved
• Invocation latency remained stable (~7 seconds average)
• Maintenance simplified through automated evaluation cycles

Conclusion & Next Steps

Through systematic testing and evaluation automation, plus Bedrock-native AI model optimization, we helped the client turn a poorly performing AI assistant into a reliable and measurable knowledge-based agent.

Next steps include:

• Expanding multilingual testing (DE, EN, FR)
• Integrating new agent personalities for domain-specific advisory roles
• Deploying the evaluation pipeline to monitor new model updates automatically

Update, October 2026: newer models such as Claude Sonnet 5.5 and Amazon Nova 2 have been released since this project. The evaluation pipeline can rerun the same test set against them, which is the reason we built it.

This kind of evaluation and optimization work is part of our AI agent development services on AWS Bedrock.

Is your AI assistant still ignoring its knowledge base?

Perfsys builds and evaluates AI agents on Amazon Bedrock, and we can start by measuring what your assistant gets wrong.

Book a Discovery Call

AWS AI Agents

Eugene Orlovsky

CEO & Founder @ Perfsys | Serverless architect with 10+ years of hands-on experience designing cloud-native architectures on AWS, backed by multiple AWS certifications. His writing bridges deep technical expertise with real-world business strategy, covering topics from AWS best practices to scaling tech-driven organizations.