Skip to content

CASE STUDY

03/Work, selected

05 / 08

ABE and One-L

Two AI agents for state procurement, recognized by NASPO in 2025.

What I ownedOwnership: Led the engineering

Joined One‑L in November 2025 and ABE in February 2026, after the 2025 NASPO awards. Since then I lead ABE’s engineering, where I built the quality monitoring and the thumbs-up-to-test-case pipeline; on One‑L I set the direction, with four co-op students on the build.

2NASPO awards for ABE and One‑L, 2025

Year
2025 to 2026
Role
Technical lead, ABE evaluation and observability
With
The Burnes Center for Social Change
Category
Government AI

01/The problem

Procurement runs on rules nobody can keep in their head.

Massachusetts buyers wade through a thicket of purchasing rules, and lawyers redline every vendor contract against state terms by hand. Both jobs are slow and easy to get wrong.

02/What I built

Two agents: one answers buyers, one redlines contracts.

  1. 01

    ABE, the Assistive Buyer Engine, is an agentic RAG chatbot. Claude on Bedrock calls four tools in a loop (knowledge base search, full-document retrieval, metadata and structured spreadsheet queries) and answers with citations.

  2. 02

    One‑L takes a vendor contract through an 11-stage Step Functions workflow. It splits the document, retrieves state requirements, has Claude 4 Sonnet flag conflicts, merges the results and writes a redlined file. A human reviews every redline.

  3. 03

    Both agents won NASPO awards in 2025, before I joined: I picked up One‑L in November 2025 and ABE in February 2026. I own ABE’s quality loop today. A teammate, Ritik Bompilwar, wrote the first RAGAS pipeline in 2025. From February 2026 I built the quality-monitoring platform around it, added the path that turns thumbs-up answers into test cases.

  4. 04

    ABE is now a white-label template. One config file sets the brand, so another agency can rebrand and deploy it from a single file.

  5. 05

    I have set the technical direction for One‑L since November 2025, working with the four Northeastern co-op students on the team.

03/Architecture

Questions get citations. Contracts get redlines.

A question hits the agent loop, Claude calls retrieval tools and answers with citations. Contracts run through the Step Functions redline workflow. Evaluation runs score the answers and alarms watch the system.

A/ABE

01Buyer02CloudFront and WAF03Cognito04WebSocket API05Chat Lambdaagent loop06Bedrock Claude07Knowledge Base onOpenSearch Serverless08RAGAS evalLambda09CloudWatch

B/One-L

01Lawyer02Cognito03Step Functions redlineworkflow04Bedrock Claude05Knowledge Baseon OpenSearchServerless06Redlined contract

04/Outcomes

Two NASPO awards, and lawyers who start from a draft.

  1. 2

    NASPO awards for ABE and One‑L, 2025

    Cronin Gold and Academic Collaboration, won by Massachusetts OSD with the Burnes Center before I joined either project. I now lead ABE’s quality monitoring and One‑L’s direction.

    Third partySource: naspo.org

  2. 83%

    Less legal review time

    From my résumé: lawyers review a redline that One‑L has already drafted, instead of marking up each contract by hand. The baseline and sample are not published here.

    Self-reported

Check these in the ledger

05/Production notes

Behind the numbers

All receipts for this project
Evaluation

Instrumented, no results

A Step Functions pipeline scores each test question on seven RAGAS metrics, including faithfulness and context recall. Test questions are curated, plus thumbs-up answers that an LLM rewrites into Q&A. The judge is the same primary Claude model that writes the answers, so the scores are not cross-family. No results are published here, and the screenshot below is sample data.

Eval time

One evaluated question takes about 60 to 90 seconds: a full chatbot answer plus six RAGAS judge calls. A run fans out one question per Lambda, three at a time.

Retrieval

Claude Opus 4.6 answers from a Bedrock Knowledge Base on OpenSearch Serverless (semantic chunking, Titan Embed v2), and Sonnet 4.6 takes the fast jobs. ABE began on Kendra in 2024 and moved off it in September 2024, before my first commit.

Guardrails

Bedrock Guardrails on the chat path, a web application firewall in front, and Cognito sign-in.

What broke
  • Evals hung as “running”. Large runs passed the 15-minute Lambda limit, and the error handler read the wrong Step Functions path and crashed itself, so the run never left “running”. Now one question per chunk, a real FAILED state, and a Failed chip in the History tab.
  • An Excel result overflowed the context window. Tool output is now capped before it reaches the model, and I trimmed the system prompt from about 4,400 to 2,500 tokens.
Reversed

Retrieval carried a 0.6 score cutoff that looked like a quality gate. Hybrid scores run about 1.3 to 1.6, so it removed nothing. I dropped it and cap chunks per document instead.

06/Stack

Built with

  • AWS CDK
  • TypeScript
  • Python
  • Bedrock
  • Claude
  • OpenSearch Serverless
  • Titan Embeddings
  • Step Functions
  • Lambda
  • DynamoDB
  • Cognito
  • CloudFront
  • RAGAS
  • React