Custom AI System Guide: From PoC to Production
Key takeaways
- A custom AI project should begin with the business task, input data, expected output, and tolerated failure — not with a model choice.
- A useful PoC leaves a representative evaluation set and measured accuracy, latency, and cost, not just an impressive demo.
- Use retrieval-augmented generation for internal knowledge and combine AI with rules, APIs, and workflows for structured operations.
- Production requires access control, data minimization, citations, human approval, regression evaluation, and cost monitoring.
When a custom AI system is useful
Custom AI is useful when company data and procedures determine the answer: internal document search, quote or report drafting, support triage, repetitive data entry, and image or audio processing. The goal is to place AI inside the existing approval, search, storage, and notification workflow rather than bolt on a separate chatbot.
Good candidates are frequent tasks with explainable inputs and outputs that people already perform using repeatable criteria. If success cannot be defined or data rights are unclear, fix the process and governance first.
Step 1: define the task and acceptance criteria
Replace “implement AI” with a sentence that names the user, input, output, and approver. Then define measurable criteria such as accuracy, omission rate, turnaround time, cost per case, and the share of drafts that humans edit.
For financial, legal, or safety-sensitive work, default to draft generation with human approval rather than automatic action. Design a fallback to the existing manual process.
Step 2: select data and architecture
Use retrieval-augmented generation when current internal documents are the source of truth. For deterministic operations such as calculation, lookup, or registration, have the model call validated APIs, database queries, or rules instead of guessing. OCR, speech recognition, and classifiers can be connected as separate stages.
Rights and quality matter more than raw volume. Confirm document owners, update cycles, stale or duplicate material, sensitive data, and per-user access, then enforce the same access rules on retrieval.
Step 3: what a PoC must prove
Build an evaluation set containing normal cases, difficult cases, and cases that must never fail. Re-run it after model or prompt changes. Measure citations, appropriate refusal, latency, and per-request cost as well as answer quality.
A PoC should produce a demo, evaluation report, unresolved failure modes, expected operating cost, security requirements, and a production scope. Those outputs support an evidence-based go or no-go decision.
Step 4: operations, security, and quality
Production controls include least privilege, encryption, sensitive-data masking, audit logs, approval gates, and retention rules. If an external AI API is used, document what leaves the environment and keep secrets out of pages and source code.
Models and knowledge change, so quality needs continuous measurement. Add failures to the evaluation set and automate regression, cost, and latency monitoring before releases.
What to prepare before speaking with IT7
Prepare the current workflow, real input and output examples, data locations and permissions, monthly volume, human approval points, and expected benefit. IT7 can stage the work from requirements and PoC through web or app interfaces, API integration, production infrastructure, and maintenance.
FAQ
Where should a custom AI project start?
Define one task’s user, input, expected output, approver, and acceptance criteria. Build a small evaluation set from real cases and validate quality, latency, and cost in a PoC.
Should we use RAG or fine-tuning?
Start with RAG when answers must be grounded in current internal knowledge. Consider fine-tuning for repeatable style or classification patterns; the approaches can also be combined.
Can AI be used with internal personal data?
Yes, after designing data minimization, masking, access control, transfer scope, retention, and audit logging. Keep human approval for high-impact outputs.