This article is a summary of the webinar ‘The role of data science in Agentic AI projects’ by Samar Deem on 15 July 2026 for the Quantic School of Business and Technology.
Designing a complete system
Agentic AI is often discussed as if the main challenge were simply to “build an agent,” but the real challenge is designing a complete system around a business process. In that system, the LLM sits at the center, interprets prompts, decides what to do next, and uses tools to retrieve information or take action. This is a contrast this with traditional RAG, where retrieval is usually more linear and less reasoning-driven.
The key idea is that an agent should not be designed in isolation. It should fit a real workflow, whether that is contract generation, customer support, financial analysis, or dashboard-based decision-making. The right orchestration pattern depends on the use case, so the architecture should follow the problem rather than the other way around.
Start with the business process
Before choosing models, tools, or orchestration patterns, it is recommended to start with the business process itself. The first questions are practical: What does the current workflow look like? What pain point are we solving? How will we know the agent has improved the process?
That means identifying baseline KPIs before deployment. Metrics such as turnaround time, report accuracy, number of clicks, report volume, or engagement levels help establish whether the agent is creating real value. Without that baseline, it becomes difficult to explain why the new system is better than the old one.
Data readiness comes first
A central theme is that data readiness is often more important than the agent itself. Teams should ask whether the available data is actually sufficient for the task the agent is supposed to perform. In many projects, the bottleneck is not the model but the quality, structure, and accessibility of the source data.
That is why exploratory data analysis on the source data matters so much. It is recommended cleaning the data, chunking it properly, enriching it where needed, and indexing it so the agent retrieves useful information instead of “garbage”. The point is not simply to collect data, but to create a reliable knowledge base that supports the intended business outcome.
Experiment and evaluate early
There is a strong case for treating evaluation and experimentation as core parts of the AI lifecycle. A good approach starts with a golden dataset: a set of representative questions or inputs that reflect how the agent will actually be used. That dataset can then be used for offline testing to check whether the agent behaves as expected.
Samar Deem also connects evaluation to engineering practice. Changes to prompts, vector indexes, or orchestration logic should be tested in an experimental branch before being promoted into production. If the new version performs better against the golden set, it is a stronger candidate for deployment; if not, the design should be revised.
Metrics that support decisions
Samar Deem warns against collecting every possible metric and hoping the numbers will make the answer obvious. Instead, metrics should support a specific decision at a specific stage of the lifecycle. Early in the project, business metrics matter most; later, model and tool metrics become more important.
For agents, tool-call precision and recall are especially important, but neither should be interpreted alone. Precision shows how often the agent’s positive calls were correct, while recall shows how many of the needed calls it actually made. She recommends using F1 when both sides matter, since it balances false positives and false negatives in a more useful way than accuracy alone.
Guardrails, rules and monitoring to establish trust
The final message is that Agentic AI must be trustworthy, not just functional. The speaker highlights the need for responsible AI practices such as bias checks, drift monitoring, explainability, and content safeguards. If an agent starts producing harmful, biased, or unreliable output, the business risks can outweigh the apparent benefits.
That is why guardrails matter and deterministic rules, security constraints, and monitoring should be built into the system from the beginning. An agent is never “done”; it must keep being measured, monitored, and improved as conditions change.
Practical takeaway
The main lesson is simple: successful Agentic AI is not just about prompts and models. It is about business design, data quality, evaluation discipline, and responsible deployment working together. When those pieces are aligned, an agent can create real business value; when they are not, it becomes just another fragile automation layer.
Acronyms and Terms
AI — Artificial Intelligence
LLM — Large Language Model
RAG — Retrieval-Augmented Generation
KPI — Key Performance Indicator
EDA — Exploratory Data Analysis
API — Application Programming Interface
Precision: Of all the cases the agent predicted as positive or relevant, how many were actually correct.
Recall: Of all the cases that were actually positive or relevant, how many the agent successfully found or identified
F1 — F1 score, the harmonic mean of precision and recall:
PII — Personally Identifiable Information
RAI — Responsible AI
CICD — Continuous Integration / Continuous Delivery