Components of AI agentic frameworks — Why you should avoid them!

How “agentic” frameworks work.

As a developer your probably been seeing the demand for building AI features but you might be confused on which framework to use. Some of the companies behind these frameworks are very good at marketing to encourage you to use their framework instead of others. When I got started building with Large language models, I was also confused and overwhelmed on which one should I use? After 16 months of building AI systems both for clients and my own projects, I would like to help you decide which one you should pick.

Building a Reliable Text-to-SQL Pipeline: A Step-by-Step Guide pt.1

What does a AI framework actually do?

Image by Author

Many developers are confused about what frameworks like Langchain actually do. The framework helps you orchestrate flow between the different parts of a language model (LLM) application.

There are three main parts to this:

1. LLM APIs: These are APIs provided by companies that let you access their language models (LLMs). You can use these APIs to interact with their models and get responses.

2. Retrievers: These are like databases that help you find the information you need. They store information from different documents and help you retrieve relevant details when you need them.

3. Utilities: These are extra tools like PDF readers, web search APIs, or any software that helps you perform specific tasks. The LLM can generate inputs for these utilities to carry out tasks.

Want someone who can see beyond jargon help you with your AI solutions?

A framework’s main job is to connect three key components: LLM APIs, Retrievers, and utilities. It helps you build more complex apps that can load documents, find information, and search the web.

You might be wondering, since each of these parts has its own API, can’t I just do the orchestration myself?

The answer is yes, you can do the orchestration yourself. In fact as I explain later in the post you should avoid some of these frameworks

To better understand here is an example:

The below code written in Python shows how you can easily create a LLM program using OpenAI Python SDK that searches the web using popular web search API and also indexes that page for further use.

# Install these
# !pip install openai requests serpapi qdrant-client beautifulsoup4

import openai
import requests
from serpapi import GoogleSearch
from bs4 import BeautifulSoup
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams
import json
# Set your OpenAI and SerpAPI keys
openai.api_key = "your_openai_api_key"
serpapi_api_key = "your_serpapi_api_key"
# Initialize Qdrant client
qdrant_client = QdrantClient(host="localhost", port=6333)  # Adjust with your Qdrant instance details
qdrant_collection_name = "web_search_results"
# Ensure Qdrant collection exists
def create_qdrant_collection():
    if not qdrant_client.get_collection(qdrant_collection_name):
        qdrant_client.create_collection(
            collection_name=qdrant_collection_name,
            vector_params=VectorParams(size=1536, distance="Cosine")  # Size depends on the OpenAI model (e.g., text-embedding-ada-002)
        )
# Search the web using SerpAPI
def search_web(query):
    params = {
        "q": query,
        "api_key": serpapi_api_key
    }
    search = GoogleSearch(params)
    results = search.get_dict()
    return results
# Extract text from HTML using BeautifulSoup
def extract_text_from_html(html):
    soup = BeautifulSoup(html, 'html.parser')
    paragraphs = soup.find_all('p')
    return " ".join([para.get_text() for para in paragraphs])
# Function to embed text using OpenAI API (text-embedding-ada-002 for embedding)
def get_openai_embedding(text):
    response = openai.Embedding.create(
        model="text-embedding-ada-002",  # You can use different models as per your requirement
        input=text
    )
    return response['data'][0]['embedding']
# Store HTML content in Qdrant
def store_in_qdrant(query, html, metadata=None):
    text_content = extract_text_from_html(html)
    embedding = get_openai_embedding(text_content)
    
    # Store the embedding along with metadata in Qdrant
    qdrant_client.upsert(
        collection_name=qdrant_collection_name,
        points=[
            {
                "id": hash(query),
                "vector": embedding,
                "payload": {
                    "query": query,
                    "html_content": html,
                    "metadata": metadata
                }
            }
        ]
    )
def main():
    create_qdrant_collection()  # Create the Qdrant collection if not already done
    query = "Python programming tutorials"  # Example search query
    search_results = search_web(query)
    
    # Loop through search results to fetch HTML content and store it in Qdrant
    for result in search_results.get("organic_results", []):
        url = result.get("link")
        metadata = {
            "title": result.get("title"),
            "url": url
        }
        
        # Fetch HTML content of the URL
        response = requests.get(url)
        if response.status_code == 200:
            store_in_qdrant(query, response.text, metadata)

Now let’s see how do so using a popular framework LangChain

from langchain_core.tools import Tool
from langchain_google_community import GoogleSearchAPIWrapper

search = GoogleSearchAPIWrapper()
tool = Tool(
    name="google_search",
    description="Search Google for recent results.",
    func=search.run,
)

The main benefit of using a framework is that it takes care of the orchestration for you. While you can still use the APIs for each of the three parts on your own, the framework simplifies the process by handling the connections and organization for you.

Why you should avoid most frameworks!

This is the controversial part of the post, but after 1.5 years of running an AI development agency, I’ve realized that many frameworks aren’t great for building high-quality, production-ready LLM programs.

What these frameworks really do is just take outputs and pass them as inputs to other parts of your program. Basically, they make it easier to move text around in your app. But for this convenience, you’re paying a pretty high price.

Reasons why some frameworks are not good

1. Un-optimized: In the web search example, LangChain provides a pre-built prompt for general use. But what if your use case is more specific, like shopping on the web? In that case, you’d get a more reliable app if you build a prompt tailored for that purpose. Similarly, the retriever you use should be optimized to store the exact content you need. If you have to customize each of these on your own, the time you saved by using the framework could end up being lost.

2. Inheriting Tech debt: Frameworks have been heavily criticized for always changing. Imagine you have an app that serves thousands of users, and you decide to upgrade to the latest version of the framework. But then, everything crashes. If you don’t upgrade, you lose support, including documentation, bug fixes, and updates. In the end, you might have no choice but to rewrite everything from scratch.

3. Complexity: Some frameworks add layers of abstraction. For example, LangChain had a tool for splitting text, but it’s actually just a simple string method in Python. By over-complicating the code, it becomes harder to fix bugs or even understand what’s going on. In software design, it’s a well-known principle that your program should avoid unnecessary complexity.

4. Loss of control: When you use a simpler framework or none at all, you have more control over your application. This means you can fix bugs more quickly, easily add or remove features, and make the app more reliable. LLM output is probabilistic, meaning it’s not always predictable. To make your program more reliable, one way is to reduce the number of LLM API calls needed. This can help minimize errors and improve consistency.

How to make more reliable reports using AI — A Technical Guide

Thank you for reading, please follow Firebird Technologies on Medium, Linkedin & substack.

Want a reliable technology partner to guide you through the complexities of AI development?