Examples of known SLMs: <\/p>\n\n\n\n
- \n
- Mistral 7B<\/strong><\/li>\n\n\n\n
- Phi-2<\/strong> from Microsoft (~2.7B parameters)<\/li>\n\n\n\n
- TinyLLaMA<\/strong><\/li>\n\n\n\n
- Gemma 2B<\/strong> of Google<\/li>\n<\/ul>\n\n\n\n
Model Quantization: Downsizing without Precise Mortality <\/h2>\n\n\n\n
One of the main obstacles to training AI models is their size and the processing power required. Here, quantization plays a key role. This technique decreases model size by converting high-precision weights (FP32, FP16) to low-precision weights (INT8, INT4) without significantly affecting model performance. Advantages include:<\/p>\n\n\n\n
- \n
- Reduced memory usage<\/strong>Allows models to be stored and managed on devices with limited capacity.<\/li>\n\n\n\n
- Increased GPU\/CPU efficiency:<\/strong> Reduces CPU load by speeding up mathematical operations.<\/li>\n\n\n\n
- Accelerated inference:<\/strong> Models can react more quickly, decreasing the accuracy of calculations.<\/li>\n<\/ul>\n\n\n\n
Comparison of SLM and LLM<\/h2>\n\n\n\n
While LLM has proven to be a powerful tool, SLM offers important advantages in situations where efficiency and privacy are paramount. Below, we compare the two approaches:<\/p>\n\n\n\n
![\"\"](\"https:\/\/exceltic.com\/wp-content\/uploads\/2025\/06\/tabla-comparacion-SLM-y-LLM.png\")
<\/figure>\n\n\n\nRecovery Augmented Generation (RAG)<\/h2>\n\n\n\n
The Retrieval Augmented Generation (RAG) technique is used to improve the accuracy and contextualisation of models. This method maximises responses by obtaining information from additional sources and improves the context prior to text generation. Its structure consists of:<\/p>\n\n\n\n
- \n
- Chunking<\/strong>fragmentation of data into manageable parts.<\/li>\n\n\n\n
- Embeddings of documents<\/strong>: conversion of text into numeric vector.<\/li>\n\n\n\n
- Vector database (VectorDB)<\/strong>: )<\/strong>is a database that stores and retrieves relevant data.<\/li>\n\n\n\n
- Information retrieval<\/strong>When faced with a query, it locates the most relevant pieces of information.<\/li>\n\n\n\n
- Generating responses<\/strong>: synthesis of contextualised information to improve the output of the model.<\/li>\n<\/ol>\n\n\n\n
Implementing SLMs + RAG: An Efficient and Secure Model<\/h2>\n\n\n\n
The combination of SLMs with the RAG strategy enables the creation of highly efficient and controllable generative AI systems. With this architecture, organisations can use optimised models that ensure greater privacy while using fewer resources.<\/p>\n\n\n\n
Key Benefits:<\/h3>\n\n\n\n
- \n
- Optimised use of data<\/strong>The inclusion of information retrieval allows for more accurate and informed responses.<\/li>\n\n\n\n
- Full control over the model:<\/strong> avoids the need for third-party services and allows customisation of AI behaviour.<\/li>\n\n\n\n
- Execution in restricted environments:<\/strong> Due to their quantifiability and smaller size, SLMs can be deployed on edge devices or local servers.<\/li>\n<\/ul>\n\n\n\n
Basic Architecture with Quantified Model, LangChain, RAG and FastAPI<\/h2>\n\n\n\n
The following architecture can be used to create an efficient SML environment:<\/p>\n\n\n\n
- \n
- Loading of the quantized model:<\/strong> a model that has been previously quantified in INT8 or INT4 is used to optimise its performance.<\/li>\n\n\n\n
- LangChain for prompts management<\/strong>LangChain: LangChain allows to structure and extend the requests of the model.<\/li>\n\n\n\n
- Use of RAG for enhanced recovery<\/strong>a: makes use of vector databases to improve the context of responses.<\/li>\n\n\n\n
- REST API with FastAPI:<\/strong> this model explains how to use an API to facilitate integration with other applications.<\/li>\n<\/ol>\n\n\n\n
Example code:<\/h3>\n\n\n\n
from fastapi import FastAPI, HTTPException<\/p>\n\n\n\n
from langchain.chains import RetrievalQA<\/p>\n\n\n\n
from langchain.vectorstores import FAISS<\/p>\n\n\n\n
from langchain.embeddings import HuggingFaceEmbeddings<\/p>\n\n\n\n
from langchain.llms import HuggingFacePipeline<\/p>\n\n\n\n
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline<\/p>\n\n\n\n
import torch<\/p>\n\n\n\n
<\/p>\n\n\n\n
# Load quantized model<\/p>\n\n\n\n
tokenizer = AutoTokenizer.from_pretrained(\"model-quantified\")<\/p>\n\n\n\n
model = AutoModelForCausalLM.from_pretrained(\"model-quantified\", torch_dtype=torch.int8)<\/p>\n\n\n\n
pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer)<\/p>\n\n\n\n
llm = HuggingFacePipeline(pipeline=pipe)<\/p>\n\n\n\n
<\/p>\n\n\n\n
# Load Vector Database for RAG<\/p>\n\n\n\n
embeddings = HuggingFaceEmbeddings(\"sentence-transformers\/all-MiniLM-L6-v2\")<\/p>\n\n\n\n
db = FAISS.load_local(\"ruta_vector_db\", embeddings)<\/p>\n\n\n\n
retriever = db.as_retriever()<\/p>\n\n\n\n
qa_chain = RetrievalQA(llm=llm, retriever=retriever)<\/p>\n\n\n\n
app = FastAPI()<\/p>\n\n\n\n
@app.post(\"\/generate\")<\/p>\n\n\n\n
def generate_response(prompt: str, max_length: int = 100):<\/p>\n\n\n\n
try:<\/p>\n\n\n\n
response = qa_chain.run(prompt)<\/p>\n\n\n\n
return {\"answer\": answer}<\/p>\n\n\n\n
except Exception as e:<\/p>\n\n\n\n
raise HTTPException(status_code=500, detail=str(e))<\/p>\n\n\n\n
if __name__ == \"__main__\":<\/p>\n\n\n\n
import uvicorn<\/p>\n\n\n\n
uvicorn.run(app, host=\"0.0.0.0.0\u2033, port=8000)<\/p>\n\n\n\n
Explanation<\/h2>\n\n\n\n
Loading the quantized model<\/h3>\n\n\n\n
tokenizer = AutoTokenizer.from_pretrained(\"model-quantified\")<\/em><\/p>\n\n\n\n
model = AutoModelForCausalLM.from_pretrained(\"model-quantified\", torch_dtype=torch.int8)<\/em><\/p>\n\n\n\n
pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer)<\/em><\/p>\n\n\n\n
llm = HuggingFacePipeline(pipeline=pipe)<\/em><\/p>\n\n\n\n
- \n
- AutoTokenizer.from_pretrained(\"model-quantified\"): <\/strong>Loads the tokeniser of the quantized model.<\/li>\n\n\n\n
- AutoModelForCausalLM.from_pretrained(\"model-quantified\", torch_dtype=torch.int8): <\/strong>Loads the quantized model in int8 precision, which reduces memory usage and speeds up inference.<\/li>\n\n\n\n
- pipeline(\"text-generation\", model=model, tokenizer=tokenizer):<\/strong> Creates a text generation pipeline based on the quantified model.<\/li>\n\n\n\n
- HuggingFacePipeline(pipeline=pipe):<\/strong> Integrate the pipeline into LangChain for further use in the RAG architecture.<\/li>\n<\/ul>\n\n\n\n
2. Retrieval-Augmented Generation (RAG) configuration:<\/h3>\n\n\n\n
embeddings = HuggingFaceEmbeddings(\"sentence-transformers\/all-MiniLM-L6-v2\")<\/em><\/p>\n\n\n\n
db = FAISS.load_local(\"ruta_vector_db\", embeddings)<\/em><\/p>\n\n\n\n
retriever = db.as_retriever()<\/em><\/p>\n\n\n\n
qa_chain = RetrievalQA(llm=llm, retriever=retriever)<\/strong><\/em><\/p>\n\n\n\n
- \n
- HuggingFaceEmbeddings(\"sentence-transformers\/all-MiniLM-L6-v2\"): <\/strong>Uses an embedding model (MiniLM-L6-v2) to convert text into vector representations.<\/li>\n\n\n\n
- FAISS.load_local(\"path_vector_db\", embeddings): <\/strong>Loads a FAISS vector database with previously generated embeddings.<\/li>\n\n\n\n
- db.as_retriever():<\/strong> It turns the database into a search engine to retrieve relevant information.<\/li>\n\n\n\n
- RetrievalQA(llm=llm, retriever=retriever)<\/strong>Combines quantified language modelling with information retrieval to improve response generation.<\/li>\n<\/ul>\n\n\n\n
3. Creating the API with FastAPI:<\/h3>\n\n\n\n
app = FastAPI()<\/em><\/p>\n\n\n\n
@app.post(\"\/generate\")<\/em><\/p>\n\n\n\n
def generate_response(prompt: str, max_length: int = 100):<\/em><\/p>\n\n\n\n
try:<\/em><\/p>\n\n\n\n
response = qa_chain.run(prompt)<\/em><\/p>\n\n\n\n
return {\"answer\": answer}<\/em><\/p>\n\n\n\n
except Exception as e:<\/em><\/p>\n\n\n\n
raise HTTPException(status_code=500, detail=str(e))<\/em><\/p>\n\n\n\n
- \n
- FastAPI(): <\/strong>Create a REST API to expose the RAG model and functionality..<\/strong><\/li>\n\n\n\n
- @app.post(\"\/generate\"):<\/strong> Defines a \/generate endpoint that accepts POST requests with an input prompt.<\/li>\n\n\n\n
- qa_chain.run(prompt): <\/strong>It uses a combination of information retrieval (RAG) and text generation to respond.<\/li>\n\n\n\n
- Exception handling:<\/strong> If an error occurs, an HTTP 500 code is returned with the error message.<\/li>\n<\/ul>\n\n\n\n
4. Server execution:<\/h3>\n\n\n\n
if __name__ == \"__main__\":<\/em><\/p>\n\n\n\n
import uvicorn<\/em><\/p>\n\n\n\n
uvicorn.run(app, host=\"0.0.0.0.0\u2033, port=8000)<\/em><\/p>\n\n\n\n
- \n
uvicorn.run(app, host=\"0.0.0.0\", port=8000)<\/strong><\/code>: Starts the server on port 8000, allowing access to the API.<\/li>\n<\/ul>\n\n\n\nConclusion<\/h3>\n\n\n\n
Companies and developers can now adopt lighter, faster and more private models thanks to SLMs (Small Language Models)<\/strong>a key breakthrough in the evolution of generative AI. The quantization of models<\/strong> -which significantly reduces the size and computational requirements without compromising basic performance - allows these models to be run in environments on-premise<\/strong> or on resource-constrained devices, while maintaining full control over data and processes.<\/p>\n\n\n\n
This approach is complemented by the architecture based on RAG (Retrieval-Augmented Generation)<\/strong>together with tools such as FastAPI and LangChain<\/strong>These strategies enable the deployment of AI solutions that are governable, auditable and tailored to specific requirements. These strategies make it possible to fully controlled AI generation<\/strong>making it a realistic and effective choice for demanding sectors such as data analysis, scientific research or customer service.<\/p>\n\n\n\n
The combination of quantified SLMs, modular architecture and autonomous deployment represents one of the most secure and efficient ways to integrate generative AI into your organisation.<\/p>\n\n\n\n
Want to see how this translates into a real case?<\/h3>\n\n\n\n
Access the full paper and find out more.<\/p>\n\n\n
- @app.post(\"\/generate\"):<\/strong> Defines a \/generate endpoint that accepts POST requests with an input prompt.<\/li>\n\n\n\n
- FAISS.load_local(\"path_vector_db\", embeddings): <\/strong>Loads a FAISS vector database with previously generated embeddings.<\/li>\n\n\n\n
- AutoModelForCausalLM.from_pretrained(\"model-quantified\", torch_dtype=torch.int8): <\/strong>Loads the quantized model in int8 precision, which reduces memory usage and speeds up inference.<\/li>\n\n\n\n
- LangChain for prompts management<\/strong>LangChain: LangChain allows to structure and extend the requests of the model.<\/li>\n\n\n\n
- Full control over the model:<\/strong> avoids the need for third-party services and allows customisation of AI behaviour.<\/li>\n\n\n\n
- Embeddings of documents<\/strong>: conversion of text into numeric vector.<\/li>\n\n\n\n
- Increased GPU\/CPU efficiency:<\/strong> Reduces CPU load by speeding up mathematical operations.<\/li>\n\n\n\n
- Phi-2<\/strong> from Microsoft (~2.7B parameters)<\/li>\n\n\n\n