I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search.
Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.
josh_p
I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN".
It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.
It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
chorizo
Why not use what USPS Addresses API? That what it’s there for. It will validate and standardize addresses including ZIP+4. Wonder if other countries have similar API’s.
polishdude20
Their API doesn't allow bulk conversions.
chorizo
Believe they have a paid access tier with much higher rate limits.
oever
What's your opinion on nominatim? I find that it gives up quickly when there's one or two typos in an address. It nails your examples.
jameshart
I think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users.
(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
comandillos
I indexed thousands of documents into a SQLite database with an FTS5 index, plugged it into DeepSeek v4 Flash and got better much better results than any other commercial solutions my company has tried in the past.
The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.
teleforce
Just wondering on the reported accuracy of the SQL queries generated by SQL based on RAG. Reports have been not encouraging and also confirmed by these results but perhaps your methodology is different from these reports [1],[2],[3].
>On this benchmark, a pure LLM generated an accuracy score of zero. Adding RAG, prompt engineering, and agentic AI raised accuracy to the 10+% range.
[1] Any text-to-SQL benchmark should address difficulties of real-world data stores (acm.org) (21 comments):
I'm now using this approach too, and it feels better than any sorting method I've used before. The only thing I'm thinking about now is: if the LLM makes a mistake, how can I provide feedback and verify it?
b112
I log all toolcalls to a file, I think others have said the same. But I'm a bit leery of letting a hallucinating LLM write SQL queries. I think most I've spoken with, agree that an LLM is like a 20 year old, eager intern. Well meaning, but left unrestrained capable of immensely inexperienced mistakes.
Before a lot of frameworks existed, you'd see DEVs taking user input on a web form, and then just throwing it directly at the MTA. So spammers could submit email@address\nCC: persontospam@address, and the like.
Now LLMs are a different beast, but you have input validation for LLMs, unique to all other validation methods. Yet there's actually no safe way to ever validate user input for a LLM, except for very rigid input validation on single words. Take the email example above. You'd need a regex to only validate an email address (and that isn't simple), but once you expand it to actually allowing sentences?
The LLM is now input validation vulnerable.
And that means no user input can be used in unvalidated commands.
And then just random hallucinations. I'm curious how the gp managed weirdo LLM behaviour, like out of the blue 'drop table' or accidental select into as opposed to just select.
comandillos
Its a read-only SQLite file. And I mean the people using the chatbot knows it uses AI so just like Google they shouldnt pick the first result, but forcing the LLM to mention the sources and not assume acronyms works amazingly well
comandillos
I have a web user interface connected to a coding agent (OMP) running inside a container, so if any of the tool calls fail or something happens, usually my model recovers autonomously from these situations. The capabilities of models like DS4 Flash are those of frontier models from months ago, so its recovery and autonomous capabilities are quite impressive.
xocp
Do you have the LLM write a query with every end-user input (i.e. during every search request)?
comandillos
Not really, I just made a really simple Python library the agent can use and modify as he wishes within a sandboxed environment to explore the SQLite database. When the user prompts the agent, the prompt goes into my coding agent, omp, and starts running Python scripts and throwing SQL sentences until coming up with the answer. It works extremely well in our experience.
ochiba
I did the exact same thing for our “company brain”. Works great
j0selit0
good one. yes, that's pretty much my experience with query rewriting as well
_the_inflator
RAG is art. I have a very straight forward setup that is highly modular.
RAG is routing and decision making.
I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.
I agree with you.
Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.
It maintains itself.
Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.
I found my niche with RAG selling and I build them myself.
I take pride in them.
So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.
And there are regulated businesses where certain information is required to be always factual correct - pricing for example.
Vector search becomes a liability for this use case.
So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.
I love this kind of stuff.
And there is personal information etc.
Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.
And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.
It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.
I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.
RAG gives me dbCORE vibes so to say.
Good luck and fun with your RAG systems.
alex-zaporozhan
I think so too. RAG with its layers and fine-tuning captures the imagination. Sometimes you even lose the thread between where it is math and when it is just intuitively obvious
andai
Re: the rube goldberg machine of diminishing returns
This is from two years ago, but I think it's still SotA?
gardnr
That is the approach I would take today. Late Interaction is worth a look. Evals are necessary.
mmargenot
And you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.
kaon_2
Can you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?
hnfong
Yes. Thank you for pointing this out.
I think there needs to be a linguist version of "what every programmer needs to know about (full?) text search"...
I'm not a linguist and I don't study languages, but I know enough to realize if a text search system is not designed for a particular language, it simply won't work. (As an example, to implement English search in a system for a hobby project, I had to import a US/UK spelling wordlist, and implement the Porter Stemming Algorithm. This is just for "one" language, and probably does not cover the other "English" dialects. Imagine doing a different workaround for every language in existence...)
RAG is actually a very language-agnostic way to work around those issues.
usernametaken29
I am a linguist and can say your view is too simplistic. This works only for things that are common enough to be mapped onto the same vector from the training set. Try searching for my company XY and it won’t work, because your embedding doesn’t have an adequate embedding, so you will end up having to build a translation table anyways..
tantalor
FTS like Elasticsearch supports cross-language (also called multi-language) search.
jon-wood
Instinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are querying do the same thing. When returning search results show them in the original language.
whilenot-dev
Why not create indexes for multiple languages, as that would also avoid double translation issues (e.g. GER [query] → ENG [index] → GER [document])?
j0selit0
you would also need to maintain multiple indexes in multiple languages. I never had to do that - but I assume it's a pain
whilenot-dev
It's a matter of running a for-loop. You'd get faster response times (no query→index translation), but the storage requirements for the indexes would be larger.
kaon_2
Yes we've tried. It works. But jargon is hard. RAG with embeddings works all the same. The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets
jameshart
Embedding search is effectively machine translation into a single common ‘language’ - embedding space - and then searching that; cleaner and less lossy than translating everything into English for searching, but harder to debug when it goes wrong.
lacedeconstruct
I thought text search was always the first thing you try, then fuzzy search, then you go for RAG
wongarsu
It's not like a simple embedding search takes that much longer to implement. Especially on short descriptions where you don't have to deal with chunking. And if you let an LLM write the code it's even less of a difference. Combine that with embedding search promising to solve all your search problems, and I understand why people often skip over full text search and go straight to embeddings
EagnaIonat
Even that is an oversimplification unless you are doing something very basic.
Volume of documents, size of documents, versioning, frequency of update, documents similar or overlapping information, how much or exactly what you need for the LLM to understand, AI friendly documents, who has access and at what level, blue teaming, red teaming, multi-lingual, does the LLM know the domain language of the user and documents.
I probably missed a few things even with that.
ozim
I think Bitwarden implemented some vector search in their password search feature ... totally annoying it gives me back all kinds of stuff that I don't care.
I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
gwerbin
Bandcamp has had legendarily bad semantic search for as long as they've been around. It's often completely impossible to find an artist or album or song even when you type the exact name.
t_mahmood
ahh now I realize why I get so much completely irrelevant search results in many sites recently. I mean I'm searching for betel and you're giving me nuts. haha
a1o
A good UI could do these and also exact match, give some point system to the results, then order them and perhaps use a bold highlight to reflect what parts of the input query reflected in each result.
itintheory
Bitwarden has lost the plot. The most recent Windows update is so bad. It has way lower information density in the UI, more buttons to click for the same use, no longer puts focus on the search field by default (this one makes me irrationally angry), and on one of my Win 11 installs can't lock the vault, manually or automatically. How could they mess up such a simple app that worked fine for so long?! What perverse incentives caused this nonsense?!
j0selit0
I wish everyone thought like you, in my experience unfortunately it's not the case
shay_ker
How long have "large scale RAG systems" really existed in the first place? I'm always surprised at this, given how new all this really is, relatively speaking.
j0selit0
my personal experience - I have been involved with such projects for the last 2 years. interestingly enough, a lot of times such initiatives didn't take off because people/stakeholders were overcomplicating things and wanting to use semantic search for everything - without having a minimum knowledge of chunking strategies, pros/cons etc
mdp2021
> people vastly underestimate full text search
It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
locknitpicker
> It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours.
As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
lopsotronic
Precisely this. The people in charge of technical direction don't understand the fundamentals of the technology. So you get the idea that LLMs can help make sense of parts data. Which . . . no, no it really can't, not without ALSO plugging in basically every other hunk of natural language you might have laying around. Unless you think PLG HT HFI is just a natural synonym of HOT PLUG INJECTOR, in which case you're just quantitatively wrong.
Vectors and LLMs are great, but there's no magic pill here. If your parts data and config management[1] is all crazy, that's an institutional problem. Buying a crapton of tokens isn't fixing it, unless you're using it to help build an actual formal solution based on good fundamentals.
[1] Such as it is.
dominotw
> particularly if your primary usecase is to search for technical keywords or codenames.
i dont believe ppl are building rag for this
eureka7
They are, I have people at work building RAG search engines for stuff that works just fine using full text search, or if you really need it, using a cheap model in codex/opencode.
You underestimate the ability of people to overengineer things.
j0selit0
this is quite nuanced. in financial markets you have a combination of natural language questions that involve technical keywords / slang / acronym. and for these specific terms, an off-the-shelf embeddings model fails miserably.
locknitpicker
> i dont believe ppl are building rag for this
What do you actually think people do when using LLMs to build AI coding agents?
andy99
Maybe I’m interpreting this differently but to me modern LLM+full text search means “agentic” - LLM gets to pick the search terms and iterate on them. The underlying LLM does know synonyms etc, better and more flexibly than an embedding model, and gets explainable feedback from failed searches.
mdp2021
That could work in a way, but it's very expensive as expressed and I do not know of prominent robust implementations.
On the other hand, your post may contain a good idea: L=instruct_LLM("provide a list of synonyms and periphrases of terms T within context C", T, C); then iter(`grep l in L`). One NN query and a `grep` collection. But again, if one wanted to order the results, it is either through a dumb crierion or through another LLM query - but this could make it extremely costly (requiring either a huge context or a quadratic number of ordering queries).
And, the above `grep` based procedure would remain keyword based and not semantic based, which means that the user must know that it will not be based on comprehension but on the possible results that keyword matching can yield.
woah
You do not know of prominent robust implementations? This is how Claude Code, GPT Codex, etc have worked for a couple years. And they do tend to be impressively good at navigating large amounts of text.
mdp2021
Thank you, no, I did not know that. Where have you found the info? Sebastian Raschka, Anthropic/OpenAI blogs?
(BTW: you made me realize - I had to take "time off" for over half a year... I am sure I missed a lot.)
--
Edit: for clarity: for "full text search" we remain on the interpretation of "searching for literal substrings" - and whether plain user provided keywords list or LLM enriched list based on the former, and whether more or less successful, it remains a syntactic search quite distinct from a semantic one. Having an LLM enrich the original keywords list can be a good idea, but the possibility of misses remains when compared to a properly working semantic search.
antonvs
> Where have you found the info?
You can see it in action if you watch the “chain of thought” text when using coding agents.
woah
Yes, they navigate with heavy use of the "grep" tool
piterrro
RAG only makes sense if you have an LLM review the results, pick the most relevant ones and iterate further if there's a need running another query and repeating the process. Raw dump of vector search (even with reranking) is asking for troubles (or rather weird user questions like 'why this crap popped up in the results?')
MarkMarine
This depends on your users and doc corpus. On finance docs and with users that use jargon and acronyms heavily, BM25 can fail on trivial queries... you end up encoding a masters in economics into the query re-writing logic.
Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)
j0selit0
I work on a really similar scenario. We ended up having to create a so-called "semantic layer" containing metadata (table and schema descriptions) and glossary terms. Still, there is a lot of work involved maintaining glossary terms since some of them are ambiguous and people have different understanding/interpretations for some of them.
rao-v
A frontier LLM based query rewriter has atleast a masters in economics, and a pretty good understanding of finance informal language. How long ago did you try this? I'd be curious if you find this still to be the case.
quijoteuniv
On my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag
idontneedcoffee
I would be really grateful if someone could battle-test my frankendb in a full-fledged RAG setup(lmdb + roaring bitmaps + to-be-removed lance with a bitmap-based virtual fs-like tree on top of your data) outside of its original narrow use-case (index for user data + workflows)
I believe the second Suggestion solves 95% of my problems. I want a system where i can describe my search and the system generated 5-15 keywords for a query
bensyverson
Yes, and don’t forget, LLMs are very good at tagging, so it’s not even that painful to backfill the corpus.
kvivek05
have you also explored graph based mechanisms to bypass some of these challenges - especially around noise generated due to similarity?
clevergadget
I don't know what level of quality is required for this site but RAG is trash its just trash. its magic beans.
jillesvangurp
RAG is basically good old information retrieval with LLMs doing the querying. This can include vector search but it works without that as well. Treating vector search as magic pixie dust that makes search great without effort is not necessarily going to work that well. Also, it can add a lot of cost and complexity to the equation. And if not tuned properly, you don't necessarily get good results.
The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).
With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.
The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.
dmix
> With search, and by extension RAG, the principle of shit in, shit out applies
Similar to SEO on marketing pages, we started rewriting product docs around the idea that it will be consumed by a RAG. Mostly by putting a lot of focus on well structured headlines, thinking more carefully about technical terminology vs common human-language questions, occasionally using variations of keywords in the text, etc. This applies to pure LLM consumption too, not just hybrid search.
Once you start tracking what users are asking you learn to adapt the documentation around it. And LLMs can also suggest improvements by comparing questions vs search results vs LLM responses.
jillesvangurp
It's a start. Where it gets tricky is companies with years/decades of highly unstructured data, duplicated documents, obsolete or draft versions of those documents, etc. And where it gets more tricky if the data is spread all over the place in weird tools, databases, spreadsheets, etc. that has some structure but is maybe a bit inconsistent, incomplete, or not that well documented.
If you flatten all that into plain text and then create embeddings, you are effectively throwing out the baby with the bathwater. But on the other hand if you put some effort into normalizing and extracting some structured meta data, you gain a flexibility to do more sophisticated querying that get you more precise results.
You can of course try to fix things at the source, which is a valid thing but usually not that practical when you have a lot of data to worry about.
IanCal
> RAG is basically good old information retrieval with LLMs doing the querying.
No - rag is doing search before you call the llm to give it context from some corpus like your helpdesk articles.
jillesvangurp
That's a very narrow definition that might have been accurate a few years ago. But of course people mostly use things like chat gpt or claude which can use tools these days, including any search tools you make available as well as web searches. Which means they typically may do multiple searches that don't just send exactly what you type but instead interpret it, adapt to what comes back and then iterate to formulate refined queries, query different search tools you might have, etc. You can still do RAG in the narrow sense you suggest but it's probably not that optimal for answering real world questions.
IanCal
It’s the type of rag the article is about however, it’s still very relevant.
jrochkind1
More LLM-generated text about LLMs.
Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
Planktonne
Your brain is incredibly adept at pattern recognition; it doesn't focus on LLM-generated text for the same reason it doesn't stare at wallpaper.
We've all learnt that it's not really communication, and so can be dispensed with.
EGreg
It’s largely because LLMs are reaching for many different types of adjectives or verbs in the same sentence, in a jarring way. While embedding it in a confidently declarative sentence. Everything sounds like some profound insight, dialed to an 11, but written as poetry. Especially those headings. With the short sentences.
allexander
I have to agree with you. Yet it is tiring, people don't even try anymore.
the biggest giveway is actually not the writing style, but the content
"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model
timedude
GPT-4o-mini is still very relevant. It is a very capable model and very very fast and dirt cheap. Excellent for this type of stuff
j0selit0
author here. at most companies I've worked for recently (F500) RAG is still quite trendy. this was what frustrated me a bit and motivated to write this article - along with other experiences that definitely relate with some of the folks in the comments above
jrochkind1
But you actually had an LLM write this article, no?
j0selit0
to be fully transparent, I used LLM to review my draft yes. and interestingly enough, this article had tens of thousands of more views and estimulated significantly more discussions than my other, non-LLM written articles. so there's that.
literallyroy
Why pretend? You mean your draft was the meat of the article and you had an LLM make it an article (sentences rather than bullet points?)
This is absolutely not a small draft review or edit.
jrochkind1
So maybe it works! They are of course models distilling the best click-baity eyeball trapping engagement processes.
cpdomina
what I implied is that in 2024 a lot was written about RAG, LLMs were trained on it, and that explains your gpt-4o-mini reference. the argument is that the article is ai slop, and that point is totally unrelated to name dropping fortune 500s references
j0selit0
sorry if it came across as name dropping - the point I wanted to make is actually that F500s are chasing RAG (as in pure semantic search without any care for chunking strategies, evals etc) instead of starting with the basics.
and for the record, my last employer was still using gpt-4o and gpt-4o mini last year. and they are an F500 (not that it means anything, just for context).
j0selit0
and for the record, my last employer was still using gpt-4o and gpt-4o mini last year. and they are an F500 (not that it means anything, just for context).
polynomial
But it's "trendy"
LukasWeber
"This is where the real trade-off lives – not cost, but speed."
That is where I stopped
alerighi
Everything that is generate from a LLM is shit, I don't know why people continue using it. I'm waiting for this bubble to explode once for all so we can return doing things in the sane way.
allexander
In the same boat here.
polynomial
The enshittification of the web, now powered by AI.
Angostura
I have a particular antipathy for articles too lazy to spell out acronyms on first use.
The audience for this piece is already very familiar with RAG. I don't want articles discussing e.g. OLED screens telling me what the acronym is - that would be a sign that the article is far below the level that I need.
triceratops
There's a middle-ground where you write out ambiguous acronyms ("rag" is an English word) and not unambiguous ones ("oled" only has one commonly-used meaning).
Angostura
I found the piece interesting, once I worked out what it was about. I strongly disagree that taking time to spell out acronyms should be taken as a signal that an article is low level.
arjie
For people familiar with the field, it would be like if you had every article about hardware read “Intel Central Processing Units (CPUs) with modern Double Data Rate 5 (DDR5) Random Access Memory (RAM) can be coupled with Nvidia Graphics Processing Units (GPUs) to run Large Language Models (LLMs) that are stored on Solid State Disks (SSDs)”. Just rapidly becomes unreadable.
The acronym constraint was valid in a pre-LLM world but now you are perhaps 3 clicks in a modern browser from learning.
If I read an article that spelled out Random Access Memory I would definitely treat that as a lay article.
vaylian
A hyperlink to Wikipedia would have solved that issue.
Lorean1
Maybe if a person can't even google RAG they are not the intended audience of that article.
ninkendo
When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”.
It’s not that I can’t or don’t know how, it’s rather that the expectation should be that a website should… link you to the information it believes to be relevant background. It’s why it’s called a “web”, linking is a core concept.
brazukadev
> When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”.
in this case there was a menu in the next empty table and you saw it but in place of getting it you want the waitress to get it for you. Which is a normal behavior but you could save your time by just getting the menu yourself.
AshleyGrant
No. It isn't. With acronyms, there's often plenty of potential things it can stand for, and if the person doesn't know enough to know which one is the correct acronym, Googling it isn't going to help them.
As OP said, simply providing a link to a Wikipedia article, or a glossary, helps widen the audience beyond "IFYKYK."
The NWS knows this and automatically links to their glossary for both acronyms as well as jargon in their discussions. <-- See what I did there? What does NWS mean in this context? If only I had provided a link that would help you know. I very easily could have. I just didn't.
serf
>As OP said, simply providing a link to a Wikipedia article, or a glossary, helps widen the audience beyond "IFYKYK."
it also serves as a minimum barrier to entry for the masses, which isn't always a bad thing.
if you're reading this stuff, and you can't figure out what kind of RAG that the search engine mentioned is being talked about through context clues, or you aren't clever enough to feed context into the search like 'hackers , computers, rag' as a query -- there is a very high probability that the person will have absolutely nothing constructive to add to the conversation that is about the topics they haven't even yet conceptualized or are aware of.
in that case that slight barrier to entry for the conversation will serve as a tool to produce less work for the moderators and derail less threads into uselessness.
(much like this stupid divergence.)
AshleyGrant
> it also serves as a minimum barrier to entry for the masses, which isn't always a bad thing.
No. Lowering the barrier of entry to those who are trying, in earnest, to learn about a new topic, to broaden their base of knowledge is NEVER a bad thing. None of us were born with the knowledge to read this (or similar) article. Trying to kick the ladder down after you have climbed it is terrible behavior and absolutely must be discouraged and stopped at all costs.
> much like this stupid divergence
The only thing stupid in this conversation is the insistence that folks who might know less than the author of the linked article or the poster do not deserve to have access to the information.
wldcordeiro
blah blah blah justifications for gatekeeping.
ninkendo
Given the title of the article is “RAG is simpler than you think”, and how I expected it to literally be an article explaining what RAG is and how it works, it’s more like going to the menu holder at the front of the restaurant, finding out it doesn’t have any menus in it, and then the waitress saying “we don’t have menus, dummy, just google it.”
serf
given the audience and the venue I think it's more like going to a restaurant as a customer and then asking the waiter to explain what a sandwich is.
ninkendo
I mean the article is entitled “RAG is simpler than you think”. I clicked the link thinking “I don’t know what RAG is, so ok I’m willing to learn something”, and the article… completely fails to explain it.
Given that the stated purpose of the article is to literally explain how simple something is, not explaining that thing seems a bit misleading, no?
I’m gonna get rich when I make a website explaining all the technical concepts in AI. Every article will just say “lol google it”, it’s gonna be great.
inigyou
I thought this would be a useless search that brought up pictures of rags, but indeed, DDG delivers a full page of results about retrieval-augmented generation for the query "rag"
mdp2021
We can confirm, RAG has been a very big thing in the past few years. It's actually bewildering that it be new to some now - but we are also getting the vibe that some are living an ""AI"-nausea" that may be shielding them from some trends.
Zambyte
Eh, a healthy web is a web. I enjoy my preferred search engine, but surfing the web is becoming a lost medium.
tux3
Hypermedia? In my hypertext markup language?
That is so not Web 5.0. Best I can offer is a support widget that pops up and keeps trying to talk to you until you interract with it.
_joel
For those times you need to Red Amber Green your BM25
alansaber
I have built systems using all of these approaches (all in tandem). For the most part, the juice is not worth the squeeze (in building a highly optimised corpus-specific information retrieval strategy) outside of a very few fringe cases. The amount of technical discussion far outstrips the use case for RAG.
seamossfet
I notice a lot of these AI written articles share this pattern where they'll present idea 1, then idea 2, and finally idea 3 which is some amalgamation of idea 1 and 2. Claude especially will present hybrid options and compromises to avoid having to make a choice then framing the hybrid option as the "best of both worlds" when they're borderline nonsensical.
"on the fly embedding" and "Sparse + dense reranking" don't really make sense how they're presented and smell like they came from a long claude-driven conversation after multiple cycles of these hybrid compromises across many turns.
ChipopLeMoral
This has Claude written all over it.
"Recipe 4: On-The-Fly Embedding (The Fresh Data Play)
The insight
If your data changes frequently, why pay to re-embed everything?"
This reads like every Claude generated presentation I've seen.
seamossfet
yeah, but I mean even prose specific claude-isms aside; the information itself is a weird patchwork of concepts
refactor_master
Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls.
Then, throw it all in BigQuery. Handles all the vector stuff natively.
Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.
I assume other vendors than Google have a similar batteries-included approach you can just plug in.
usernametaken29
> embed everything the first time
This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”
robrorcroptrer
What about splitting bigger content into chunks before embedding?
freakynit
How are you gonna handle the relations that span across individual chunks... if a later chunk refers something from 2 chunks before using `it`, rather than proper name, how will you handle that? Because at query time, that later chunk would not match.
gf000
Absolutely a novice in this topic, but I would imagine that by simply having sufficiently big chunks it's simply not a problem? You surely have enough information in like a couple of paragraphs to denote in vector space roughly what it is about. So that both chunks would get found by a vector search, and then whatever is the logic it may put the whole original text of those chunks into context, but in any case enough so that an LLM can "reason" about the references in-between the two.
harlanji
Chunks can only be as large as the embedding model’s token limit, about 512-1024 tokens usually. Anything longer gets truncated.
Natural language processing could expanded references, but it starts to get tricky. Do you use Graph RAG, embed another version of the chunk that is distinct from the full text version, etc.. Another layer of processing and data to keep in sync if the source dan be updated.
gf000
(assuming English text)
512 tokens ≈ 350–400 words ≈ a long paragraph or two.
1024 tokens ≈ 700–800 words ≈ about a page and a half to two pages.
I would be very surprised if that amount of text is not enough to encode a general topic into the embedding (otherwise, what would be the whole point of them?).
So if there is a meaningful reference in C referring to A (assuming A-B-C consecutive 1-2 paragraphs), I would expect that the content of the two at least superficially resemble each other, and a vector query for one would return both. (And also, if I am including A in the context after retrieval, then I better give some context before-after as well -- and references tend to be local).
But feel free to prove me wrong, I'm mostly just guessing, not even an educated (in the given topic) guess here.
refactor_master
Humans usually have ways around that in longer documents eg page numbers, paragraphs, links.
If someone gave me a report, in my hands, that said “see ‘it’” I’d also be confused.
mdp2021
What member freakynit said nearby about chunks and relations between chunks, plus the storage and information efficiency problem: make some calculations about storing vectors - for paragraphs and for collections of paragraphs -, then compare the needed space with the original data...
Because you could have clever ideas about vectors related to more paragraphs related in the document structure - but that would multiply the vectors. The index can become much bigger than the corpus.
khalic
you won't get anything out of a whole book embedding anyway, even a structured page is too much
j0selit0
I'm sorry is this ironic or not? doesn't sounds simple at all
cpursley
Yep, lock into some vendor from day 1. Great idea!
orisho
Vendor lock in is 2025. Porting became trivial with LLMs advancing like they have.
cpursley
What I'm saying is pick transportable tech from day 1 so you can easily move if they shut down, hike prices, decide they don't like you, etc.
jankovicsandras
If someone has a Postgres db and want very simple RAG:
The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.
7734128
There have been many blogs like this over the last years.
Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.
90% of "document" based RAG projects should view semantic search with embeddings as their primary method.
It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.
petesergeant
Embeddings are reasonably simple, but it’s a journey to get there, and I am very proud of the dog-heavy explainer I wrote on them: https://sgnt.ai/p/embeddings-explainer/
dotancohen
This is terrific, thank you! There's a typo in the following sentence:
> we don’t especially want to say that books on forestry and similar to books on puppies
^and^are
rglover
Started reading and will have to finish later but thank you for sharing. Very helpful post.
dizhn
This is very good. Thanks.
pantsforbirds
I think it's VERY project specific. If you are looking for anything technical at all, then keyword search almost always does better (in my experience). I'd actually recommend starting with keyword search, and then expanding with embeddings after you have a better idea of what your users are trying to determine.
aitchnyu
How is pgvector with Sentence Transformers, a CPU-only embedding model, compared to a model hosted by OpenAI?
bob1029
Agentic query rewrite on top of good old fashioned Lucene is the end game. This is effectively providing a lot of the same magic you get with the semantic approach. Allowing the agent to query the document store iteratively is where the capabilities become unbounded.
Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.
[deleted]
Comment removed.
pixelbro
I've not seen such a clipped cadence out of an LLM. I would not automatically suspect the GP. Maybe there's better ways to spend your time?
jrochkind1
Maybe people are just learning to write in that style LLMs learned to write from statistical people? "is where the capabilities become unbounded" is a weird thing to say and not really true. "is the end game", "add non determinism on top of non determinism", there are a lot of AI-isms in this short comment. But it's possible people are just learning to write this way now, I am curious if that's so too!
As far as uses of time, you are engaging in this dialog too, if you find it not a good way to spend time I recommend ceasing!
bob1029
It would seem AI psychosis flows both ways.
Otterly99
Althought I agree with the first point of the author that FTS is underrated in this new RAG-first framework, the whole article really hides all the problems with RAG-pipeline and kind of hand wave everything.
If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/
Note: I am not the author, just got her article in my newsletter and found it useful.
klm127
RAG stands for Retrieval Augmented Generation. The purpose is to search a corpus of text by meaning rather than exact match.
I had to look it up.
spunker540
That sounds more like semantic search and vector db.
RAG is simply fetching external data (retrieval) and adding it to LLM context (augmenting) prior to generating a final response.
Any time LLMs do a grep or a web search to answer the query, it’s RAG. Many people use vector db for their own RAG implementation bc of the semantic search benefits.
0x457
Because people writing about RAG never explained what RAG is and exclusively wrote about embeddings and vector dbs, for most people RAG became "embeddings + vector db".
People don't understand that any sort of retrieval before generation is RAG.
gabosarmiento
I would like to see how each recipe performs against its corresponding evals. Some sort of ranking would be useful.
Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.
autogn0me
It seems not many RAG compare themselves across the same benchmarks. https://ggozad.github.io/haiku.rag/ Does an ok job. The part I don’t see being discuss is the whole RL agents writing code to perform RAG queries. It’s one thing haiku-rag does that’s interesting and would like to know what other RAG have that agentic querying with benchmarks
j0selit0
author here - that's an amazing idea. would be an insanely large article though - maybe will write up a series
waximabbax
We removed retrieval from our coding agent a while back. What convinced us wasn’t a benchmark, we found that the retrieval path had been returning zero results for quite some time because of a technical bug, still nobody noticed, indeed it was working better than before.
After doing some rigorous A/B testing, we dropped indexing. For coding, I think the reason is that a repo is already searchable. Imports, call sites, file and test names, grep gives you cheap yet reliable version of what indexing would do, and the agent can read around a hit to verify it. Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source. Another thing that I noticed was the most intelligent models like Opus 5 and Fable ignored chunks anyway most of the time for some reason. Possibly perhaps they are trained around not trusting similarity checks for codebases.
Extremely large codebases with docs feel different. You can’t grep for a concept you can’t name. That’s the case where I’d still use retrieval.
(I work on TheGitAI, for disclosure.)
simianwords
OT but its interesting that none of the harnesses today use embeddings but just simple grep. I would not have predicted this
anthonypasq
cursor still uses embeddings and theyve found it works better than just grep
their data use policy from july 2026 explicitly mentions embeddings
imtringued
Ok? I'm not seeing how that is interesting, you're exclusively focusing on coding which requires precise substring locations. Google is basically almost entirely driven by embedding models now.
marginalia_nu
A lot of this is due the size of the corpus.
Grep falls apart for severely underspecified queries, which is the difficult part of web search. For any given query in web search there can be several millions of candidate results. You can get good results with FTS as well, but just finding phrase matches is inadequate, you need more ranking signals to find relevant results.
When Claude is looking for a function in your code base, it needs to sift through dozens of matches. This is not hard, and anything beyond grep is likely not worth the effort.
simianwords
And why do you think coding didn’t benefit from embeddings? It was attempted many times and the industry gave up.
I find this interesting because practically no one is doing RAG on thier personal data which is something I wouldn’t have expected.
jmutex
Chunk size matters way more than the retrieval model in my experience. Get that wrong and nothing else helps.
esafak
Don't leave us hanging! How do you set it?
seanspradlin0
But over-engineering things is fun.
RAG is one of those things where I can hyper optimize to an absolutely needless degree.
apavlinovic
The article sounds like AI slop with some predictable tells like short punctual sentences, bizarre jargon, and titles like "Recipe 4: On-The-Fly Embedding (The Fresh Data Play)"
Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"
dsego
You are right, now I noticed "Real talk" and "Why this is underrated" and I can't unsee it.
7734128
They're absolutely right – and this is is why it's a load bearing observation that cuts to the heart of the issue.
khalic
> Why this is more flexible than embeddings
Oh boy...
saltysalt
If like me you run models locally, it's pretty easy to run your own RAG locally also using a Vector Database like Qdrant for persistence, and a middle-layer like Mem0 for realtime retrial and updates. I documented the set-up steps here: https://leadprompt.sh/a/739-Building-an-Infinite-Memory-Loca...
bityard
Thanks for the nice article.
If you're looking for feedback, I'd suggest adding a short demo at the end. It would be nice to see you send it a prompt that says, "hey, remember this" and then tell it to recall that memory. Or show what the memories look like on their way to the model. Are the memories added to the context on every turn or only once per conversation?
saltysalt
Thanks for the feedback, and that's a great idea I should have done that! They are extracted and added each turn, all handled by the same middleware proxy that also handles the retrievals. I put the full code for that (memory_proxy.py) in the article.
nilirl
Maybe I'm old but where exactly are the "dragons"?
How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?
If so, I'd like to see more design patterns around existing search problems:
- Correcting or backtracking based on feedback.
- Measuring relevance.
- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?
brabel
The whole embedding thing which converts “tokens” to vectors, which you then store in a vector database so that you can later query by vector distance, seems to be LLM specific technology, no? As far as I know the vectors look a lot like the weights in a LLM itself which is why the vector search also works with some level of intelligence.
triangle
Vector embeddings predate LLMs. They have been used as far back as the early 2000s. They are a general machine learning technique, rather than LLM specific
ozim
Unfortunately LLMs made vector search more popular so it seems like something LLM specific.
What makes it worse, a lot of people in the thread equate vector search with RAG, whereas RAG is the name for anything that model can query so a user doesn't have to copy/paste feed it to the model manually like access to text files is RAG.
nilirl
Sure and that's a new technique for indexing and querying.
Where's the new design tension? Indexes always had to be monitored for freshness and queries have always needed cleaning or parsing.
KaseyKim
right, it is the foundation of machine learning.
ewidar
not really, vectorising text/books is old school ML by this point.
I wonder who was doing doing semantic search in the last century!
"The future is already here—It's just not very evenly distributed..."
vintermann
Sure, the idea of making a vector embedding for words, sentences, documents etc. is old, but the meat is in how you construct this embedding. I think embeddings have gotten quite a bit better since word2vec.
TudorAndrei
It's just information retrieval packaged as something new.
mdp2021
It's just information retrieval through a new NN based technology that allows to map concepts and ideas as the compression of long text into points in a multidimensional space that manages to compress even more dimensions than the given ones, through non-transparent engines that give different mappings and results, and still (the information retrieval) requires many more clever tricks than the simple idea of vector distance ordering because things do not quite work as they should.
Let's say it's just "computation packaged as something new". "Trivial things".
kachnuv_ocasek
And you can't fundraise on some old "information retrieval".
ivansavz
Does anyone have experience using SMLs for RAG (either as query rewriter or as generator for the final answer)?
I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.
I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."
Any info or advice would be appreciated.
ShinyLeftPad
You can just ask your LLM about it and get this article. This should've been a prompt.
trivet
Start with BM25 and only add embeddings when keyword search actually fails you. Saves a lot of pain.
Wren_ops
Agreed, simpler is almost always better. The hard part is resisting the urge to over-engineer it.
Tycho
I don’t understand the 4th option, “on the fly”. It didn’t seem to be explained properly.
j0selit0
author here - apologies for it not being clear. the idea here is:
step 1: sparse index retrieval (FTS/BM25) - say with k = 10
step 2: re-rank the 10 records using embeddings
the difference in this approach is during step 2, you convert text to embeddings on the fly - when you're running the retrieval pipeline, meaning you don't need to have all of your corpus pre-embedded in a vector db
respectattentio
I believe embedding-based RAG, everybody is using, will end. As chips advance, you would use a big llm instead of word embedding for retrieval. It's much more accurate and extensive covering every topic.
Still need ~2 years to be replaced.
inigyou
How would you use a big LLM for retrieval?
respectattentio
As simple as a prompting it with structural output or restrictions for your criteria.
With agents, the prompting could be dynamic for maximum accuracy for every retrieval.
This absolutely would beat the best of the best embedding-based RAG models.
Nobody uses this now mainly due to speed. An llm retrieval would be 10x or more slower than embedding.
You can try that now
Take some failing cases or bad retrieval from your current system
Prompt an llm wisely like a perfect prompt to get what you want and provide it the context to it.
And see the results.
For context, you are limited now by models contexts (1m), so mostly you would need to split what you have and prompt twice....or more...and so on
inigyou
So uh ... Where's the retrieval part? You know RAG is used to implement that, right? You're basically saying "we don't need an ALU, we can just use the Windows calculator"
respectattentio
The only difference is using LLMs instead of Embedding models
maxrumpf
The easiest way to strip complexity is to expose simple tools to an agent model like SID-1 that can use them well. It makes more of an effort for hard questions, and little effort for easy ones.
(found of sid.ai so obv biased)
yipinwong
Only those who mastered the craft makes their work look simple.
The AI that wrote this might be the master not the writer, as this looks written by AIs.
I will use the author's agents, not read his articles or use him for the job.
j0selit0
author here - thanks, I'm honored you would use my agents :)
yipinwong
I'd still pay for your brain (expertise, experience for built them), which is separate from your agent :)
j0selit0
that makes me even more humbled :)
garn810
Yet people sell vector DBs solutions as if it's a guarded magic knowledge
Whole LLM agent tool call with ripgrep gives 99% use cases right lol
pioneerjeff
What RAG means for AI is what a library means for human beings.
It's necessary and would be good for you if you want to learn something systematically.
But for most of the normal issues, we can not rely a lot on it.
Silasdev
Very little of this is RAG but rather just FTS with clever reformulation and re-ranking.
RAG is about providing an grounded response, given the actual data in the corpus.
Great article and content, nonetheless!!
j0selit0
author here - thank you!
Silasdev
I apologize for write "just" FTS. I know there is a lot of work involved and your article summed it up pretty damn well, in a way that makes it approachable for someone new to the topic.
I will keep a note of this article for next time I am asked about this topic.
KaseyKim
i want to ask that, if a user want to search sth, but he doesnt know the exact name(keywords), just some description. at this moment, whether the text serach fail?
timedude
Text search is not ideal for that. I such cases embedding works better
Alifatisk
I skimmed through the article and it seemed okay. But then I lost my enticement when reading the comments saying this is an LLM written article.
akshay_akula
Agreed. Embeddings are cheap to try and hard to mess up. Most projects can do plain semantic search first and see if they ever need more.
sangwook
Im curious whether the $10,000 figure includes unstated migration costs, since the raw embedding API cost under the earlier assumptions comes to $10.
maxweylandt
> Elasticsearch. Postgres full-text search. The stuff that existed before “embedding” became a verb.
Noun, no?
_pdp_
All computer primitives are relatively straightforward in pure form and vastly more complicated in real-world scenarios.
hn58622tsf
Bookmarked, thanks again
zabriel_goss
Helpful framing, thanks for sharing!
geniium
yet harder to implement proplery than you think
bewareofscams
RAG is so 2024.
ufocia
Wow! Terrible layout. Shouldn't fully justify on a small screen.
It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.
It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.
>On this benchmark, a pure LLM generated an accuracy score of zero. Adding RAG, prompt engineering, and agentic AI raised accuracy to the 10+% range.
[1] Any text-to-SQL benchmark should address difficulties of real-world data stores (acm.org) (21 comments):
https://news.ycombinator.com/item?id=49013995
[2] If You Think You Can Do Real-World Text-to-SQL:
https://cacm.acm.org/blogcacm/if-you-think-you-can-do-real-w...
[3] BEAVER: An Enterprise Benchmark for Text-to-SQL:
https://beaverbench.github.io/
Before a lot of frameworks existed, you'd see DEVs taking user input on a web form, and then just throwing it directly at the MTA. So spammers could submit email@address\nCC: persontospam@address, and the like.
Now LLMs are a different beast, but you have input validation for LLMs, unique to all other validation methods. Yet there's actually no safe way to ever validate user input for a LLM, except for very rigid input validation on single words. Take the email example above. You'd need a regex to only validate an email address (and that isn't simple), but once you expand it to actually allowing sentences?
The LLM is now input validation vulnerable.
And that means no user input can be used in unvalidated commands.
And then just random hallucinations. I'm curious how the gp managed weirdo LLM behaviour, like out of the blue 'drop table' or accidental select into as opposed to just select.
RAG is routing and decision making.
I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.
I agree with you.
Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.
It maintains itself.
Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.
I found my niche with RAG selling and I build them myself.
I take pride in them.
So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.
And there are regulated businesses where certain information is required to be always factual correct - pricing for example.
Vector search becomes a liability for this use case.
So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.
I love this kind of stuff.
And there is personal information etc.
Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.
And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.
It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.
I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.
RAG gives me dbCORE vibes so to say.
Good luck and fun with your RAG systems.
https://www.anthropic.com/engineering/contextual-retrieval
This is from two years ago, but I think it's still SotA?
I think there needs to be a linguist version of "what every programmer needs to know about (full?) text search"...
I'm not a linguist and I don't study languages, but I know enough to realize if a text search system is not designed for a particular language, it simply won't work. (As an example, to implement English search in a system for a hobby project, I had to import a US/UK spelling wordlist, and implement the Porter Stemming Algorithm. This is just for "one" language, and probably does not cover the other "English" dialects. Imagine doing a different workaround for every language in existence...)
RAG is actually a very language-agnostic way to work around those issues.
Volume of documents, size of documents, versioning, frequency of update, documents similar or overlapping information, how much or exactly what you need for the LLM to understand, AI friendly documents, who has access and at what level, blue teaming, red teaming, multi-lingual, does the LLM know the domain language of the user and documents.
I probably missed a few things even with that.
I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours.
As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
Vectors and LLMs are great, but there's no magic pill here. If your parts data and config management[1] is all crazy, that's an institutional problem. Buying a crapton of tokens isn't fixing it, unless you're using it to help build an actual formal solution based on good fundamentals.
[1] Such as it is.
i dont believe ppl are building rag for this
You underestimate the ability of people to overengineer things.
What do you actually think people do when using LLMs to build AI coding agents?
On the other hand, your post may contain a good idea: L=instruct_LLM("provide a list of synonyms and periphrases of terms T within context C", T, C); then iter(`grep l in L`). One NN query and a `grep` collection. But again, if one wanted to order the results, it is either through a dumb crierion or through another LLM query - but this could make it extremely costly (requiring either a huge context or a quadratic number of ordering queries).
And, the above `grep` based procedure would remain keyword based and not semantic based, which means that the user must know that it will not be based on comprehension but on the possible results that keyword matching can yield.
(BTW: you made me realize - I had to take "time off" for over half a year... I am sure I missed a lot.)
--
Edit: for clarity: for "full text search" we remain on the interpretation of "searching for literal substrings" - and whether plain user provided keywords list or LLM enriched list based on the former, and whether more or less successful, it remains a syntactic search quite distinct from a semantic one. Having an LLM enrich the original keywords list can be a good idea, but the possibility of misses remains when compared to a properly working semantic search.
You can see it in action if you watch the “chain of thought” text when using coding agents.
Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)
https://github.com/canvas-ui/canvas-synapsd
The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).
With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.
The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.
Similar to SEO on marketing pages, we started rewriting product docs around the idea that it will be consumed by a RAG. Mostly by putting a lot of focus on well structured headlines, thinking more carefully about technical terminology vs common human-language questions, occasionally using variations of keywords in the text, etc. This applies to pure LLM consumption too, not just hybrid search.
Once you start tracking what users are asking you learn to adapt the documentation around it. And LLMs can also suggest improvements by comparing questions vs search results vs LLM responses.
If you flatten all that into plain text and then create embeddings, you are effectively throwing out the baby with the bathwater. But on the other hand if you put some effort into normalizing and extracting some structured meta data, you gain a flexibility to do more sophisticated querying that get you more precise results.
You can of course try to fix things at the source, which is a valid thing but usually not that practical when you have a lot of data to worry about.
No - rag is doing search before you call the llm to give it context from some corpus like your helpdesk articles.
Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
We've all learnt that it's not really communication, and so can be dispensed with.
"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model
This is absolutely not a small draft review or edit.
and for the record, my last employer was still using gpt-4o and gpt-4o mini last year. and they are an F500 (not that it means anything, just for context).
That is where I stopped
So: https://en.wikipedia.org/wiki/Retrieval-augmented_generation
The acronym constraint was valid in a pre-LLM world but now you are perhaps 3 clicks in a modern browser from learning.
If I read an article that spelled out Random Access Memory I would definitely treat that as a lay article.
It’s not that I can’t or don’t know how, it’s rather that the expectation should be that a website should… link you to the information it believes to be relevant background. It’s why it’s called a “web”, linking is a core concept.
in this case there was a menu in the next empty table and you saw it but in place of getting it you want the waitress to get it for you. Which is a normal behavior but you could save your time by just getting the menu yourself.
As OP said, simply providing a link to a Wikipedia article, or a glossary, helps widen the audience beyond "IFYKYK."
The NWS knows this and automatically links to their glossary for both acronyms as well as jargon in their discussions. <-- See what I did there? What does NWS mean in this context? If only I had provided a link that would help you know. I very easily could have. I just didn't.
it also serves as a minimum barrier to entry for the masses, which isn't always a bad thing.
if you're reading this stuff, and you can't figure out what kind of RAG that the search engine mentioned is being talked about through context clues, or you aren't clever enough to feed context into the search like 'hackers , computers, rag' as a query -- there is a very high probability that the person will have absolutely nothing constructive to add to the conversation that is about the topics they haven't even yet conceptualized or are aware of.
in that case that slight barrier to entry for the conversation will serve as a tool to produce less work for the moderators and derail less threads into uselessness.
(much like this stupid divergence.)
No. Lowering the barrier of entry to those who are trying, in earnest, to learn about a new topic, to broaden their base of knowledge is NEVER a bad thing. None of us were born with the knowledge to read this (or similar) article. Trying to kick the ladder down after you have climbed it is terrible behavior and absolutely must be discouraged and stopped at all costs.
> much like this stupid divergence
The only thing stupid in this conversation is the insistence that folks who might know less than the author of the linked article or the poster do not deserve to have access to the information.
Given that the stated purpose of the article is to literally explain how simple something is, not explaining that thing seems a bit misleading, no?
I’m gonna get rich when I make a website explaining all the technical concepts in AI. Every article will just say “lol google it”, it’s gonna be great.
That is so not Web 5.0. Best I can offer is a support widget that pops up and keeps trying to talk to you until you interract with it.
"on the fly embedding" and "Sparse + dense reranking" don't really make sense how they're presented and smell like they came from a long claude-driven conversation after multiple cycles of these hybrid compromises across many turns.
"Recipe 4: On-The-Fly Embedding (The Fresh Data Play)
The insight If your data changes frequently, why pay to re-embed everything?"
This reads like every Claude generated presentation I've seen.
Then, throw it all in BigQuery. Handles all the vector stuff natively.
Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.
I assume other vendors than Google have a similar batteries-included approach you can just plug in.
This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”
Natural language processing could expanded references, but it starts to get tricky. Do you use Graph RAG, embed another version of the chunk that is distinct from the full text version, etc.. Another layer of processing and data to keep in sync if the source dan be updated.
512 tokens ≈ 350–400 words ≈ a long paragraph or two. 1024 tokens ≈ 700–800 words ≈ about a page and a half to two pages.
I would be very surprised if that amount of text is not enough to encode a general topic into the embedding (otherwise, what would be the whole point of them?).
So if there is a meaningful reference in C referring to A (assuming A-B-C consecutive 1-2 paragraphs), I would expect that the content of the two at least superficially resemble each other, and a vector query for one would return both. (And also, if I am including A in the context after retrieval, then I better give some context before-after as well -- and references tend to be local).
But feel free to prove me wrong, I'm mostly just guessing, not even an educated (in the given topic) guess here.
If someone gave me a report, in my hands, that said “see ‘it’” I’d also be confused.
Because you could have clever ideas about vectors related to more paragraphs related in the document structure - but that would multiply the vectors. The index can become much bigger than the corpus.
https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )
The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.
Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.
90% of "document" based RAG projects should view semantic search with embeddings as their primary method.
It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.
Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.
As far as uses of time, you are engaging in this dialog too, if you find it not a good way to spend time I recommend ceasing!
If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/
Note: I am not the author, just got her article in my newsletter and found it useful.
I had to look it up.
RAG is simply fetching external data (retrieval) and adding it to LLM context (augmenting) prior to generating a final response.
Any time LLMs do a grep or a web search to answer the query, it’s RAG. Many people use vector db for their own RAG implementation bc of the semantic search benefits.
People don't understand that any sort of retrieval before generation is RAG.
Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.
After doing some rigorous A/B testing, we dropped indexing. For coding, I think the reason is that a repo is already searchable. Imports, call sites, file and test names, grep gives you cheap yet reliable version of what indexing would do, and the agent can read around a hit to verify it. Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source. Another thing that I noticed was the most intelligent models like Opus 5 and Fable ignored chunks anyway most of the time for some reason. Possibly perhaps they are trained around not trusting similarity checks for codebases.
Extremely large codebases with docs feel different. You can’t grep for a concept you can’t name. That’s the case where I’d still use retrieval.
(I work on TheGitAI, for disclosure.)
https://cursor.com/blog/semsearch
https://cursor.com/data-use
their data use policy from july 2026 explicitly mentions embeddings
Grep falls apart for severely underspecified queries, which is the difficult part of web search. For any given query in web search there can be several millions of candidate results. You can get good results with FTS as well, but just finding phrase matches is inadequate, you need more ranking signals to find relevant results.
When Claude is looking for a function in your code base, it needs to sift through dozens of matches. This is not hard, and anything beyond grep is likely not worth the effort.
I find this interesting because practically no one is doing RAG on thier personal data which is something I wouldn’t have expected.
RAG is one of those things where I can hyper optimize to an absolutely needless degree.
Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"
Oh boy...
If you're looking for feedback, I'd suggest adding a short demo at the end. It would be nice to see you send it a prompt that says, "hey, remember this" and then tell it to recall that memory. Or show what the memories look like on their way to the model. Are the memories added to the context on every turn or only once per conversation?
How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?
If so, I'd like to see more design patterns around existing search problems:
- Correcting or backtracking based on feedback.
- Measuring relevance.
- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?
What makes it worse, a lot of people in the thread equate vector search with RAG, whereas RAG is the name for anything that model can query so a user doesn't have to copy/paste feed it to the model manually like access to text files is RAG.
Where's the new design tension? Indexes always had to be monitored for freshness and queries have always needed cleaning or parsing.
at least to me that seems the same as https://en.wikipedia.org/wiki/Word2vec for e.g.
"The future is already here—It's just not very evenly distributed..."
Let's say it's just "computation packaged as something new". "Trivial things".
I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.
I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."
Any info or advice would be appreciated.
step 1: sparse index retrieval (FTS/BM25) - say with k = 10 step 2: re-rank the 10 records using embeddings
the difference in this approach is during step 2, you convert text to embeddings on the fly - when you're running the retrieval pipeline, meaning you don't need to have all of your corpus pre-embedded in a vector db
Still need ~2 years to be replaced.
With agents, the prompting could be dynamic for maximum accuracy for every retrieval.
This absolutely would beat the best of the best embedding-based RAG models.
Nobody uses this now mainly due to speed. An llm retrieval would be 10x or more slower than embedding.
You can try that now
Take some failing cases or bad retrieval from your current system Prompt an llm wisely like a perfect prompt to get what you want and provide it the context to it. And see the results.
For context, you are limited now by models contexts (1m), so mostly you would need to split what you have and prompt twice....or more...and so on
(found of sid.ai so obv biased)
The AI that wrote this might be the master not the writer, as this looks written by AIs.
I will use the author's agents, not read his articles or use him for the job.
Whole LLM agent tool call with ripgrep gives 99% use cases right lol
It's necessary and would be good for you if you want to learn something systematically.
But for most of the normal issues, we can not rely a lot on it.
RAG is about providing an grounded response, given the actual data in the corpus.
Great article and content, nonetheless!!
I will keep a note of this article for next time I am asked about this topic.
Noun, no?