I reached a similar conclusion: LLMs should only really sit at the terminals of request fulfilment.
1. User request understanding: natural language -> a more rigorous representation, in my case Datalog.
2. Result interpretation: facts and derived facts -> natural language.
Between those terminals, the work should be mechanical reasoning over some ontology or formal knowledge structure.
That connects to another principle I've been thinking about, which I call Weathering: useful reasoning should change the shape of the system. If an LLM has already had to infer a relation, mapping, rule, or abstraction, repeated use should wear that inference into the system so that the next similar request doesn't require discovering it again from scratch.
With continued use, a weathering-capable system should therefore require less and less probabilistic intelligence for recurring work. Put another way, there should be a declining marginal cost of cognition since the products of intelligence harden into structure that can subsequently be reused and evaluated mechanically.
abhgh
What you call "Weathering" has been a constant gripe of mine. We have LLM-driven softwares toward that almost seem to start from scratch every time a request comes in - there are mechanisms to learn or generalize, like writing out a memory, but they are not reliable or reliable in general. There is no convenient lever to be able to say "yes this is in the memory but the request seems like it needs a fresh scan of data, so ignore your memory", or the opposite "you can infer this from stuff in the memory - don't re-reason!". There is some work like Dynamic Cheatsheets [1] and Agentic Context Engineering [2] that have studied this aspect, but we are far from a generally reliable solution. And till we have that, I think the system variations for systems trying to solve this problem are going to be (a) LLM-leaning: create unstructured memory files, with human in the loop as a filter to reject inaccurate responses (b) LLM-as-a-layer: what you describe and the article kind of is doing.
I’ve been thinking about this.. the same idea behind Dynamic cheat sheets paper and the ACE paper. Thanks for those links! Nice to see those ideas academically studied.
My problem is the context of today’s models (that I’ve been using - Claude Opus 5 and GPT-Sol) are a black box to a user like me. I cannot tell what they already hold in their context over the duration of a coding session. I cannot tell influencers a little by using/not using MCP tools, AGENTS.md like MD files, but what they _actually_ use at inference time is still opaque.
For example, if I insist that GPT-Sol use a size bounded dynamic cheat sheet for my project, and even have those reflector&curator agents refine them concurrently, what is to say GPT-Sol doesn’t have an intrinsic cheat sheet like that already, and it is just reading/curating/managing this one in addition and only getting confused needlessly along the way?
Similarly, all those codegraph like tools and semantic search addons.. what is to say the model genuinely uses them instead of relying on its own inference techniques and just “perform” for the user audience as if they are using these fancy tools?
This is not to say we shouldn’t try more ways of managing context and just give up to our AI provider overlords. Just thinking through the logic of what we are working with.
throwup238
> My problem is the context of today’s models (that I’ve been using - Claude Opus 5 and GPT-Sol) are a black box to a user like me.
Would it even help if it weren’t a blackbox? Maybe to the frontier labs with resources, but even trying to read through the thinking traces of an open model doing a nontrivial task easily takes me an entire day to do correctly (i.e. actually looking at the code it read in) and much more if I’m trying to tune a specific repeatable task.
Applying that to agentic coding seems like a losing proposition. The only way to manage it seems to be more agents and that’s its own rabbit hole of compounding errors (like Claude’s useless comments).
alansaber
Theoretically but practically any LLM generated infra/classification set is going to drift due to inaccuracy and harm IR/whatever logical process you're using. I am a big fan of using a loose taxonomy but it's not been revolutionary.
sirtaj
I guess efforts like Google's OKF, which teach the llm to manage the ontology directly, are the opposite of what you want.
The weathering need is real. An llm equivalent of dbms analysing usage pattern to tune indexes.
PcChip
When i hear “weathering” i think of something slowly eroding away
akk0
I feel like the thing eroding away in this case is a blocker, a friction. Think of a river cutting a path through the mountains.
So he's using an LLM to generate data stored in an "is_a" representation.
That's so classic AI.
Soon, he'll discover that he needs quantifiers. Then that "for all" is too strong sometimes, and he needs "for most". That way lies Cyc.
It's not a bad idea. But it does have a history.
mentalgear
In general, what all the big LLM providers are doing is moving towards classical & neural (neuro-symbolic) AI - even though they dont publicly admit it because that would counter their claims for years of "scale is all you need" (which has vanished with diminishing returns, see $MS / altman's GPT-5 bet).
doginasuit
It seems like the two approaches compliment each other nicely. Human intelligence also relies on parallel information processing. LLMs are like a massive working memory, incredibly effective but with a similar set of limitations. What they lack is a symbolic model of reality, something that they can build and refine.
RussianBot9580
It's strange to frame this as classical vs scale. Us humans have a powerful inference engine in our heads. We also use a calendar to avoid re-deriving everything from first principles before we've had our morning coffee. Businesses couple many creative (human) agents together. They also have processes and rules.
mentalgear
It isn't, and it is not what was described: its about unbounded imagination (neural / LLM) that needs reality constraints (symbols / rules) to produce useful output.
Think of it like human imagination may do anything (flying cars), but the real world has constraints and we use language/writing (symbols) and rules bound to them to simulate and reconcile our imagination with reality to actually flow our energy into something that may work in the real world.
nz
The various advances in LLM technology tend to rhyme with the advances in computer programming in general. For example, the stunts that involved getting LLMs to create compilers and browsers are really just extremely expensive[0] versions of genetic programming (none of it would have worked without using the test-suite as a fitness-function). The recent news of migrations from one test-framework to another (featuring Asana, I believe), was something that we could always do trivially in a language that was based on S-Expressions (Lisp, Scheme, etc).
In fact, both Cyc and the "AI" Labs have the _same basic thesis_: Intelligence is, primarily, a data entry problem. They just disagree about what kinds of heuristics should be run over that data (logic-programs, neural-nets).
Whenever I read about someone using LLMs to write code, it _very closely_ resembles how Lenat was using Eurisko/Cyc to solve problems: they let the system run continuously, and they "nudge" it in "interesting" directions, "when it gets stuck", or "runs out of steam". (Quotes indicate their phrasing, not mine)
Even Lee Spector noticed something analogous with his genetic programming system. When he tried to get it to discover optimal data structures (or maybe it was sorting algorithms, I forget), the system would quickly "run out of steam", without a solution. But when they added new verbs/opcodes to the system, that were a better fit for that domain (e.g. index-based memory loads + stores), it converged on a solution very quickly (even for GP, domain specific languages keep delivering unreasonable wins). You will note that this rhymes with the "micro-theories" of Cyc, which in turn rhyme with the SLMs of the AI labs.
In my personal experience, most of the "silver bullets" do not work (obviously), but some of them do nudge you towards being a better programmer (by refining your intuition about the problem specifically, and computers more generally).
EDIT: just remembered something. LLMs tend to produce larger and larger programs over time, and most people (IIRC) interpret this as a kind of entropy. This happens to rhyme with a similarly observed behavior in GP. Most genetic programs that do not have a fitness function that rewards smaller size, tend to grow in an unbounded way. The reason for this, is that most of the code/genes are useless, and random mutations do not lobotomize the program under evolution. I suspect that the coding LLMs tend to grow their code for similar reasons.
[0]: I suspect that, this was mostly a triumph of enormous amounts of hardware, more than the actual LLM technology. I further suspect that a traditional GP approach, on the same quantity of hardware, could have gotten there faster (if not better as well).
z0r
I find your perspective on LLM generated programs to be an interesting and fresh take
akkad33
How do you know this?
pegasus
They're probably referring to tech like reasoning models, or agent harnesses for example, which are arguably slowly moving things towards the symbolic end of the spectrum.
mentalgear
LLMs using REPL are one instance of symbols to "bounce" their prediction against domain constraints for verification. Also shout out to Gary Marcus who was right after all (and LLM companies wasting 100s of billions of dollars for years in-between on pure scaling).
IsTom
With validity intervals mentioned it'll also be nice to have LTL's "next" and "until" too.
jarboot
I encountered this with trying to have LLMs populate facts about electoral campaigns. Like when a candidate drops out, when endorsements happen, but also if a candidate is un-endorsed or drops and rejoins. It also needed to handle if any of these facts were incorrect.
I settled on a knowledge graph in Postgres and downloading/storing the source documents so it could iterate on past results without more scraping or network calls.
This blog post helped me understand security analysis in this context! A lot of the important systems around malware analysis or large scale system security (the parts people really care about) clicked for me. So thanks for writing it.
Anyways I hope we can find some pattern to converge on with this wrt "facts management" since I feel this is currently something a lot of people and LLMs are struggling with. In practice current LLMs working with episodic memory feels similar to a grandparent with dementia scrawling things down in notebooks, crossing things out, and getting very confused.
mirekrusin
I had similar problems to solve for trading advice, complex project reasoning etc.
I settled on simplicity, extended it to serve clear, useful purpose.
Started with markdown database, single fact per line, structured/parseable (subject VERB object). This can be easily diffed/reviewed etc in git. Then added optional metadata (confidence, tags, persisted comments for natural language, uncertainty for numeric values, context, timestamps/spans), querying, alternative sqlite3 backend, self describing VERBS, z3 solver etc.
Those kind of graph information systems are great to quickly structure knowledge in a way that LLMs and humans can use/act on/loop on. Creating ontology, linking, some rules and actions and kicking it so it all munches and spits out results that feed back in so it self evolves. It's very natural for llms to query/update/restructure those graphs (also good for humans because it's very terse, essential information only).
There is no need to create k8s style complexity/services/what-not, it all works well from single sqlite db file or bunch of markdown (.cave in my case) files.
It's also interesting to see how well local open weight models are dealing with information arranged this way.
keeda
Very cool. I recall an HN submission (which I can't find offhand unfortunately) that did something similar -- it used an LLM to decompose articles into a set of statements which were used to construct an entity-relationship graph of facts and events. It then queried that using conventional graph query methods, much like DataLog / Lemmalog is doing here. I remember it was particularly effective at answering timeline-based queries that LLMs (back then) sucked at.
I think approaches like this are going to be (or maybe already are?) the basis of effective grounding of LLM responses in authoritative data sources. It should be possible to pinpoint any error to an incorrect traversal or an incorrect "fact." This would work best for concrete, unambiguous facts, however; fuzzy, ambiguous or opinion-based information will probably remain the purview of LLMs.
This is great for evidence grounding, but doesn't produce a large memory/reasoning improvement (in most cases)
manmal
My problem with memory is that it goes stale, and updates to facts are often not changing all locations of that fact. Such a system should make it easier to maintain a single source of truth, and versioning, no?
alansaber
Yes you need a VCS and to recalculate relationships at certain thresholds. It adds a lot of complexity and computarional overhead, which is why there's no uniform widespread graph use
iamflimflam1
This really matches up to my experience on long research projects with Claude.
It’s very hard to remove information - Claude has a habit of recording things all over the place and will happily treat things as facts even after they’ve been disproved.
What is currently true can get easily contaminated with old “facts”.
trinsic2
Something of this capacity would be useful in investigating obscure hardware failures in the logs that I couldn't confirm because the problem was not being observed while the device was in my shop. the problem was surfacing in another location probably due to some set of circumstances in the software that I could recreate, or some particular peripherals that were attached.
I ran into the very same problem of the LLM forgetting that we ruled out a conclusion that was verified not to be the cause as it came up further in the conversation history while I was exploring possibilities.
I had to keep reminding we ruled out that conclusion prior.. I just carried on with having the LLM capture some of the supporting sources of other people experiencing the same problem and kept having to refine those sources because it was focused only on summaries, but eventually i got the sources to a point where they were good enough hypothesis that we could formulate a better conclusion on what the potential cause was.
coder-pm
This is the fact I’ve been struggling with for quite some time. It’s not because it forgets the facts, it’s because the invalidation doesn’t propagate.
My way of handling that is a decision log. For every project since I started doing that it’s working great. My CLAUDE.md instruct the agent to store my every decision to the file with a metadata when I made this decision and what was the context. The agent is using this file as an index of decisions and rarely lose a track. It also helps team members to find out more about the development phases.
Does your system invalidate the parts of the memory if these are not valid or relevant anymore or just store/retrieve?
alexpotato
I recently stumbled upon the technique of asking the LLM to create a Dot Viz (or mermaid) flowchart of the program flow.
The LLMs are great at:
- understanding the flow
- making diagrams
- running the code with logging they add to to even better understand the flow
The flow being in Dot (or other machine readable format) makes it even easier for the LLM to use that as a reference going forward.
schmuhblaster
Great work! If anyone is looking for a way to integrate something like this into their own harness or the pi coding agent, then you might be interested in DeepClause [0]. It comes with a Prolog-like language implemented on top of SWI-Prolog (WASM Version). The purpose of the project is to allow for broad experimentation around the intersection of LLMs/Agents and GOFAI. So you could use it to build memory systems like OP did, create executable specs, define graphs and loops for agents and subagents... It also comes with a pi extension that greatly simplifies getting started with it.
Opposed to OP, DeepClause uses Prolog semantics, so running some more complex queries on knowledgebases might cause some issues (which is the use case where a Datalog might be more useful). For smaller scales it should be fine though.
I'm not sure if this is the right direction, but it's certainly momentarily helpful. I think the right direction would be to enable the model itself do dynamic program analysis, deterministically and dynamically via runtime-inference.
btw. your comment is grayed out, not sure what it means. However, thanks for sharing, I'll look into it.
apt-apt-apt-apt
Is this the kind of thing that works when you have a tight domain-specific language, but devolves into natural language (English) eventually?
E.g. A -> B, B -> C so A -> C (works, great). A -> B when A is sort of red and blue, also A is intermittent (what to do now?)
sigbottle
Yeah, philosophy of science & analytic philosophy (especially with Quine) has been down this rabbit hole before.
Formalisms should be in service to higher-level intelligences, not the other way around. It's pretty clear that LLM's are intelligent inherently; the lean doesn't just "prove math itself". (Admittedly, I'm not full blown AGI pilled either - there are some structural constraints that do make me think there is room to be gained in intelligence. And formalism will play a part in that. But it's not the end-all be-all to it).
frumiousirc
Datalog seems like a way to "spell" knowledge graph (KG).
The article touches on Datalog statements changing over time. One ingredient I think would be good to add to the system is to make every statement carry "providence" metadata. The providence should be sufficient to enable later confirmation that a statement is still valid or if the statement needs to be reformed without the need to remake the entire graph from scratch.
I would make at least some forms of providence follow a strict schema that is defined for the subject matter that is being captured. For example, statements about a code base should refer to the source files and their version (file modification date, content hash) from which the statements were concluded. When a source file is modified we may then find all statements made from them and reevaluate just those statements.
The next level would be to keep statements even if reevaluation breaks them and add a method to derive a subgraph for a given state of the subject. For example, over many releases of a code base, a lot of statements would not change, some would. Having a graph that spans all conclusions about all releases of a code base and a way to form the subgraph for a specific release would allow the system to efficiently target queries for a particular release.
grey-area
I think you mean provenance, not providence
pixelsort
The OP rediscovered that frontier models prefer to reason over logical scaffolding for complex tasks. They excel at technical work with many constraints as they can perfectly maintain the references, flow graph, and evidence states while they works through your conformance gates.
Many people here might disagree; but they're holding it wrong. Those folks should ask:
1. Am I using free tier tokens?
2. Am I working on trivial software?
3. Am I expecting models to adapt to my ways of thinking?
Anyone affect by any of these three mistakes will maintain an impenetrable filter of perpetual ignorance about model capabilities. Since the OP came with receipts, I'm reproducing an example graph below.
if you aren't familiar with the literature, then good on you for getting the right insight. provenance and support are staples of the datalog world.
the other really handy thing here is that it very straightforward to differentiate datalog rule sets. so if you change a fact, you can run a smaller solution that tells you which consequences are affected by the change. under the assumption that the evaluator is pure, we don't need to keep the support graph as you suggest, we can just generate the deltas from evolution of the differentiated form.
detailed provenance can be of real application utility, but if you just care about the accounting, keeping track of the number of supports for each fact is sufficient.
another model which is fun is to make the version of the database (monotonic time) an explicit field in your base facts, assuming you can afford to keep the whole history. a deletion then is just a negative-support, and you can ask questions about the state of the world at any time.
linguae
This summer I’ve been investigating agentic coding with local LLMs, and while I’m far from an expert, one thought that has been on my mind is leveraging techniques from “old-school” AI such as heuristic search to guide agents when it comes to planning. The use of Datalog in this article resonates with me, since logic programming was a major part of old-fashioned symbolic AI. I’m very curious about this combination of “old-school” AI and LLMs.
ianhorn
That’s practically what i’m working on! But i’m training the models to do this natively. It gets complex given the inherent relevant uncertainties (aka making it resilient to their bullshit). Luckily, datalog and monotonic logic in general is reasonably amenable to packed representations of the possible knowledge bases, and reasoning from them in a way that lends itself to decent UX, but it’s been a huge amount of elbow grease to get working with reasonable complexity end-to-end.
Goofy_Coyote
What a great write up.
One of the few long form contents that I just opened and read from top to bottom without planning for it or keeping the tab open to read later.
Great job.
I’ve been battling the same problem, and I solved it by keeping the state in my brain, long focus hours, and breaking down the problem to smaller chunks that agents could almost one-shot.
That made me the bottleneck, and although I can do it for codebases I’m familiar with, working on totally new projects has been very painful.
I’m going to test it in my own vuln research workflow.
jnpnj
Isn't that why we created models and graphs before ? Maybe now the uml to source roundtrip can be solved with LLMs. Back in 2010s that's where it was stuck.
Note: I wonder how many people reified their codebase as logical facts to query or investigate it more deterministically. I've been trying for a while, still not far but getting there.
vatsachak
Eventually lambda prolog will rise again
mirekrusin
You should checkout cave lang [0] - terse language that explores this area of knowledge/graph/ontology/provenance/querying/confidence/solver etc.
I couldn't quite figure out Lemmalog's latest benchmark stats from the README but it sounds cool!
I heard of Scallop, that also uses Datalog for neurosymbolic programming, I wonder where Lemmalog fits with Scallop? I know little about Datalog so sorry if this is a dumb question!
akkad33
I had tried to get long term memory out of Claude by indexing my notes with keywords and putting that in a sqllite database and Claude queries using full text search. Don't know how good it is, it seems to find things alright. My goal was to keep context small and only get Claude to ask for what it needs. Datalog seems like a great idea, will definitely try it out
kaeluka
I’ve been toying with using an llm to compile to smtlib and solving with z3. It works quite well, although I’m not quite sure how practical (large programs, reliably good runtime) I can make it.
mic_sm
Sounds interesting for me, could you share a link of POC?
akkad33
Has anyone tried formal verification with AI generated code? I can't convince my company to use it but I realise it's very easy to ask Claude to add a verification step locally on my own PRs
insanitybit
At one point I was using TLA+ but it just made the problem "is the spec right?" or "does the code match the spec?". I could ensure that the properties defined in the spec were valid, but that didn't seem to translate into confidence that my code was correct. Maybe I was holding it wrong, it was just an experiment in an area I'm unfamiliar with.
Ultimately I have stuck to the informal verification of defining my expectations and ensuring that tests cover them.
jjp
Interesting and potentially has applicability in deriving logical rules from regulation, contracts etc. Are there already formal languages that can be used to codify, that sort of information.
mgr86
And here I am just having them author org-files for me and them to refer back too. For fun I even ask for DITA sometimes.
4b11b4
Hmm DITA..
est
Very cool article. I had a similar idea where "fact checking" should be real programs for logic correctness.
But IRL it's too vague. The exploit hunting is a better use case.
ikari_pl
I was trying to connect to wifi on a fresh macOS install without only a keyboard connected last week.
After googling for an hour, I gave up.
onoesworkacct
yooo.. this is exactly what I wanted to do... I'm so happy because (a) I hadn't figured out how to do it (b) it seemed kind of difficult in general and (c) now I don't have to, lol.
bbeonx
It seems like you might be inventing a form of non-monotonic logic. Check out answer set programming, it actually does exactly what you want of "unlearning" facts that you've learned. Not sure if it helps in your particular instance, but it's very cool stuff and IIRC there is an implementation that extends datalog. https://en.wikipedia.org/wiki/Answer_set_programming
alexpotato
> The LLM handles the fuzzy part:
> And Lemmalog handles the deterministic part:
There seems to be this view in some circles that the LLM should do EVERYTHING. The most extreme version of this was "just commit the prompt, bro".
The more I've used LLMs, the more I think that the LLM should do either:
1. the fuzzy parts as mentioned in the post
2. helping to write deterministic tools to expand the "non-fuzzy" part
For #2, we invented code to run the same instructions the same way over and over again for very, very low cost. The code is also easy to read and modify as needed.
Why we would replace the above with a smart but stochastic system still seems strange to me.
ande-mnoc
Ctrl-F “prolog”: 0 result. :-/
skybrian
Search on datalog instead.
cookiengineer
This was a pretty awesome read, I liked it a lot!
What I found out during malware analysis is that LLM agents have a couple of quirks that you can solve by:
- optimize for short lived agent workflows
- use symbols as function contracts
- maintain decision and discovery state
- give LLMs CLI linters
- give LLMs access to knowledge bases
The linter part is mindblowing. I built linters that validate HTML or markdown or docx or Go or C files, for example, and they output what kind of structure is expected instead of useless token based errors (e.g. h4 inside h1? Must be h1 > h2 ...).
With linters the output quality of agents is just soo much better.
For program analysis, I'm currently exploring the idea of using an external ebpf daemon that programs can be observed with via a public API (which is the tool for the agent to use). Not sure if it'll do the trick yet, but I think it has lots of potential.
Limitation of LLM for and toward reverse engineering; it's the LLM innate error of forgetting states thru agentic recursion by overflow of context or prior premises being optimized away due to not using ternary-state (uninit/written/read) memory state.
Once again, on LLM being: a digital librarian, at its finest; logic a logic analyst, not so much.
fizx
Is this sort of re-inventing Graph RAG from another angle, or does it feel novel?
processunknown
It seems more like a handrolled CodeQL
tptacek
It's an agent system that basically embeds the core idea of CodeQL (Datalog extraction from codebases) and then allows a model to pose questions and answer them.
1. User request understanding: natural language -> a more rigorous representation, in my case Datalog.
2. Result interpretation: facts and derived facts -> natural language.
Between those terminals, the work should be mechanical reasoning over some ontology or formal knowledge structure.
That connects to another principle I've been thinking about, which I call Weathering: useful reasoning should change the shape of the system. If an LLM has already had to infer a relation, mapping, rule, or abstraction, repeated use should wear that inference into the system so that the next similar request doesn't require discovering it again from scratch.
With continued use, a weathering-capable system should therefore require less and less probabilistic intelligence for recurring work. Put another way, there should be a declining marginal cost of cognition since the products of intelligence harden into structure that can subsequently be reused and evaluated mechanically.
[1] https://aclanthology.org/2026.eacl-long.333/ [2] https://openreview.net/pdf?id=eC4ygDs02R
My problem is the context of today’s models (that I’ve been using - Claude Opus 5 and GPT-Sol) are a black box to a user like me. I cannot tell what they already hold in their context over the duration of a coding session. I cannot tell influencers a little by using/not using MCP tools, AGENTS.md like MD files, but what they _actually_ use at inference time is still opaque.
For example, if I insist that GPT-Sol use a size bounded dynamic cheat sheet for my project, and even have those reflector&curator agents refine them concurrently, what is to say GPT-Sol doesn’t have an intrinsic cheat sheet like that already, and it is just reading/curating/managing this one in addition and only getting confused needlessly along the way?
Similarly, all those codegraph like tools and semantic search addons.. what is to say the model genuinely uses them instead of relying on its own inference techniques and just “perform” for the user audience as if they are using these fancy tools?
This is not to say we shouldn’t try more ways of managing context and just give up to our AI provider overlords. Just thinking through the logic of what we are working with.
Would it even help if it weren’t a blackbox? Maybe to the frontier labs with resources, but even trying to read through the thinking traces of an open model doing a nontrivial task easily takes me an entire day to do correctly (i.e. actually looking at the code it read in) and much more if I’m trying to tune a specific repeatable task.
Applying that to agentic coding seems like a losing proposition. The only way to manage it seems to be more agents and that’s its own rabbit hole of compounding errors (like Claude’s useless comments).
The weathering need is real. An llm equivalent of dbms analysing usage pattern to tune indexes.
Soon, he'll discover that he needs quantifiers. Then that "for all" is too strong sometimes, and he needs "for most". That way lies Cyc.
It's not a bad idea. But it does have a history.
Think of it like human imagination may do anything (flying cars), but the real world has constraints and we use language/writing (symbols) and rules bound to them to simulate and reconcile our imagination with reality to actually flow our energy into something that may work in the real world.
In fact, both Cyc and the "AI" Labs have the _same basic thesis_: Intelligence is, primarily, a data entry problem. They just disagree about what kinds of heuristics should be run over that data (logic-programs, neural-nets).
Whenever I read about someone using LLMs to write code, it _very closely_ resembles how Lenat was using Eurisko/Cyc to solve problems: they let the system run continuously, and they "nudge" it in "interesting" directions, "when it gets stuck", or "runs out of steam". (Quotes indicate their phrasing, not mine)
Even Lee Spector noticed something analogous with his genetic programming system. When he tried to get it to discover optimal data structures (or maybe it was sorting algorithms, I forget), the system would quickly "run out of steam", without a solution. But when they added new verbs/opcodes to the system, that were a better fit for that domain (e.g. index-based memory loads + stores), it converged on a solution very quickly (even for GP, domain specific languages keep delivering unreasonable wins). You will note that this rhymes with the "micro-theories" of Cyc, which in turn rhyme with the SLMs of the AI labs.
In my personal experience, most of the "silver bullets" do not work (obviously), but some of them do nudge you towards being a better programmer (by refining your intuition about the problem specifically, and computers more generally).
EDIT: just remembered something. LLMs tend to produce larger and larger programs over time, and most people (IIRC) interpret this as a kind of entropy. This happens to rhyme with a similarly observed behavior in GP. Most genetic programs that do not have a fitness function that rewards smaller size, tend to grow in an unbounded way. The reason for this, is that most of the code/genes are useless, and random mutations do not lobotomize the program under evolution. I suspect that the coding LLMs tend to grow their code for similar reasons.
[0]: I suspect that, this was mostly a triumph of enormous amounts of hardware, more than the actual LLM technology. I further suspect that a traditional GP approach, on the same quantity of hardware, could have gotten there faster (if not better as well).
I settled on a knowledge graph in Postgres and downloading/storing the source documents so it could iterate on past results without more scraping or network calls.
This blog post helped me understand security analysis in this context! A lot of the important systems around malware analysis or large scale system security (the parts people really care about) clicked for me. So thanks for writing it.
Anyways I hope we can find some pattern to converge on with this wrt "facts management" since I feel this is currently something a lot of people and LLMs are struggling with. In practice current LLMs working with episodic memory feels similar to a grandparent with dementia scrawling things down in notebooks, crossing things out, and getting very confused.
I settled on simplicity, extended it to serve clear, useful purpose.
Started with markdown database, single fact per line, structured/parseable (subject VERB object). This can be easily diffed/reviewed etc in git. Then added optional metadata (confidence, tags, persisted comments for natural language, uncertainty for numeric values, context, timestamps/spans), querying, alternative sqlite3 backend, self describing VERBS, z3 solver etc.
Those kind of graph information systems are great to quickly structure knowledge in a way that LLMs and humans can use/act on/loop on. Creating ontology, linking, some rules and actions and kicking it so it all munches and spits out results that feed back in so it self evolves. It's very natural for llms to query/update/restructure those graphs (also good for humans because it's very terse, essential information only).
There is no need to create k8s style complexity/services/what-not, it all works well from single sqlite db file or bunch of markdown (.cave in my case) files.
It's also interesting to see how well local open weight models are dealing with information arranged this way.
(See also Cyc: https://en.wikipedia.org/wiki/Cyc)
I think approaches like this are going to be (or maybe already are?) the basis of effective grounding of LLM responses in authoritative data sources. It should be possible to pinpoint any error to an incorrect traversal or an incorrect "fact." This would work best for concrete, unambiguous facts, however; fuzzy, ambiguous or opinion-based information will probably remain the purview of LLMs.
https://news.ycombinator.com/item?id=41445445
It’s very hard to remove information - Claude has a habit of recording things all over the place and will happily treat things as facts even after they’ve been disproved.
What is currently true can get easily contaminated with old “facts”.
I ran into the very same problem of the LLM forgetting that we ruled out a conclusion that was verified not to be the cause as it came up further in the conversation history while I was exploring possibilities.
I had to keep reminding we ruled out that conclusion prior.. I just carried on with having the LLM capture some of the supporting sources of other people experiencing the same problem and kept having to refine those sources because it was focused only on summaries, but eventually i got the sources to a point where they were good enough hypothesis that we could formulate a better conclusion on what the potential cause was.
My way of handling that is a decision log. For every project since I started doing that it’s working great. My CLAUDE.md instruct the agent to store my every decision to the file with a metadata when I made this decision and what was the context. The agent is using this file as an index of decisions and rarely lose a track. It also helps team members to find out more about the development phases.
Does your system invalidate the parts of the memory if these are not valid or relevant anymore or just store/retrieve?
The LLMs are great at:
- understanding the flow
- making diagrams
- running the code with logging they add to to even better understand the flow
The flow being in Dot (or other machine readable format) makes it even easier for the LLM to use that as a reference going forward.
Opposed to OP, DeepClause uses Prolog semantics, so running some more complex queries on knowledgebases might cause some issues (which is the use case where a Datalog might be more useful). For smaller scales it should be fine though.
[0] https://github.com/deepclause/deepclause-sdk [1] https://github.com/deepclause/deepclause-pi
btw. your comment is grayed out, not sure what it means. However, thanks for sharing, I'll look into it.
E.g. A -> B, B -> C so A -> C (works, great). A -> B when A is sort of red and blue, also A is intermittent (what to do now?)
Formalisms should be in service to higher-level intelligences, not the other way around. It's pretty clear that LLM's are intelligent inherently; the lean doesn't just "prove math itself". (Admittedly, I'm not full blown AGI pilled either - there are some structural constraints that do make me think there is room to be gained in intelligence. And formalism will play a part in that. But it's not the end-all be-all to it).
The article touches on Datalog statements changing over time. One ingredient I think would be good to add to the system is to make every statement carry "providence" metadata. The providence should be sufficient to enable later confirmation that a statement is still valid or if the statement needs to be reformed without the need to remake the entire graph from scratch.
I would make at least some forms of providence follow a strict schema that is defined for the subject matter that is being captured. For example, statements about a code base should refer to the source files and their version (file modification date, content hash) from which the statements were concluded. When a source file is modified we may then find all statements made from them and reevaluate just those statements.
The next level would be to keep statements even if reevaluation breaks them and add a method to derive a subgraph for a given state of the subject. For example, over many releases of a code base, a lot of statements would not change, some would. Having a graph that spans all conclusions about all releases of a code base and a way to form the subgraph for a specific release would allow the system to efficiently target queries for a particular release.
Many people here might disagree; but they're holding it wrong. Those folks should ask:
1. Am I using free tier tokens? 2. Am I working on trivial software? 3. Am I expecting models to adapt to my ways of thinking?
Anyone affect by any of these three mistakes will maintain an impenetrable filter of perpetual ignorance about model capabilities. Since the OP came with receipts, I'm reproducing an example graph below.
From a plan in my active research project:
the other really handy thing here is that it very straightforward to differentiate datalog rule sets. so if you change a fact, you can run a smaller solution that tells you which consequences are affected by the change. under the assumption that the evaluator is pure, we don't need to keep the support graph as you suggest, we can just generate the deltas from evolution of the differentiated form.
detailed provenance can be of real application utility, but if you just care about the accounting, keeping track of the number of supports for each fact is sufficient.
another model which is fun is to make the version of the database (monotonic time) an explicit field in your base facts, assuming you can afford to keep the whole history. a deletion then is just a negative-support, and you can ask questions about the state of the world at any time.
I’ve been battling the same problem, and I solved it by keeping the state in my brain, long focus hours, and breaking down the problem to smaller chunks that agents could almost one-shot. That made me the bottleneck, and although I can do it for codebases I’m familiar with, working on totally new projects has been very painful.
I’m going to test it in my own vuln research workflow.
Note: I wonder how many people reified their codebase as logical facts to query or investigate it more deterministically. I've been trying for a while, still not far but getting there.
[0] https://mirekrusin.com/cave
I heard of Scallop, that also uses Datalog for neurosymbolic programming, I wonder where Lemmalog fits with Scallop? I know little about Datalog so sorry if this is a dumb question!
Ultimately I have stuck to the informal verification of defining my expectations and ensuring that tests cover them.
But IRL it's too vague. The exploit hunting is a better use case.
After googling for an hour, I gave up.
There seems to be this view in some circles that the LLM should do EVERYTHING. The most extreme version of this was "just commit the prompt, bro".
The more I've used LLMs, the more I think that the LLM should do either:
1. the fuzzy parts as mentioned in the post
2. helping to write deterministic tools to expand the "non-fuzzy" part
For #2, we invented code to run the same instructions the same way over and over again for very, very low cost. The code is also easy to read and modify as needed.
Why we would replace the above with a smart but stochastic system still seems strange to me.
What I found out during malware analysis is that LLM agents have a couple of quirks that you can solve by:
- optimize for short lived agent workflows
- use symbols as function contracts
- maintain decision and discovery state
- give LLMs CLI linters
- give LLMs access to knowledge bases
The linter part is mindblowing. I built linters that validate HTML or markdown or docx or Go or C files, for example, and they output what kind of structure is expected instead of useless token based errors (e.g. h4 inside h1? Must be h1 > h2 ...).
With linters the output quality of agents is just soo much better.
For program analysis, I'm currently exploring the idea of using an external ebpf daemon that programs can be observed with via a public API (which is the tool for the agent to use). Not sure if it'll do the trick yet, but I think it has lots of potential.
My stuff in case you're interested:
[1] https://github.com/cookiengineer/exocomp
[2] https://github.com/cookiengineer/gobayashi
[3] https://github.com/cookiengineer/gonano
Once again, on LLM being: a digital librarian, at its finest; logic a logic analyst, not so much.