Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
j / k
|
Down or up a comment |
h / l
|
Previous or next link |
?
|
This help |
https://github.com/emmilco/us-tax-model
In more detail:
- every line of every form gets a YAML representation which includes the source text, a data type, and an expression built from a small set of lean atoms, plus a citation pointing to the exact location of the rule in the original official document.
- underneath the yaml layer there's a lean formalization of the atoms showing that they behave as specified, that all comuptations use only the allowed atoms and operations, etc. There's also a typescript evaluator that's used to cross check the lean for agreement, so we have the arithmetic represented two ways.
The critical failure point for an LLM-produced formalization like this is the interpretation of the form text. Since I'm not a tax lawyer and don't intend to become one, this is an explicitly owned weakness and one I've structured the project around. Publicly available tax calculators (PolicyEngine, UsTaxes, OpenTaxSolver, etc.) are used to cross check derived numbers from this system. Mutation testing ensures the test suite is robust against bugs and arithmetic errors. Interpretations are cross checked by different model families (Claude, GPT, Deepseek). Etc.
The idea is to derive confidence from the formal properties of the system itself, and to allow theoretical future users of such a system to derive confidence in the interpretation of specific forms by making it easy to cross check line-level formalizations against the source documents.
There's a lot more detail in the (alas, Claude-authored) README. When I have time I'm going to write my own version of the README expressing my vision of the project, but I'm not there yet.
And to be clear: my vision here isn't "have an agent do your taxes", it's to have agents build a hardened and thoroughly verified deterministic tool that can help you navigate the (somewhat bewildering) forms and instructions decision tree when filing your own taxes.
https://www.irs.gov/filing/where-to-file-paper-tax-returns-w...
Anyway, my point is that for 95% of people they have all the information already and you shouldn’t have to file.
(If a painter knew that the camera was coming out I would not want them to throw their canvas away)
That's not a fair comparison because the painter produces different work, whereas tax computations will be the same.
There's an article about it here https://biggo.com/news/202510160717_IRS-Fact-Graph-Public-Re...
These were artifacts from the IRS Direct File program (RIP)
Maybe your project could (re)productionize this dataset.
- It's oriented toward the IRS's internal XML API in a way that's not really useful for this project, since no consumer of my code is going to digitally submit to the IRS.
- The semantics of FactGraph are largely undefined, and the maintainers themselves seem to acknowledge that its development trajectory went off in a bad direction and it needs major divisions.
- It's super verbose.
- It's the year of Lean, so I wanted to find an excuse to use Lean.
Other work has been done in this domain, though. The biggest example is Catala[0], though Prof. Sarah Lawsky at Northwestern seems to have made some strides toward formalizing the Internal Revenue Code.
[0] https://catala-lang.org/
I feel like we're going to get to a point where it's mandatory to use AI for certain tasks - humans just can't be trusted to get things right that require a level of precision.
He amortized the goodwill of businesses I acquired over 5 years instead of 15 (incredibly basic stuff, which Claude flagged with *CRITICAL ERROR*). When I asked about it, he said he had caught the error in his review but forgot to save after he fixed it. I don't know if there's a good response to making that kind of a mistake, but if there is, it's definitely not that one.
Obviously total bullshit.
Did you fire him?
Ask for damages?
Assume yes, fired.
Feed previous year returns into Astra and find out! It's not too late for amended returns, you have 3? years from original date of filing
Top tier tax professionals can do pretty well on this benchmark (better than agents) but won't hit 100%.
The kind of people spending time preparing oodles of returns would not do very well on this metric, though (especially midway through that pile of returns, as opposed to "think carefully about this trick question").
1. found errors, e.g. where I used the incorrect exchange rate (and managed to work out how I had made that error, which I was impressed by)
2. found items which would appear to be errors except for context it did not know
3. flagged areas where it disagreed with me as to the correct tax treatment, but accepted that the law is ambiguous (and in some cases I preferred its interpretation after additional research and in other cases I did not)
4. flagged areas where it disagreed with IRS guidance, and only acknowledged said guidance after I provided links to it
5. provided highly inaccurate estimated tax calculations, and not been able to fix its estimates even when challenged
So my takeaway is that it is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check. (Possibly it is if you just have a W-2 and no other substantial income, but I can't speak for that scenario.)
I think what helped is giving it 100+ pdf and csv documents of my full situation so it knew everything about me and could cross-reference with all my bank statements.
I have foreign income and foreign businesses, my partner has foreign income from two countries. The tax law is then based on tax treaties, most tax preparers do not have sufficient knowledge to complete a return, and tax software doesn't handle these cases.
The tax treaties are not clear, tax professionals will disagree on interpretation, and tax benefits in one country will not apply to another.
Then there is a thing called AMT, alternative minimum tax, which is a second way to calculate your taxes that you need to submit as well.
I no longer do consulting on the side so I just use online tax software. It is about $150 but I do that more out of laziness as it integrates with my brokerage and payroll company to import everything automatically. Also I have carry forward losses and such that it already knows about.
Most people’s taxes are simpler than mine but I am sure there are lots of folks here with much more complex tax situations
Same situation as you. I have a local accountant in the country I'm in plus my US accountant. Claude/ChatGPT have been a huge help for checking, explaining, and finding issues. They also found websites with pertinent information that Google did not, which surprised me.
I had it prepare an analysis and then found a professional to confirm everything and then handle doing the amendment.
Definitely useful as a second pair of eyes.
Try this, it uses a tax engine as the backbone and provides feedback for Claude if it goes off the rails. Albeit it doesn't support every tax scenario.
(And as I said to another commenter, I think it would be advisable in this thread for people to expressly acknowledge or disclaim any interest in or prior experience with any particular product)
it's the most ai-native tax experience for US citizens abroad
(It would also help to know if you had a financial interest in or prior experience with this company)
What do you mean by "cost you $20K"? That your tax liability was calculated by the accountant to be higher than it actually should have been? Or that the accountant calculated your liability too low, which was caught and billed by the IRS?
In the first case, you can still file for a refund within 3 years (I'm omitting some technical details), but in the latter case, as mentioned by others, you still owe the tax whether it was correctly reported on your original return or not.
The real issue is that accountant most likely has a track record of those kinds of mistakes - but you have no info on other clients. So you weren’t as informed at the time you chose them vs the new one.
I think the red flag should have been when he printed out all my tax returns and made me pick them up in person, and didn't even send a digital copy.
It's not the accountant that paid $20k for his mistake but the accountant's client.
Seems like simplify or just erradicting the tax code would be more just at this point.
In some jurisdictions, it may help with criminal but not civil liability.
I am not aware of setups available to me that would transfer liability onto accountant.
So in either case I am taking liability.
(also, I am pretty sure that if corporation loses 10 millions over accountant making bad decisions, they are not going to recover that funds either)
And there was no real path to recovering funds, except small part, after long court case.
At least they were able to fire them without severance pay.
https://blog.nawaz.org/posts/2022/Nov/do-you-trust-your-tax-...
There is no such thing as "failing" an IRS audit. You either under reported income, overstated deductible expenses, or your "accountant" changed the numbers you provided and committed fraud, which would have resulted in immediate legal proceedings by the IRS against the preparer. So did you "fail" because your preparer committed federal tax fraud or did you underreport your income?
The statement that the one time you used an accountant you got audited implies that you are confusing causation with correlation. There are about 6 ways that the IRS selects returns for audit but only 3 are actually relevant here.
- Information Matching (ie. if your W2 does not match the W2 submitted by your employer). This mostly used for your basic W2, 1099DIV, etc. kind of taxpayer and is usually just a letter from the IRS telling you your numbers are wrong and you need to amend your return.
- If they are examining a related return (ie. you own a business and another partner in that business is being audited)
- Computer scoring which breaks down into the Discriminant Function System (DIF), basically, how does your return compare to similar returns, and Unreported Income DIF, what is the potential that your return underreported income. This is what is commonly used for businesses and self employed individuals' Schedule C. Basically, when you prepare a business return (1040 Sch. C, 1041, 1065, 1120, 1120S, or 990) you have to report your business industry's NAICS code, and provide a single line explanation what your business does. Using this and the return data, the IRS is able to score a return on the likelihood that profit is understated by comparing a businesses income and expenses to the norm for a similarly sized businesses in that industry. High scoring returns are then reviewed by IRS personnel.
One of those three things happened to you and none of them are the fault of a preparer that is using accurate information provided by the taxpayer.
Finally, you keep saying things like "accountant". You disparaging an entire industry without saying what type of tax preparer you used. Was it a CPA, licensed by a state with experience in tax? An Enrolled Agent, licensed by the IRS directly? Or a tax preparer, an unlicensed person preparing your return either on their own or through something like HR Block or Turbotax? Because if you used an unlicensed tax preparer and are blaming licensed individuals that is like hiring an unlicensed electricians and then when your house burns down from an electrical fire, turning around and saying all electricians are bad
And that's true if all you do is have W-2 income, and RSUs/ESPPs. ESPPs were the most complex thing I had to deal with until I stupidly decided to do things and get K-1s.
Well, you are an accountant :-)
With my K1, the issue was that it was for a real estate investment, and you have to be careful about things like bonus depreciation/cost segregation etc. One tax accountant I used was not carrying over the loss (or perhaps not applying the prior year's loss, or something like that). Another accountant was deducting that loss from my W-2 income, which I can't do as I'm not a real estate professional.
When I finally brought these points up and got all these corrected, I studied the final result and then continued doing them myself. But it was a pain getting the online tax SW (TurboTax, TaxAct, etc) to mimic what they did. I had to be really careful what things I said "Yes" to and what I didn't. I would never have figured it out on my own by reading the relevant tax publications (which I did peruse).
And then later, I formed a partnership with a friend so we could both invest in something. Now I also have to issue K-1s! Not fun! RSUs are much easier.
When launching my financial business, I spent half a year learning accounting to be sure to understand everything, then I implemented my accounting software, specialized for my activity. And as luck had it, I made it be a CLI. This last few months, I've been using Codex(lately with GPT-6.1 Sol) to do the annoying work - attaching proofs to entries, fixing the dates and amounts issues, etc. It works on on a database copy, generate a script of commands using my CLI to fix and close the previous month, that I then review before executing on my prod database, sometimes asking questions to the agent about why this or how that. … And it's been flawless so far, both on my professional accounting and my personal one. Knowing a bit about what we ask the model to do can go a long way.
Can't be good getting caught with mistakes and losing customers?
Additionally, I think the industry and individual practices are probably still figuring out contracts, accountability, and legality. Will their insurance policies cover mistakes made by an AI?
I'd also guess that its one thing for you to feed your financial documents to Claude; would you feel as comfortable if your accountant were feeding it to an unknown model of their choosing? I'm not so sure I'd be.
Has anyone verified their figures to see if they hold up?
https://www.accountingtoday.com/news/new-bill-calls-for-irs-...
There is also an IRS form (14157) to report a return preparer complaint, which of course does not include an option to report a faulty AI.
Pretty sure the LLMs could implement tax calculations better than the average accountant, were they given the rules.
Most calculations are trivial.
I’m not sure the evidence is on the side of the CPAs. Maybe CPAs with LLM assistance.
I literally gave it your question. What models did you test?
Cursor uses Grok so far as I can tell, which is.. not great and I hope you did not try ChatGPT free or 5.6 or Luna.. you cannot unfortunately just say “Claude” or “ChatGPT”. Their lineup is sufficiently complex - each with their own failure modes - to warrant some further specification.
I think you are “directionally” correct, as these clankers like to say, because language models are indeed not known for their arithmetic wizardry. However, as my correct answers show, they can do it just fine.
Size, unfortunately, matters.
These types of detail-oriented professional tasks seem like they would perform better with task-specific harnesses. The example they give, of Astra missing the property excluded gains requirement, seems easily fixed by scaffolding ensuring that requirements are compiled first, then checked.
If the scaffolding is absent, is this benchmark testing that a model can self-scaffold, or track requirements in-context?
0 - https://huggingface.co/datasets/surgeai/DAYJOB-tax/tree/main...
I've found it so far more effective for preparation than for planning/advice.
I've got a plan to feed the laws and rules and instructions into the machine before the prompts - is RAG still cool? Maybe I should small-language this?
Admittedly, it could be promting issue, lots of context for these decisions.
I downloaded 1700 of my transactions and use AI to build an app for it, so I can analyze the transactions and make sure that everything is properly categorized. It used to take me whole day because of the Quickbooks. It took me less than an hour to go through all the transactions.
EDIT: good point that paying accountant does not ensure avoiding it
wonderful, you made a good point that make me dislike situation even more
The price of convenience and massive time savings is taking some risks on that front. We run a small business. We heavily depend on codex doing all sorts of administrative stuff that in the past either wasn't happening at all, wasn't happening properly, too late, or was being outsourced to people with various degrees of flakiness/incompetence/ineptness. We got rid of all of those and freed up a lot of our time.
As a small business you are always reacting to incoming demands on your time (accountants, taxes, invoices, insurances, etc.) for all sorts of reasons that boil down to having to do a lot of detective work in documents, filling in lots of forms, responding to queries, etc. AI tools can do essentially all of that now faster and better than most people can. Even the complicated stuff that involves fiddling with difficult tools, poorly designed websites, convoluted processes, etc.
Half the success here is getting organized on how and where you store data. A lot of SAAS tools actually make this harder, not easier. It really matters which bank you use, for example. Some banks have decent APIs and tools and export options. Others don't. Same with accountants. Some are more forward thinking than others.
It's a bit of a leap of faith when you start delegating important stuff to LLMs. And then you realize it's mostly fine and that it's doing a better job than you would. You still should double check the output. And you probably should still use an accountant to sign off on things and do some final checks. But don't fill in forms manually or go hunting through your drive for some document or information.
The plan now is to retrieve the current year's tax forms then have an LLM compare them to stored versions from last year, and look for variances (e.g. new field, new form requirements, etc) all of which an LLM would excel at (whilst basic math it historically did not).
- someone who hates me personally will get access to it (see case where someone put their ex-spouse on no-fly-list etc)
- leak my private info via new version of LLM was trained on it
- sell it to everyone interested (maybe sold after their bankruptcy)
- give it to NSA and whoever else
from obvious ones
I started my career fixing the mistakes that my clients' former tax advisers made. It looks like the next generation of tax lawyers will get to do the same except this time the former advisers will be llms.
I guess my point is these benchmarks unless they are constantly refreshed with new scenarios and not ever having their outcomes written about publicly or available on a public server, will always be prone to being gamed
I'd have honestly never been able to do it myself - its just too tedious.
Some accountants need to make judgement calls. LLMs are not great at many judgement calls yet.