Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
sandeepkd
Its a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part of system design interviews.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
switchbak
“changing data once its running in production is hard, changing the data structure is even harder”
100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
necovek
The fact that it took them 4 months to roll out does not mean this is the hard part: it's simply a coordinated rollout with incremental, staggered deployments and rate-limited migrations.
Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.
However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.
So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.
tyre
> This is a misconception
Pretty sure they were joking.
brabel
For anything you do, no matter how novel and difficult, someone on HN will tell you it’s trivial and done since the 70s.
paretolaw
Would you rather be doing GB with 70s hardware or TB today?
I would not understimate what folks in 70s had done.
hinkley
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
<strange high pitched equivocal sounds>
I do a lot of tuning work that others either can't be arsed to do, or more often are scared shitless of touching (learned helplessness). I tend to work on things that have to go right the first time in general, but I firmly believe that I learned that skill doing optimization work early in my career, not the other way around. Through a series of unfortunate events, my first specialization was performance, before I got good enough at anything else for people to listen to me.
Ultimately, the single thing that fucks up performance work, far and above any other concerns or sources, is that the vast majority of developers want to skip from Make it Work to Make it Fast without going through Make it Right. Then they have to use dodgy 'optimization' tricks that embrittle the entire thing, destabilize production and make every new feature feel like walking a minefield.
So what happens is that when someone who actually knows what the fuck they are doing comes along and wants to build a superhighway, they find the route littered with abandoned 'vehicles' that have sunk into the mud up to their door handles and left to cement in place. Clearing the proverbial route becomes a bigger technical challenge than building the road. And so you'll find that your peers have a bunch of ideas about what needs to be done but none of the stomach to do any of it.
Which is why I say "stubbornness" is the most important tool in my performance toolbox, far above cunning.
brainless
I do not think Cloudflare was a less-than-peers optimized product when they launched. This is one of their blog posts which describes taking one aspect even further.
I think Cloudflare became big only because they were so much more optimized than others that they offered some services for free that others were not offering. If running costs are high, you only burn (VC) cash and then you exit.
majke
First version was a three layer VM monster, with the main entrance done as HTTP proxy written in PHP.
Voultapher
There is the entirely plausible option of the NSA indirectly bankrolling them to counteract the growing number of TLS connections. We terminate your TLS for free in our server and you don't have to change anything was a sweet deal for website operators _and_ those that want to look at unencrypted traffic. Given enough sustained funding they could undercut competitors and grow to what they are today.
vanviegen
Or optimize a bit earlier and prevent having to scale out to a bazillion systems.
bcrosby95
The way I usually prevent having to scale out to a bazillion systems is never getting more than 10 users.
bigbuppo
And that's why I charge $10,000,000/user/mo.
steve_adams_86
I wonder why Cloudflare didn’t think of this
kevin_thibedeau
This is Broadcom's business model
gerdesj
Quite. I was a VMware fanboi (25+ years, man and boy)
I still look after a few VMware estates and a lot of Proxmox ones (that used to run VMware).
Hilariously, VMware is described as "enterprise class", which I can only conclude means MVP and a bit wanky.
Today I repaired a Proxmox HA + Ceph node using boring old normal Linux skills and as it turns out I have 30 years of those. Part way through a remote v8 to 9 upgrade I think I lost comms due to using OpenvSwitch for networking and despite using tmux for the upgrade session. Anyway, the Proxmox ISO was useless for rescue but the classic systemrescuecd worked nicely and I could run dpkg in a chroot.
VMware "used" Linux and never really gave back. I don't miss fixing vCentres and all the other nonsense that "Enterprise" wankery has foisted on me over the years.
taneq
When I was first getting into software dev, I thought 'enterprise' meant 'industrial grade', 'powerful', 'high-performing'.
Then I actually met some enterprise software, and realised that it means 'expensive', 'bespoke', 'one-off', and usually 'janky'.
phoghed
Enterprise means it has SSO and a support contract
jacquesm
It means you are paying for a support contract. Whether you actually have one time will tell.
PeterStuer
Only seasoned engineers will understand you are not joking.
phoghed
Sorry, the very simple thing you’re trying to do is too complex and non-standard for our support team to handle. We’ll gladly sell you a consultant for $400/hr. He’ll work on modifying our system, and then we’ll sell those modifications to everyone else.
valenterry
And in-transparent pricing and this annoying dude that asks you once in a while if everything is okay, usually before the renewal.
esseph
Lol, so true
bigbuppo
Enterprise quality software is a just a DOS application, probably written for dBase III, that has been rolled forward to the present day.
elcritch
Those old school systems are often much more stable than any newer systems. Autozone looks to use something like that and I’ve never seen them have issues as a customer.
Nursie
To me it always meant needlessly complex and overspecced for what's needed. I think probably due to Java's enterprise years.
Why solve the problem directly when you can abstract everything away into FactoryFactoryImplementationInterfaceFactorys, and have something that is both a memory-hog and completely unassailable to any normal programmer seeking to understand it or make changes?
dboreham
"Enterprise" mainly describes the sales and support process, not the nature of the product itself. That said one does influence the other.
bch
The Art of Production
forgot-my-pw
The art of premature optimizations
smolder
The art is in knowing how to write software that doesn't perform like shit without doing all the work of measuring and refining. If you can save $100k in hardware costs in a couple days by just knowing what you're doing, that optimization is not premature.
jodrellblank
And yet when Prof. Donald Knuth wrote that in 1974 paper[1] it was in this context:
> "The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal"
also:
> "In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by
virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable"
In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals.
In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".
> And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".
Oof that hit hard. The last project I worked on suffered from a very similar disease, and it has really taken a toll on me psychologically. To work day in and day out on something that you can prove cannot work is unbelievably demoralizing. From an organizational standpoint, it makes a lot of sense to have an "internal consultant" who can deliver bad news like this. I tried to do it from the "inside" which was a huge mistake--got a negative performance review saying I had a "communication problem" because nobody wants to hear "negativity". I can come off online as kind of an asshole, so this may not seem credible, but I did actually deliver this news in a professional, measured manner. It's just that organizations are allergic to it, and their antibody response kicks in. You need someone who is not affected by the organizational hierarchy (or at least not that branch of the tree) to step in and deliver the bad news without fear of retaliation.
froh
pro move. made my evening.
scottlamb
You're never going to get promoted with that attitude!
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
Dylan16807
It was already reasonably lean. If they had 10 bazillion systems, they now need somewhere between 6 and 8 bazillion systems.
otterley
Remember that everything has an opportunity cost. Running a lot of servers might cost $10 million annually, but if the product team had to choose between a project that would recoup $5 million of that vs. an opportunity to earn $50 million ARR for the same amount of work, the logical answer would be obvious.
Retric
In my experience optimizations actually preformed tend to have ridiculously high ROI because they are so rarely prioritized.
Better performance = saving money + better user experience.
mtrovo
Depends on how you measure your ROI and how it could be different from how your company measure their ROI.
The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.
In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
Retric
You’re assuming performance has zero impact on customer retention, spending, etc which is demonstrably false. Further future costs aren’t bound by the current customer base or fiscal quarter.
Insufficiently optimized code kills companies in highly competitive markets.
otterley
Nobody’s assuming that. A well-functioning business compares opportunity costs with all those considerations in mind.
Retric
“Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.”
27183
> The problem with optimizations is that you are competing in prioritization with other features.
I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users.
I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost.
> In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
vanviegen
That's assuming the ops team has infinite capacity.
megous
You're assuming faster/more resource efficient software would result in the same ARR as the laggy slow one, but that's not a given.
otterley
Nobody’s assuming that. A well-functioning business compares opportunity costs with all those considerations in mind.
hinkley
This is perhaps the hardest nut to crack.
We give perverse incentives to the marketing department to bring us the most customers they can, instead of the most appropriate customers.
Someone told me at my second lead position that they had landed a top-tier customer for our demo-ware and the first words out of my mouth were, "FUCK ME". Not the response they were expecting, but then that guy never did end up understanding me the entire time we worked together. My bosses did though.
When your system is new you're a loss leader for all of your customers. Every dollar they bring in costs you two, possibly more. You have to get to positive MRR before the venture capital runs out, and over a long enough time horizon, you will go for a new round and find out that a recession is about to start and the VC guys getting cagey is the first clue it's coming.
Meanwhile if you spend all of your time and energy on reducing costs instead of increasing revenue, then your competitors catch up with you. Particularly if their funding rounds are half a cycle off from yours - one of you will be the last one to have gotten a cash infusion before the money dried up.
Most recently I worked at a place that didn't understand my calls for sobriety until it was too late. So I got put in charge of a rear-guard action that was too little too late, and our customers all fled to much cheaper competitors who could do 80% of what we could for half the price. I learned some good stuff, but the company got bought by a competitor who scrapped those systems.
dakolli
You can build foundations that aren't extermely optimal but have future optimisations in mind.
nine_k
This assumes that you have plenty of cash to burn in the process, which is approximately correct for VC-backed ventures, and for offshoots of large corporations that play a lomg game.
hinkley
I have a mountain of bad things to say about Microsoft, but they definitely had loss leader products that never made a dime but pulled you into their ecosystem. So they funneled money from a couple of cash cows back into the other projects to keep it enticing.
chii
> start generating profit, and then you can start optimizing your costs.
this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit.
Or, in the current era of ai, the cost of the capital investment is far exceeding the ability to generate profit off it. The optimization in how the resources gets used will be needed to cut the costs down, and allow increase in the scale of usage for the same hardware. That's where profits would lay.
Of course, in order to achieve any of this, you'd need the runway to survive until such times. A small scale operator won't have this runway, and so die before they can accomplish anything profitable (or get big by begging for investor money to grow large - as we've seen in the past 20 years of tech).
hinkley
I've seen a lot of people relearn the same couple of lessons over an over again.
One, most of the things you believe scale linearly scale at nlogn because of physics. Storage is much worse, scaling at n^1.5. What fit in a single 4U case is orders of magnitude slower when it no longer fits in a single data center. Inserting records into a table with a billion rows is much slower than a table with ten thousand. Wipe the surprised Pikachu look off of your face.
Second, every architectural choice needs to be reviewed every time traffic increases by an order of magnitude, and preferably before so that you have time to stew on possible solutions and allow serendipity the time it needs to find you a better answer. If you have made too many architectural decisions, you will find there aren't enough hours in a month to keep up, even if you never sleep. That's the real reason we practice Principle of Least Power. Not security, but job security. Keep the system small enough that you have time to keep the plates spinning while also implementing new features to attract or retain customers.
adornKey
What I find surpring here is this being about DNS. Simple optimization should have been done maybe 1997. Letting it build up to 100 TB is noteworthy, but on the other hand for IT that's common. It's surprising but also totally expected...
gbin
This is true if you can scale out (ie you can add resources to your system). But for a robot for example, just adding a GPU can just flop your product completely: you need more battery, more weight, suddenly your unit economics is out of the window... Your next hardware iteration will be very slow to come and very expensive.
So here, you better not have a system wasting too much resources pretty early on after the prototype phase.
casey2
This reasoning assumes you have access to infinite runway. You don't.
lpapez
Exactly, and you need to start turning a profit before the end of that runway. Even if that means running code that is suboptimal.
ramon156
i suppose you could say the same about buying a house. just make that initial 300.000 and from there on out its easy. everything looks better on paper
topato
My house is a ~700 sqft. condominium, gov. subsidized for lower income individuals, and even my mortgage is more than 300k… maybe I’m just basing my info off of coastal city prices, but is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
esseph
That would buy you thousands of square feet and often several acres within 20 minute drive to a lot of US city downtown areas.
AlotOfReading
Can you ground the discussion by mentioning what you think these cities are? Taking Columbus, OH as the most average of American cities and a 20m isochrone map from city center, there are currently 0 parcels for sale with 3+ ("several") acres under $300k. There are a few within 30m drive, one of which even has a possibly habitable structure. The rest are bare agricultural land you'd need additional investment to actually live on.
esseph
I'm just looking at Kansas City as an example right now.
Tons in the $200s btw, and this is in the city less than 20 minutes from the Nelson-Atkins Museum, WW I museum, the river, farmers market, KC Chiefs / Royals / Sporting KC / KC Current teams, parks, schools, newly rebuilt international airport, etc.
AlotOfReading
I picked the convention center as city center (union station would also be a reasonable choice) and I was only able to find 2 improved lots under $300k. Notably, neither is quite 3+ acres or within 20 minutes of the things you listed, and both are on septic. And this is in a metro where home prices are 36% below national median.
First if you us the convention center, you place the housing center off of where most of the housing actually is. Maybe 40% is in Kansas, with another 60% in the metro being on the Missouri side, roughly. The population density of the metro drags further south though if you look at it on a heatmap.
Note 2: Your second link is way out in Independence, MO but is still only 29 minutes away from the WW I museum.
AlotOfReading
First if you us the convention center, you place the housing center off of where most of the housing actually is.
Convention centers are usually downtown. Mean population centers reflect the surrounding geography and can drift pretty far from an "idealized" circular urban area. My area's is out on an island for example, nowhere near downtown. Jackson county's is east of Raytown. I'm not sure why you'd specifically want to live within 20 minutes of the mean population center, so it wasn't the obvious interpretation of your original statement.
Between $175k and $350k center on KCMO alone there are over 760 results.
No, there are 17 results once you filter for 2+ acres. Most of them are mislabeled or unimproved property. If you filter those out too (because this is a housing discussion), what remains within 20-30ish minutes are the two I linked.
I realize multiple acres of land is a bit of a silly requirement, but you're the one who suggested there are tons of houses with "several acres within 20 minute drive to a lot of US city downtown areas". That's why I was surprised.
esseph
> That would buy you thousands of square feet and often several acres within 20 minute drive to a lot of US city downtown areas.
This is my direct quote.
Let me break this down:
1. That would buy you thousands of square feet
True, clearly seen in the search results.
2. and often several acres
True, you just filtered down to several results currently on the market.
> I realize multiple acres of land is a bit of a silly requirement
Why? I live slightly further than 20min outside of town. There are massive, beautiful homes with tons of land out here, and tons have gigabit fiber from GFiber or several of the other providers in the area.
But that's not a requirement if you don't want it to be! There are clearly tons of homes available in the city with 3-4+ bedrooms and 2+ bathrooms in the $200 and $300ks is the point.
8n4vidtmkvmk
Not from what I've seen. The desirable neighborhoods are 500-600k. Suburbs outside major city
catlifeonmars
Desirable neighborhoods are by definition expensive. The trick is to find a neighborhood you like where your home can just be a home and not a top-heavy investment.
Acknowledging this isn’t always easy or possible, but just pointing out that this is a self reinforcing problem.
hdjrudni
> The trick is to find a neighborhood you like where your home can just be a home and not a top-heavy investment
I meant desirable for me to live there, not as an investment. Who wants to buy a home in place they don't want to live?
dboreham
Assuming you are not married.
esseph
You didn't read what I said.
I mentioned acres of land. You normally don't have multiple acres of land in the suburbs.
hdjrudni
Well then you didn't read parent's comment
> is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
Who wants acreage? We want homes.
esseph
> is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
What does "city" mean to you? For some, it's 500 people, or 5,000. For some, it's 5 million. Define that first. The US is a big place, and I know people that don't live within 50 miles of another human.
4 bed, 4 bath, 2,295 sq ft, $365,000, built in 2022.
anon48293
That’s 350, which you can’t buy for 300..
irishcoffee
If the extra $200/month means you can’t afford it, you probably shouldn’t be buying a house…
Yeah that sounds cruel and messed up, because it is. I don’t like how expensive real estate is and I wish we could have another 2008 without a bailout. But the point stands. If you hafta pick between eating and paying your mortgage, you made a bad financial decision. Blame whoever you want, I am of the opinion that people still have agency and are responsible for their decisions, as unpopular as that seems to be these days.
esseph
These are midrange. Most of the homes in those areas are around $225k.
You'd be surprised how much house and land you can get for that price.
rcxdude
This reasoning is largely centered around the runway being finite. You obviously can't have costs so high you are making a huge loss, but also there's little value in improving margins past profitability until you actually have a stable segment of the market.
inopinatus
we are all perfectly smooth, round, and filled with an incompressible liquid
rexpop
Every startup is one bet in a Martingale strategy played by the class of people who remain solvent when you bust.
robocat
The median return for a startup is $0. Take care when trying to extrapolate cause and effect.
DanielHB
You underestimate the cost of "optimization", sometimes it means actually rebuilding large parts of the system. I would not say it is the "easiest" part, but it is usually not what will kill your business though.
MuffinFlavored
> Produce working product first, validate the idea, stabilize the business, start generating profit,
not everybody is so lucky to be able to go in that order? The first part requires upfront capital/investment?
phoghed
So obviously you start at optimization
aperture147
> This is the right way to deliver software.
It's only right if you are not the one who pay the bills.
suncore
Not the right order if the optimization is a prerequisite for a positive business case. That happens more often than people think...
saidnooneever
right or wrong, good or bad are all taste.
from a business perspective, this might be considered the only way, but it is not. at large volume scale it becomes more, but often large scale is lacking optimisations in the first place.
its not wrong in my eyes, but definitely not the only path to take.
compounding_it
>optimization is by far the easiest part of the process
Not if the whole thing is architected poorly but was a requirement of the hour so it became big. Then optimisation becomes an art, but definitely not the ‘easiest part of the process’
Sarkie
Make it work, make it fast, refactor
aeonfox
> start generating profit, and then you can start optimizing your costs
Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...
I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.
Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
mkeeter
For the curious, this is technically possible in Rust using a dynamically sized type [1], but in practice is difficult and doesn't really play nice with the rest of the language. The nomicon entry concludes with "Yes, custom DSTs are a largely half-baked feature for now." [2]
> putting the record data right after the CacheEntry members
I assumed they couldn't do that because they're using it with some kind of generic HashMap<K, V>. In that situation, can "V" be dynamically sized?
A dynamically sized "V" would mean you can't have an array of them, which might preclude some hash map implementations.
esterna
> All type parameters have an implicit bound of Sized. The special syntax ?Sized can be used to remove this bound if it’s not appropriate.
, which HashMap does not do, i.e. the keys and values have to have a statically known size.
f311a
Unfortunately, Rust is not a good choice for this kind of tricks. This is where Zig shines. In Rust, you can’t even use proper arenas, which can help a ton with allocations.
Cloudflare started to pick Zig recently, for projects, that have memory constraints.
afdbcreid
> In Rust, you can’t even use proper arenas
You definitely can and this is done a lot. What you might mean is that you can't use standard library's collections with them (this is getting stabilized soon!) and have to use third-party, but that is a different thing than "can't use arenas".
> Rust is not a good choice for this kind of tricks.
Rust can do those tricks, but it's true that it is hard than in C or Zig. That said there are often crates to help.
f311a
Stabilized soon, really? They did not stabilize it after 10 years and were thinking about different approach.
I thought it’s dead.
I'd like to know why I can't use arenas in rust? Especially considering that I have used them before in rust.
f311a
You can’t allocate collections without nightly or without reimplementing them in the library. Every implementation uses it’s own set of trade offs to provide safety in unsafe implementation.
kibwen
Rust supports arenas just fine ( https://crates.io/crates/bumpalo ), and if you mean the support for using custom allocators in the standard library collections, that's as stable as Zig is.
f311a
There are more than 10 crates for arenas with different tradeoffs, I'm well aware of them. They are still very limited compared to C/Zig.
sdcfgy
System programming always matters. Things are cheap until they aren't one day.
tehlike
things are cheap until you reach a scale.
9dev
Things are cheap until they are someone else’s problem, I say!
jiggawatts
I wish more programming languages implemented record types as seen in databases, where dynamically sized fields are packed into a contiguous area of memory.
The CloudFlare manually implemented a clumsy version of this.
Wouldn’t it be nice for the compiler to manage this for you in the same way that your database engine does when it saves a “row”?
anitil
> dynamically sized fields are packed into a contiguous area of memory
Are you able to explain this? Do you mean an N sized array where each entry is either a value or a pointer to a value where the 'pointed-to' values are after the end of the array?
I'm trying to underatnd how you'd do this without having to parse M-1 elements to get the Mth entry if you did a [{size0, value0}, ....., {sizeN, valueN}] arrangement
toast0
I think they mean the cache entry is a collection of dynamically sized fields. It would be nicer to store that as a single contiguous allocation, rather than a bunch of pointers to individually allocated dynamically sized items. At least in this case, it might.
In a row oriented database, you get a contiguous spot for the whole row even when there are multiple variable width fields.
jiggawatts
There are various ways of implementing this, someone from a C programming background mentioned on option where the heap-allocated record objects aren't fixed size structs, but instead the allocated space is dynamically sized and the struct is just a prefix.
Physically the compiler would generate something like:
struct FooRecord {
long __length__;
int fixed_sized_field;
char some_other_field;
char* first;
char* last;
char* title;
}
Where 'first', 'last', and 'title' are sequentially stored after the struct in the heap memory.
There are variants of the above, of course. Instead of pointers the compiler could use lengths, offsets, or a pointer to the end of the variable length field -- this works because the beginning of the first field is at a fixed offset, and then pairs of pointers delimit the rest.
You can rely on the heap allocator to track the "__length__" instead, or you can encode it into the record explicitly to make "dynamic sized copies" simple.
Windows APIs generally work this way! You create a buffer, put a length in the first field, and then the API call writes a fixed-sized prefix followed by the dynamic-sized fields into the buffer. The 'length' is replaced too, so you know how many bytes to copy out without having to understand the structure.
Database engines go one step further and pack multiple "records" into a single "row". They typically store the fields "packed" at the start of the row with 16-bit length or offset markers at the end for the various dynamic sizes.
Something like:
fixed_sized_field // Row #0
some_other_field
first
last
title
fixed_sized_field // Row #1
some_other_field
first
last
title
... empty space ...
next_offset // always populated
row#1_title_offset
row#1_last_offset
row#1_first_offset
row#1_offset
row#0_title_offset
row#0_last_offset
row#0_first_offset
row#0_offset // typically the constant zero
The idea here is that every length is the difference between pairs of sequential offsets. I.e. row#1_title has length (next_offset-row#1_title_offset).
listeria
Depends on how the CacheEntry is stored, it's probably stored in a slice of &[CacheEntry] which precludes storing the record data alongside it as the size of each entry must be fixed.
irdc
This is where hand-rolled intrusive data structures, as are traditional in C, really shine.
jmalicki
Even in C, if you want differently sized data to be indexable in O(1), you're stuck leaving them as pointers. You definitely could just have a variable-sized area for this, but that level of optimization is pretty seldomly done in C.
cobalt
less ergonomic, but still totally doable
strenholme
With my own MaraDNS, I aggressively optimized the memory usage of blacklist entries by having a single really big malloc() to allocate the memory for the entries, then traversing that memory block for potentially blacklisted entries.
When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.
Why do I always find interesting new Twitter accounts just as the person is leaving :)
strenholme
Twitter has become a cesspool, and there’s a lot of reasons why people are leaving it in droves.
My personal issue is the misogynists who have created a hateful completely false narrative that 80% of the women sleep with 20% of the men (including the very demeaning and hurtful notion that all women are sexually promiscuous, but only if you’re one of the 20% of supposedly “Alpha” men) [1] Twitter is also full of—let’s call a spade a spade—racists who constantly post some video from years before showing some random Black person doing a criminal act, and then a bunch of racists comment that that’s how all Black people are and it’s the “evil left wing media” suppressing this supposed “truth”.
Just as Twitter has become a right-wing cesspool, Reddit has become a leftist cesspool, so I also avoid Reddit, which, like Twitter, is also becoming a closed walled garden—they just this month started clamping down on people reading old.reddit.com anonymously, so now you have to log in to have a usable interface with Reddit. Excuse me, no.
These seem like some fairly standard approaches for reducing memory usage. I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.
If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.
It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
ratorx
I think it’s more of a time vs code tradeoff, if done properly.
For example in the Vec case, you could theoretically build an alternative which encodes the “three sections” property internally, and ensures correctness at construction time for the pointers. Not as completely safe as a Vec, but you can still get similar benefits for the “business logic”.
But I agree, just having a custom structure that does not provide a safe wrapper around this would be sacrificing standard guarantees.
vsgherzi
you could always do a .get into the vector and handle the error, it doesn't necessarily need to panic.
Thank being said in this case it should be impossible to index out of bounds so maybe a panic is warented.
pocksuppet
It's the exact thing Rust is made to protect against, on a more local scale. Every memory corruption bug is just an out-of-bounds index that wasn't protected against.
bvanheu
is dangling pointers reuse memory corruption bug from out of bound index?
IshKebab
No. He's wrong that every memory error is out-of-bounds access (spatial memory safety). There's also use-after-free (temporal memory safety). Arguably type confusion too but that's a grey area.
NIckGeek
You can make a wrapper type that abstracts the offset lookup logic with a safe interface. If it's a transparent struct then rust will compile it away into nothing but you still get the abstraction in your code.
afdbcreid
> I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.
Not really. You just need to make the underlying fields private and provide methods to get slices to the data you need.
FpUser
Tools exist to serve us, not the other way around.
asgraham
Sure, and usually one of the ways Rust serves us is with safety guarantees.
Which isn’t to say this optimization is a bad idea, just to say it’s sort of a straw man to imply coding in Rust to take advantage of safety guarantees is “serving Rust”
grep_it
This reminds me how you can save a bunch of bytes just by making sure your structs are aligned. In go for example:
type Wasteful struct {
a int16
b int
c byte
}
type Aligned struct {
b int
a int16
c byte
}
Will have sizes of 24bytes and 16bytes (on a 64bit system). Same data 8bytes more. If you are storing millions of those objects, then it adds up.
masklinn
Rust does that automatically unless you switch to the C layout.
In langages that don’t there’s a tension between memory use and human readability / consistency of the layout.
There are also other domains which can be affected e.g. databases, it’s a concern / issue when using postgres for instance as it uses aligned columns and stores them in schema order.
jordiburgos
Why this is not done automatically by the compiler? That seems something quite easy to calculate to me.
JeromeLon
There is no way in C to express that you don't care about the orde. When you express a struct in C, you list what you want in the struct and (sometimes without wanting it) exactly in what order you want it.
Interestingly, there is also no way to write a loop on i for all the values between 0 and 99 without specifying the order. Luckily, in this case, the compiler is allowed to prove that the order has no impact (because it's local), and to decide that it will scan the values in a different order for optimisation purposes.
So the compiler could do it on a structure as well, as soon as it's able to prove that the structure is not exposed in any way to any code that it doesn't control, but that's much more difficult than proving that variable i is not visible outside of a tight loop.
cestith
It could be a new keyword rather than counting on the compiler to prove certain access patterns don't exist. That's a bit of a messy tradeoff. Maybe something like 'unordered struct' or 'packed struct' works, but it would be a nonstandard extension for some time.
masklinn
It is not very hard, but it is additional complexity, and it then requires the language to have a way to opt out so you can handle things like FFI or explicit ordering (usually for padding to avoid false sharing and friends).
So most languages opt to follow what their predecessors did: do nothing and task developers with reordering the structure if they want to minimise its size.
grep_it
Sibling comment touched on it, and I guess rust does offer just that, but void casting and doing explicit offset checks for a field is one reason. You can kind of think of it like a tuple in that way and a db engine might use a similar technique.
Agentlien
One of my proudest professional moments was when me and three others managed to reduce memory load of the game Wavetale from 20+GiB to under 3GiB so we could port it to Nintendo Switch.
The 100 TiB number almost gives me vertigo. Though in this context it was "just" 50%
adzm
I'd love to hear what was taking up that 17GiB if you can share, even if it is commonly optimized things like packing, still fun to hear about
Agentlien
It's an indie open world game originally released for Stadia and in its original form simply loaded the whole world into memory at boot.
We had to implement a streaming system and figure out a good way of chunking the world. This was a challenge because everything was on water and you could see nearby islands quite far away. An intern called Tommi did a great job identifying a good strategy and writing the system.
We also had to reduce the density and model complexity of a lot of environmental details such as rocks, vegetation, and stuff like pots and clotheslines. This was done largely by sorting things by memory size and frequency of use and identifying outliers.
One of the biggest issues was actually really silly: the journal fetched Portrait images and names of characters by referencing the actual NPC and having them embedded there. Meaning the journal, which was always loaded, would pull every NPC involved in a quest into memory including their behaviors, textures, and models.
On my end I also found a lot of silly details wasting hundreds of megabytes. Special render passes using huge textures and render targets, poor structuring of the render pipeline caused memory increases, several key shaders referenced huge textures which weren't necessary, ...
My blog posts about the project[0] mainly focus on rendering performance because that's where I spent more time and it contains more interesting content for discussion. But reducing memory was an ongoing concern with countless little improvements over the 1.5 year porting process
Not sure what they use to hold the cache key and entry. If a hashmap is used, then a radix tree (adaptive radix tree) would be better in saving memory space. Most of content of the qname field of the CacheKey is hostname, like www.site.com. The reverse version com.site.www fits nicely in navigation path of a radix tree. The common prefixes like "com." are shared and compressed in the parent nodes of the tree.
Even a BTree with compressed prefix keys can save space in the qname.
Tuna-Fish
At this scale, moving from one pointer chase to multiple is almost certainly a huge loss, even if radix tree would save a lot of memory.
ww520
Unless the keys are completely random, compressed keys shorten the tree height and cause fewer pointer jumps. Hostnames are highly compressible. Plus the root and the upper levels of the tree are always hot, most likely in L1/L2/L3 all the times. OTOH collisions in hash table cause pointer chase as well.
BikiniPrince
Funny thing about cloudflare. I have a dns warming script that uses their top 1k or 10k addresses. Then when my master starts up it warms the entire cache. Everything else uses memcache so the cluster is nice and toasty. As far as I can tell no one else releases domain statistics like them.
edflsafoiewq
General theme: A programming language's native in-memory object format is typically optimized for random access, uniformity, and mutability (fields at fixed offsets, etc). Serialization formats for network or disk tend to be designed explicitly to be more compact. But you can design your own in-memory representation too, with the properties you need.
kccqzy
That’s the old school of thought. These days, designers of newer serialization formats realize that designing a more compact format doesn’t really buy much on modern CPUs and modern networks. See for example Cap’n Proto (whose inventor, kentonv, also works at Cloudflare) and flatbuffers.
inigyou
That's also the ancient school of thought, before compaction was viable and before portability was needed.
0xAstro
It's weird that it took so long for these trivial optimizations but it might just be that they were working on optimizing other stuff.
sergq
this applies to more than DNS caches. In 1998 I mailed Microsoft a proposal to replace search engine crawlers with a push-based filesystem monitor (detect change → extract → compress → push to index). Got a 5-line rejection letter. They built the same thing 20 years later as IndexNow. Full story with the original letter: https://dev.to/andrew_vl/in-1998-i-proposed-push-based-searc...
rfgplk
Sadly it could be a result of xenophobia and discrimination due to your nation of origin. Very common in tech circles. If someone from San Francisco proposes it, it's a glorious idea, if not, it's junk.
bhouston
I've run into issues with using public wifi when I override my MacBook's DNS server to 1.1.1.1 or 8.8.8.8. I believe this is because captive portals require custom resolution of the name captive.apple.com. And external DNS servers will not resolve that correctly to the local gateway's authorization page.
MayeulC
AFAIK (at least it worked like that some 10 years ago) the captive portal just intercepts the HTTP page load and inserts its own content (most often a 302). So it just has to be a http web page. Firefox uses http://detectportal.firefox.com/canonical.html
Edit: ah, yes, DNS can be hijacked too (requires intercepting outgoing traffic on port 53 therefore incompatible with DoH), that may require fewer computing resources. Still need http otherwise the server cannot use the correct cert chain.
Edit 2: Wikipedia says both methods are used: https://en.wikipedia.org/wiki/Captive_portal and also mentions RFC 8910. I suspected something like that existed, hence my initial disclaimer.
My point was: that domain is not treated any differently from other domains.
comprev
I've had reliable success by using http://neverssl.com to force a basic HTTP connection for kickstarting a public WiFi portal login, although I have to disable NextDNS (iOS) too.
bpicolo
I use good ol example.org
Sohcahtoa82
The ironic part of neverssl.com is that it does indeed now support SSL.
fc417fc802
Can we take a minute to appreciate how utterly broken this state of affairs is? The dogged over centralization of DNS is an endless source of problems.
brians
That’s a Mac bug if so—it should be always using dumb udp/53 for captive detection, not some fancy DoH thing.
deathanatos
Dumb captive portals, which do still exist in some places, usually do MitM attacks on the connection, so you need some http(no-s) site that you can abuse as "yeah, this can get attacked by the WiFi" to then answer the portal.
The right way is that there's DHCP option for the network to signal "I have a captive portal", that's been standardized for over a decade.
… or … IDK … just stop shoving ads down people's throats just because they want WiFi.
jcgl
I’ve never even heard of the DHCP option for this. How widely adopted is it? It’s surprising to hear it called “the right way” when it wouldn’t work for a single-stack IPv6 (or IPv6-mostly, probably) network.
deathanatos
DHCP works over IPv6, but there's also a corresponding RA option, too, if your v6-only network doesn't use DHCP.
Ah great, thanks for the link! A very clearly-written RFC.
1saadcodes
We're finally seeing more appreciation for this kind of engineering. Not everything needs to be solved by throwing more hardware at the problem
xyst
With memory costs soaring due to AI bubble, this type of engineering has become "profitable".
9bot
The most interesting result to me is that the richer parsed representation was not necessarily the faster one. If the hot path is mostly “read from cache and serialize back to DNS,” parsing everything upfront only to serialize it again can become unnecessary work and hurt locality....
The intermediate level Rust dogma is to try your hardest to avoid the heap, and to tear your hair out at the throne of monomorphization. While both are broadly true, it's articles like this that show that a single pointer (or call) indirection can sometimes be better.
kibwen
I'd say that boxing large enum variants is itself an intermediate level Rust topic, and a well-accepted practice. Clippy will even point out places where you might benefit from boxing an enum variant: https://rust-lang.github.io/rust-clippy/master/index.html?se...
rfgplk
Frankly weird that they were resorting to high level containers for this in the first place. Also, this line struck me as odd
> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.
jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?
cobalt
Rust supports punning via pointer casting, but you'll want to use #[repr(C)] on any data types used
senderista
I would also have instinctively reached for a large VM reservation to exploit demand paging. I have used that pattern a lot in C++ but not in Rust, so I don't know how difficult it would be to implement there.
jandrese
An approach like that would be at constant war with the borrow checker in Rust. Apparently it is possible but there is enough friction that these guys went a different route.
kevincox
It sounds like you are just writing your own allocator? That sounds great, but why is it going to be better than jemalloc?
There are many reasons a specialized allocator can be better, but just saying "write your own" doesn't really add much value to the discussion.
foltik
If you squint, even their TLV encoding in a Box<[u8]> is kind of a specialized arena allocator.
Dylan16807
So they optimized from Vec to Box, but they're still using Box all over and spending 16 bytes on it? The things they're boxing need 2 bytes for length, and their memory use is low enough that they could cram the pointers into 4 bytes. Trying to pack that into 6 bytes is probably too much fuss for the benefit, but I see no reason to use more than 8 bytes.
didgetmaster
Why do people seem to think that optimization is something you only have to deal with once the software scales so much that 100s of TB of memory or disk space (or thousands of hours of processing time) are being wasted.
It is almost like nobody even thought during the design phase about what might happen down the road.
This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).
jacquesm
It can be quite hard to predict where particular usage patterns will take a piece of software under extreme load, especially with things that have lots of internal state. Obviously when you get to spend 100 T or more the pay off of an optimization is much larger than what it is in the case of 1T or less, and your typical developer is not going to have that kind of memory even in aggregate to play with. I tend to be forgiving when it comes to watching software bloat that I did not cause myself (and yet, I'm frustrated that Ubuntu's start-up greeting message takes a whopping 500 M).
In the case of internet infrastructure I don't think there was anybody even up to the year 2000 who had any idea of how bit this was going to be. And even now we have IPV4 and lots of legacy to deal with. Cloudflare is not my favorite company, let's put it like that, but in this case they show how the sausage is made and I think that should be applauded. Much better than 'why were down again for X hours'.
philippta
I recently heard a great analogy for this exact problem under the premise of "Make it work, make it right, make it fast".
Assume a sorting algorithm as a metaphor for your whole program.
You make it work by implementing the simplest thing you know how to write: bubble sort. Works.
At the end you notice it's way too slow and replace with something much better: quicksort.
Now, how much of your program is surviving? Almost nothing, perhaps except for the "greater than" comparison.
If you apply that idea to a real world program, we're pretty much talking about a full rewrite.
jacquesm
You could have picked a better example, sorting algorithms are typically drop-in replacements.
ManBeardPc
The Record struct contains rtype and data where RecordData is a tagged union. Aren’t those two always in sync? Not a DNS expert, just wondering if this is redundant or there is a reason both are there. Doesn’t matter anymore if they store it already serialized but I would be interested why it was this way.
fulafel
Where are their users coming from? Besides the few manually putting 1.1.1.1 in their settings.
kibwen
Firefox's default DNS-over-HTTPS provider is Cloudflare.
cristaloleg
Obvious question: why wasn’t this done earlier? It looks like all the data was already available. At THAT scale, reducing memory usage is a must-have, not a nice-to-have. Weird.
yieldcrv
probably agents going through tech debt or finding wins
every dept knows what they could do with more budget, the budget for those things just never comes
now agents have utilized budget more effeftively, unbottlenecking many things, including engineering blogs
sophacles
Cloudflare talks about having datacenters in 300+ cities. Presumably they have at least a few servers per datacenter. They saved 130 servers worth of memory... not even the minimum number of servers they have (seriously though, they probably have a LOT of servers)... a few GBs of memory per server running the service. At that scale this is a nice-to-have.
superze
How much is this in euro or do we measure money in ram now?
inigyou
Currently $15 per GB, he saved Cloudflare $1,500,000 and got exactly $0 bonus. He must really believe in cloudflare's vision (global enshittification). In related news, three times today Cloudflare told me that I'm a bot and shall not pass - not that it needs to check if I'm a bot before it lets me pass.
winrid
I saved an employer $4m/yr, entirely myself. I think I got a 10k bonus.
OptionOfT
> we store the records as a single Box<[u8]> containing each record encoded as a 2-byte length prefix followed by its raw bytes.
You start, get the type & length, and then that is how many bytes you read.
Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.
In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.
So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.
As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
jandrewrogers
This kind of encoding[0] is ubiquitous in networking protocols. It scales down to small silicon well and enables the receiver to estimate resource requirements or skip parts of a serial byte stream without storing it in memory first. These encodings usually aren't aligned by design.
It's called TLV encoding - tag/length/value. It's very common in all sorts of network protocols and serialisation formats. It allows you to skip unidentified tags. Sometimes, like in the PNG file format, there's a fixed bit in the tag that tells you whether it's safe to skip or if you have to reject the whole thing because you don't understand this tag.
Hey dang can I get my rate limit turned off pretty please?
squirrellous
I wonder at their scale, why wouldn’t it make sense to store the entries lightly compressed in memory?
winrid
Yeah with snappy they probably could get like 30% savings (I bet domains don't compress well) without much more cpu usage.
Alternatively a btree somehow they can take advantage of prefix compression
squirrellous
Ah yes, something like prefix tree would work well storing reverse domain names like com.abc.www (although this is such a well known thing I feel like I must be missing things).
akoboldfrying
Definitely. Even a custom compression/encoding that knows to treat the different fields differently -- just a 4-byte binary IP address for A records, normal LZ-based text compression of the domain name (perhaps using a custom starting dictionary and/or Huffman table), etc.
ricudis
REWRITE IT IN C!
dshat
I'll buys some spare RAM you now have. I only need 64GB.
fudgy73
Are they selling that RAM?
CyberDildonics
Great article, but I'm surprised they waited until they were using $2 million USD of memory before shaving off all the unused bytes at the end of a vector.
empiricus
my first thought was if is this will impact the memory market prices :)
this is so interesting, memory and storage is cheap until it isn't and then you optimize.
varispeed
Now put the 100 terabytes of memory back to the market. Stop hoarding RAM.
HDBaseT
DNS is important and 100TB of RAM is effectively nothing.
varispeed
Internet worked fine without 1.1.1.1. Corporate hoarding of resources to deprive working class from accessing it is worse.
NahImG
sees cloudflare
leaves
mannyv
One question the article doesn't answer is: why are they cacheing at all? If your cache is that big it isn't a cache. How much bigger is the dataset in question? There are 250 billion entries. Assuming 80/20, that implies 1.25 trillion records?
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
It's an interesting problem.
bastawhiz
Maybe I'm misunderstanding, but this powers 1.1.1.1, it doesn't front an internal dataset. A cache miss hits a nameserver. Which is to say, the dataset is "every DNS record in the world"
auspiv
I think the question is probably more along the lines of - why not do a database with 100 TB of storage/records instead of a cache? tomato / tomato.. especially with smart caching in front of database. 100TB of flash is a good bit cheaper than 100TB of memory
ecnahc515
Because it would be slower and have different scaling requirements than the ones they want.
pocksuppet
It's not 100TB of data. It's probably 50 GB of data on each of 2000 servers. Because it's a cache. What is the point of a central cache if it's as slow to access as the original data?
inigyou
TFA gives numbers closer to 5GB.
fc417fc802
I'm no expert but presumably all of throughout, latency, and churn. DNS is approximately a giant KV store where the typical record has a TTL of ~5 minutes.
robotresearcher
This is smart, task-specific caching in front of database.
seiferteric
You have to cache, cloudflare doesn't know all the records ahead of time, they have to do recursive lookups to the authoritative servers that own the records and that is only good for the period of the TTL of the record. There is no "global" DNS record database or something like that.
pbhjpbhj
>that is only good for the period of the TTL of the record.
Not really, TTLs are often short, but IPs might not change for years.
You can probably generate your own TTL, at scale, and avoid many DNS requests.
fc417fc802
Why would anyone want to use a DNS resolver that tampered with records on a large scale? The TTL is intentionally set by the originator of the record.
Or alternatively, if you don't tamper why would I want to use a service that serves stale data?
robotresearcher
Every distributed system serves stale data.
You can define away ‘stale’ by picking a consistency model, but look inside the consistency machinery and you will see fresher data you aren’t allowed to have yet.
fc417fc802
That's like refusing to use a fridge, insisting on letting your produce gradually spoil on the counter because "all harvested produce is in a state of decay anyway".
robotresearcher
No, it's using a fridge in the understanding of the fundamental limitations of fridges.
You asked ‘why would
I want to use a service that serves stale data’. The answer is you don’t want to. But if you have a distributed system your only choices are about the nature of the possible staleness, since global instantaneous consistency is known to be impossible due to fundamental features of the universe.
fc417fc802
In that case you're (intentionally?) misinterpreting what I originally said. The person I replied to was metaphorically suggesting turning the temperature dial on the fridge way up and I metaphorically asked why I would want to eat at his house if he's going to be serving me spoiled food.
My original question could also be described as asking why I would choose to shop at a grocer where items on the shelf are routinely far past the printed expiry date. That's perhaps a more direct analogy.
robotresearcher
Your additional explanation does help clarify your meaning.
Serving up records with TTLs in the past is so deeply weird that I didn't even consider that was what the grandparent had in mind.
otterley
In DNS, the owner of each record has full control over its TTL. Intermediary DNS servers are required to honor them and are not permitted to replace TTLs with their own.
ButlerianJihad
Actually that is not true. The IETF has expanded the definition of “TTL” and explicitly permits resolvers to serve “stale” RRs beyond their expiration time.
As a corollary, there is obviously no floor on refetching unexpired RRs, of course, except for efficiency concerns.
seiferteric
That's only when the authoritative server cant be reached though
dizhn
DNS servers do in fact do that but it would not be a good look for the world's largest DNS provider.
inigyou
I don't think cloudflare cares about how it looks, also I think Google is bigger.
pbhjpbhj
You are obliged to pass on the TTL, you're not obliged to cache according to it.
At least in my country (UK) I know of no law relating to DNS caching.
Why throwaway perfectly good data every few minutes that is only modified every couple of years, just so someone can move their domain quickly when they eventually wish to? It is my contention that a [caching] DNS service can do far better. Trusting user (domain owner) input blindly is not for me.
otterley
It's not some sort of public law with public enforcement, but it is in the RFCs that govern the protocol.
I should be a bit clearer here; the TTL is an upper bound on how long it can be cached. Caches are free to consult more frequently but not less frequently. That said, out of respect for upstream cache operators and authoritative servers, most DNS caches honor TTLs as best they can.
inigyou
The IETF isn't the internet police. You don't have to follow its advice.
otterley
No, but if you didn’t, the internet wouldn’t really work all that well. It was the fact that participants, despite being independent, all agreed (either explicitly or implicitly) to adhere to the standards that it became a global network. If they hadn’t, the result would have been more of the same: independent networks that only had narrow interoperability at best. It’s a textbook example of how global cooperation can yield incredible results.
Advocating to do things against agreed-upon standards without a compelling reason and without giving due consideration to the adverse consequences is one of the hallmarks of a bad engineer. Even Microsoft played nice with Internet standards for the most part (although with some notable exceptions at the application layer that got them well-deserved criticism).
inigyou
It was the fact that adhering to the standard was in the best interest of each participant. When it isn't, they don't.
otterley
It's impossible to know how often participants chose to stick to the standards anyway despite it not being in their best interest. Do you have any specific examples?
inigyou
Linux routers responding ICMP from 192.0.0.8 when not having an IP address (currently being standardized). Clouds using class E address space (240.0.0.0-255.255.255.254). Routers that drop IP packets with options. The entire concept of NAT until the existing behavior was eventually standardized. Unnumbered interfaces.
fc417fc802
I think all of those are examples of "[doing] things against agreed-upon standards [with] a compelling reason and [with] due consideration to the adverse consequences".
inigyou
And the compelling reason is always profit.
fc417fc802
"The entire concept of NAT" was for profit? "Linux routers responding ICMP from 192.0.0.8 when not having an IP address" was for profit? What?
The grocery store down the road only sells me things for profit but I hope you're not trying to imply that's somehow nefarious.
otterley
No it isn’t. Sometimes it’s security, or making the most out of limited resources.
Besides, not all profitable ventures are bad.
seiferteric
then they would be breaking DNS at scale.
toast0
It's a recursive resolver. The global DNS dataset is not something you could collect to serve directly vs caching from observations.
The data source is authoritative name servers operated by third parties, some of which are slow on their own, some of which are behind slow or lossy networks. Origin response times vary between probably 1 ms and 2 seconds +/- origins that never respond.
eggnet
They’re adding the cache consumed across all of their servers. It’s not one giant deep cache.
otterley
The simple answer is that if you didn't cache, DNS traffic would skyrocket, and the load would pile up on the authoritative servers, which were intended to be small, and during the early days of the Internet, were frequently on bandwidth-constrained links.
DNS is designed to distribute query load to the edge as much as possible, and that's enabled by caching. It just so happens that "the edge" is now becoming concentrated among a small set of providers because they wanted to make a business out of it.[1] They knew that this would be expensive going in, though.
[1] Nobody has to use 8.8.8.8 or 1.1.1.1. Most people can use their ISP's cache or a local cache instead without any noticeable difference in behavior.
fragmede
The problem is there is a noticable difference in behavior because the ISP cache is overloaded so queries take longer. Sure, that's not everyone's experience, but there's a reason people chose to use alternate servers.
BowBun
> If your cache is that big it isn't a cache.
This is an incorrect statement. Caches do not have a requirement of being smaller than their source data set. CDN is an example of a cache that generally matches the size of the source data.
eviks
> Once we store a DNS response in the cache, however, we never modify it again. The capacity field serves no purpose, but still costs 8 bytes per Vec
Were there no design discussions/reviews when the system was setup to catch trivial things like this?
r3trohack3r
Rob Pikes 5 Rules of Programming:
Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.
Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)
Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.
Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.
> Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Genuine question, is software performance really linear like that, that one can and should only fight the tightest bottleneck, one workload at a time? Never really sounded right.
It also sounds like the typical sleight of hand where the difficult bit is simply laundered a layer up, in this case the choice of what workload one investigates.
toast0
It can be. Sometimes you take a profile and there's a big smoking gun and nothing else matters.
Sometimes it's a lot of small things everywhere and you can pick up significant performance after a lot of small value fixes. In this case, caching wire data instead of structured data is almost one of these, because the contribution to response time for serving a cache hit is small... otoh it happens so often than a small improvement matters; but this is a pretty focused use case, you usually hit the many smalln improvement issue in a less focused application where there are many code paths.
Sometimes the whole code structure / data structures are so wrong, but it works and perf is bad and profiling will never tell you. This article is not that case; these data structures only needed refinement.
GeneralMaximus
In many cases, yes. A software pipeline can only achieve as much throughput as its slowest stage, and much of the software we write can be modeled as a sequence of processing stages.
eviks
> Data structures, not algorithms, are central to programming
So you agree that they should've designed the system to use the appropriate data structure from the beginning?
ecnahc515
Notice rules are ordered. You don't optimize until you know you need it. They started with a data structure they though would be fine. Clearly it was fine since it worked and they decided it was later worth optimizing.
x-complexity
Rule 5 is superseded by Rules 1 & 2. Without the measurements to back it up, you're chasing phantoms.
eviks
Rules 1&2 are about speed, not memory. But also, what do you think the issue with measurement is in this case??
win311fwg
The existence of 1.1.1.1 speaks to a much larger design problem. If you want to talk about what should have been done, you need to step much, much further back.
eviks
I don't want to step back and go off topic
win311fwg
"Should" cannot be evaluated in a vacuum. The only thing that would be off-topic is pretending that it can be.
lbriner
It is often not worth optimising in the early days. You don't know how popular it will become, you might not know how many DNS records you will hold, it was possibly written in an earlier language and ported as-is.
At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
eviks
It's also often that you can avoid all those future migration/fallback risks and pains if you invest a little bit of design thinking upfront.
So how would you decide which path to take in situations like this?
suriyaG
It only looks super obvious in hindsight and the well explained blog post. when a team of 5 is tasked with getting a completely new DNS up at the scale and integrate well with cloudflare.
if you spend cycles on nitty gritty opinions like this time to market goes out further and further out. some napkin math, 130 gen13 servers cost "only" ~$2.6M. relative to the importance of the 1.1.1.1 and the market at the time. that is nothing to cloudflare.
this is not to say good system design does not matter. it very much does, but making that call at that time would've butchered the prodcut very much similar to google+, youtube etc.
eviks
This one also looks pretty obvious "in foresight" (using the same tools that existed back then. Maybe owner dedupe might be less obvious and require a bit of knowledge and probing into actual data, but for rw vs ro you are fine knowing nothing?) and you forgot the napkin math re. how much your precious "time to market" would have been delayed by.
It's also not nothing, otherwise it would never be optimized away now, but left as is. After all, wasting time on optimization delays "time to market" for other useful features.
I also don't get the reference to YouTube, it's a very successful product, how was it butchered by good system design???
sophacles
Imagine you're an engineer at cloudflare, an 8 year old (at the time of launch of 1.1.1.1) company. The company is wildly popular and any service launched is going to have a lot of traffic and a lot of attacks right away. Any problems with it are going to embarass the company a lot.
You're tasked with making a DNS caching recursive resolver that can operate at a large scale and will be run on thousands of servers each of which has a lot of GBs of ram.
You are given some period of time to build this and make it production ready. How do you spend your time:
* Focusing on making sure that the resolver works correctly?
* Focusing on make sure that it actually provides improved DNS performance for internet users?
* Handles an very large number of record requests/s?
* Saves a few GB of ram per server?
There are tradeoffs to consider. RAM is cheap, even at today's prices RAM is not the most expensive thing that can go wrong in such a scenario. Having the responses be slow or incorrect is a far more expensive problem. A good engineer would pick a simple data structure that has the right shape but might not be optimal in footprint to focus on correctness and response time. The few extra GBs of RAM per server can be dealt with later.
When building things at scale you want to make sure it works correctly, fails correctly, and does the thing quickly before worrying about reducing resource consumption. I've never seen a project fail on Vec<T> vs Box<[T]> memory differeneces, or even on a few GBs of RAM usage per instance. I have seen them fail on "one wierd corner case of correctness" though, and on poorly thought through failure modes.
toast0
> The company is wildly popular and any service launched is going to have a lot of traffic and a lot of attacks right away.
Doesn't this also inform you that your cache will be very large, so you shouldn't use growable structures with slack space when cache entries won't grow; slop space reduces the size of your cache. And also that the query volume will be high so the cached data should require as little work as possible before returning data; spending time marshalling response data on every cache hit increases response time and decreases capacity.
sophacles
RAM is cheap. I'd find myself far far more concerned with:
* unbounded growth of the cache and properly invalidating after TTL expires (a few GBs of slop is nothing on a server with 64 or more GBs of ram, unbounded growth is a problem).
* making sure the DNS implementation works correctly on both the serving side and recursive resolution side.
* What strategy is best for deduping recursive requests across machines (if something a few miliseconds away has a live result, why do a full lookup taking hundreds or thousands of milliseconds?). This potentially improves RAM usage across the datacenter too from not having a given record on dozens (or more) machines' local cache. I don't know exactly how they do it, but naively I'd look at some sort of DHT shaped solution to look for records in peers within the datacenter. Or maybe some sort of tiered caching with the upper tier being sharded on domain name or the like.
* The biggest performance gains cloudflare can provide in Web and DNS cache come from a cache hit. This is on the order of 10s or 100s of ms due to having a big cache and short distance to the requesting machine. A suboptimal lookup algorithm that is a few microseconds slower in local compute and ram access is just not as important as the other concerns for dedup and cache sharing. That's not to say it's unimportant, just that it's not the top priority when you're trying to deliver this much larger performance gains from other aspects of the system. Thats why they are getting to it several years after release.
Cloudflare writes a lot about distributed systems solutions to various problems. They likely don't think as hard about single machine performance as much as whole datacenter performance when approaching problems.
Keep in mind that the per-server cost of the whole program pre-optimization seems to be about 10GB (from the graph in the post). IME that's not bad for a big busy caching service.
toast0
> The biggest performance gains cloudflare can provide in Web and DNS cache come from a cache hit.
Using twice as much ram per cache entry makes the cache half as large, assuming your cache is bounded by ram, unless the queried, unexpired result set is less than the ram budget (which I would tend to doubt... lots of randomized queries out there; maybe I'm wrong if the cache size dropped).
When you're storing billions of records, it makes sense to spend a few minutes to consider how they're used and make a good choice about how to store them.
When you're getting a cache hit tons of times per second, it makes sense to consider every step and which ones don't need to happen every time. You have to consider every step while you're pursing correctness anyway, so might as well have the performance lens active too.
I'm not asking for heroic optimization: I didn't ask for vectorized stuff or kernel/nic offloading or kernel bypass networking... Just you have to use some data structures, you might as well not use ones that are expensive for features you don't need; and you have to store something in your cache, you may as well store something that requires less munging on the way out.
If this were a small local cache, that didn't want to use something already existing like unbound for some reason then yeah, data structures don't make a huge difference, extra marshalling doesn't make a huge difference, just don't reimplement all the CVEs that BIND had in the 90s. But if you're going to allocate 100 TB of ram, make it count. Even if you do use twice the ram but you get value from it, maybe that's fine... I've run wacky systems with bloated storage when there was a benefit. Vec doesn't give any value over a Box<[]> in this case; convenience or lazyness would be fine except that the sheer number of objects makes it worth the few minutes it takes to do something better.
sophacles
> Using twice as much ram per cache entry makes the cache half as large, assuming your cache is bounded by ram, unless the queried, unexpired result set is less than the ram budget (which I would tend to doubt... lots of randomized queries out there; maybe I'm wrong if the cache size dropped).
This is true. I'm arguing that its unlikely this was ever bound by available RAM. Cloudflare is a DDoS protection company that absorbs attacks. They have a lot of available capacity at any moment. When you're building a service in a sitaution where you have more capacity than you'll likely need.
The savings were 100 TB across >300 data centers. The savings were on the order of 50%. So prior to this reduction the service was using something less than 2/3 of TB per datacenter. The service ram usage was about 10GB per instance according to the graph in post. IDK how cloudflare divides thier stuff between machines, but assuming they don't run less than 64 GB per server that's less than 12 servers per datacenter of ram for a flagship product, and they likely run it spread across 65 of the machines in the datacenter that are also doing other stuff. The per-instance RAM likely isn't the the concerning limit.
Overall RAM usage is proabably a bigger concern. Thats why I would think about dedup between instances and distributed caching strategy first. I could focus on redudcing the ram needed per service instance and get a 50% reduction per machine. Or I could focus on deduping 1/n (where n > 2) reduction in total memory usage across all instances. Personally if I was worried about reducing RAM I'd put more energy into growing N.
However all this is a red herring. The assumption people are making is that the cache was always read-only, and it's obvious that Box<[T]> was the best decision because in a RO cache smaller entries hold more things.
The 1.1.1.1 service advertises improved DNS performace. That's its value add. The biggest performance gain you can have from a cache is not having a cache miss, and in DNS a cache miss means a very expensive recursive lookup. So there's concerns about how to minimize those lookups. If one instance has does a lookup, it makes sense to share that result to the other instances that may need to do a lookup [1]. I don't know off the top of my head if it makes sense to get those updates and modify the existing record or just replace it in the local cache. That comes down to locking strategies and reading patterns in the specific code and service traffic patterns. Until i have hard evidence one way or another I'd like my cache to be able to do both and keep the data structs modifiable until that's nailed down. If per-isntance ram ever becomes the issue, there's easy wins there to buy me time to find better large scale solutions to the problem.
No one is disagreeing that the larger datastructures are larger. No one is disagreeing that they take more RAM, and or even if RAM was the the problem reducing it would be good.
The thing people are pointing out is that this isn't a homework problem about an optimal cache structure in a vacuum. We're pointing our that engineering real large scale solutions has a lot more to consider than a homework problem, and that the thing you're harping about likely didn't have any real budgetary or noticable performance impact on bulding that system. The reduction in ram is just a smallish improvement in operating costs after all the more expensive stuff was figured out.
Put another way 100TB of RAM is ~$350K. Thats one engineer year for a mid-level engineer.[2] Would you rather spend that money to save an equivalent amount of money somewhere, or... would you spend that money putting the engineer on something that saved $700K elsewhere (alternately that generated $700K)?
[1] I talked a lot about dedup and the simple gotcha is "hahah then its not deduped so you need smaller objects". But on a service that is running on a few dozen instances having a few redundant copies to deal with loss of a machine and/or load can still result in 1/(n>2) savings in total ram.
[2] I'm not saying someone worked on this for a year btw, a couple people likely spent a couple months on the code, validation and testing of it. A manager spent time overseeing it. Operations people spent time understaning any effects it had on running systems. Costs add up and it wouldn't suprise me if this didn't end up being roughly break-even for the year.
mhitza
Premature optimization argument fits right in. Now that memory is up to 10x more expensive it is worth considering optimizing programs with large memory footprint.
toast0
Using obviously better data structures the first time isn't premature optimization.
mannyv
There was a reason for that field, but that reason never panned out.
eviks
Could you point to that reason?
mayli
Maybe it's not that obviously better when it's impltd.
eviks
How does that fit? What would be the evil of not wasting memory for many years at 1x?
jgrahamc
One of the "evils" of premature optimization is how much time you spend on the optimization vs. the benefit you get from it. If your goal is correctness and shipping fast and you're not memory constrained then spending time using the least amount of memory is a waste of time specifically because you want to ship fast.
Another interesting thing that happens is you don't necessarily know what form your actual optimizations will need to take. Later when your systems grow you discover the suboptimal parts you hadn't optimized for.
Very early on at Cloudflare I worked on part of the DNS infrastructure that took DNS records from the UI and got them in a state for actual authoritative serving. The system had been constructed anticipating Cloudflare having millions of customers with unique domains, but it had not been constructed for a single customer with a single domain with millions of records. This caused a periodic slow down in DNS record updating while the system churned on that one customer.
In a different job I worked on a piece of optimization software that needed to keep track of "node" A is reachable from node "B". This had been implemented as a matrix (literally a malloced NxN matrix of ints storing 0 or 1) which worked really well for small systems. But you'd be out of memory really fast on a large project. I replaced the matrix with a hash table and all was good because the matrix was actually really sparse.
stickfigure
Absolutely true, but I will say that LLMs have changed the equation somewhat.
With a rather short prompt, claude/codex will take your code, write a harness, profile it, build experiments, profile those, and give some pretty solid advice which one to pick. Then integrate the changes. It's the kind of goal-directed, bite-sized job that LLMs excel at. Extremely low-commitment.
Except for the whole "making changes in production at scale" problem, of course.
gbear605
Engineers are expensive, especially good system engineers who are trained in your code base. Very possible that this just hadn't gotten to the top of the priority list.
eviks
I don't understand why you need training on your code base to design a cache format for read only vs rw workloads, but anyway yours is a comment about neglect, not the "evil" that would happen if you did that design
Spooky23
I see your point but disagree. Engineering is about constraints. Time, materials, labor, scope.
The “evil” of premature optimization is that it’s a misapplication of priority. If I have an acute medical problem that needs attention, it’s not the right time to talk about chloresterol and statins, get my broken leg set.
There’s always a tension between engineering management who needs to deliver a solution to the business and engineers who want to deliver a beautiful object.
win311fwg
> I don't understand why you need training on your code base to design a cache format
Because anyone willing to come in just to design your cache format is going to expect payment that is many multiples more than the engineers you already cannot afford? Long-term employees cost less, which brings them closer to being affordable, but you have to be able to keep them busy for long periods of time to realize that reduction in cost. A engineer who doesn't understand your codebase isn't going to be useful for very long.
eviks
You explained why it's beneficial for other workloads, but the original point was about this specific design
scott_meyer
Discussing trivial optimizations is a waste of valuable design time. You're never going to "forget" an optimization. The running system will remind you when the optimization is actually needed.
ratmice
Boxed slice isn't really the most well known type/optimization,
There usually aren't that many vec's that it makes a big difference.
Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.
However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.
So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.
Pretty sure they were joking.
I would not understimate what folks in 70s had done.
<strange high pitched equivocal sounds>
I do a lot of tuning work that others either can't be arsed to do, or more often are scared shitless of touching (learned helplessness). I tend to work on things that have to go right the first time in general, but I firmly believe that I learned that skill doing optimization work early in my career, not the other way around. Through a series of unfortunate events, my first specialization was performance, before I got good enough at anything else for people to listen to me.
Ultimately, the single thing that fucks up performance work, far and above any other concerns or sources, is that the vast majority of developers want to skip from Make it Work to Make it Fast without going through Make it Right. Then they have to use dodgy 'optimization' tricks that embrittle the entire thing, destabilize production and make every new feature feel like walking a minefield.
So what happens is that when someone who actually knows what the fuck they are doing comes along and wants to build a superhighway, they find the route littered with abandoned 'vehicles' that have sunk into the mud up to their door handles and left to cement in place. Clearing the proverbial route becomes a bigger technical challenge than building the road. And so you'll find that your peers have a bunch of ideas about what needs to be done but none of the stomach to do any of it.
Which is why I say "stubbornness" is the most important tool in my performance toolbox, far above cunning.
I think Cloudflare became big only because they were so much more optimized than others that they offered some services for free that others were not offering. If running costs are high, you only burn (VC) cash and then you exit.
I still look after a few VMware estates and a lot of Proxmox ones (that used to run VMware).
Hilariously, VMware is described as "enterprise class", which I can only conclude means MVP and a bit wanky.
Today I repaired a Proxmox HA + Ceph node using boring old normal Linux skills and as it turns out I have 30 years of those. Part way through a remote v8 to 9 upgrade I think I lost comms due to using OpenvSwitch for networking and despite using tmux for the upgrade session. Anyway, the Proxmox ISO was useless for rescue but the classic systemrescuecd worked nicely and I could run dpkg in a chroot.
VMware "used" Linux and never really gave back. I don't miss fixing vCentres and all the other nonsense that "Enterprise" wankery has foisted on me over the years.
Then I actually met some enterprise software, and realised that it means 'expensive', 'bespoke', 'one-off', and usually 'janky'.
Why solve the problem directly when you can abstract everything away into FactoryFactoryImplementationInterfaceFactorys, and have something that is both a memory-hog and completely unassailable to any normal programmer seeking to understand it or make changes?
> "The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal"
also:
> "In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable"
In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals.
In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".
[1] https://dl.acm.org/doi/pdf/10.1145/356635.356640
Oof that hit hard. The last project I worked on suffered from a very similar disease, and it has really taken a toll on me psychologically. To work day in and day out on something that you can prove cannot work is unbelievably demoralizing. From an organizational standpoint, it makes a lot of sense to have an "internal consultant" who can deliver bad news like this. I tried to do it from the "inside" which was a huge mistake--got a negative performance review saying I had a "communication problem" because nobody wants to hear "negativity". I can come off online as kind of an asshole, so this may not seem credible, but I did actually deliver this news in a professional, measured manner. It's just that organizations are allergic to it, and their antibody response kicks in. You need someone who is not affected by the organizational hierarchy (or at least not that branch of the tree) to step in and deliver the bad news without fear of retaliation.
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
Better performance = saving money + better user experience.
The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.
In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
Insufficiently optimized code kills companies in highly competitive markets.
I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users.
I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost.
> In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
We give perverse incentives to the marketing department to bring us the most customers they can, instead of the most appropriate customers.
Someone told me at my second lead position that they had landed a top-tier customer for our demo-ware and the first words out of my mouth were, "FUCK ME". Not the response they were expecting, but then that guy never did end up understanding me the entire time we worked together. My bosses did though.
When your system is new you're a loss leader for all of your customers. Every dollar they bring in costs you two, possibly more. You have to get to positive MRR before the venture capital runs out, and over a long enough time horizon, you will go for a new round and find out that a recession is about to start and the VC guys getting cagey is the first clue it's coming.
Meanwhile if you spend all of your time and energy on reducing costs instead of increasing revenue, then your competitors catch up with you. Particularly if their funding rounds are half a cycle off from yours - one of you will be the last one to have gotten a cash infusion before the money dried up.
Most recently I worked at a place that didn't understand my calls for sobriety until it was too late. So I got put in charge of a rear-guard action that was too little too late, and our customers all fled to much cheaper competitors who could do 80% of what we could for half the price. I learned some good stuff, but the company got bought by a competitor who scrapped those systems.
this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit.
Or, in the current era of ai, the cost of the capital investment is far exceeding the ability to generate profit off it. The optimization in how the resources gets used will be needed to cut the costs down, and allow increase in the scale of usage for the same hardware. That's where profits would lay.
Of course, in order to achieve any of this, you'd need the runway to survive until such times. A small scale operator won't have this runway, and so die before they can accomplish anything profitable (or get big by begging for investor money to grow large - as we've seen in the past 20 years of tech).
One, most of the things you believe scale linearly scale at nlogn because of physics. Storage is much worse, scaling at n^1.5. What fit in a single 4U case is orders of magnitude slower when it no longer fits in a single data center. Inserting records into a table with a billion rows is much slower than a table with ten thousand. Wipe the surprised Pikachu look off of your face.
Second, every architectural choice needs to be reviewed every time traffic increases by an order of magnitude, and preferably before so that you have time to stew on possible solutions and allow serendipity the time it needs to find you a better answer. If you have made too many architectural decisions, you will find there aren't enough hours in a month to keep up, even if you never sleep. That's the real reason we practice Principle of Least Power. Not security, but job security. Keep the system small enough that you have time to keep the plates spinning while also implementing new features to attract or retain customers.
Tons in the $200s btw, and this is in the city less than 20 minutes from the Nelson-Atkins Museum, WW I museum, the river, farmers market, KC Chiefs / Royals / Sporting KC / KC Current teams, parks, schools, newly rebuilt international airport, etc.
[0] https://www.realtor.com/realestateandhomes-detail/3711-N-67t...
[1] https://www.realtor.com/realestateandhomes-detail/20118-E-Tr...
That said, look around:
https://www.zillow.com/kansas-city-mo/?category=SEMANTIC&sea...
Between $175k and $350k center on KCMO alone there are over 760 results.
Note: your first link https://www.realtor.com/realestateandhomes-detail/3711-N-67t... is 23 minutes from the WW I museum.
Note 2: Your second link is way out in Independence, MO but is still only 29 minutes away from the WW I museum.
I realize multiple acres of land is a bit of a silly requirement, but you're the one who suggested there are tons of houses with "several acres within 20 minute drive to a lot of US city downtown areas". That's why I was surprised.
This is my direct quote.
Let me break this down:
1. That would buy you thousands of square feet
True, clearly seen in the search results.
2. and often several acres
True, you just filtered down to several results currently on the market.
> I realize multiple acres of land is a bit of a silly requirement
Why? I live slightly further than 20min outside of town. There are massive, beautiful homes with tons of land out here, and tons have gigabit fiber from GFiber or several of the other providers in the area.
But that's not a requirement if you don't want it to be! There are clearly tons of homes available in the city with 3-4+ bedrooms and 2+ bathrooms in the $200 and $300ks is the point.
Acknowledging this isn’t always easy or possible, but just pointing out that this is a self reinforcing problem.
I meant desirable for me to live there, not as an investment. Who wants to buy a home in place they don't want to live?
I mentioned acres of land. You normally don't have multiple acres of land in the suburbs.
> is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
Who wants acreage? We want homes.
What does "city" mean to you? For some, it's 500 people, or 5,000. For some, it's 5 million. Define that first. The US is a big place, and I know people that don't live within 50 miles of another human.
Otherwise:
https://www.zillow.com/homedetails/424-Olive-St-Kansas-City-...
4 bed, 3 bath, 1,580 sq ft, beautiful! $342,500, built in 1900.
https://www.zillow.com/homedetails/3508-N-College-Ave-Kansas...
4 bed, 4 bath, 2,295 sq ft, $365,000, built in 2022.
Yeah that sounds cruel and messed up, because it is. I don’t like how expensive real estate is and I wish we could have another 2008 without a bailout. But the point stands. If you hafta pick between eating and paying your mortgage, you made a bad financial decision. Blame whoever you want, I am of the opinion that people still have agency and are responsible for their decisions, as unpopular as that seems to be these days.
You'd be surprised how much house and land you can get for that price.
not everybody is so lucky to be able to go in that order? The first part requires upfront capital/investment?
It's only right if you are not the one who pay the bills.
from a business perspective, this might be considered the only way, but it is not. at large volume scale it becomes more, but often large scale is lacking optimisations in the first place.
its not wrong in my eyes, but definitely not the only path to take.
Not if the whole thing is architected poorly but was a requirement of the hour so it became big. Then optimisation becomes an art, but definitely not the ‘easiest part of the process’
Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...
I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.
¹ https://timeline.www.cloudflare.com/
Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
[1] https://doc.rust-lang.org/reference/dynamically-sized-types....
[2] https://doc.rust-lang.org/nomicon/exotic-sizes.html
I assumed they couldn't do that because they're using it with some kind of generic HashMap<K, V>. In that situation, can "V" be dynamically sized?
A dynamically sized "V" would mean you can't have an array of them, which might preclude some hash map implementations.
, which HashMap does not do, i.e. the keys and values have to have a statically known size.
Cloudflare started to pick Zig recently, for projects, that have memory constraints.
You definitely can and this is done a lot. What you might mean is that you can't use standard library's collections with them (this is getting stabilized soon!) and have to use third-party, but that is a different thing than "can't use arenas".
> Rust is not a good choice for this kind of tricks.
Rust can do those tricks, but it's true that it is hard than in C or Zig. That said there are often crates to help.
The CloudFlare manually implemented a clumsy version of this.
Wouldn’t it be nice for the compiler to manage this for you in the same way that your database engine does when it saves a “row”?
Are you able to explain this? Do you mean an N sized array where each entry is either a value or a pointer to a value where the 'pointed-to' values are after the end of the array?
I'm trying to underatnd how you'd do this without having to parse M-1 elements to get the Mth entry if you did a [{size0, value0}, ....., {sizeN, valueN}] arrangement
In a row oriented database, you get a contiguous spot for the whole row even when there are multiple variable width fields.
So logically you'd have the equivalent of:
Physically the compiler would generate something like: Where 'first', 'last', and 'title' are sequentially stored after the struct in the heap memory.There are variants of the above, of course. Instead of pointers the compiler could use lengths, offsets, or a pointer to the end of the variable length field -- this works because the beginning of the first field is at a fixed offset, and then pairs of pointers delimit the rest.
You can rely on the heap allocator to track the "__length__" instead, or you can encode it into the record explicitly to make "dynamic sized copies" simple.
Windows APIs generally work this way! You create a buffer, put a length in the first field, and then the API call writes a fixed-sized prefix followed by the dynamic-sized fields into the buffer. The 'length' is replaced too, so you know how many bytes to copy out without having to understand the structure.
Database engines go one step further and pack multiple "records" into a single "row". They typically store the fields "packed" at the start of the row with 16-bit length or offset markers at the end for the various dynamic sizes.
Something like:
The idea here is that every length is the difference between pairs of sequential offsets. I.e. row#1_title has length (next_offset-row#1_title_offset).When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.
https://samboy.github.io/blog/entries/MaraDNS.html#BlogEntry...
My personal issue is the misogynists who have created a hateful completely false narrative that 80% of the women sleep with 20% of the men (including the very demeaning and hurtful notion that all women are sexually promiscuous, but only if you’re one of the 20% of supposedly “Alpha” men) [1] Twitter is also full of—let’s call a spade a spade—racists who constantly post some video from years before showing some random Black person doing a criminal act, and then a bunch of racists comment that that’s how all Black people are and it’s the “evil left wing media” suppressing this supposed “truth”.
Just as Twitter has become a right-wing cesspool, Reddit has become a leftist cesspool, so I also avoid Reddit, which, like Twitter, is also becoming a closed walled garden—they just this month started clamping down on people reading old.reddit.com anonymously, so now you have to log in to have a usable interface with Reddit. Excuse me, no.
[1] This annoyed me to the point I researched the claims to verify it’s a bunch of bullshit. https://samboy.github.io/blog/80-20-myth.html
- TigerBeetle: A database without dynamic memory allocation, https://news.ycombinator.com/item?id=33192288 (2022).
- Succinct Data Structures: Cramming 80,000 words into a Javascript file, https://news.ycombinator.com/item?id=2348619 (2011).
If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.
It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
For example in the Vec case, you could theoretically build an alternative which encodes the “three sections” property internally, and ensures correctness at construction time for the pointers. Not as completely safe as a Vec, but you can still get similar benefits for the “business logic”.
But I agree, just having a custom structure that does not provide a safe wrapper around this would be sacrificing standard guarantees.
Thank being said in this case it should be impossible to index out of bounds so maybe a panic is warented.
Not really. You just need to make the underlying fields private and provide methods to get slices to the data you need.
Which isn’t to say this optimization is a bad idea, just to say it’s sort of a straw man to imply coding in Rust to take advantage of safety guarantees is “serving Rust”
In langages that don’t there’s a tension between memory use and human readability / consistency of the layout.
There are also other domains which can be affected e.g. databases, it’s a concern / issue when using postgres for instance as it uses aligned columns and stores them in schema order.
Interestingly, there is also no way to write a loop on i for all the values between 0 and 99 without specifying the order. Luckily, in this case, the compiler is allowed to prove that the order has no impact (because it's local), and to decide that it will scan the values in a different order for optimisation purposes.
So the compiler could do it on a structure as well, as soon as it's able to prove that the structure is not exposed in any way to any code that it doesn't control, but that's much more difficult than proving that variable i is not visible outside of a tight loop.
So most languages opt to follow what their predecessors did: do nothing and task developers with reordering the structure if they want to minimise its size.
The 100 TiB number almost gives me vertigo. Though in this context it was "just" 50%
We had to implement a streaming system and figure out a good way of chunking the world. This was a challenge because everything was on water and you could see nearby islands quite far away. An intern called Tommi did a great job identifying a good strategy and writing the system.
We also had to reduce the density and model complexity of a lot of environmental details such as rocks, vegetation, and stuff like pots and clotheslines. This was done largely by sorting things by memory size and frequency of use and identifying outliers.
One of the biggest issues was actually really silly: the journal fetched Portrait images and names of characters by referencing the actual NPC and having them embedded there. Meaning the journal, which was always loaded, would pull every NPC involved in a quest into memory including their behaviors, textures, and models.
On my end I also found a lot of silly details wasting hundreds of megabytes. Special render passes using huge textures and render targets, poor structuring of the render pipeline caused memory increases, several key shaders referenced huge textures which weren't necessary, ...
My blog posts about the project[0] mainly focus on rendering performance because that's where I spent more time and it contains more interesting content for discussion. But reducing memory was an ongoing concern with countless little improvements over the 1.5 year porting process
[0] https://agentlien.github.io
Even a BTree with compressed prefix keys can save space in the qname.
Relevant support page, though light in details: https://support.mozilla.org/en-US/kb/captive-portal
Edit: ah, yes, DNS can be hijacked too (requires intercepting outgoing traffic on port 53 therefore incompatible with DoH), that may require fewer computing resources. Still need http otherwise the server cannot use the correct cert chain.
Edit 2: Wikipedia says both methods are used: https://en.wikipedia.org/wiki/Captive_portal and also mentions RFC 8910. I suspected something like that existed, hence my initial disclaimer.
My point was: that domain is not treated any differently from other domains.
The right way is that there's DHCP option for the network to signal "I have a captive portal", that's been standardized for over a decade.
… or … IDK … just stop shoving ads down people's throats just because they want WiFi.
https://www.rfc-editor.org/info/rfc8910/ is, I think, the latest version of the standard.
And they say nobody uses IPV6.
https://www.google.com/intl/en/ipv6/statistics.html
> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.
jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?
There are many reasons a specialized allocator can be better, but just saying "write your own" doesn't really add much value to the discussion.
It is almost like nobody even thought during the design phase about what might happen down the road.
This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).
In the case of internet infrastructure I don't think there was anybody even up to the year 2000 who had any idea of how bit this was going to be. And even now we have IPV4 and lots of legacy to deal with. Cloudflare is not my favorite company, let's put it like that, but in this case they show how the sausage is made and I think that should be applauded. Much better than 'why were down again for X hours'.
Assume a sorting algorithm as a metaphor for your whole program.
You make it work by implementing the simplest thing you know how to write: bubble sort. Works.
At the end you notice it's way too slow and replace with something much better: quicksort.
Now, how much of your program is surviving? Almost nothing, perhaps except for the "greater than" comparison.
If you apply that idea to a real world program, we're pretty much talking about a full rewrite.
every dept knows what they could do with more budget, the budget for those things just never comes
now agents have utilized budget more effeftively, unbottlenecking many things, including engineering blogs
Interestingly this is exactly how netlink works-ish: https://manpages.ubuntu.com/manpages/focal/man3/netlink.3.ht...
You start, get the type & length, and then that is how many bytes you read.
Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.
In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.
So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.
As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
[0] https://en.wikipedia.org/wiki/Type–length–value
Hey dang can I get my rate limit turned off pretty please?
Alternatively a btree somehow they can take advantage of prefix compression
leaves
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
It's an interesting problem.
Not really, TTLs are often short, but IPs might not change for years.
You can probably generate your own TTL, at scale, and avoid many DNS requests.
Or alternatively, if you don't tamper why would I want to use a service that serves stale data?
You can define away ‘stale’ by picking a consistency model, but look inside the consistency machinery and you will see fresher data you aren’t allowed to have yet.
You asked ‘why would I want to use a service that serves stale data’. The answer is you don’t want to. But if you have a distributed system your only choices are about the nature of the possible staleness, since global instantaneous consistency is known to be impossible due to fundamental features of the universe.
My original question could also be described as asking why I would choose to shop at a grocer where items on the shelf are routinely far past the printed expiry date. That's perhaps a more direct analogy.
Serving up records with TTLs in the past is so deeply weird that I didn't even consider that was what the grandparent had in mind.
https://www.rfc-editor.org/info/rfc8767/
As a corollary, there is obviously no floor on refetching unexpired RRs, of course, except for efficiency concerns.
At least in my country (UK) I know of no law relating to DNS caching.
Why throwaway perfectly good data every few minutes that is only modified every couple of years, just so someone can move their domain quickly when they eventually wish to? It is my contention that a [caching] DNS service can do far better. Trusting user (domain owner) input blindly is not for me.
I should be a bit clearer here; the TTL is an upper bound on how long it can be cached. Caches are free to consult more frequently but not less frequently. That said, out of respect for upstream cache operators and authoritative servers, most DNS caches honor TTLs as best they can.
Advocating to do things against agreed-upon standards without a compelling reason and without giving due consideration to the adverse consequences is one of the hallmarks of a bad engineer. Even Microsoft played nice with Internet standards for the most part (although with some notable exceptions at the application layer that got them well-deserved criticism).
The grocery store down the road only sells me things for profit but I hope you're not trying to imply that's somehow nefarious.
Besides, not all profitable ventures are bad.
The data source is authoritative name servers operated by third parties, some of which are slow on their own, some of which are behind slow or lossy networks. Origin response times vary between probably 1 ms and 2 seconds +/- origins that never respond.
DNS is designed to distribute query load to the edge as much as possible, and that's enabled by caching. It just so happens that "the edge" is now becoming concentrated among a small set of providers because they wanted to make a business out of it.[1] They knew that this would be expensive going in, though.
[1] Nobody has to use 8.8.8.8 or 1.1.1.1. Most people can use their ISP's cache or a local cache instead without any noticeable difference in behavior.
This is an incorrect statement. Caches do not have a requirement of being smaller than their source data set. CDN is an example of a cache that generally matches the size of the source data.
Were there no design discussions/reviews when the system was setup to catch trivial things like this?
Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.
Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)
Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.
Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.
https://web.archive.org/web/20260314210910/https://users.ece...
Genuine question, is software performance really linear like that, that one can and should only fight the tightest bottleneck, one workload at a time? Never really sounded right.
It also sounds like the typical sleight of hand where the difficult bit is simply laundered a layer up, in this case the choice of what workload one investigates.
Sometimes it's a lot of small things everywhere and you can pick up significant performance after a lot of small value fixes. In this case, caching wire data instead of structured data is almost one of these, because the contribution to response time for serving a cache hit is small... otoh it happens so often than a small improvement matters; but this is a pretty focused use case, you usually hit the many smalln improvement issue in a less focused application where there are many code paths.
Sometimes the whole code structure / data structures are so wrong, but it works and perf is bad and profiling will never tell you. This article is not that case; these data structures only needed refinement.
So you agree that they should've designed the system to use the appropriate data structure from the beginning?
At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
So how would you decide which path to take in situations like this?
if you spend cycles on nitty gritty opinions like this time to market goes out further and further out. some napkin math, 130 gen13 servers cost "only" ~$2.6M. relative to the importance of the 1.1.1.1 and the market at the time. that is nothing to cloudflare.
this is not to say good system design does not matter. it very much does, but making that call at that time would've butchered the prodcut very much similar to google+, youtube etc.
It's also not nothing, otherwise it would never be optimized away now, but left as is. After all, wasting time on optimization delays "time to market" for other useful features.
I also don't get the reference to YouTube, it's a very successful product, how was it butchered by good system design???
You're tasked with making a DNS caching recursive resolver that can operate at a large scale and will be run on thousands of servers each of which has a lot of GBs of ram.
You are given some period of time to build this and make it production ready. How do you spend your time:
* Focusing on making sure that the resolver works correctly?
* Focusing on make sure that it actually provides improved DNS performance for internet users?
* Handles an very large number of record requests/s?
* Saves a few GB of ram per server?
There are tradeoffs to consider. RAM is cheap, even at today's prices RAM is not the most expensive thing that can go wrong in such a scenario. Having the responses be slow or incorrect is a far more expensive problem. A good engineer would pick a simple data structure that has the right shape but might not be optimal in footprint to focus on correctness and response time. The few extra GBs of RAM per server can be dealt with later.
When building things at scale you want to make sure it works correctly, fails correctly, and does the thing quickly before worrying about reducing resource consumption. I've never seen a project fail on Vec<T> vs Box<[T]> memory differeneces, or even on a few GBs of RAM usage per instance. I have seen them fail on "one wierd corner case of correctness" though, and on poorly thought through failure modes.
Doesn't this also inform you that your cache will be very large, so you shouldn't use growable structures with slack space when cache entries won't grow; slop space reduces the size of your cache. And also that the query volume will be high so the cached data should require as little work as possible before returning data; spending time marshalling response data on every cache hit increases response time and decreases capacity.
* unbounded growth of the cache and properly invalidating after TTL expires (a few GBs of slop is nothing on a server with 64 or more GBs of ram, unbounded growth is a problem).
* making sure the DNS implementation works correctly on both the serving side and recursive resolution side.
* What strategy is best for deduping recursive requests across machines (if something a few miliseconds away has a live result, why do a full lookup taking hundreds or thousands of milliseconds?). This potentially improves RAM usage across the datacenter too from not having a given record on dozens (or more) machines' local cache. I don't know exactly how they do it, but naively I'd look at some sort of DHT shaped solution to look for records in peers within the datacenter. Or maybe some sort of tiered caching with the upper tier being sharded on domain name or the like.
* The biggest performance gains cloudflare can provide in Web and DNS cache come from a cache hit. This is on the order of 10s or 100s of ms due to having a big cache and short distance to the requesting machine. A suboptimal lookup algorithm that is a few microseconds slower in local compute and ram access is just not as important as the other concerns for dedup and cache sharing. That's not to say it's unimportant, just that it's not the top priority when you're trying to deliver this much larger performance gains from other aspects of the system. Thats why they are getting to it several years after release.
Cloudflare writes a lot about distributed systems solutions to various problems. They likely don't think as hard about single machine performance as much as whole datacenter performance when approaching problems.
Keep in mind that the per-server cost of the whole program pre-optimization seems to be about 10GB (from the graph in the post). IME that's not bad for a big busy caching service.
Using twice as much ram per cache entry makes the cache half as large, assuming your cache is bounded by ram, unless the queried, unexpired result set is less than the ram budget (which I would tend to doubt... lots of randomized queries out there; maybe I'm wrong if the cache size dropped).
When you're storing billions of records, it makes sense to spend a few minutes to consider how they're used and make a good choice about how to store them.
When you're getting a cache hit tons of times per second, it makes sense to consider every step and which ones don't need to happen every time. You have to consider every step while you're pursing correctness anyway, so might as well have the performance lens active too.
I'm not asking for heroic optimization: I didn't ask for vectorized stuff or kernel/nic offloading or kernel bypass networking... Just you have to use some data structures, you might as well not use ones that are expensive for features you don't need; and you have to store something in your cache, you may as well store something that requires less munging on the way out.
If this were a small local cache, that didn't want to use something already existing like unbound for some reason then yeah, data structures don't make a huge difference, extra marshalling doesn't make a huge difference, just don't reimplement all the CVEs that BIND had in the 90s. But if you're going to allocate 100 TB of ram, make it count. Even if you do use twice the ram but you get value from it, maybe that's fine... I've run wacky systems with bloated storage when there was a benefit. Vec doesn't give any value over a Box<[]> in this case; convenience or lazyness would be fine except that the sheer number of objects makes it worth the few minutes it takes to do something better.
This is true. I'm arguing that its unlikely this was ever bound by available RAM. Cloudflare is a DDoS protection company that absorbs attacks. They have a lot of available capacity at any moment. When you're building a service in a sitaution where you have more capacity than you'll likely need.
The savings were 100 TB across >300 data centers. The savings were on the order of 50%. So prior to this reduction the service was using something less than 2/3 of TB per datacenter. The service ram usage was about 10GB per instance according to the graph in post. IDK how cloudflare divides thier stuff between machines, but assuming they don't run less than 64 GB per server that's less than 12 servers per datacenter of ram for a flagship product, and they likely run it spread across 65 of the machines in the datacenter that are also doing other stuff. The per-instance RAM likely isn't the the concerning limit.
Overall RAM usage is proabably a bigger concern. Thats why I would think about dedup between instances and distributed caching strategy first. I could focus on redudcing the ram needed per service instance and get a 50% reduction per machine. Or I could focus on deduping 1/n (where n > 2) reduction in total memory usage across all instances. Personally if I was worried about reducing RAM I'd put more energy into growing N.
However all this is a red herring. The assumption people are making is that the cache was always read-only, and it's obvious that Box<[T]> was the best decision because in a RO cache smaller entries hold more things.
The 1.1.1.1 service advertises improved DNS performace. That's its value add. The biggest performance gain you can have from a cache is not having a cache miss, and in DNS a cache miss means a very expensive recursive lookup. So there's concerns about how to minimize those lookups. If one instance has does a lookup, it makes sense to share that result to the other instances that may need to do a lookup [1]. I don't know off the top of my head if it makes sense to get those updates and modify the existing record or just replace it in the local cache. That comes down to locking strategies and reading patterns in the specific code and service traffic patterns. Until i have hard evidence one way or another I'd like my cache to be able to do both and keep the data structs modifiable until that's nailed down. If per-isntance ram ever becomes the issue, there's easy wins there to buy me time to find better large scale solutions to the problem.
No one is disagreeing that the larger datastructures are larger. No one is disagreeing that they take more RAM, and or even if RAM was the the problem reducing it would be good.
The thing people are pointing out is that this isn't a homework problem about an optimal cache structure in a vacuum. We're pointing our that engineering real large scale solutions has a lot more to consider than a homework problem, and that the thing you're harping about likely didn't have any real budgetary or noticable performance impact on bulding that system. The reduction in ram is just a smallish improvement in operating costs after all the more expensive stuff was figured out.
Put another way 100TB of RAM is ~$350K. Thats one engineer year for a mid-level engineer.[2] Would you rather spend that money to save an equivalent amount of money somewhere, or... would you spend that money putting the engineer on something that saved $700K elsewhere (alternately that generated $700K)?
[1] I talked a lot about dedup and the simple gotcha is "hahah then its not deduped so you need smaller objects". But on a service that is running on a few dozen instances having a few redundant copies to deal with loss of a machine and/or load can still result in 1/(n>2) savings in total ram.
[2] I'm not saying someone worked on this for a year btw, a couple people likely spent a couple months on the code, validation and testing of it. A manager spent time overseeing it. Operations people spent time understaning any effects it had on running systems. Costs add up and it wouldn't suprise me if this didn't end up being roughly break-even for the year.
Another interesting thing that happens is you don't necessarily know what form your actual optimizations will need to take. Later when your systems grow you discover the suboptimal parts you hadn't optimized for.
Very early on at Cloudflare I worked on part of the DNS infrastructure that took DNS records from the UI and got them in a state for actual authoritative serving. The system had been constructed anticipating Cloudflare having millions of customers with unique domains, but it had not been constructed for a single customer with a single domain with millions of records. This caused a periodic slow down in DNS record updating while the system churned on that one customer.
In a different job I worked on a piece of optimization software that needed to keep track of "node" A is reachable from node "B". This had been implemented as a matrix (literally a malloced NxN matrix of ints storing 0 or 1) which worked really well for small systems. But you'd be out of memory really fast on a large project. I replaced the matrix with a hash table and all was good because the matrix was actually really sparse.
With a rather short prompt, claude/codex will take your code, write a harness, profile it, build experiments, profile those, and give some pretty solid advice which one to pick. Then integrate the changes. It's the kind of goal-directed, bite-sized job that LLMs excel at. Extremely low-commitment.
Except for the whole "making changes in production at scale" problem, of course.
The “evil” of premature optimization is that it’s a misapplication of priority. If I have an acute medical problem that needs attention, it’s not the right time to talk about chloresterol and statins, get my broken leg set.
There’s always a tension between engineering management who needs to deliver a solution to the business and engineers who want to deliver a beautiful object.
Because anyone willing to come in just to design your cache format is going to expect payment that is many multiples more than the engineers you already cannot afford? Long-term employees cost less, which brings them closer to being affordable, but you have to be able to keep them busy for long periods of time to realize that reduction in cost. A engineer who doesn't understand your codebase isn't going to be useful for very long.