Very interesting read, these bugs that only happen after a lot of time in production are the worse to troubleshoot
nasretdinov
Go was born into 64-bit era, I remember seeing many more bugs that only occur on 32-bit systems. I'm pretty certain the observation in the article is correct that no one at Google tests Go on anything apart from 64-bit ARM/x86 Linux
typical182
FWIW, every commit for the Go project is tested on multiple 32-bit architectures, including pre-commit tests and post-commit tests.
(That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)
I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.
orphereus
Very cool site. Makes me want to do some system programming project just to make a site with all the tests like this.
ReactiveJelly
So a stronger type system in the boundary between Go and Linux would have caught this?
fn-mote
My reading is that this is entirely Go’s fault. In Go, there was a union of two different types without metadata to distinguish the two. The disambiguation was done by looking to see if the data looked like a certain pointer.
I think the reason it is a problem only in 32 bit systems is that the assignment setting the lower half of the data is a 32 bit op, which can overflow the 16 bits used, but won’t affect the other half of a 64 bit value. (Corrections welcome.)
inigyou
It's 64 bits on both but only 32 bits were read on 32-bit systems. Two 64-bit values that were different were considered the same because only 32 bits were checked.
BlackRabbit1
Not directly related to your embedded Go bug, but maybe helpful:
What really sky-rocketed my debugging skills in embedded devices was learning to utilize the Python plugin interface of 'gdb'.
budman1
Isn't this something that software developers do everyday?
apitman
The way I look at these types of difficult debugging problems has completely changed.
If it's a system I want to understand more deeply and I have the time, I try to force myself to do it by hand. If not, the agent does it.
The trap is that it's easy to convince myself that I never have the time.
budman1
Not being a user of agents, my question is 'does the agent always find the root cause and remove the defect'?
The way I got good at finding the hard ones was finding hard ones. Without that practice, I would be helpless.
k4rli
Reason why it's super hard for juniors at the moment. For "agents" to produce good results, the operator must have the experience to direct it in the right direction and properly validate the output.
I've had some weird bug investigations where it has suggested wrong fixes confidently multiple times before arriving at the actual fix. Only with extra details from manual testing did we arrive at the real bug. The ~5 first suggested fixes would've hidden the root cause even more.
DarmokTanagra
I wonder what other 32 bit bugs are waiting to be discovered.
You can see a list here:
https://build.golang.org/
(That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)
I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.
I think the reason it is a problem only in 32 bit systems is that the assignment setting the lower half of the data is a 32 bit op, which can overflow the 16 bits used, but won’t affect the other half of a 64 bit value. (Corrections welcome.)
What really sky-rocketed my debugging skills in embedded devices was learning to utilize the Python plugin interface of 'gdb'.
If it's a system I want to understand more deeply and I have the time, I try to force myself to do it by hand. If not, the agent does it.
The trap is that it's easy to convince myself that I never have the time.
The way I got good at finding the hard ones was finding hard ones. Without that practice, I would be helpless.
I've had some weird bug investigations where it has suggested wrong fixes confidently multiple times before arriving at the actual fix. Only with extra details from manual testing did we arrive at the real bug. The ~5 first suggested fixes would've hidden the root cause even more.