When I was struggling to figure out why so many of my devices on my gigabit network were only getting 100 megabits, I learned that a lot more of my devices were on 10/100 ethernet than I expected. Including some recent purchases (TV's especially).
It took me a while because I assumed gigabit had been kind for far longer than it actually was in practice.
Aurornis
Gigabit is more expensive, consumes more power, and generates more heat. If a device doesn't need the speed, 100M is a better choice.
Even the most high bitrate streaming services have peak bitrates under 80Mbps, with averages being much lower than that.
justsomehnguy
Didn't know we are in 2003 now. Should go experience Splinter Cell and Beyond Good And Evil the first time again.
toast0
TV stuff doesn't tend to need more than 100Mbps, unless you're playing blu-ray 4k discs which have higher peak rates. Saving a buck or two on the SoC adds up for the manufacturer.
retired
Exactly. You spend $3000 on a TV and now you have a crap experience playing BluRay because the manufacturer saved 12 cents on a Gigabit chip. Good way to lose customers.
adastra22
It is so frustrating that this can bring down the rest of the network to the lowest common denominator. I know it is the fault of a cheap switch, not the TV, but still.
toast0
> It is so frustrating that this can bring down the rest of the network to the lowest common denominator.
It shouldn't? I have 10M to 10G in the same ethernet collision domain everything works at appropriate speeds (my 10G hosts don't always hit 10G cause some of them have anemic cpus, but that's a different story). Most of my switches are cheap gigE switches, not the 10G stuff obviously, so cheap isn't the problem.
Only thing is if some equipment thought ethernet pause frames were a good idea... Send too much traffic to a low speed switch port and get a pause frame will tend to stop all outbound traffic which is not helpful. Not a lot of things send or respect pause frames though...
blahlabs
Pause frames are wild. I have a cheap Legion Go dock and if I leave it with power/Ethernet connected and no legion in the dock, it brings down my whole network after about 30min or so.
KetoManx64
The only reason this would be the case is if you are using network hubs or some ancient early 200's network equipmemt.
adastra22
Buffer exhaustion can cause it to happen pretty quickly with some normal, non-contrived usage patterns. A 10G host sending large data to a 100M peer will cause all packets between all other peers to drop once switch memory is filled up. Ideally the switch is smarter about this, but cheap switches in the real world…
windexh8er
This is generally not related to the switch at all. It's upstream egress to the Internet and the most common cause is buffer bloat. Traffic on a 10Gb link between local hosts won't be impacted by a 100Mb connection going out to the Internet and also a 10Gb connection sending to a 100Mb connection won't either. TCP congestion control matches the speed of the sender and during an initial connection burst the switch will likely buffer for a short period of time (which is exactly why the buffer exists) and drains at line rate. This is normal and all during this TCP is signaling congestion / back off. However, if your egress router doesn't handle buffer bloat you can end up with poor connections leaving the network, but generally this isn't because of local Ethernet speed mismatch or related to the switch at all.
And even in cheap switches (think Realtek / Broadcom chips in $30 switches) they have dynamic shared pools. An elephant flow can exhaust some of that pool, but even the cheap stuff won't allow a single port to exhaust all resources as claimed.
10Gb to 100Gb UDP? Sure... Maybe in this scenario you end up with some odd behavior, but this isn't going to be normal usage. We've got PAUSE frames, congestion control and hardware that's been doing this for decades. Even on the cheap switching. It is imperative to have a router, however that actually deals with buffer bloat appropriately and as your Internet pipe scales you want to be cognizant your router can handle queuing appropriately at those line rates.
But ultimately...
> A 10G host sending large data to a 100M peer will cause all packets between all other peers to drop once switch memory is filled up.
...is not true. Switches don't work like that.
iso1631
You always get that no matter
Imagine 5 devices that all push at 100mbit in 100 milliseconds (1gig linerate) to the target 1gig connected device, then nothing for 900ms.
On paper they are feeding 100mbit a second each to a target receiving 500mbit a second. Easily fits in a 1gig link.
But because of the timing you've got to transmit 500mbit in 100ms, or a peak speed of 5gbit.
So instead you have to buffer and delay the packets. Your near-zero latency increases to 400ms, and your switch needs about 400mbit (50MB) of buffer, or over 30,000 packets of buffer space.
You'd need a very high end switch to avoid any drops in that situation.
adastra22
I’m not an expert on switching algorithms, but there are various algos they can use for dropping or rejecting packets, or QoS prioritization that are supposed to self limit even a poorly implemented network protocol and provide fair service and reasonable latency even in such a situation.
Again, a cheapo non reputable switch probably runs with whatever settings got it working the first time with no follow up testing. Networking algorithms are an area where the simple dumb thing mostly works except in edge cases and adversarial situations, and the right/proper way to do things is fiendishly complex.
justsomehnguy
If you claim
> I’m not an expert on switching algorithms
then you surely shouldn't tell about 'various algos'
And no, no amount of software magik would help if your L1 can't handle the traffic.
rcxdude
And hubs like that wouldn't work with 1G anyway. (Apparently theoretically some versions of gigabit ethernet could work with hubs but it's not clear any were ever made)
tshaddox
Even UHD Blu-Ray isn't likely to need sustained transfer rates over 100 Mbps. The spec allows transfer rates of about 128 Mbps, and some discs reportedly have momentary spikes above that, but I suspect the vast majority of titles would be playable over Fast Ethernet, especially if given a few seconds of buffer.
mvanbaak
> especially if given a few seconds of buffer.
This is the biggest problem with those cheap network interfaces in tvs. They dont have any form of usable buffer, so a spike instantly creates problems.
using an usb network dongle (or if your setup is good enough wifi, but this is very unstable)
Also, most tv's are not able to push the full 100Mbps over their network interface.
tshaddox
Will TVs stream UHD Blu-Ray rips off the network anyway? I guess modern TVs all have H.265 decoders now, so there’s no reason they couldn’t as long as the network interface is quick enough and they know how to handle common video containers.
iknowstuff
Totally. It's only inconvenient for a total rounding error of a customer who's trying to stream games from cloud or sunshine/moonlight, where it can cost up to 4ms of latency for higher bitrates.
somehnguy
Yes - I think in cases where you have a good wireless setup at home it's actually preferable to not hardwire the TV. It's easy to get better performance over wifi when the port is only 100mbps.
xattt
Some devices do better throughput-wise with either wifi, or a USB Ethernet dongle that you buy based on community advice.
For USB ether, you’re still going to be limited by USB 2 speeds.
> had its receive buffers at the default 512, although it supported up to 4096. I increased them. I love an increased buffer.
Dave Täht is rolling in his grave
FatherOfCurses
Is there a similar diagnostic that can be run on a Playstation 5? I generally have good download speeds on my network but it seems to take forever to install games on my PS5.
toast0
Take a pcap on your router while you're downloading things and it should be enlightening.
If you don't have a router you can capture pcaps with, setup a box with two nics, bridged, between your ps5 and your router and take pcaps on that.
I'd expect slow downloads to be either a pMTU problem (but I over index on that), the tcp window is too small, or the download server is far away/high latency. Too small window and far away download server combine to make it worse. Sibling's suggestion of adding a squid proxy likely helps because squid is highly likely to have large tcp windows. Other sibling's suggestion of disable ipv6 is likely to help if Sony didn't bother to do happy eyeballs and their ipv6 infrastructure chooses far away mirrors or is otherwise partially broken.
Could also be using a download server with high path loss, that would show up in the pcap as lots of out of order packets.
robotmay
If it's anything like the PS4, you might be able to fix this by running a HTTP proxy using Squid and configuring that on the PlayStation. There's something really funky about Sony's networking on them, and configuring a squid proxy for my PS4 vastly sped up my downloads.
I tried to find an article covering this but I've had no luck, so I'm just going off memory here, but I'm pretty certain that's all that was required.
pfyra
I had problems where the download ran at max speed for about 3 seconds and then practically idle for a minute, over and over. Took forever to download games. The solution was to disable the newfangled ipv6 in my router; now I get full speed at the ps5.
iso1631
ip6 tends to cause far more problems than it solves. But mention that in any networking circles and it's a brave network engineer that agrees.
It was badly designed in the first place, CLAT should have been mandated as part of the spec rather than dual stack nonsense.
icedchai
Many “IT” people disable ipv6 because they still don’t understand it after 2 decades. It works fine.
deberon
My guess would be that this has more to do with disk speed if your download speeds are decent. Especially if it’s wired to the internet rather than WiFi. It’s always easy to spot the console user in cross-platform lobbies because of the loading times.
tredre3
The write speed of the PS5 SSD is indeed quite bad but it's still way faster than gigabit ethernet, it can't be the bottleneck. It also can't be the bottleneck with load times, it still reads at 5000MB/s even though it writes at 1/10 that on a good day.
OptionOfT
Playstation's networking capabilities are an absolute mess.
In anticipation for GTA6 I actually bought a PS5 Pro 2 weeks ago. I had to create a separate IPv4-only network because it would not complete the user-sign-up on the device on my IPv4/IPv6 network.
Absolutely insane.
ValdikSS
Another example: PCIe ASPM exit latency. Fast LAN speed (because the card does not have time to go into sleep) but low internet speed (manages to enter ASPM if the host is at least 3ms away)
Awesome writeup, thanks. I wasn't aware of iperf3 as a tool. This helped me confirm that the 10GbE lines I ran a while ago are in good shape.
I didn't need to change any NIC settings, but I'm on an X540 instead of an E610-XT2
anonymousiam
I upgraded my entire home network to 10GBase-T earlier this year, and I get 10Gbps speeds everywhere, but I run Linux everywhere, except for a few Windows VMs, which also get 10Gbps. I think the TLDR for this article comes down to "Windows networking sucks", which everybody should already know anyway.
toast0
Windows networking doesn't suck that much. Microsoft does do some good things with networking (they originated receive side scaling, which is really useful for high volume servers), this is just bad defaults and poor feedback.
Other OSes also suffer from bad defaults and poor feedback, so whatcha gonna do.
iso1631
10-gbase-t sucks, massive power draw, massive heat generation. Use fibre.
You'll still run into issues with USB connections and ensuring you have a high speed through them as not all usb connectors and dongles will run reliably at 10g bidirectional.
anonymousiam
I agree 100%, but my relatively new house already had Cat6 everywhere, and I did not think it was worthwhile to start tearing up walls. Also, lots of the connected goodies use PoE++ which you cannot do over fiber.
(I'm aware of some instances where the light in the fiber has been used to power local electronics, but AFAIK there's no standard for this, and it's probably not an economical solution for me.)
TacticalCoder
> I think the TLDR for this article comes down to "Windows networking sucks", which everybody should already know anyway.
You can even safely remove the word "networking" from that sentence.
alexnewman
Recently I found my bilateral
Bandwidth was limited from the switch to host. Turns out I didn’t plug the sfp+ all the way in. Amazing it worked at all
chasd00
> Turns out I didn’t plug the sfp+ all the way in.
I don't know why I know this but back in 2011 something not being plugged in all the way was the root reason why neutrinos were detected traveling faster than the speed of light.
"In March 2012 an LNGS seminar was held, confirming the fiber cable was not fully screwed in during data gathering"
anyone old enough from dialup modems and RS232 to remember the trick of replacing the 16450 UART with 16550 if you were lucky enough to have it on a socket instead of soldered?
that took it from a 1-byte buffer to 16-bytes, instant speed boost
same idea going from X-modem to Y-modem for transfers, 128bytes vs 1024bytes before checksum
surprised ethernet by now doesn't constantly try to negotiate larger buffer windows
toast0
A lot of these drivers were written in a different era and the defaults are less appropriate now.
Having too few receive buffers leaves a trail in diagnostics (if you know to look for it!), having too many gives more subtle problems. But with 10G ethernet, you can get a lot of packets really quickly and the buffer default was probably set for a card doing 100M or 1G and the driver was updated to handle 10G without new defaults. Interrupt modulation is also common now, which is great for throughput, but means packets buffers will get more full.
I feel like segmentation offload is a nice idea, but weirdness always lurks in the shadows and I don't hesitate to turn it off if any weirdness appears. I haven't noticed a perf difference when I turn it off, but I also haven't run great tests.
Scoundreller
Took wayyyy too many hours of tech support to find someone that clued in that I had an 8250 uart. A surprisingly crappy piece of hardware on an 486. and that’s why my $$$ 33.6k performed like garbage until I bought a 16550 card.
I also only learned like 5 years ago that crossover cables are unnecessary nowadays. The 10mbit one I made with a bad soldering iron worked surprisingly well.
It took me a while because I assumed gigabit had been kind for far longer than it actually was in practice.
Even the most high bitrate streaming services have peak bitrates under 80Mbps, with averages being much lower than that.
It shouldn't? I have 10M to 10G in the same ethernet collision domain everything works at appropriate speeds (my 10G hosts don't always hit 10G cause some of them have anemic cpus, but that's a different story). Most of my switches are cheap gigE switches, not the 10G stuff obviously, so cheap isn't the problem.
Only thing is if some equipment thought ethernet pause frames were a good idea... Send too much traffic to a low speed switch port and get a pause frame will tend to stop all outbound traffic which is not helpful. Not a lot of things send or respect pause frames though...
And even in cheap switches (think Realtek / Broadcom chips in $30 switches) they have dynamic shared pools. An elephant flow can exhaust some of that pool, but even the cheap stuff won't allow a single port to exhaust all resources as claimed.
10Gb to 100Gb UDP? Sure... Maybe in this scenario you end up with some odd behavior, but this isn't going to be normal usage. We've got PAUSE frames, congestion control and hardware that's been doing this for decades. Even on the cheap switching. It is imperative to have a router, however that actually deals with buffer bloat appropriately and as your Internet pipe scales you want to be cognizant your router can handle queuing appropriately at those line rates.
But ultimately...
> A 10G host sending large data to a 100M peer will cause all packets between all other peers to drop once switch memory is filled up.
...is not true. Switches don't work like that.
Imagine 5 devices that all push at 100mbit in 100 milliseconds (1gig linerate) to the target 1gig connected device, then nothing for 900ms.
On paper they are feeding 100mbit a second each to a target receiving 500mbit a second. Easily fits in a 1gig link.
But because of the timing you've got to transmit 500mbit in 100ms, or a peak speed of 5gbit.
So instead you have to buffer and delay the packets. Your near-zero latency increases to 400ms, and your switch needs about 400mbit (50MB) of buffer, or over 30,000 packets of buffer space.
You'd need a very high end switch to avoid any drops in that situation.
Again, a cheapo non reputable switch probably runs with whatever settings got it working the first time with no follow up testing. Networking algorithms are an area where the simple dumb thing mostly works except in edge cases and adversarial situations, and the right/proper way to do things is fiendishly complex.
> I’m not an expert on switching algorithms
then you surely shouldn't tell about 'various algos'
And no, no amount of software magik would help if your L1 can't handle the traffic.
This is the biggest problem with those cheap network interfaces in tvs. They dont have any form of usable buffer, so a spike instantly creates problems.
using an usb network dongle (or if your setup is good enough wifi, but this is very unstable)
Also, most tv's are not able to push the full 100Mbps over their network interface.
For USB ether, you’re still going to be limited by USB 2 speeds.
Dave Täht is rolling in his grave
If you don't have a router you can capture pcaps with, setup a box with two nics, bridged, between your ps5 and your router and take pcaps on that.
I'd expect slow downloads to be either a pMTU problem (but I over index on that), the tcp window is too small, or the download server is far away/high latency. Too small window and far away download server combine to make it worse. Sibling's suggestion of adding a squid proxy likely helps because squid is highly likely to have large tcp windows. Other sibling's suggestion of disable ipv6 is likely to help if Sony didn't bother to do happy eyeballs and their ipv6 infrastructure chooses far away mirrors or is otherwise partially broken.
Could also be using a download server with high path loss, that would show up in the pcap as lots of out of order packets.
I tried to find an article covering this but I've had no luck, so I'm just going off memory here, but I'm pretty certain that's all that was required.
It was badly designed in the first place, CLAT should have been mandated as part of the spec rather than dual stack nonsense.
In anticipation for GTA6 I actually bought a PS5 Pro 2 weeks ago. I had to create a separate IPv4-only network because it would not complete the user-sign-up on the device on my IPv4/IPv6 network.
Absolutely insane.
https://lore.kernel.org/intel-wired-lan/803760bf-04ff-4b23-8...
I didn't need to change any NIC settings, but I'm on an X540 instead of an E610-XT2
Other OSes also suffer from bad defaults and poor feedback, so whatcha gonna do.
You'll still run into issues with USB connections and ensuring you have a high speed through them as not all usb connectors and dongles will run reliably at 10g bidirectional.
(I'm aware of some instances where the light in the fiber has been used to power local electronics, but AFAIK there's no standard for this, and it's probably not an economical solution for me.)
You can even safely remove the word "networking" from that sentence.
I don't know why I know this but back in 2011 something not being plugged in all the way was the root reason why neutrinos were detected traveling faster than the speed of light.
"In March 2012 an LNGS seminar was held, confirming the fiber cable was not fully screwed in during data gathering"
https://en.wikipedia.org/wiki/2011_OPERA_faster-than-light_n...
anyone old enough from dialup modems and RS232 to remember the trick of replacing the 16450 UART with 16550 if you were lucky enough to have it on a socket instead of soldered?
that took it from a 1-byte buffer to 16-bytes, instant speed boost
same idea going from X-modem to Y-modem for transfers, 128bytes vs 1024bytes before checksum
surprised ethernet by now doesn't constantly try to negotiate larger buffer windows
Having too few receive buffers leaves a trail in diagnostics (if you know to look for it!), having too many gives more subtle problems. But with 10G ethernet, you can get a lot of packets really quickly and the buffer default was probably set for a card doing 100M or 1G and the driver was updated to handle 10G without new defaults. Interrupt modulation is also common now, which is great for throughput, but means packets buffers will get more full.
I feel like segmentation offload is a nice idea, but weirdness always lurks in the shadows and I don't hesitate to turn it off if any weirdness appears. I haven't noticed a perf difference when I turn it off, but I also haven't run great tests.
I also only learned like 5 years ago that crossover cables are unnecessary nowadays. The 10mbit one I made with a bad soldering iron worked surprisingly well.