Hi,
I will soon get an 100Mbps Internet connection so I have tried the throughput on some different firmwares on my WNR3500L to make sure that I don't get a bottleneck in the router and the results were a bit surprising. (WAN to LAN results):
Original Netgear firmware V1.0.2.50_31.1.25: 240Mbps.
Tomato K26USB-1.27.9047MIPSR2-beta16: 225Mbps.
DD-WRT v24-14826-NEWD-2_K2.6_big: 80Mbps.
OpenWRT Release 21May10 Kamikaze 8.09.2: 40Mbps.
Tomato is OK but why are DD-WRT and OpenWRT so slow? Is it possible to do some tuning to boost the throughput? I could clearly see that the CPU-load was the problem on OpenWRT.
The test were not very scientific, just copying a 200MB file using FTP between ram-disks on two gigabit connected hosts. The measured speed for the same copy between the hosts without the router was about 850Mbps.
The 200MB file was transfered from WAN to LAN (like if I would download a large file from the Internet) and the the results are the transfer-speed reported by the FTP client (multiplied by 8 sine it reports bytes per second).
I have now tried a few different DD-WRT firmwares with the same disappointing result :(
Sending files LAN->WAN is actually a bit faster (120 Mbps) so that would be sufficient for my 100 Mbps line but there is no way to push the WAN->LAN speed above 80 Mbps :(
And to be able to handle 100 Mbps in full duplex I guess it has to be able to transfer in 200 Mbps in one direction (since the CPU is the bottleneck).
In other words if I want to fully utilize my 100 Mbps full duplex line my only options are Netgear or Tomato firmware (or possibly buy a WNDR3700 router with faster CPU).
The sad thing is that my first pick in firmware probably would have been OpenWRT (I like the modular design) but that is by far the slowest firmware available (and it does not look as "WNR3500L ready" as the other ones :(
Hi bosa, did you deactivate Logging?
Security-Firewall-LogManagement-Log disable
Did you also test with SPI Firewall disabled?
I have not yet checked for performance if it comes to wan-lan speeds but if tomato does fine than it is probably only a minor config difference.
Openwrt by default uses b43 instead of broadcom binary wl, did you use a standard openwrt build or did you compile it your self using binary broadcom driver
Yes I run without logging and I also tried to deactivate SPI Firewall but it did not help (DD-WRT).
I used the binary OpenWRT firmware listed in the beginning of this post.
Today I also tried the latest tomato-K26USB-1.28.9048MIPSR2-beta18-Ext and to my big disappointment the download speed was cut in half (using the same config) not as bad as DD-WRT but close...
So something was changed between 1.27 and 1.28...
Hi bosa,
ok, then we know where this speed comes from:-)
Tomato introduced broadcoms fast nat in 1.27 but removed it again in the latest builds since it caused a lot of problems. I guess Netgear official firmware uses the fast_nat module and due to a different featureset doesn't see the bugs that were seen in tomato. DD-WRT does not use the fast nat module
The fast nat module also speeds up usb read/write speeds, but since it is not stable enough currently it doesn't make sense to include it.
Here is some info on the topic and why this causes problems:
http://security.maruhn.com/iptables-tutorial/c3163.html
Actually it would be interesting to see how much performance increases if one disables connection traffic using NOTRACK target in iptables rules
CAN you redo your test with some iptables rules:
iptables -t raw -A OUTPUT -p TCP -s 0/0 --dport 80 -j NOTRACK
iptables -t raw -A PREROUTING -p TCP -s 0/0 --dport 80 -j NOTRACK
iptables -t raw -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -t raw -I PREROUTING 1 -m state --state UNTRACKED -j ACCEPT
iptables -I INPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
In the above case you would use port 80 to do your transfers, just change it for your needs you could also set destination, source ips etc., see iptables man page
Hi Kong, thank you for explaining this!
Unfortunately the iptables rules did not work on DD-WRT. After entering the commands all TCP traffic stoped working completely through the router (for all ports not just the specified ones).
I tried it a few times even using cut&paste for port 80 but with the same result.
Hi bosa NOTRACK definitely works, at least on my build. I tested the rule yesterday with port 445 and symba but only with traffic in my LAN.
Make sure you have no other special rules set that may interfere with those rules.
One can check if the NOTRACK rule is in place with:
iptables -t raw -L
Once the rule is working and traffic passing through you will notice that you don't see conntrack entries for these connections in:
cat /proc/net/ip_conntrack
You can also define rules by looking at ip_conntrack table
Ok, after some more checking I realize that I was wrong about the "all TCP traffic" part, it's just the specified protocol that doesn't work.
After entering the iptables rules it looks like this:
And I can also see the "state UNTRACKED" rule on top of both the INPUT and OUTPUT chain but still no HTTP request goes through.
I am using this firmware (downloaded from dd-wrt.com):
DD-WRT v24-sp2 (04/23/10) mega (SVN revision 14311)
I guess this is because you do wan-lan transfers I did lan-lan transfers, so you probably have to tell ip tables to also forward "UNTRACKED" packets. I'll test this tonight.
Kong, I have uploaded your new firmware to my 3500L and everything works great, including what I was looking for, NAS/media server.
What I did notice and also noticed with the 'normal' dd-wrt firmware that my N speed has dropped to half speed.
I've configured it as N-only, mixed etc, also played with the advanced settings (afterburner on/off) but nothing. I have a 80Mbit internet connection and only get 30/40, whereas Tomato I get 79/80Mbit.
The problem with the Tomato is that I only get 11M upstairs (I can live with that as I don't use that as much as in the living room). I'm looking for the perfect setup (whether Tomato or dd-wrt) where it has a media server and full speed in the living room and upstairs. I know it's not the channel or the advanced settings as I have set them the same as in Tomato, but somehow only half speed in dd-wrt.
Just to make this clear you only get 40Mbps via WLAN or LAN? 40Mbps would be what I expect if you use N with single stream. To get 80Mbps you need dual stream, for that you have to configure it to run in N-only , Channel 6, Channel Width 40Mhz
I also recommend to lower the TX Power as the dd-wrt default of 71 is to high, the maximum that can be set on the wnr is 84 not as the gui claims.
I have it on N only, channel 6 +4 (width 40MHz), lowered the output to 42mW (as it is with Tomato) but it's still around 40, sometimes 30 and sometimes 50. As soon as I upload the stock Netgear or Tomato and configure it with 40MHz it's full speed. I had that with all the dd-wrt firmwares (started half a year ago or so).
Hmmm, if you use my build and enable ftp with a connected drive how fast can you read or write from the drive?
What download speeds do you get using a cable connection to the router?
I would also make sure, that you don't run into TCP Retransmissions, due different packet size settings in dd-wrt/tomato, you can verify this using a network sniffer like ethereal, if you use windows, there is a portable version which doesn't have to be installed. Once you capture and you see lots of Black/Red Lines with TCP Retransmissions you have a setup problem.
Wireless FTP speed begins at 3.4MB/s and drops steady to 1.8MB/s (using filezilla).
When using a cable the download speed is 79.9Mbit, thats the normal speed which I also get with Netgear/Tomato firmware.
I've changed the Basic Rate in Advanced Settings to All (instead of default) and now it gets to 60Mbit. It's fast I now but still misses 20Mbit. :-)
Okay, then try following options:
Frameburst ->Enable
Preamble ->Short
WMM Support -> Disable
Now I'm curious.
Ah and by the way you should have speeds of about 6MB/s to a connected drive I guess you are using either an ntfs formatted drive or a slow usb stick
For now I've gone back to Tomato.
I've tried Frameburst enabled and even wmm support off but it didn't matter.
I'm okay with a little slower usb speed as the streamed movies to the ps3 works wonderfully. It's an ext3 drive btw. (I'm all linux).
btw: it seems you can now go to whatever firmware you like, whether from tomato to dd-wrt or back. I remember not long ago you had to go to stock firmwar before you could go to tomato but now dd-wrt to tomato works in one go. Great (saves time).
With ext3 you definitely should have faster speeds. I did transfer tests a few weeks ago and I got 6MB/s constant read speed (also filezilla) connected at 244Mpbs with an Intel 5100 AGN. Therefore it looks like there is some problem with your setup.
Since the FTP test only measures the speed of the actual data-bytes being transfered and not counting overhead such as IP and TCP headers as well as FTP protocol overhead it's as I pointed out earlier not that accurate.
I have now run some "real" tests using a benchmarking software (TPtest) which gives more correct measurements.
I guess the conclusion of these tests are: If transfer speed is your main concern stick with the Netgear firmware or possibly Tomato 1.27 (if you can live with the fast_nat bugs).
The free firmwares can almost fill a 100 Mbps full duplex internet connection.
In the test results below, Half Duplex means that the transfer goes in one direction at the time and Full Duplex means that it pushes data in both directions simultaneously.
The tptest server runs on the WAN side and the client on the LAN side so receive means WAN->LAN and send LAN->WAN. (No WLAN in these tests.)
DD-WRT v24-sp2 build 14311 (SPI Disabled) TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes Half Duplex Receive = Throughput: 103409114 bps (103.41 Mbit/s) Send = Throughput: 125354791 bps (125.35 Mbit/s) Full Duplex Receive = Throughput: 73741418 bps (73.74 Mbit/s) Send = Throughput: 51659250 bps (51.66 Mbit/s) DD-WRT v24-sp2 build 14311 (SPI Enabled) TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes Half Duplex Receive = Throughput: 81991541 bps (81.99 Mbit/s) Send = Throughput: 124700980 bps (124.70 Mbit/s) Full Duplex Receive = Throughput: 69220199 bps (69.22 Mbit/s) Send = Throughput: 51817808 bps (51.82 Mbit/s) Tomato USB v1.27 build 9047 TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes Half Duplex Receive = Throughput: 250627822 bps (250.63 Mbit/s) Send = Throughput: 274621170 bps (274.62 Mbit/s) Full Duplex Receive = Throughput: 242672209 bps (242.67 Mbit/s) Send = Throughput: 130494608 bps (130.49 Mbit/s) Tomato USB v1.28 build 9048 TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes Half Duplex Receive = Throughput: 102954643 bps (102.95 Mbit/s) Send = Throughput: 123144660 bps (123.14 Mbit/s) Full Duplex Receive = Throughput: 102915519 bps (102.92 Mbit/s) Send = Throughput: 54901252 bps (54.90 Mbit/s) Netgear V1.0.2.50_31.1.25 TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes Half Duplex Receive = Throughput: 262655738 bps (262.66 Mbit/s) Send = Throughput: 276394652 bps (276.39 Mbit/s) Full Duplex Receive = Throughput: 256632671 bps (256.63 Mbit/s) Send = Throughput: 133234828 bps (133.23 Mbit/s) OpenWRT Kamikaze 8.09.2 build 4838 TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes Half Duplex Receive = Throughput: 66640632 bps (66.64 Mbit/s) Send = Throughput: 160550559 bps (160.55 Mbit/s) Full Duplex Send = Throughput: 104659041 bps (104.66 Mbit/s) Receive = Throughput: 57134564 bps (57.13 Mbit/s) Reference test Local LAN (without WNR) TCP Speed test, host=bilbo, transfer size=50000000 bytes Half Duplex Receive = Throughput: 903477031 bps (903.48 Mbit/s) Send = Throughput: 935631082 bps (935.63 Mbit/s) Full Duplex Send = Throughput: 595713953 bps (595.71 Mbit/s) Receive = Throughput: 750707072 bps (750.71 Mbit/s)Checking the duplex on the nics? How is that going to help the speed with the wireless, whereas it is full speed with netgear and tomato firmware but not the dd-wrt. It surely must be a configuration in dd-wrt.
It must be something very simple but I can't find it. I've double checked all the configuration items in all firmwares and they are all the same, but no full speed in dd-wrt.
(wow, those are quite some differences in speed in the above reports, thanks for posting them).
Hi bosa,
great Benchmark comparison!!!
I checked the netgear sources yesterday and it does indeed use the fast nat module. One thing that would be too cool. Could you do another test with my build here:
http://www.myopenrouter.com/download/file/19806-133
This build is optimized by the compiler for speed, not for size like official dd-wrt builds.
This build shows faster transferrates for samba ftp etc. so it should also provide better throughput.
Hi Kong,
I have tried your new build now and the optimization did indeed make a noticeable difference! Specially when SPI is enabled.
Great!!!
a few weeks ago I checked what could be optimized in the tcp/iptable settings, but I just tested it with samba and ftp access, not throughput. These things could speed it up a bit more:
These two lines avoid to process already established and related connections again
iptables -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
Ans some tcp stack opts:
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 0 > /proc/sys/net/ipv4/tcp_sack
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
The iptables rules above should definitely speed up things with SPI enabled.
Yes Kong, so it did!
Almost as good results as with SPI disabled now!
Yeah !!! The rule problem is already a known problem which will eventually be fixed in all builds including official.
Now to sum this up for folks that only read parts of the thread:
compared to the current official build the compile optimization along with rule optimization boosts your download speed from 82Mbps to 130Mbps
According to some info on the net the broadcom 4718(4716A) is a mips32r2 chip, so it could be compiled with -mtune=mips32r2 instead of now mips32 this along with a few more gcc opts could improve the performance even more. I'm going to see if this works.
Yes this is really good news since >= 100 Mbps was what I was looking for!
I guess I can live without 100 Mbps in both directions at the same time, maybe that will be possible some day when someone clever has solved the bugs in the fast_nat module...
Should be interesting to see what those extra optimizations can do.
Thank you for all your help with this Kong!
So what you basicaly saying is I need to take Tomato Firmware with USB (Compatible with NETGEAR WNR3500L)
BETA 16 build of Tomato 1.27 with USB, compatible with NETGEAR WNR3500L, updated 06/14/2010 if Im looking for 100/100Mbit
Yes, either that or Netgear firmware.
Excellent work! Very informative. Thanks to both of you (bosa & kong).
My next build will have a fix for the performance issue. Should be out within the next 7 days
Pages