WNR3500L WAN to LAN throughput using different firmwares

61 posts / 0 new
Last post
bosa
bosa's picture
WNR3500L WAN to LAN throughput using different firmwares

Hi,

I will soon get an 100Mbps Internet connection so I have tried the throughput on some different firmwares on my WNR3500L to make sure that I don't get a bottleneck in the router and the results were a bit surprising. (WAN to LAN results):

Original Netgear firmware V1.0.2.50_31.1.25: 240Mbps.

Tomato K26USB-1.27.9047MIPSR2-beta16: 225Mbps.

DD-WRT v24-14826-NEWD-2_K2.6_big: 80Mbps.

OpenWRT Release 21May10 Kamikaze 8.09.2: 40Mbps.

Tomato is OK but why are DD-WRT and OpenWRT so slow? Is it possible to do some tuning to boost the throughput? I could clearly see that the CPU-load was the problem on OpenWRT.

The test were not very scientific, just copying a 200MB file using FTP between ram-disks on two gigabit connected hosts. The measured speed for the same copy between the hosts without the router was about 850Mbps.

bosa
bosa's picture
The 200MB file was transfered

The 200MB file was transfered from WAN to LAN (like if I would download a large file from the Internet) and the the results are the transfer-speed reported by the FTP client (multiplied by 8 sine it reports bytes per second).

bosa
bosa's picture
I have now tried a few

I have now tried a few different DD-WRT firmwares with the same disappointing result :(

Sending files LAN->WAN is actually a bit faster (120 Mbps) so that would be sufficient for my 100 Mbps line but there is no way to push the WAN->LAN speed above 80 Mbps :(

And to be able to handle 100 Mbps in full duplex I guess it has to be able to transfer in 200 Mbps in one direction (since the CPU is the bottleneck).

In other words if I want to fully utilize my 100 Mbps full duplex line my only options are Netgear or Tomato firmware (or possibly buy a WNDR3700 router with faster CPU).

The sad thing is that my first pick in firmware probably would have been OpenWRT (I like the modular design) but that is by far the slowest firmware available (and it does not look as "WNR3500L ready" as the other ones :(

Kong
Kong's picture
Hi bosa, did you deactivate

Hi bosa, did you deactivate Logging?

Security-Firewall-LogManagement-Log disable

Did you also test with SPI Firewall disabled?

I have not yet checked for performance if it comes to wan-lan speeds but if tomato does fine than it is probably only a minor config difference.

Openwrt by default uses b43 instead of broadcom binary wl, did you use a standard openwrt build or did you compile it your self using binary broadcom driver

bosa
bosa's picture
Yes I run without logging and

Yes I run without logging and I also tried to deactivate SPI Firewall but it did not help (DD-WRT).

I used the binary OpenWRT firmware listed in the beginning of this post.

Today I also tried the latest tomato-K26USB-1.28.9048MIPSR2-beta18-Ext and to my big disappointment the download speed was cut in half (using the same config) not as bad as DD-WRT but close...

So something was changed between 1.27 and 1.28...

Kong
Kong's picture
Hi bosa,

Hi bosa,

ok, then we know where this speed comes from:-)

Tomato introduced broadcoms fast nat in 1.27 but removed it again in the latest builds since it caused a lot of problems. I guess Netgear official firmware uses the fast_nat module and due to a different featureset doesn't see the bugs that were seen in tomato. DD-WRT does not use the fast nat module

The fast nat module also speeds up usb read/write speeds, but since it is not stable enough currently it doesn't make sense to include it.

Here is some info on the topic and why this causes problems:

http://security.maruhn.com/iptables-tutorial/c3163.html

Actually it would be interesting to see how much performance increases if one disables connection traffic using NOTRACK target in iptables rules

Kong
Kong's picture
CAN you redo your test with

CAN you redo your test with some iptables rules:

iptables -t raw -A OUTPUT -p TCP -s 0/0 --dport 80 -j NOTRACK
iptables -t raw -A PREROUTING -p TCP -s 0/0 --dport 80 -j NOTRACK

iptables -t raw -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -t raw -I PREROUTING 1 -m state --state UNTRACKED -j ACCEPT

iptables -I INPUT 1 -m state --state UNTRACKED -j ACCEPT
iptables -I OUTPUT 1 -m state --state UNTRACKED -j ACCEPT

In the above case you would use port 80 to do your transfers, just change it for your needs you could also set destination, source ips etc., see iptables man page

bosa
bosa's picture
Hi Kong, thank you for

Hi Kong, thank you for explaining this!

Unfortunately the iptables rules did not work on DD-WRT. After entering the commands all TCP traffic stoped working completely through the router (for all ports not just the specified ones).

I tried it a few times even using cut&paste for port 80 but with the same result.

Kong
Kong's picture
Hi bosa NOTRACK definitely

Hi bosa NOTRACK definitely works, at least on my build. I tested the rule yesterday with port 445 and symba but only with traffic in my LAN.

Make sure you have no other special rules set that may interfere with those rules.
One can check if the NOTRACK rule is in place with:

iptables -t raw -L

Once the rule is working and traffic passing through you will notice that you don't see conntrack entries for these connections in:

cat /proc/net/ip_conntrack

You can also define rules by looking at ip_conntrack table

bosa
bosa's picture
Ok, after some more checking

Ok, after some more checking I realize that I was wrong about the "all TCP traffic" part, it's just the specified protocol that doesn't work.

After entering the iptables rules it looks like this:

root@DD-WRT:~# iptables -t raw -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     0    --  anywhere             anywhere            state UNTRACKED 
NOTRACK    tcp  --  anywhere             anywhere            tcp dpt:www 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     0    --  anywhere             anywhere            state UNTRACKED 
NOTRACK    tcp  --  anywhere             anywhere            tcp dpt:www 

And I can also see the "state UNTRACKED" rule on top of both the INPUT and OUTPUT chain but still no HTTP request goes through.

I am using this firmware (downloaded from dd-wrt.com):
DD-WRT v24-sp2 (04/23/10) mega (SVN revision 14311)

Kong
Kong's picture
I guess this is because you

I guess this is because you do wan-lan transfers I did lan-lan transfers, so you probably have to tell ip tables to also forward "UNTRACKED" packets. I'll test this tonight.

MarcelE
MarcelE's picture
Kong, I have uploaded your

Kong, I have uploaded your new firmware to my 3500L and everything works great, including what I was looking for, NAS/media server.
What I did notice and also noticed with the 'normal' dd-wrt firmware that my N speed has dropped to half speed.
I've configured it as N-only, mixed etc, also played with the advanced settings (afterburner on/off) but nothing. I have a 80Mbit internet connection and only get 30/40, whereas Tomato I get 79/80Mbit.
The problem with the Tomato is that I only get 11M upstairs (I can live with that as I don't use that as much as in the living room). I'm looking for the perfect setup (whether Tomato or dd-wrt) where it has a media server and full speed in the living room and upstairs. I know it's not the channel or the advanced settings as I have set them the same as in Tomato, but somehow only half speed in dd-wrt.

Kong
Kong's picture
Just to make this clear you

Just to make this clear you only get 40Mbps via WLAN or LAN? 40Mbps would be what I expect if you use N with single stream. To get 80Mbps you need dual stream, for that you have to configure it to run in N-only , Channel 6, Channel Width 40Mhz
I also recommend to lower the TX Power as the dd-wrt default of 71 is to high, the maximum that can be set on the wnr is 84 not as the gui claims.

MarcelE
MarcelE's picture
I have it on N only, channel

I have it on N only, channel 6 +4 (width 40MHz), lowered the output to 42mW (as it is with Tomato) but it's still around 40, sometimes 30 and sometimes 50. As soon as I upload the stock Netgear or Tomato and configure it with 40MHz it's full speed. I had that with all the dd-wrt firmwares (started half a year ago or so).

Kong
Kong's picture
Hmmm, if you use my build and

Hmmm, if you use my build and enable ftp with a connected drive how fast can you read or write from the drive?

What download speeds do you get using a cable connection to the router?

I would also make sure, that you don't run into TCP Retransmissions, due different packet size settings in dd-wrt/tomato, you can verify this using a network sniffer like ethereal, if you use windows, there is a portable version which doesn't have to be installed. Once you capture and you see lots of Black/Red Lines with TCP Retransmissions you have a setup problem.

MarcelE
MarcelE's picture
Wireless FTP speed begins at

Wireless FTP speed begins at 3.4MB/s and drops steady to 1.8MB/s (using filezilla).
When using a cable the download speed is 79.9Mbit, thats the normal speed which I also get with Netgear/Tomato firmware.
I've changed the Basic Rate in Advanced Settings to All (instead of default) and now it gets to 60Mbit. It's fast I now but still misses 20Mbit. :-)

Kong
Kong's picture
Okay, then try following

Okay, then try following options:

Frameburst ->Enable
Preamble ->Short
WMM Support -> Disable

Now I'm curious.

Ah and by the way you should have speeds of about 6MB/s to a connected drive I guess you are using either an ntfs formatted drive or a slow usb stick

MarcelE
MarcelE's picture
For now I've gone back to

For now I've gone back to Tomato.
I've tried Frameburst enabled and even wmm support off but it didn't matter.
I'm okay with a little slower usb speed as the streamed movies to the ps3 works wonderfully. It's an ext3 drive btw. (I'm all linux).
btw: it seems you can now go to whatever firmware you like, whether from tomato to dd-wrt or back. I remember not long ago you had to go to stock firmwar before you could go to tomato but now dd-wrt to tomato works in one go. Great (saves time).

Kong
Kong's picture
With ext3 you definitely

With ext3 you definitely should have faster speeds. I did transfer tests a few weeks ago and I got 6MB/s constant read speed (also filezilla) connected at 244Mpbs with an Intel 5100 AGN. Therefore it looks like there is some problem with your setup.

bosa
bosa's picture
Since the FTP test only

Since the FTP test only measures the speed of the actual data-bytes being transfered and not counting overhead such as IP and TCP headers as well as FTP protocol overhead it's as I pointed out earlier not that accurate.

I have now run some "real" tests using a benchmarking software (TPtest) which gives more correct measurements.

I guess the conclusion of these tests are: If transfer speed is your main concern stick with the Netgear firmware or possibly Tomato 1.27 (if you can live with the fast_nat bugs).

The free firmwares can almost fill a 100 Mbps full duplex internet connection.

In the test results below, Half Duplex means that the transfer goes in one direction at the time and Full Duplex means that it pushes data in both directions simultaneously.

The tptest server runs on the WAN side and the client on the LAN side so receive means WAN->LAN and send LAN->WAN. (No WLAN in these tests.)

DD-WRT v24-sp2 build 14311 (SPI Disabled)

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 103409114 bps (103.41 Mbit/s)
Send    = Throughput: 125354791 bps (125.35 Mbit/s)

Full Duplex
Receive = Throughput: 73741418 bps (73.74 Mbit/s)
Send    = Throughput: 51659250 bps (51.66 Mbit/s)


DD-WRT v24-sp2 build 14311 (SPI Enabled)

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 81991541 bps (81.99 Mbit/s)
Send    = Throughput: 124700980 bps (124.70 Mbit/s)

Full Duplex
Receive = Throughput: 69220199 bps (69.22 Mbit/s)
Send    = Throughput: 51817808 bps (51.82 Mbit/s)


Tomato USB v1.27 build 9047

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes
    
Half Duplex
Receive = Throughput: 250627822 bps (250.63 Mbit/s)
Send    = Throughput: 274621170 bps (274.62 Mbit/s)

Full Duplex
Receive = Throughput: 242672209 bps (242.67 Mbit/s)
Send    = Throughput: 130494608 bps (130.49 Mbit/s)
    

Tomato USB v1.28 build 9048

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 102954643 bps (102.95 Mbit/s)
Send    = Throughput: 123144660 bps (123.14 Mbit/s)

Full Duplex
Receive = Throughput: 102915519 bps (102.92 Mbit/s)
Send    = Throughput: 54901252 bps (54.90 Mbit/s)


Netgear V1.0.2.50_31.1.25

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 262655738 bps (262.66 Mbit/s)
Send    = Throughput: 276394652 bps (276.39 Mbit/s)

Full Duplex
Receive = Throughput: 256632671 bps (256.63 Mbit/s)
Send    = Throughput: 133234828 bps (133.23 Mbit/s)


OpenWRT Kamikaze 8.09.2 build 4838

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 66640632 bps (66.64 Mbit/s)
Send    = Throughput: 160550559 bps (160.55 Mbit/s)

Full Duplex
Send    = Throughput: 104659041 bps (104.66 Mbit/s)
Receive = Throughput: 57134564 bps (57.13 Mbit/s)


Reference test Local LAN (without WNR)

TCP Speed test, host=bilbo, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 903477031 bps (903.48 Mbit/s)
Send    = Throughput: 935631082 bps (935.63 Mbit/s)

Full Duplex
Send    = Throughput: 595713953 bps (595.71 Mbit/s)
Receive = Throughput: 750707072 bps (750.71 Mbit/s)
MarcelE
MarcelE's picture
Checking the duplex on the

Checking the duplex on the nics? How is that going to help the speed with the wireless, whereas it is full speed with netgear and tomato firmware but not the dd-wrt. It surely must be a configuration in dd-wrt.
It must be something very simple but I can't find it. I've double checked all the configuration items in all firmwares and they are all the same, but no full speed in dd-wrt.
(wow, those are quite some differences in speed in the above reports, thanks for posting them).

Kong
Kong's picture
Hi bosa,

Hi bosa,

great Benchmark comparison!!!

I checked the netgear sources yesterday and it does indeed use the fast nat module. One thing that would be too cool. Could you do another test with my build here:

http://www.myopenrouter.com/download/file/19806-133

This build is optimized by the compiler for speed, not for size like official dd-wrt builds.

This build shows faster transferrates for samba ftp etc. so it should also provide better throughput.

bosa
bosa's picture
Hi Kong,

Hi Kong,

I have tried your new build now and the optimization did indeed make a noticeable difference! Specially when SPI is enabled.

DD-WRT O2 SPI Enabled

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 114048851 bps (114.05 Mbit/s)
Send    = Throughput: 139370187 bps (139.37 Mbit/s)

Full Duplex
Send    = Throughput: 63836711 bps (63.84 Mbit/s)
Receive = Throughput: 71885416 bps (71.89 Mbit/s)


DD-WRT O2 SPI Disabled

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 132416481 bps (132.42 Mbit/s)
Send    = Throughput: 144225872 bps (144.23 Mbit/s)

Full Duplex
Receive = Throughput: 84639425 bps (84.64 Mbit/s)
Send    = Throughput: 67289825 bps (67.29 Mbit/s)
Kong
Kong's picture
Great!!!

Great!!!

a few weeks ago I checked what could be optimized in the tcp/iptable settings, but I just tested it with samba and ftp access, not throughput. These things could speed it up a bit more:

These two lines avoid to process already established and related connections again

iptables -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

Ans some tcp stack opts:

echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 0 > /proc/sys/net/ipv4/tcp_sack
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling

The iptables rules above should definitely speed up things with SPI enabled.

bosa
bosa's picture
Yes Kong, so it did!

Yes Kong, so it did!

Almost as good results as with SPI disabled now!

DD-WRT O2 SPI Enabled with some tuneing

TCP Speed test, host=192.168.0.30, transfer size=50000000 bytes

Half Duplex
Receive = Throughput: 130754163 bps (130.75 Mbit/s)
Send    = Throughput: 139560752 bps (139.56 Mbit/s)

Full Duplex
Send    = Throughput: 63735789 bps (63.74 Mbit/s)
Receive = Throughput: 73580160 bps (73.58 Mbit/s)
Kong
Kong's picture
Yeah !!! The rule problem is

Yeah !!! The rule problem is already a known problem which will eventually be fixed in all builds including official.

 

Now to sum this up for folks that only read parts of the thread:

compared to the current official build the compile optimization along with rule optimization boosts your download speed from 82Mbps to 130Mbps

 

According to some info on the net the broadcom 4718(4716A) is a mips32r2 chip, so it could be compiled with -mtune=mips32r2 instead of now mips32 this along with a few more gcc opts could improve the performance even more. I'm going to see if this works.

bosa
bosa's picture
Yes this is really good news

Yes this is really good news since >= 100 Mbps was what I was looking for!

I guess I can live without 100 Mbps in both directions at the same time, maybe that will be possible some day when someone clever has solved the bugs in the fast_nat module...

Should be interesting to see what those extra optimizations can do.

Thank you for all your help with this Kong!

Sawyer
Sawyer's picture
So what you basicaly saying

So what you basicaly saying is I need to take Tomato Firmware with USB (Compatible with NETGEAR WNR3500L)
BETA 16 build of Tomato 1.27 with USB, compatible with NETGEAR WNR3500L, updated 06/14/2010 if Im looking for 100/100Mbit

bosa
bosa's picture
Sawyer said: So what you

Sawyer said: So what you basicaly saying is I need to take Tomato Firmware with USB (Compatible with NETGEAR WNR3500L) BETA 16 build of Tomato 1.27 with USB, compatible with NETGEAR WNR3500L, updated 06/14/2010 if Im looking for 100/100Mbit

Yes, either that or Netgear firmware.

taekwon3dan
taekwon3dan's picture
Excellent work! Very

Excellent work! Very informative. Thanks to both of you (bosa & kong).

Kong
Kong's picture
Sawyer said: So what you

Sawyer said: So what you basicaly saying is I need to take Tomato Firmware with USB (Compatible with NETGEAR WNR3500L) BETA 16 build of Tomato 1.27 with USB, compatible with NETGEAR WNR3500L, updated 06/14/2010 if Im looking for 100/100Mbit

  My next build will have a fix for the performance issue. Should be out within the next 7 days

Pages