R7000 Kong's firmware - shared USB storage

34 posts / 0 new
Last post
poketix
poketix's picture
R7000 Kong's firmware - shared USB storage

Dear Kong, first of all let me express my deep appreciation and respect for your amazing doing. Your firmware beats Netgear's miserable creation in almost every aspect. Thank you for your dedication.

I have several issues around the shared USB storage. I've recently flashed your latest release r23655. My observations are as follows:

1. Samba shares work properly (I'm using public access) but only when accessing from the Windows 7 computers. My LG streamer instantly hangs up and reboots just during the initial handshaking (when I'm just trying to see the root folder structure via its UI). Since it was working with the official FW I assume that the streamer dislikes something about smbd configuration in your release. My wild guess is for password encryption or whatsoever. Could you please shed some light on how can I tune the Samba configuration. I'd kindly appreciate any ideas and suggestions how to resolve my streamer issue.

2. DLNA service does not start properly after the router's reboot. It shows no folders on th TV and streamers. The workaround is as follows (little bit annoying indeed) - after every reboot turn off the DLNA via appropriate Web configuration page. Then DLNA processes must be explicitly killed via the telnet connection. After this enabling DLNA via the Web interface results in working system (it takes some time for miniDLNA to re-index the folders however). Is it any way to keep DLNA instantly working after the reboots?

3. DLNA permanent index database functionality unfortunately is not perfect yet. In fact it must be explicitly disabled as otherwise TV sees the videos but can't play them (gets stuck or reports the invalid file format). Without database retaining, everything works under some restrictions (see the above paragraph).

4. USB 3 performance is really poor. I’ve read your comments about the drivers not exposed by Netgear (what a shame indeed). Do you have any ideas/suggestions about what can be done on subject?

Sorry for the long list and thanks again for your great work.

Poketix

Kong
Kong's picture
The dlna problem is most

The dlna problem is most likely a regression, we introduced IGMP snooping in 23655 that breaks multicast traffic and thus breaks some dlna traffic.

Regarding usb you get the best speeds with either ext2/ext3 or FAT32, but if you want to make use of optware packages, then you need a linux filesystem.

poketix
poketix's picture
Kong said: The dlna problem

Kong said: The dlna problem is most likely a regression, we introduced IGMP snooping in 23655 that breaks multicast traffic and thus breaks some dlna traffic. Regarding usb you get the best speeds with either ext2/ext3 or FAT32, but if you want to make use of optware packages, then you need a linux filesystem.

Thanks for your commenting. I'm afraid we are not completely aligned concerning the DLNA. After I'm restarting it manually forcing the database cleanup it further works properly, thus I doubt it is about any regression. It looks more like some kind of race conditions when the system starts. Please note also that miniDLNA does not recognize adding or removal of the media files. Again force cleanup and complete daemon restart resolv the issue.

May I kindly remind about my first question in the list?

Thanks in advance.

Kong
Kong's picture
poketix said:

poketix said:

Kong said: The dlna problem is most likely a regression, we introduced IGMP snooping in 23655 that breaks multicast traffic and thus breaks some dlna traffic. Regarding usb you get the best speeds with either ext2/ext3 or FAT32, but if you want to make use of optware packages, then you need a linux filesystem.

Thanks for your commenting. I'm afraid we are not completely aligned concerning the DLNA. After I'm restarting it manually forcing the database cleanup it further works properly, thus I doubt it is about any regression. It looks more like some kind of race conditions when the system starts. Please note also that miniDLNA does not recognize adding or removal of the media files. Again force cleanup and complete daemon restart resolv the issue. May I kindly remind about my first question in the list? Thanks in advance.

If you look at the generated minidlna.conf file you will see:

 

notify_interval=300

 

means it will discover newly added files after 5 min. Of course if you stop/start it will discover them right away. Regarding your smb issue, I have no idea why your LG reboots, but sounds to me like a problem in your LG firmware. If it works with netgear firmware, then it might be because netgear uses a rather old samba version while we are using the latest samba version ,due to security reasons.

poketix
poketix's picture
Is it any way to change the

Is it any way to change the default smbd configuration of the router? What is the difference between the old and new samba versions and can be old implementation behavior reproduced with the new one?

Kong
Kong's picture
Regarding minidlna I have

Regarding minidlna I have found an issue in minidlna that allows the multiple minidlna startup + I have optimized service startup in order to reduce the calls to start services.

This will be included in my next build - 23700 R01

poketix
poketix's picture
Indeed using "top" I always

Indeed using "top" I always observed the two dlna processes running. As I mentioned above after disabling the DLNA via Webif it is necessary to kill the remaining process explicitly.

May I ask you to point me to some manuals relevant to the new smbd used in your build? Is it any way to change some Samba parameters permanently specifically for my router? I'm asking it just in case I find some combination that allows me using of the LG streamer again.

Thanks again.

Kong
Kong's picture
poketix said: Indeed using

poketix said: Indeed using "top" I always observed the two dlna processes running. As I mentioned above after disabling the DLNA via Webif it is necessary to kill the remaining process explicitly. May I ask you to point me to some manuals relevant to the new smbd used in your build? Is it any way to change some Samba parameters permanently specifically for my router? I'm asking it just in case I find some combination that allows me using of the LG streamer again. Thanks again.

To try different settings you need to manually kill and start smbd e.g.:

 

killall smbd

then modify:

vi /tmp/smb.conf

then start it again:

 /usr/sbin/smbd -D --configfile=/tmp/smb.conf

poketix
poketix's picture
I made some experiments with

I made some experiments with the DLNA. Despite the fact that "notify_interval=300" line exists in the minidlna.conf index of the folders is never renewed. I added several video files into the different folders on my disk but minidlna has not discovered them even though I gave it the whole 20 hours to complete. After cleaing the index and restarting the minidlna form the Webif disk content was re-indexed properly. Is this consistent with the issue you found?

Yet another question - I'm instantly experiencing the "403 forbidden" error when trying to access your firmware download page (link). Is it any particular reason for this? Something related to my ISP?

Kong
Kong's picture
Looks like you found another

Looks like you found another bug in minidlna, I just tested it and inotify did not work, because minidlna's own config generator that sets up defines for compile includes had a typo and therefore uses the wrong inotify includes for the kernel.

I'm also enabling minidlnas logging mechanism in my build now.

poketix
poketix's picture
Great, I'm looking forward to

Great, I'm looking forward to your next release.

Update: DLNA works like a charm now. New media files are automatically indexed and shown on TV - well done as usual, Kong.

I'm sorry to be the "messenger" but something went wrong in this and previous version with the USB3.0 or smbd. Now when I'm copying large files from PC to the attached shared disk the throughput is very low (maximum 6 MB/s) while in the February releases it was around 20-24 MB/s.

Still any idea about the "403 forbidden" problem? I'm able to access the FW download location from my office thus I assume that there are some country-based restrictions on your side. Namely it works when I'm browsing from UK (corporate) and blocks me when I'm trying to access from Israel (home). Could it be fixed somehow?

poketix
poketix's picture
 

 

Kong said:

To try different settings you need to manually kill and start smbd e.g.: killall smbd then modify: vi /tmp/smb.conf then start it again:  /usr/sbin/smbd -D --configfile=/tmp/smb.conf

I'm trying to play with the parameters - without actual progress by now.

Please note that the throughput when uploading large files from a PC to the external disk connected to the R7000 USB3.0 port is observed to be very unstable in the latest releases. It instantly varies between the 0.5MBps and 7.5MBps. In very rare cases it climbs up to 19MBps but then drops again.

Both PC and router are connected via the 1GB LAN ports.

 

Kong
Kong's picture
poketix said:

poketix said:

Kong said:

To try different settings you need to manually kill and start smbd e.g.: killall smbd then modify: vi /tmp/smb.conf then start it again:  /usr/sbin/smbd -D --configfile=/tmp/smb.conf

I'm trying to play with the parameters - without actual progress by now. Please note that the throughput when uploading large files from a PC to the external disk connected to the R7000 USB3.0 port is observed to be very unstable in the latest releases. It instantly varies between the 0.5MBps and 7.5MBps. In very rare cases it climbs up to 19MBps but then drops again. Both PC and router are connected via the 1GB LAN ports.

You are probably using a slow usb flash drive.

The optimization in latest build works best with fast flash drives or regular hard disk drives.

poketix
poketix's picture
Kong said:

Kong said:

You are probably using a slow usb flash drive. The optimization in latest build works best with fast flash drives or regular hard disk drives.

Sorry if it looks like I'm insisting, but I use the same USB 3.0 external WD hard drive that with the factory firmware demonstrates stable 36MBps throughput. Unfortunately Netgear's firmware tends to ruin the file structure of my disc thus I can't use it. Your firmware looks perfect in almost every aspect except for the mentioned throughput. Just to stress it again, your versions from the last February were stable and gave me ~20MBps. Something was changed in the latest March releases.

Kong
Kong's picture
Try this, logon via telnet

Try this, logon via telnet/ssh and run command:

echo 128 >/sys/block/sda/queue/read_ahead_kb
echo 128 >/sys/block/sdb/queue/read_ahead_kb

Now do your transfer test. Back to old performance?

poketix
poketix's picture
Kong said: Try this, logon

Kong said: Try this, logon via telnet/ssh and run command: echo 128 >/sys/block/sda/queue/read_ahead_kb echo 128 >/sys/block/sdb/queue/read_ahead_kb Now do your transfer test. Back to old performance?

Could it be just "/sys/block/sda/queue/read_ahead_kb"? I guess I do not have "/sys/block/sdb/queue/read_ahead_kb" as nothing is plugged into the second USB port.

Kong
Kong's picture
poketix said:

poketix said:

Kong said: Try this, logon via telnet/ssh and run command: echo 128 >/sys/block/sda/queue/read_ahead_kb echo 128 >/sys/block/sdb/queue/read_ahead_kb Now do your transfer test. Back to old performance?

Could it be just "/sys/block/sda/queue/read_ahead_kb"? I guess I do not have "/sys/block/sdb/queue/read_ahead_kb" as nothing is plugged into the second USB port.

Yes, I just wanted to make sure it is also set in case you attached two drives:-)

poketix
poketix's picture
 

 

Kong said:

Yes, I just wanted to make sure it is also set in case you attached two drives:-)

 

Thank you very much for your kind support. I can definitely report an improvement when reducing the read_ahead size (I've got up to 18MBps on the freshly started system, but still 6-7MBps after running for a while - see below).

It is pretty strange that read ahead value effects on the write speed, indeed. Still while copying the huge files the throughput varies all the time (but changing amplitude is less with the new value). BTW, is it a way to keep this parameter among the reboots? I see that it restores to the default value 1024 after the system restart. Besides, is it anything else to change that might improve the uploading speed?

Update: after R7000 is running for half an hour the throughput degrades dramatically to the 6-7MBps. Could it be related to the fact that I have huge amount of the music and video files on the attached disk and miniDLNA running in parallel? The latter operates fluently after your latest fix.

P.S. I'm using the version with the old Broadcom drivers if it does matter.

Thank you very much for your kind support. I can definitely report some improvement when reducing the read_ahead size (I've got up to 18MBps when system is fresh after the reboot, will check whether it reduces or not after running for a while).
It is pretty strange that read ahead value effects on the write speed, indeed. Still while copying the huge files the throughput varies all the time (but changing amplitude is less with the new value). BTW, is it a way to keep this parameter among the reboots? I see that it restores to the default value 1024 after the system restart. Besides, is it anything else to change that might improve the uploading speed?
poketix
poketix's picture
Well, it looks like I'm

Well, it looks like I'm mostly communicating with myself. Still two important observations to mention. I've upgraded to the recent r23884 version (old drivers, standard):
- It looks like new Samba greatly improved the performance issue - uploading throughput is solid stable around 19MBps even after R7000 is up and running for a long time. Great job again, Kong.
- Unfortunately DLNA seems to malfunction in couple of aspects:
o Automatic re-indexing works only partially - it adds some files but ignores others. Old workaround (disable/enable with DB erasing) works as expected
o Folders now include one more definitely redundant level - "sda1" is shown above the actual folders. Besides the same "/mnt/sda1" share is shown as two consequent "sda1" folders on TV. It is unclear why DLNA server decided to split the same folder structure into two.
Any ideas how to fix it?

Kong
Kong's picture
poketix said: Well, it looks

poketix said: Well, it looks like I'm mostly communicating with myself. Still two important observations to mention. I've upgraded to the recent r23884 version (old drivers, standard): - It looks like new Samba greatly improved the performance issue - uploading throughput is solid stable around 19MBps even after R7000 is up and running for a long time. Great job again, Kong. - Unfortunately DLNA seems to malfunction in couple of aspects: o Automatic re-indexing works only partially - it adds some files but ignores others. Old workaround (disable/enable with DB erasing) works as expected o Folders now include one more definitely redundant level - "sda1" is shown above the actual folders. Besides the same "/mnt/sda1" share is shown as two consequent "sda1" folders on TV. It is unclear why DLNA server decided to split the same folder structure into two. Any ideas how to fix it?

It is probably the way you setup the paths, what I do is, mount the partition that holds the media files to /jffs or /opt and then specify /jffs or /opt as path in minidlna.

poketix
poketix's picture
Kong said:It is probably the

Kong said:It is probably the way you setup the paths, what I do is, mount the partition that holds the media files to /jffs or /opt and then specify /jffs or /opt as path in minidlna.

I would agree but I changed nothing in my setup and just upgraded to the latest release that seems to have different minidlna version according to the release notes. Using the same /mnt/sda1 path as a root folder for DLNA results in the different appearance on DLNA client (TV). Generally I'd like to have an option to add just separate directories (including their subdirectories) and not a single huge directory tree. Presently WebUI prevents me from doing this as it shows /mnt/sd1 as the only valid option for the DLNA. Am I missing something here?

Kong
Kong's picture
poketix said:

poketix said:

Kong said:It is probably the way you setup the paths, what I do is, mount the partition that holds the media files to /jffs or /opt and then specify /jffs or /opt as path in minidlna.

I would agree but I changed nothing in my setup and just upgraded to the latest release that seems to have different minidlna version according to the release notes. Using the same /mnt/sda1 path as a root folder for DLNA results in the different appearance on DLNA client (TV). Generally I'd like to have an option to add just separate directories (including their subdirectories) and not a single huge directory tree. Presently WebUI prevents me from doing this as it shows /mnt/sd1 as the only valid option for the DLNA. Am I missing something here?

Subdirs like in samba paths are not yet supported. I have it on my todo list.

poketix
poketix's picture
Dear Kong,

Dear Kong,
After I flashed the latest version 24345M I'm experiencing the serious issues around the connected USB drive:
1. File access is very slow. In case two separate computers try to copy anything to/from the shared disk everything freezes and copying take a while.
2. DLNA streaming instantly drops - TV loses the server completely. After several minutes it recovers. Some movies can't be played at all (higher resolution).
3. DLNA does not index new movies automatically.
Reverting to the old version 24045M restores everything to the working condition. Could you please suggest something to fix this (indeed except for staying with the working old version :-))?
Thanks in advance.

Kong
Kong's picture
I can redroduce the issue

I can redroduce the issue with not indexing, the problem is the fact, that you can't set any subdirs currently and minidlna can only handle inotify for different dirs, thus if you have audio and video, it will only reindex audio. If you have Video and Images, then it will only reindex videos:-)

To fix this we need at least a subdir option in the gui, maybe change the gui completely to avoid beeing able to set pic,video,audio indexing for the same path.

A workaround would be to create different partitions on the drive and then use each partition for a certain type, e.g. video partition, image partition

poketix
poketix's picture
Thanks for the suggestion - I

Thanks for the suggestion - I'll indeed separate the partitions. Have you by chance reproduced the USB slow/failing access issues (#1 and #2 in my report)?

Kong
Kong's picture
Are you using a cheap usb

Are you using a cheap usb flash drive?

For these drives the read ahead optimization does not work well, in this case try to add this to your startup commands:

echo 128 > /sys/block/sda/queue/read_ahead_kb

poketix
poketix's picture
 .

 .

poketix
poketix's picture
I'm using the expansive and

I'm using the expansive and very fast external 2TB hard-drive by WD. I've tried your solution in the past - it somehow improves the situation but just for some short period of time.

Generally I've noticed the following behavior (relevant for DD-WRT every version that I tried). Fresh system starts with the full theoretical throughput of the USB connection. Since DD-WRT does not use the USB3 drivers it is able to provide just 20MBps compared to the 35MBps of the factory FW. However after 20-30 minutes of the routers operation USB throughput reduces significantly and becomes unstable. Sometimes it fluctuates around 12-15MBps, sometimes even worse - 8-9MBps. In more rare cases it is stuck around 1-2MBps. Rebooting the router returns everything back to the higher throughput. All the measurements are for the writing long files from the PC to the R7000 via the wired 1GB LAN connection.

It looks like over the time firmware accumulates something (threads, lack of memory, etc.) affecting the USB access performance (sorry for such a non-precise description). Any idea where it comes from?

Thanks.

Kong
Kong's picture
poketix said: I'm using the

poketix said: I'm using the expansive and very fast external 2TB hard-drive by WD. I've tried your solution in the past - it somehow improves the situation but just for some short period of time. Generally I've noticed the following behavior (relevant for DD-WRT every version that I tried). Fresh system starts with the full theoretical throughput of the USB connection. Since DD-WRT does not use the USB3 drivers it is able to provide just 20MBps compared to the 35MBps of the factory FW. However after 20-30 minutes of the routers operation USB throughput reduces significantly and becomes unstable. Sometimes it fluctuates around 12-15MBps, sometimes even worse - 8-9MBps. In more rare cases it is stuck around 1-2MBps. Rebooting the router returns everything back to the higher throughput. All the measurements are for the writing long files from the PC to the R7000 via the wired 1GB LAN connection. It looks like over the time firmware accumulates something (threads, lack of memory, etc.) affecting the USB access performance (sorry for such a non-precise description). Any idea where it comes from? Thanks.

No, as I don't experience these problems. In my last test I copied around 60GB via USB 3.0 to different ext3 partitions and I get constant speeds of ~35MB/s. Differences in USB speed has nothing to do with USB drivers. For NTFS netgear uses an performance optimized closed source ntfs linux driver, while it is fast it corrupts the ntfs table under certain circumstances.

I could optimize the kernel driver too in order to get these speeds, but then a write is not reliable anymore. You can't magically be 75% faster compared to a filesystem that exists for years and has been optimized by linux kernel professionals, so it comes at a price. If you want to win benchmarks it is cool, as most of these benchmarks just copy a stream of data and thus it is easy to look good in such a test.

poketix
poketix's picture
Benchmarking the firmware is

Benchmarking the firmware is definitely not what I'm trying to do. I'm holding pretty huge set of the media files of different types that are widely reused over the home network. All my problems come out when I'm copying the new content to the shared disk connected to R7000. I'm indeed considering switching to ext3 from time to time. However this solution has a serious impact on my configuration as it affects a possibility of using the external HD locally without the router. I'll need to maintain a Linux PC just because of it (existing Windows solutions for ext3 are not reliable enough).
May I ask you to reconsider the approach related to the NTFS? As far as I can see there is yet another issue reported for the latest versions claiming that writing the large files to ntfs disk crashes the wifi. It looks more like a specific bug rather generic driver issue. Sorry for my insisting, I greatly appreciate your amazing work.

Law
Law's picture
Hi Poketix and Kong, first of

Hi Poketix and Kong, first of all, sorry about jumping into your post, but I've spent quite a bit of time looking on the internet and I couldn't find a solution to my problem.

I'm a total newbie on DD-WRT and I just got a new external HD (WD My Book 4TB USB 3.0/USB 2.0) today. Hooked it up to my R7000 with netgear's default firmware and it works right away, but I realized that as soon as I connected the drive to the R7000, my wifi became very unstable. So after doing some research online, I found that others also reported wifi issue with usb3 on the default firmware, so I decided to try out Kong's firmware.

After reading a bit about Kong's firmware, I flashed the R7000.chk firmware and then updated it to the latest 247100M build. After rebooting the router, I was able to see the USB drive showed up under Services > USB > Disk Info (it's still in NTFS format, I did not touch it but just plugged into the router directly after I got it.)

Disk Info:
--- /dev/sda1
Block device, size 3.639 TiB (4000751550464 bytes)
Windows NTLDR boot loader
NTFS file system
UUID A2CA0AEBCA0ABC13
Volume size 3.639 TiB (4000751546368 bytes, 976745983 sectors of 4 KiB)
/dev/sda1 mounted to /tmp/mnt/sda1

Then I went to the NAS tab, Enabled Samba Server, I'm not sure what to put in the Server String field, so I just put R7000. Left the Workgroup field blank. Under File Sharing, I added an item with the following information:

Path: /mnt/sda1 (selected from the dropdown list)
Subdir: blank
Name: ShareDrive (I just made this name up)
Public: checked
Access: Read/Write

After saving the configuration, I still couldn't see the drive come up in the Window Explorer. Any advice is appreciated. Thank you!

Pages