Kong 24500 external USB HDD causes router crash

26 posts / 0 new
Last post
Venturis
Venturis's picture
Kong 24500 external USB HDD causes router crash

Just updated to build 24500 today and noticed that when copying file to my external HDD attached to the front USB 3.0 port of the R7000 that the router spontaneously reboots after the first few seconds of the file transfer.

Seems not to be an issue when copying files over Wifi to the router attached USB HDD but is reproducable fault when coping files to the USB HDD from a computer connected directly to the router lan port.

File copy from local PC to the USB HDD starts off ok and then within 3-4 seconds freezes and the router reboots.

Anyone have any experience with this issue?

Not having extensively tested USB HDD connectivity in the previous builds I can't say if this is a new problem but I do recall doing a PC backup to the USB HDD on an earlier build that seemed to go without incident.

Venturis
Venturis's picture
Should add that it was an

Should add that it was an NTFS formatted 500G USB 3.0 external portable HDD.

Tried coping a single 8.5Gig file over when the router crashes and reboots.

Venturis
Venturis's picture
Hi Guys,

Hi Guys,

Anyone had this issue with the router rebooting when trying to copy files to an USB external HDD.

I really want this to work.

My R7000 is on release 24345 of Kong's DD-WRT build and this is really causing me to re-think going back to Netgear's stock firmware.

I've seached the forums and Google and can't find any mention on other sites about this issue so I can only assume it's a unique issue to me?

poketix
poketix's picture
I would like to add my two

I would like to add my two cents into the thread. I have upgraded my R7000 to the latest Kong's firmware 24865M. Even though Kong's software is very solid and stable I cannot rely on it completely so R7000 is used as a sophisticated Access Point connected to the main router via 1GB LAN. I have 2TB WD USB3.0 hard drive connected to R7000. My disk is formatted with NTFS and I have no intention to re-format it to Ext4 for numerous reasons.

All this long preamble is just to help explaining my observations. I believe that either NTFS drivers or USB drivers (or both) used in the Kong's firmware are terribly buggy and probably should be replaced with some better alternative (if exists of course). Following issues seem to be related to the NTFS/USB drivers:

1. Copying large amounts of data to the FS disk consumes a lot of router's CPU and memory. As the result other processes become stuck and sometimes watchdog mechanism decides to restart the R7000 (this seems to be a main reason for the issues reported in the beginning of this thread)

2. USB3.0 performance is critically degraded (it is significantly less than expected even for the USB2.0 connection)

3. Fresh (just booted) system performs disk operations much faster. After 5-10 minutes, it degrades slightly and after hour or two, it degrades dramatically. This usually points to the memory leakage problem when driver/thread/process consumes much more memory than system can provide it without complicated memory management procedures.

I’ve experimented with the different external hard disks and my conclusion is that the above issues are common for the different manufacturers and thus not resulted by the disks. I indeed have no experience of dealing with Ext4 however I’ve heard about numerous issues of shared folders not being recognized by the Windows and Mac environments thus I prefer to stay with NTFS.

Any thoughts? Kong, what is your opinion on subject?

Venturis
Venturis's picture
 

 

poketix said: I would like to add my two cents into the thread. I have upgraded my R7000 to the latest Kong's firmware 24865M. Even though Kong's software is very solid and stable I cannot rely on it completely so R7000 is used as a sophisticated Access Point connected to the main router via 1GB LAN. I have 2TB WD USB3.0 hard drive connected to R7000. My disk is formatted with NTFS and I have no intention to re-format it to Ext4 for numerous reasons. All this long preamble is just to help explaining my observations. I believe that either NTFS drivers or USB drivers (or both) used in the Kong's firmware are terribly buggy and probably should be replaced with some better alternative (if exists of course). Following issues seem to be related to the NTFS/USB drivers: 1. Copying large amounts of data to the FS disk consumes a lot of router's CPU and memory. As the result other processes become stuck and sometimes watchdog mechanism decides to restart the R7000 (this seems to be a main reason for the issues reported in the beginning of this thread) 2. USB3.0 performance is critically degraded (it is significantly less than expected even for the USB2.0 connection) 3. Fresh (just booted) system performs disk operations much faster. After 5-10 minutes, it degrades slightly and after hour or two, it degrades dramatically. This usually points to the memory leakage problem when driver/thread/process consumes much more memory than system can provide it without complicated memory management procedures. I’ve experimented with the different external hard disks and my conclusion is that the above issues are common for the different manufacturers and thus not resulted by the disks. I indeed have no experience of dealing with Ext4 however I’ve heard about numerous issues of shared folders not being recognized by the Windows and Mac environments thus I prefer to stay with NTFS. Any thoughts? Kong, what is your opinion on subject?

 

To rule out any issues with my NTFS formatted hard drive I formated my external 2.5" USB drive to EXT4 and tried to copy some files over.

Alas, same results....initally tried a single 8.5GB archive file and within the first few seconds the transfer stopped and the computer seemed to wait....shortly thereafter the router rebooted.

Tried to be a little less adventerous and only tried to copy over a 700MB movie file this time....started off with promise...but shortly thereafter came to a grinding halt and the router rebooted.

Maybe I just have a faulty router....my only solution now is to revert back to the Netgear OEM firmware and try using the external HDD again and see if I have more success.

Its a real shame because every other aspect of Kong's DD-WRT is simply brilliant, except for the fact that I cant' use the newest builds due to my router rebooting due to the broadcom driver issue but I understand that Kong is waiting for new drivers before that can be fixed.

Venturis
Venturis's picture
I reverted back to Netgear's

I reverted back to Netgear's latest OEM firmware and tried out my USB HDD again.

This time, no problems at all coping files back and forth between computers and USB storage connected to the router. Used CrystalMark to stress test the drive. No issues occurred.

That the very least it rules out a hardware problem.

For the time being I am going to stick with the OEM firmware....yes it has its own problems and the interface is god awful but it just mostly works.

phinn
phinn's picture
After reading this in hope of

After reading this in hope of having a solid USB 3.0 NAS box with the R7000 on a custom firmware isn't happening for now, whether it's driver issues, configuration, etc. As such I'm considering returning this and getting the WRT1900AC (yes I know Linksys lied and there is no full open-source).

Regardless did you guys try Shibby's Tomato builds yet? Any luck with USB 3.0 / NAS to see if he did anything different? Or does it only work properly on the OEM R7000 firmware?

Thanks

Venturis
Venturis's picture
phinn said: After reading

phinn said: After reading this in hope of having a solid USB 3.0 NAS box with the R7000 on a custom firmware isn't happening for now, whether it's driver issues, configuration, etc. As such I'm considering returning this and getting the WRT1900AC (yes I know Linksys lied and there is no full open-source). Regardless did you guys try Shibby's Tomato builds yet? Any luck with USB 3.0 / NAS to see if he did anything different? Or does it only work properly on the OEM R7000 firmware? Thanks

I've unfortunately had to revert back to the OEM firmware for the time being.  Lets hope this issue gets sorted out in a later release.

Haven't tried any other firmware releases other than DD-WRT.

Having said that, the USB support on the OEM firmware works well.  No issues encountered so far with the same HDDs that I was trying to use with DD-WRT.  Both NTFS and EXT4 formatted partitions were recognised by the R7000.

I'm using the USB attached HDD to back up my home server.  Transfer speeds from my locally connected home server to the USB HDD are about 30MB/s read/write speed on USB3.0 with a USB3.0 HDD.

I'd still much prefer to be using Kong's DD-WRT but I think I will settle for the OEM firmware for now.

poketix
poketix's picture
Venturis said:

Venturis said:

  Both NTFS and EXT4 formatted partitions were recognised by the R7000. I'm using the USB attached HDD to back up my home server.  Transfer speeds from my locally connected home server to the USB HDD are about 30MB/s read/write speed on USB3.0 with a USB3.0 HDD.  

Do not be too optimistic. The initial reason why I ceased dealing with the Netgear's original firmware was the fact that it simply destroys data on the attached disk (mine was NTFS). Despite the long months of my intensive interacting with their miserable support they did not succeed to address this major problem. I suggest you to not count on your backups too much. Most chances you won't be able to use them afterward unless you switch back to the open source firmware.

phinn
phinn's picture
poketix said: I suggest you

poketix said: I suggest you to not count on your backups too much. Most chances you won't be able to use them afterward unless you switch back to the open source firmware.

Wow, scary. Just like Kong said they do some NTFS hacks to get better benchmarks at the cost reliability... I wonder if others do that too. The WRT1900AC is getting even better USB 3.0 performance it might use similar hacks.

I'd have no problem formatting my drive with Ext4 to get better performance, however I assume this means Windows and other devices won't read the drive without special software, or does Samba somehow convert the Ext4 to a format they support (I highly doubt it). This would be no good since my work laptop is Windows, Linux is only on my home box.

Still hoping someone gives Shibby's Tomato builds a shot just to see if he accomplished anything different.

poketix
poketix's picture
It looks like I found the

It looks like I found the root cause of my problems. It is minidlna. I attached to R7000 my 2TB harddrive full of the media files. As the result minidlna was simply eating the entire router's memory. This perfectly explains the crashes and terrible slowing down.
My solution is as follows (use telnet to log into the R7000):
1. Kill the minidlna by the "killall minidlna"
2. Edit the /tmp/minidlna.conf file (there is a vi editor for this purpose) and add the line configuring the database location. In my case I added the following line:
db_dir=/mnt/sda1/_minidlna_
to locate the database on my harddrive
3. Restart the minidlna using the following command:
minidlna -R -f /tmp/minidlna.conf &

Now minidlna uses the disk instead of the RAM to store the database. I'll monitor the status and update this thread with the longer term results.

Kong, if you read this by chance I guess it would be useful to add the database location configuration to the web interface. It might avoid many issues related to the lack of RAM. Thanks.

phinn
phinn's picture
Very interesting poketix and

Very interesting poketix and it seems like good fix for something that's plagued these builds for a while. I haven't been using minidlna, just an ext4 formatted usb 3.0 drive (Seagate Backup Plus Slim 1TB) and performance is decent.

Great work though, hope Kong is listening and can verify/fix this :)

poketix
poketix's picture
Just a brief update. After 5

Just a brief update. After 5 hours router is still up and running. CPU loading is around 6%, available memory varies between 12% and 25% (before the fix it was always around 11%).
DLNA is completely functional - video, music and pictures.
The most interesting part - database folder size is 211MB, most of which are the preview thumbnails. When those reside in RAM R7000 definitely runs out of resources.

phinn
phinn's picture
Great job. Did you message

Great job. Did you message this thread to Kong, it would be nice to get it added to his builds. He might not have seen this thread. I think he has been MIA for a little bit though.

poketix
poketix's picture
I'm not sure I know how to

I'm not sure I know how to contact Kong directly. Is it something build into the forum tools? I'd appreciate your suggestion or even forwarding the information to Kong.

phinn
phinn's picture
Yep just go to his profile:

Yep just go to his profile: http://www.myopenrouter.com/profile/4822/
And click "Send Private Message" below his photo. Maybe just link him this thread and tell him you think you fixed it. I'm going to try this out on my R7000 tonight by the way.

poketix
poketix's picture
I sent the message to Kong

I sent the message to Kong but it looks like he is away for a while.

Unfortunately my solution is not complete. R7000 self-rebooted after 12 hours of normal operating. Upon rebooting it entered into the endless loop of more rebooting. I was able to restore it only by disconnecting the disk, scanning it for errors from Windows (no errors found) and plugging back into the R7000's USB port.

I used startup script mechanism to make my solution semi-permanent. I'll monitor the status and report further.

phinn
phinn's picture
Looks like kong is back and

Looks like kong is back and pumped out a couple of new builds. Any luck with 25015M? I noticed he updated minidlna.

Btw the minidlna update is very minor, so it won't improve things unless there are some config changes:
http://sourceforge.net/projects/minidlna/files/minidlna/1.1.4/

poketix
poketix's picture
I sent my suggestion to Kong

I sent my suggestion to Kong and he advised me with the simpler solution. The moment DD-WRT is configured to use JFFS, minidlna (as same as some other components) stores its database within the /jffs partition. Kong explained that his firmware takes appropriate care of the potential caveats when external disk containing the /jffs is being unmounted.

Basing on that I can suggest the following steps replacing my original solution:

1. Find a redundant USB drive. I use the 4GB flash disk. Create one or two ext3/ext4 partitions on it. Second partition might be useful if you decide to install some extra SW packages (check http://www.dd-wrt.com/wiki/index.php/Optware). If not then one partition is enough. I used the free EaseUS Partition Master to create and format the partitions from Windows.
2. Disable DLNA via WebUI
3. Plug the formatted flash disk into the rear USB2 connector of R7000.
4. Configure JFFS as explained there: http://www.dd-wrt.com/wiki/index.php/JFFS
(Note that Kong's WebUI allows mounting of the partition to JFFS from the Services->USB page. JFFS2 configuration is on the Administration page.)
5. Enable DLNA and clear database. After your media disk is scanned minidlna stores database on the external flash disk.

This solution is tested and found very stable. As for the new minidlna version 1.1.14 it seems improving folder mapping and adds the new "Recently Added" category to the media folders.

phinn
phinn's picture
Wow keep us updated on how

Wow keep us updated on how this works out longer-term. I'll find a spare USB flash and try this out myself. How does this end up working better than a separate partition on the hard drive, or would that work too? Think this is a work-around or permanent solution? Good work!

poketix
poketix's picture
I started from the kind of a

I started from the kind of a patch solution locating the minidlna database on the same drive and partition where my whole media is retained. I needed to add the startup script that "manually" starts minidlna that uses the non-standard configuration location (DLNA is disabled via WebUI). This workaround has a caveat however. If one decides to use the DLNA in the regular way (via WebUI) it is must to remove the database folder from the media disk. Otherwise minidlna instantly crashes and router restarts just after it is brought up. The reason for this issue is too long names and too deep sub-folder layers that minidlna database uses. It is simply not suitable for the NTFS (ext3/4 seems to be okay with it).

Summarizing the above I vote for the Kong's solution (JFFS). It is absolutely inherent and requires only minimal arrangement. It also frees even more memory as other components also utilize the /jffs.

P.S. My R7000 is till up and running since yesterday, showing only 50% of the memory utilization. Shared disk access is fluent at maximum of the theoretical rate (~19MBps). Kudos to Kong again.

phinn
phinn's picture
Sounds like you guys found a

Sounds like you guys found a legit fix then. 19MBps is a little low, is this because your NTFS formatted and not Ext4? I know the stock firmware got 35+MBps but yea it has been known to trash partitions. I formatted my drive Ext4 but haven't used Minidlna yet. You've inspired me to try it out perhaps this weekend.

Just a heads up there are some R7000 tips over at DD-WRT forum too, I posted your solution there (crediting you and Kong):
http://www.dd-wrt.com/phpBB2/viewtopic.php?t=264152

poketix
poketix's picture
19MBps is more or less what

19MBps is more or less what default DD-WRT USB driver can reach. Kong has mentioned many times that Netgear does not release to public Broadcom's USB3 drivers and thus his builds can't provide the theoretical throughput of USB3. Stock firmware indeed uses the dedicated drivers. However Netgear is totally unable to release a minimally working stock firmware even year after the R7000 was announced. They instantly demonstrate lack of responsibility and respect to the customers. Kong does his amazing job alone despite the blocking efforts of Netgear.

phinn
phinn's picture
Hey Poketix just checking up

Hey Poketix just checking up on your solution for minidlna, I haven't yet set this up on mine so curious if it' still working well.

Also I saw this over at DD-WRT's forum about someone's hard drive never spinning down with Kong's build, not sure how he did with stock firmware. Have you noticed that? I haven't with mine.
http://www.dd-wrt.com/phpBB2/viewtopic.php?t=268964

poketix
poketix's picture
JFFS solution seems to

JFFS solution seems to resolve numerous issues. My router runs 10 days without any reboot or any degradation of the shared storage access speed. I've never seen it so stable. It looks like lack of memory is the root cause of most of the issues so swapping it out (using JFFS) makes the firmware happy.

As for the spinning down - mine WD indeed spins after 30 minutes of the timeout (I can see it by the LED indication). This is how I configured it by plugging into Windows and using the WD dedicated software. I guess people in the forum should do the same.

phinn
phinn's picture
Looks like an updated OEM

Looks like an updated OEM firmware released that includes a new NTFS driver and bug fix for using Samba. I wonder if these were the file corruption issues people were having all along and the improvements could be included with Kong's build:
http://kb.netgear.com/app/answers/detail/a_id/25737

They also released updated source code for the new build:
http://kb.netgear.com/app/answers/detail/a_id/2649