Fetch Softworks

Problem downloading very large number of files (>1000) (9 posts)

• Started 17 years ago by Wayne Connor
• Latest reply 17 years ago from Scott McGuire

• Wayne Connor Member

  I am trying to mirror my whole website to my hard drive as a backup.
  It is probably approx 800Mb and in the order of 5 - 10 thousand of files.

  Approx 10-20 min in or so of downloading it will stop at a file, a different file each time. The file may be small or big.

  I am using 10.4.3 and fetch 5, but had the same problems under 10.3 with fetch 4.

  I have the same problem with passive mode on or off.
  Both work for probably 100 or 200 or 300 files downloaded, then abort.
  When I restart the mirror operation, it gets a bit more done.

  I tried cuteftp (30 day trial) and no problems, so I don't think it's with the server, although this may have been a fluke.

  It's reproducible, If I click my mirror icon, it will start mirroring, then die probably 10 - 20 min into the mirror operation.

  From the gui, it appears to just freeze on the file, the dog keeps running, but the filename just stays there, then after a couple of minutes it aborts.

  here is the last entry in the log,
  example 1

  226-File successfully transferred
  226 0.000 seconds (measured here), 77.21 Mbytes per second
  Transfer of wp-settings.php (7,988 bytes, 7,988 bytes/sec, 0:01 elapsed) completed at 22/4/06 9:48:38 AM
  MDTM wp-trackback.php
  213 20060217115256
  SIZE wp-trackback.php
  213 3211
  MDTM wp-trackback.php
  213 20060217115256
  PORT 10,0,0,19,203,161
  ABOR
  Unsuccessful transfer of wp-trackback.php (0 bytes, 0 bytes/sec, 1:45 elapsed) stopped at 22/4/06 9:50:23 AM
  ftp_retrieve: -30028 (state == RGET_RETRIEVING)

  example 2
  Transfer of header_06-over.jpg (1,167 bytes, 1,167 bytes/sec, 0:01 elapsed) completed at 21/4/06 11:32:41 PM
  SIZE header_06.gif
  213 369
  MDTM header_06.gif
  213 20060201021146
  PORT 10,0,0,19,172,51
  200 PORT command successful
  RETR header_06.gif
  Unsuccessful transfer of header_06.gif (0 bytes, 0 bytes/sec, 0:01 elapsed) stopped at 21/4/06 11:32:43 PM
  ABOR

  example 3
  Transfer of theme-editor.php (4,270 bytes, 4,270 bytes/sec, 0:01 elapsed) completed at 21/4/06 11:40:38 PM
  MDTM themes.php
  213 20060201104229
  SIZE themes.php
  213 4387
  MDTM themes.php
  213 20060201104229
  PORT 10,0,0,19,33,9
  ABOR
  Unsuccessful transfer of themes.php (0 bytes, 0 bytes/sec, 1:45 elapsed) stopped at 21/4/06 11:42:24 PM
  ftp_retrieve: -30028 (state == RGET_RETRIEVING)

  Posted 17 years ago #

• Scott McGuire Administrator

  Hi,

  Sorry you're having problems.

  We'd like some more information.

  Could you please post the top part of the transcript, up until the Mirror operation begins?

  Also, do you see the same problem if you try to download the folder containing all the files using the Get command instead of the Mirror command in Fetch 5?

  Thanks,

  Scott McGuire
  Fetch Softworks

  Posted 17 years ago #

• Scott McGuire Administrator

  Hi,

  Here's another thing to try:

  * Go to Fetch Preferences, and click the General tab.
  * Make sure "Use passive mode transfers (PASV)" is checked.

  If it was unchecked, check it and then try the Mirror operation again. Let us know if this helps.

  Thanks,

  Scott McGuire
  Fetch Softworks

  Posted 17 years ago #

• Wayne Connor Member

  Sure,
  First part of transcript:

  Fetch 5.0.5 (5A835) PowerPC running on Mac OS X 10.4.6 (8I127) PowerPC English
  StuffIt Engine 0x702, StuffIt SDK Version StuffIt version 7.0.3
  Partial serial FETCHED001-LN8A-K007 T
  Update check skipped at 25/04/2006 04:05 PM (next check after 28/04/2006 07:04 AM)
  Connecting to mac-host.com port 21 (OS X firewall is off) (25/4/06 4:05:58 PM)
  Connected to 70.103.189.80 port 21 (25/4/06 4:05:58 PM)
  220---------- Welcome to Pure-FTPd [TLS] ----------
  220-You are user number 4 of 50 allowed.
  220-Local time is now 00:05. Server port: 21.
  220-IPv6 connections are also welcome on this server.
  220 You will be disconnected after 15 minutes of inactivity.
  USER machostc
  331 User machostc OK. Password required
  PASS
  230-User machostc has group access to: machostc
  230 OK. Current restricted directory is /
  SYST
  215 UNIX Type: L8
  PWD
  257 "/" is your current location
  MACB ENABLE
  500 Unknown command
  CWD public_html/
  250 OK. Current directory is /public_html
  PWD
  257 "/public_html" is your current location
  PWD
  257 "/public_html" is your current location
  TYPE A
  200 TYPE is now ASCII
  PORT 10,0,0,11,230,216
  200 PORT command successful
  LIST -al
  150 Connecting to port 35907
  drwxr-xr-x 40 800 machostc 1456 Apr 18 18:51 .
  drwx--x--- 15 800 12 872 Apr 25 00:02 ..
  -rw-r--r-- 1 800 machostc 0 Jan 30 17:52 .htaccess
  -rw-r--r-- 1 800 machostc 130 Jan 23 11:45 400.shtml
  -rw-r--r-- 1 800 machostc 130 Feb 22 17:44 400.shtml-Copy
  -rw-r--r-- 1 800 machostc 162 Jan 23 11:45 401.shtml
  -rw-r--r-- 1 800 machostc 201 Jan 23 11:45 403.shtml
  -rw-r--r-- 1 800 machostc 175 Feb 22 17:44 404.shtml
  -rw-r--r-- 1 800 machostc 363 Jan 23 11:45 500.php
  -rw-r--r-- 1 800 machostc 71 Jan 23 11:45 500.shtml
  drwxr-xr-x 3 800 machostc 104 Feb 1 18:25 7barracks
  drwxr-xr-x 4 800 machostc 128 Feb 1 17:56 7barrackscl
  drwxr-xr-x 11 800 machostc 528 Feb 1 21:13 agcamp
  drwxr-xr-x 4 800 machostc 280 Mar 1 05:41 avillion
  drwxr-xr-x 3 800 machostc 192 Jan 31 19:18 avillion.dubbo.org
  drwxr-xr-x 3 800 machostc 72 Feb 1 14:40 barracks
  drwxr-xr-x 2 800 machostc 80 Jan 31 19:29 blog.dubbo.org
  -rw-r--r-- 1 800 machostc 12497 Feb 1 17:57 blog.html
  drwxr-xr-x 2 800 machostc 80 Feb 1 18:25 blog2
  drwxr-xr-x 4 800 machostc 456 Feb 1 17:53 blogwp
  drwxr-xr-x 9 800 machostc 672 Feb 5 04:36 calendar-old bad
  drwxr-xr-x 2 800 machostc 48 Feb 2 15:36 cgi-bin
  drwxr-xr-x 3 800 machostc 104 Feb 2 02:15 crazytest
  -rw-r--r-- 1 800 machostc 1172 Jan 23 11:45 default.html
  drwxr-xr-x 11 800 machostc 952 Mar 29 21:50 dpc
  drwxr-xr-x 3 800 machostc 888 Apr 17 16:26 dragonboat
  drwxr-xr-x 9 800 machostc 2040 Mar 27 15:46 dubbo
  drwxr-xr-x 5 800 machostc 184 Mar 26 22:03 dubbo.nsw
  drwxr-xr-x 9 800 machostc 1512 Feb 1 18:04 dubbonew
  drwxr-xr-x 3 800 machostc 136 Feb 26 00:31 early
  drwxr-xr-x 5 800 machostc 280 Feb 19 23:06 evening
  -rw-r--r-- 1 800 machostc 822 Feb 16 20:18 favicon.ico
  drwxr-xr-x 4 800 machostc 136 Jan 31 18:43 feva
  drwxr-xr-x 11 800 machostc 840 Feb 20 03:08 forums
  drwxr-xr-x 11 800 machostc 776 Feb 6 15:21 forumsold
  drwxr-xr-x 2 800 machostc 264 Feb 22 17:45 henty
  drwxr-xr-x 6 800 machostc 1080 Mar 23 06:34 howto
  drwxr-xr-x 6 800 machostc 936 Feb 2 16:55 howto.old
  drwxr-xr-x 5 800 machostc 880 Feb 2 18:02 howto.old2
  drwxr-xr-x 3 800 machostc 880 Feb 2 15:33 howtoold
  drwxr-xr-x 5 800 machostc 672 Feb 1 18:54 iblog
  drwxr-xr-x 2 800 machostc 208 Mar 1 05:32 images
  -rw-r--r-- 1 800 machostc 3987 Mar 1 05:32 index.html
  drwxr-xr-x 3 800 machostc 344 Feb 17 04:20 korah
  drwxr-xr-x 2 800 machostc 80 Feb 1 18:19 media
  drwxr-xr-x 4 800 machostc 160 Feb 22 00:27 morning
  drwxr-xr-x 13 800 machostc 2336 Feb 23 03:34 photos
  -rw-r--r-- 1 800 machostc 24109 Jan 23 11:45 php.ini
  drwxr-xr-x 27 800 machostc 896 Apr 24 05:05 sermons
  drwxr-xr-x 5 800 machostc 208 Apr 20 22:00 talua
  drwxr-xr-x 3 800 machostc 72 Mar 1 05:33 themes
  drwxr-xr-x 5 800 machostc 216 Feb 17 19:11 themob
  drwxr-xr-x 4 800 machostc 192 Mar 5 05:19 wpcc
  226-Options: -a -l
  226 53 matches total
  MDTM .htaccess
  213 20060131005256
  Remote file .htaccess modified 31/01/2006 12:52 AM, local file .htaccess modified 31/01/2006 12:52 AM, will not download
  MDTM 400.shtml
  213 20060123184529
  Remote file 400.shtml modified 23/01/2006 06:45 PM, local file 400.shtml modified 23/01/2006 06:45 PM, will not download
  MDTM 400.shtml-Copy
  213 20060223004436
  Remote file 400.shtml-Copy modified 23/02/2006 12:44 AM, local file 400.shtml-Copy modified 23/02/2006 12:44 AM, will not download
  MDTM 401.shtml
  213 20060123184529

  etc...

  Posted 17 years ago #

• Wayne Connor Member

  Yes, same if I download using the get command.

  Yes, I had the passive mode box checked, but the same issue arises with the box checked or unchecked.

  When the current download crashes, of which I just posted the start of the transcript, I'll post the end too.
  Thanks,
  Wayne

  Posted 17 years ago #

• Wayne Connor Member

  Here is the end of that log.

  257 "/public_html/evening/blog/2005/12" is your current location
  PASV
  227 Entering Passive Mode (70,103,189,80,121,9)
  ABOR
  500 ?
  PASV
  227 Entering Passive Mode (70,103,189,80,66,35)
  LIST -al
  150 Accepted data connection
  drwxr-xr-x 4 800 machostc 96 Feb 1 03:11 .
  drwxr-xr-x 8 800 machostc 224 Feb 1 03:11 ..
  drwxr-xr-x 2 800 machostc 80 Feb 1 03:11 21
  drwxr-xr-x 2 800 machostc 80 Feb 1 03:11 5
  226-Options: -a -l
  226 4 matches total
  CWD 21
  250 OK. Current directory is /public_html/evening/blog/2005/12/21
  PWD
  257 "/public_html/evening/blog/2005/12/21" is your current location
  PASV
  227 Entering Passive Mode (70,103,189,80,66,193)
  ABOR
  500 ?
  PASV
  227 Entering Passive Mode (70,103,189,80,210,196)
  ABOR
  500 ?
  PASV
  227 Entering Passive Mode (70,103,189,80,92,212)
  ABOR
  500 ?
  PORT 10,0,0,11,116,46
  501 Syntax error in IP address
  PORT 10,0,0,11,213,182
  501 Syntax error in IP address
  PORT 10,0,0,11,255,97
  501 Syntax error in IP address
  ftp_list: -30027 (state == RGET_GET_LIST)
  PWD
  421 Timeout - try typing a little faster next time
  ftp_pwd: -30001 (state == GETTING_WD)

  The actual fetch alert says:
  Fetch could not get the file list because there was a timeout trying to establish a passive mode data connection.

  Posted 17 years ago #

• Scott McGuire Administrator

  Hi,

  I'm sorry to say that we're not sure what's going on. It may be an issue with your NAT/router.

  If you can connect to the server using SFTP instead of FTP, that would most likely solve the problem.

  You could also use this AppleScript to repeatedly retry the Mirror connection until the download is completely successful:

  set done to false
  repeat while not done
  try
  with timeout of 3600 seconds
  tell application "Fetch"
  mirror url sourceURL to destFolder
  end tell
  end timeout
  set done to true
  end try
  end repeat

  Replace "sourceURL" with the URL to the remote folder you're trying to download, and "destFolder" with the full path to the local folder you're trying to download.

  Sorry not to have a better answer for you. Please let us know if you have further questions.

  Scott McGuire
  Fetch Softworks

  Posted 17 years ago #

• Wayne Connor Member

  If you can connect to the server using SFTP instead of FTP, that would most likely solve the problem.

  You are right this did solve the problem.

  Why would that be?

  Thanks very much.

  Nice script too.
  Although I won't have to use it thankfully.

  Wayne

  Posted 17 years ago #

• Scott McGuire Administrator

  Hi Wayne,

  We're glad to hear that fixed the problem.

  Differences in the way SFTP and FTP use the network tend to make SFTP more reliable; as a bonus it's also more secure.

  Best,

  Scott McGuire
  Fetch Softworks

  Posted 17 years ago #