Your best friend for file transfer.

Fetch application logoFetch

Problem downloading very large number of files (>1000) (9 posts)

  • Started 17 years ago by Wayne Connor
  • Latest reply 17 years ago from Scott McGuire
  • Wayne Connor Member

    I am trying to mirror my whole website to my hard drive as a backup.
    It is probably approx 800Mb and in the order of 5 - 10 thousand of files.

    Approx 10-20 min in or so of downloading it will stop at a file, a different file each time. The file may be small or big.

    I am using 10.4.3 and fetch 5, but had the same problems under 10.3 with fetch 4.

    I have the same problem with passive mode on or off.
    Both work for probably 100 or 200 or 300 files downloaded, then abort.
    When I restart the mirror operation, it gets a bit more done.

    I tried cuteftp (30 day trial) and no problems, so I don't think it's with the server, although this may have been a fluke.

    It's reproducible, If I click my mirror icon, it will start mirroring, then die probably 10 - 20 min into the mirror operation.

    From the gui, it appears to just freeze on the file, the dog keeps running, but the filename just stays there, then after a couple of minutes it aborts.

    here is the last entry in the log,
    example 1

    226-File successfully transferred
    226 0.000 seconds (measured here), 77.21 Mbytes per second
    Transfer of wp-settings.php (7,988 bytes, 7,988 bytes/sec, 0:01 elapsed) completed at 22/4/06 9:48:38 AM
    MDTM wp-trackback.php
    213 20060217115256
    SIZE wp-trackback.php
    213 3211
    MDTM wp-trackback.php
    213 20060217115256
    PORT 10,0,0,19,203,161
    ABOR
    Unsuccessful transfer of wp-trackback.php (0 bytes, 0 bytes/sec, 1:45 elapsed) stopped at 22/4/06 9:50:23 AM
    ftp_retrieve: -30028 (state == RGET_RETRIEVING)

    example 2
    Transfer of header_06-over.jpg (1,167 bytes, 1,167 bytes/sec, 0:01 elapsed) completed at 21/4/06 11:32:41 PM
    SIZE header_06.gif
    213 369
    MDTM header_06.gif
    213 20060201021146
    PORT 10,0,0,19,172,51
    200 PORT command successful
    RETR header_06.gif
    Unsuccessful transfer of header_06.gif (0 bytes, 0 bytes/sec, 0:01 elapsed) stopped at 21/4/06 11:32:43 PM
    ABOR

    example 3
    Transfer of theme-editor.php (4,270 bytes, 4,270 bytes/sec, 0:01 elapsed) completed at 21/4/06 11:40:38 PM
    MDTM themes.php
    213 20060201104229
    SIZE themes.php
    213 4387
    MDTM themes.php
    213 20060201104229
    PORT 10,0,0,19,33,9
    ABOR
    Unsuccessful transfer of themes.php (0 bytes, 0 bytes/sec, 1:45 elapsed) stopped at 21/4/06 11:42:24 PM
    ftp_retrieve: -30028 (state == RGET_RETRIEVING)

    Posted 17 years ago #

  • Scott McGuire Administrator

    Hi,

    Sorry you're having problems.

    We'd like some more information.

    Could you please post the top part of the transcript, up until the Mirror operation begins?

    Also, do you see the same problem if you try to download the folder containing all the files using the Get command instead of the Mirror command in Fetch 5?

    Thanks,

    Scott McGuire
    Fetch Softworks

    Posted 17 years ago #

  • Scott McGuire Administrator

    Hi,

    Here's another thing to try:

    * Go to Fetch Preferences, and click the General tab.
    * Make sure "Use passive mode transfers (PASV)" is checked.

    If it was unchecked, check it and then try the Mirror operation again. Let us know if this helps.

    Thanks,

    Scott McGuire
    Fetch Softworks

    Posted 17 years ago #

  • Wayne Connor Member

    Sure,
    First part of transcript:

    Fetch 5.0.5 (5A835) PowerPC running on Mac OS X 10.4.6 (8I127) PowerPC English
    StuffIt Engine 0x702, StuffIt SDK Version StuffIt version 7.0.3
    Partial serial FETCHED001-LN8A-K007 T
    Update check skipped at 25/04/2006 04:05 PM (next check after 28/04/2006 07:04 AM)
    Connecting to mac-host.com port 21 (OS X firewall is off) (25/4/06 4:05:58 PM)
    Connected to 70.103.189.80 port 21 (25/4/06 4:05:58 PM)
    220---------- Welcome to Pure-FTPd [TLS] ----------
    220-You are user number 4 of 50 allowed.
    220-Local time is now 00:05. Server port: 21.
    220-IPv6 connections are also welcome on this server.
    220 You will be disconnected after 15 minutes of inactivity.
    USER machostc
    331 User machostc OK. Password required
    PASS
    230-User machostc has group access to: machostc
    230 OK. Current restricted directory is /
    SYST
    215 UNIX Type: L8
    PWD
    257 "/" is your current location
    MACB ENABLE
    500 Unknown command
    CWD public_html/
    250 OK. Current directory is /public_html
    PWD
    257 "/public_html" is your current location
    PWD
    257 "/public_html" is your current location
    TYPE A
    200 TYPE is now ASCII
    PORT 10,0,0,11,230,216
    200 PORT command successful
    LIST -al
    150 Connecting to port 35907
    drwxr-xr-x 40 800 machostc 1456 Apr 18 18:51 .
    drwx--x--- 15 800 12 872 Apr 25 00:02 ..
    -rw-r--r-- 1 800 machostc 0 Jan 30 17:52 .htaccess
    -rw-r--r-- 1 800 machostc 130 Jan 23 11:45 400.shtml
    -rw-r--r-- 1 800 machostc 130 Feb 22 17:44 400.shtml-Copy
    -rw-r--r-- 1 800 machostc 162 Jan 23 11:45 401.shtml
    -rw-r--r-- 1 800 machostc 201 Jan 23 11:45 403.shtml
    -rw-r--r-- 1 800 machostc 175 Feb 22 17:44 404.shtml
    -rw-r--r-- 1 800 machostc 363 Jan 23 11:45 500.php
    -rw-r--r-- 1 800 machostc 71 Jan 23 11:45 500.shtml
    drwxr-xr-x 3 800 machostc 104 Feb 1 18:25 7barracks
    drwxr-xr-x 4 800 machostc 128 Feb 1 17:56 7barrackscl
    drwxr-xr-x 11 800 machostc 528 Feb 1 21:13 agcamp
    drwxr-xr-x 4 800 machostc 280 Mar 1 05:41 avillion
    drwxr-xr-x 3 800 machostc 192 Jan 31 19:18 avillion.dubbo.org
    drwxr-xr-x 3 800 machostc 72 Feb 1 14:40 barracks
    drwxr-xr-x 2 800 machostc 80 Jan 31 19:29 blog.dubbo.org
    -rw-r--r-- 1 800 machostc 12497 Feb 1 17:57 blog.html
    drwxr-xr-x 2 800 machostc 80 Feb 1 18:25 blog2
    drwxr-xr-x 4 800 machostc 456 Feb 1 17:53 blogwp
    drwxr-xr-x 9 800 machostc 672 Feb 5 04:36 calendar-old bad
    drwxr-xr-x 2 800 machostc 48 Feb 2 15:36 cgi-bin
    drwxr-xr-x 3 800 machostc 104 Feb 2 02:15 crazytest
    -rw-r--r-- 1 800 machostc 1172 Jan 23 11:45 default.html
    drwxr-xr-x 11 800 machostc 952 Mar 29 21:50 dpc
    drwxr-xr-x 3 800 machostc 888 Apr 17 16:26 dragonboat
    drwxr-xr-x 9 800 machostc 2040 Mar 27 15:46 dubbo
    drwxr-xr-x 5 800 machostc 184 Mar 26 22:03 dubbo.nsw
    drwxr-xr-x 9 800 machostc 1512 Feb 1 18:04 dubbonew
    drwxr-xr-x 3 800 machostc 136 Feb 26 00:31 early
    drwxr-xr-x 5 800 machostc 280 Feb 19 23:06 evening
    -rw-r--r-- 1 800 machostc 822 Feb 16 20:18 favicon.ico
    drwxr-xr-x 4 800 machostc 136 Jan 31 18:43 feva
    drwxr-xr-x 11 800 machostc 840 Feb 20 03:08 forums
    drwxr-xr-x 11 800 machostc 776 Feb 6 15:21 forumsold
    drwxr-xr-x 2 800 machostc 264 Feb 22 17:45 henty
    drwxr-xr-x 6 800 machostc 1080 Mar 23 06:34 howto
    drwxr-xr-x 6 800 machostc 936 Feb 2 16:55 howto.old
    drwxr-xr-x 5 800 machostc 880 Feb 2 18:02 howto.old2
    drwxr-xr-x 3 800 machostc 880 Feb 2 15:33 howtoold
    drwxr-xr-x 5 800 machostc 672 Feb 1 18:54 iblog
    drwxr-xr-x 2 800 machostc 208 Mar 1 05:32 images
    -rw-r--r-- 1 800 machostc 3987 Mar 1 05:32 index.html
    drwxr-xr-x 3 800 machostc 344 Feb 17 04:20 korah
    drwxr-xr-x 2 800 machostc 80 Feb 1 18:19 media
    drwxr-xr-x 4 800 machostc 160 Feb 22 00:27 morning
    drwxr-xr-x 13 800 machostc 2336 Feb 23 03:34 photos
    -rw-r--r-- 1 800 machostc 24109 Jan 23 11:45 php.ini
    drwxr-xr-x 27 800 machostc 896 Apr 24 05:05 sermons
    drwxr-xr-x 5 800 machostc 208 Apr 20 22:00 talua
    drwxr-xr-x 3 800 machostc 72 Mar 1 05:33 themes
    drwxr-xr-x 5 800 machostc 216 Feb 17 19:11 themob
    drwxr-xr-x 4 800 machostc 192 Mar 5 05:19 wpcc
    226-Options: -a -l
    226 53 matches total
    MDTM .htaccess
    213 20060131005256
    Remote file .htaccess modified 31/01/2006 12:52 AM, local file .htaccess modified 31/01/2006 12:52 AM, will not download
    MDTM 400.shtml
    213 20060123184529
    Remote file 400.shtml modified 23/01/2006 06:45 PM, local file 400.shtml modified 23/01/2006 06:45 PM, will not download
    MDTM 400.shtml-Copy
    213 20060223004436
    Remote file 400.shtml-Copy modified 23/02/2006 12:44 AM, local file 400.shtml-Copy modified 23/02/2006 12:44 AM, will not download
    MDTM 401.shtml
    213 20060123184529

    etc...

    Posted 17 years ago #

  • Wayne Connor Member

    Yes, same if I download using the get command.

    Yes, I had the passive mode box checked, but the same issue arises with the box checked or unchecked.

    When the current download crashes, of which I just posted the start of the transcript, I'll post the end too.
    Thanks,
    Wayne

    Posted 17 years ago #

  • Wayne Connor Member

    Here is the end of that log.

    257 "/public_html/evening/blog/2005/12" is your current location
    PASV
    227 Entering Passive Mode (70,103,189,80,121,9)
    ABOR
    500 ?
    PASV
    227 Entering Passive Mode (70,103,189,80,66,35)
    LIST -al
    150 Accepted data connection
    drwxr-xr-x 4 800 machostc 96 Feb 1 03:11 .
    drwxr-xr-x 8 800 machostc 224 Feb 1 03:11 ..
    drwxr-xr-x 2 800 machostc 80 Feb 1 03:11 21
    drwxr-xr-x 2 800 machostc 80 Feb 1 03:11 5
    226-Options: -a -l
    226 4 matches total
    CWD 21
    250 OK. Current directory is /public_html/evening/blog/2005/12/21
    PWD
    257 "/public_html/evening/blog/2005/12/21" is your current location
    PASV
    227 Entering Passive Mode (70,103,189,80,66,193)
    ABOR
    500 ?
    PASV
    227 Entering Passive Mode (70,103,189,80,210,196)
    ABOR
    500 ?
    PASV
    227 Entering Passive Mode (70,103,189,80,92,212)
    ABOR
    500 ?
    PORT 10,0,0,11,116,46
    501 Syntax error in IP address
    PORT 10,0,0,11,213,182
    501 Syntax error in IP address
    PORT 10,0,0,11,255,97
    501 Syntax error in IP address
    ftp_list: -30027 (state == RGET_GET_LIST)
    PWD
    421 Timeout - try typing a little faster next time
    ftp_pwd: -30001 (state == GETTING_WD)

    The actual fetch alert says:
    Fetch could not get the file list because there was a timeout trying to establish a passive mode data connection.

    Posted 17 years ago #

  • Scott McGuire Administrator

    Hi,

    I'm sorry to say that we're not sure what's going on. It may be an issue with your NAT/router.

    If you can connect to the server using SFTP instead of FTP, that would most likely solve the problem.

    You could also use this AppleScript to repeatedly retry the Mirror connection until the download is completely successful:

    set done to false
    repeat while not done
    try
    with timeout of 3600 seconds
    tell application "Fetch"
    mirror url sourceURL to destFolder
    end tell
    end timeout
    set done to true
    end try
    end repeat

    Replace "sourceURL" with the URL to the remote folder you're trying to download, and "destFolder" with the full path to the local folder you're trying to download.

    Sorry not to have a better answer for you. Please let us know if you have further questions.

    Scott McGuire
    Fetch Softworks

    Posted 17 years ago #

  • Wayne Connor Member

    If you can connect to the server using SFTP instead of FTP, that would most likely solve the problem.

    You are right this did solve the problem.

    Why would that be?

    Thanks very much.

    Nice script too.
    Although I won't have to use it thankfully.

    Wayne

    Posted 17 years ago #

  • Scott McGuire Administrator

    Hi Wayne,

    We're glad to hear that fixed the problem.

    Differences in the way SFTP and FTP use the network tend to make SFTP more reliable; as a bonus it's also more secure.

    Best,

    Scott McGuire
    Fetch Softworks

    Posted 17 years ago #

Reply

  • Or nickname, if you prefer.
  • This will be kept confidential.
  • This is to ensure that you’re a person, not a spambot.