Your best friend for file transfer.

Problem downloading very large number of files (>1000) (9 posts)
- Started 17 years ago by Wayne Connor
- Latest reply 17 years ago from Scott McGuire
-
Wayne Connor Member
-
Scott McGuire Administrator
Hi,
Sorry you're having problems.
We'd like some more information.
Could you please post the top part of the transcript, up until the Mirror operation begins?
Also, do you see the same problem if you try to download the folder containing all the files using the Get command instead of the Mirror command in Fetch 5?
Thanks,
Scott McGuire
Fetch Softworks -
Scott McGuire Administrator
Hi,
Here's another thing to try:
* Go to Fetch Preferences, and click the General tab.
* Make sure "Use passive mode transfers (PASV)" is checked.If it was unchecked, check it and then try the Mirror operation again. Let us know if this helps.
Thanks,
Scott McGuire
Fetch Softworks -
Wayne Connor Member
Sure,
First part of transcript:Fetch 5.0.5 (5A835) PowerPC running on Mac OS X 10.4.6 (8I127) PowerPC English
StuffIt Engine 0x702, StuffIt SDK Version StuffIt version 7.0.3
Partial serial FETCHED001-LN8A-K007 T
Update check skipped at 25/04/2006 04:05 PM (next check after 28/04/2006 07:04 AM)
Connecting to mac-host.com port 21 (OS X firewall is off) (25/4/06 4:05:58 PM)
Connected to 70.103.189.80 port 21 (25/4/06 4:05:58 PM)
220---------- Welcome to Pure-FTPd [TLS] ----------
220-You are user number 4 of 50 allowed.
220-Local time is now 00:05. Server port: 21.
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
USER machostc
331 User machostc OK. Password required
PASS
230-User machostc has group access to: machostc
230 OK. Current restricted directory is /
SYST
215 UNIX Type: L8
PWD
257 "/" is your current location
MACB ENABLE
500 Unknown command
CWD public_html/
250 OK. Current directory is /public_html
PWD
257 "/public_html" is your current location
PWD
257 "/public_html" is your current location
TYPE A
200 TYPE is now ASCII
PORT 10,0,0,11,230,216
200 PORT command successful
LIST -al
150 Connecting to port 35907
drwxr-xr-x 40 800 machostc 1456 Apr 18 18:51 .
drwx--x--- 15 800 12 872 Apr 25 00:02 ..
-rw-r--r-- 1 800 machostc 0 Jan 30 17:52 .htaccess
-rw-r--r-- 1 800 machostc 130 Jan 23 11:45 400.shtml
-rw-r--r-- 1 800 machostc 130 Feb 22 17:44 400.shtml-Copy
-rw-r--r-- 1 800 machostc 162 Jan 23 11:45 401.shtml
-rw-r--r-- 1 800 machostc 201 Jan 23 11:45 403.shtml
-rw-r--r-- 1 800 machostc 175 Feb 22 17:44 404.shtml
-rw-r--r-- 1 800 machostc 363 Jan 23 11:45 500.php
-rw-r--r-- 1 800 machostc 71 Jan 23 11:45 500.shtml
drwxr-xr-x 3 800 machostc 104 Feb 1 18:25 7barracks
drwxr-xr-x 4 800 machostc 128 Feb 1 17:56 7barrackscl
drwxr-xr-x 11 800 machostc 528 Feb 1 21:13 agcamp
drwxr-xr-x 4 800 machostc 280 Mar 1 05:41 avillion
drwxr-xr-x 3 800 machostc 192 Jan 31 19:18 avillion.dubbo.org
drwxr-xr-x 3 800 machostc 72 Feb 1 14:40 barracks
drwxr-xr-x 2 800 machostc 80 Jan 31 19:29 blog.dubbo.org
-rw-r--r-- 1 800 machostc 12497 Feb 1 17:57 blog.html
drwxr-xr-x 2 800 machostc 80 Feb 1 18:25 blog2
drwxr-xr-x 4 800 machostc 456 Feb 1 17:53 blogwp
drwxr-xr-x 9 800 machostc 672 Feb 5 04:36 calendar-old bad
drwxr-xr-x 2 800 machostc 48 Feb 2 15:36 cgi-bin
drwxr-xr-x 3 800 machostc 104 Feb 2 02:15 crazytest
-rw-r--r-- 1 800 machostc 1172 Jan 23 11:45 default.html
drwxr-xr-x 11 800 machostc 952 Mar 29 21:50 dpc
drwxr-xr-x 3 800 machostc 888 Apr 17 16:26 dragonboat
drwxr-xr-x 9 800 machostc 2040 Mar 27 15:46 dubbo
drwxr-xr-x 5 800 machostc 184 Mar 26 22:03 dubbo.nsw
drwxr-xr-x 9 800 machostc 1512 Feb 1 18:04 dubbonew
drwxr-xr-x 3 800 machostc 136 Feb 26 00:31 early
drwxr-xr-x 5 800 machostc 280 Feb 19 23:06 evening
-rw-r--r-- 1 800 machostc 822 Feb 16 20:18 favicon.ico
drwxr-xr-x 4 800 machostc 136 Jan 31 18:43 feva
drwxr-xr-x 11 800 machostc 840 Feb 20 03:08 forums
drwxr-xr-x 11 800 machostc 776 Feb 6 15:21 forumsold
drwxr-xr-x 2 800 machostc 264 Feb 22 17:45 henty
drwxr-xr-x 6 800 machostc 1080 Mar 23 06:34 howto
drwxr-xr-x 6 800 machostc 936 Feb 2 16:55 howto.old
drwxr-xr-x 5 800 machostc 880 Feb 2 18:02 howto.old2
drwxr-xr-x 3 800 machostc 880 Feb 2 15:33 howtoold
drwxr-xr-x 5 800 machostc 672 Feb 1 18:54 iblog
drwxr-xr-x 2 800 machostc 208 Mar 1 05:32 images
-rw-r--r-- 1 800 machostc 3987 Mar 1 05:32 index.html
drwxr-xr-x 3 800 machostc 344 Feb 17 04:20 korah
drwxr-xr-x 2 800 machostc 80 Feb 1 18:19 media
drwxr-xr-x 4 800 machostc 160 Feb 22 00:27 morning
drwxr-xr-x 13 800 machostc 2336 Feb 23 03:34 photos
-rw-r--r-- 1 800 machostc 24109 Jan 23 11:45 php.ini
drwxr-xr-x 27 800 machostc 896 Apr 24 05:05 sermons
drwxr-xr-x 5 800 machostc 208 Apr 20 22:00 talua
drwxr-xr-x 3 800 machostc 72 Mar 1 05:33 themes
drwxr-xr-x 5 800 machostc 216 Feb 17 19:11 themob
drwxr-xr-x 4 800 machostc 192 Mar 5 05:19 wpcc
226-Options: -a -l
226 53 matches total
MDTM .htaccess
213 20060131005256
Remote file .htaccess modified 31/01/2006 12:52 AM, local file .htaccess modified 31/01/2006 12:52 AM, will not download
MDTM 400.shtml
213 20060123184529
Remote file 400.shtml modified 23/01/2006 06:45 PM, local file 400.shtml modified 23/01/2006 06:45 PM, will not download
MDTM 400.shtml-Copy
213 20060223004436
Remote file 400.shtml-Copy modified 23/02/2006 12:44 AM, local file 400.shtml-Copy modified 23/02/2006 12:44 AM, will not download
MDTM 401.shtml
213 20060123184529etc...
-
Wayne Connor Member
Yes, same if I download using the get command.
Yes, I had the passive mode box checked, but the same issue arises with the box checked or unchecked.
When the current download crashes, of which I just posted the start of the transcript, I'll post the end too.
Thanks,
Wayne -
Wayne Connor Member
Here is the end of that log.
257 "/public_html/evening/blog/2005/12" is your current location
PASV
227 Entering Passive Mode (70,103,189,80,121,9)
ABOR
500 ?
PASV
227 Entering Passive Mode (70,103,189,80,66,35)
LIST -al
150 Accepted data connection
drwxr-xr-x 4 800 machostc 96 Feb 1 03:11 .
drwxr-xr-x 8 800 machostc 224 Feb 1 03:11 ..
drwxr-xr-x 2 800 machostc 80 Feb 1 03:11 21
drwxr-xr-x 2 800 machostc 80 Feb 1 03:11 5
226-Options: -a -l
226 4 matches total
CWD 21
250 OK. Current directory is /public_html/evening/blog/2005/12/21
PWD
257 "/public_html/evening/blog/2005/12/21" is your current location
PASV
227 Entering Passive Mode (70,103,189,80,66,193)
ABOR
500 ?
PASV
227 Entering Passive Mode (70,103,189,80,210,196)
ABOR
500 ?
PASV
227 Entering Passive Mode (70,103,189,80,92,212)
ABOR
500 ?
PORT 10,0,0,11,116,46
501 Syntax error in IP address
PORT 10,0,0,11,213,182
501 Syntax error in IP address
PORT 10,0,0,11,255,97
501 Syntax error in IP address
ftp_list: -30027 (state == RGET_GET_LIST)
PWD
421 Timeout - try typing a little faster next time
ftp_pwd: -30001 (state == GETTING_WD)The actual fetch alert says:
Fetch could not get the file list because there was a timeout trying to establish a passive mode data connection. -
Scott McGuire Administrator
Hi,
I'm sorry to say that we're not sure what's going on. It may be an issue with your NAT/router.
If you can connect to the server using SFTP instead of FTP, that would most likely solve the problem.
You could also use this AppleScript to repeatedly retry the Mirror connection until the download is completely successful:
set done to false
repeat while not done
try
with timeout of 3600 seconds
tell application "Fetch"
mirror url sourceURL to destFolder
end tell
end timeout
set done to true
end try
end repeatReplace "sourceURL" with the URL to the remote folder you're trying to download, and "destFolder" with the full path to the local folder you're trying to download.
Sorry not to have a better answer for you. Please let us know if you have further questions.
Scott McGuire
Fetch Softworks -
Wayne Connor Member
If you can connect to the server using SFTP instead of FTP, that would most likely solve the problem.
You are right this did solve the problem.
Why would that be?
Thanks very much.
Nice script too.
Although I won't have to use it thankfully.Wayne
-
Scott McGuire Administrator
Hi Wayne,
We're glad to hear that fixed the problem.
Differences in the way SFTP and FTP use the network tend to make SFTP more reliable; as a bonus it's also more secure.
Best,
Scott McGuire
Fetch Softworks
- Page 1
I am trying to mirror my whole website to my hard drive as a backup.
It is probably approx 800Mb and in the order of 5 - 10 thousand of files.
Approx 10-20 min in or so of downloading it will stop at a file, a different file each time. The file may be small or big.
I am using 10.4.3 and fetch 5, but had the same problems under 10.3 with fetch 4.
I have the same problem with passive mode on or off.
Both work for probably 100 or 200 or 300 files downloaded, then abort.
When I restart the mirror operation, it gets a bit more done.
I tried cuteftp (30 day trial) and no problems, so I don't think it's with the server, although this may have been a fluke.
It's reproducible, If I click my mirror icon, it will start mirroring, then die probably 10 - 20 min into the mirror operation.
From the gui, it appears to just freeze on the file, the dog keeps running, but the filename just stays there, then after a couple of minutes it aborts.
here is the last entry in the log,
example 1
226-File successfully transferred
226 0.000 seconds (measured here), 77.21 Mbytes per second
Transfer of wp-settings.php (7,988 bytes, 7,988 bytes/sec, 0:01 elapsed) completed at 22/4/06 9:48:38 AM
MDTM wp-trackback.php
213 20060217115256
SIZE wp-trackback.php
213 3211
MDTM wp-trackback.php
213 20060217115256
PORT 10,0,0,19,203,161
ABOR
Unsuccessful transfer of wp-trackback.php (0 bytes, 0 bytes/sec, 1:45 elapsed) stopped at 22/4/06 9:50:23 AM
ftp_retrieve: -30028 (state == RGET_RETRIEVING)
example 2
Transfer of header_06-over.jpg (1,167 bytes, 1,167 bytes/sec, 0:01 elapsed) completed at 21/4/06 11:32:41 PM
SIZE header_06.gif
213 369
MDTM header_06.gif
213 20060201021146
PORT 10,0,0,19,172,51
200 PORT command successful
RETR header_06.gif
Unsuccessful transfer of header_06.gif (0 bytes, 0 bytes/sec, 0:01 elapsed) stopped at 21/4/06 11:32:43 PM
ABOR
example 3
Transfer of theme-editor.php (4,270 bytes, 4,270 bytes/sec, 0:01 elapsed) completed at 21/4/06 11:40:38 PM
MDTM themes.php
213 20060201104229
SIZE themes.php
213 4387
MDTM themes.php
213 20060201104229
PORT 10,0,0,19,33,9
ABOR
Unsuccessful transfer of themes.php (0 bytes, 0 bytes/sec, 1:45 elapsed) stopped at 21/4/06 11:42:24 PM
ftp_retrieve: -30028 (state == RGET_RETRIEVING)
Posted 17 years ago #