Fetch Softworks

FTP Transfer Order (12 posts)

• Started 19 years ago by emjbork
• Latest reply 19 years ago from emjbork

• emjbork Member

  I must transfer a large number of files (,000) using Fetch FTP. I have done some preliminary tests as to the reliability of getting through all 40,000 and have found that it will be very unlikely to get them all without experiencing some type of error (connection lost, invalid file/directory name, path not found, etc.). Based upon my tests, I will only be able to get through about 700 - 1,000 at a time. But I cannot reliably anticipate where a failure might occur. This forces me to have to resend the entire batch ... as opposed to only those that were not successfully transferred. It appears that Fetch transfers the files randomly (As opposed to ordered by name, date, etc.) ... I am hoping this is not the case.

  Is there a transfer scheme that I can count on so that if there is an error I can look at the last file transferred and know which files are remaining?

  Is there a way to have Fetch continue to try and process the rest of the files and just log the files/folders it could not transfer?

  I would prefer to have Fetch transfer by name in order ... is there a preference that will allow me to do this?

  Any help would be appreciated.

  [This message has been edited by emjbork (edited 07-09-2004).]

  Posted 19 years ago #

• Jim Matthews Administrator

  Fetch will transfer the files in the order that the filesystem offers them (Fetch basically says to the OS: give me the first thing in this folder, now give me the next thing, etc.). That should be alphabetical, but I'm not sure.

  I would recommend using the Mirror Folder command in the Remote menu, and save the command as a mirror document (using the Save... button in the Mirror window). That way you can easily re-start the transfer if and when it fails, and Fetch will only transfer the files that have changed or haven't been transfered yet.

  Thanks,

  Jim Matthews
  Fetch Softworks

  Posted 19 years ago #

• emjbork Member

  I will try the mirror option. I don't see it in the 3.03 version. I will check our 4.x version later.

  Based upon your comment I am assuming that if a failure occurs, the transfer will display an error message and stop - similar to what it does with a regular transfer. I can then clear the message (correct the problem) and then restart the "mirror" process. All the files that were previously saved will be skipped and the transfer will pick up where it left off.

  Is there a log that is also created that will identify the error folder/files? It would be nice to be able to just clear the error and continue - fixing all the folders/files with issues at the very end and then making one more pass.

  Thanks for your suggestions.

  Posted 19 years ago #

• Jim Matthews Administrator

  There isn't a log, or a way to skip over a troublesome file. I agree that it would be useful to have those features.

  Thanks,

  Jim Matthews
  Fetch Softworks

  Posted 19 years ago #

• emjbork Member

  I have had a chance to try the Mirror feature in 4.0.3 and it does work to some extent. I have run several tests moving approximately 3,500 files, 750 folders (1.5G). These are my results:

  - Mirror does seem to ignore most files that have not changed.

  - For some reason there are a few blocks of files that it will download over and over again regardless (maybe 2% of files)??

  - As it is passing by the unchanged files, it will sometimes "hang" and I end up having to stop the Mirror process and restart (I have waited maybe 5 minutes waiting for Fetch to give an error message, but I have not seen one). Sometimes it will get through the entire list of files without hanging, but many times it will hang again before getting to the point of starting where it left off. Obviously this is more common the further into the process that it gets. I found that after around 650 files, this was quite common. Even during the download process it would hang now and again. This can make the transfer process excruciating … any help with configurations that might help would be a biggie since I have ,000 files to transfer.

  In future versions, it would be very helpful if Fetch maintained a list of files that failed (after x attempts based upon preference). If Fetch just continued through these failures and transferred everything else, then you could go back to the list and reinitiate the FTP transfer for those that failed (or delete them from the list). I have used other Windows FTP programs that have this type of functionality and it is a great feature.

  [This message has been edited by emjbork (edited 07-21-2004).]

  Posted 19 years ago #

• Jim Matthews Administrator

  Thanks for the feedback. We are trying to improve Fetch's mirror functionality, and your suggestions are very helpful.

  Thanks,

  Jim Matthews
  Fetch Softworks

  Posted 19 years ago #

• emjbork Member

  Thank you for this consideration. Is there anything we can configure now that might improve this "hanging" state?

  Posted 19 years ago #

• Jim Matthews Administrator

  Could you open the "Fetch Transcript" window when you are in this hung state and post the last 10 or 20 lines?

  Thanks,

  Jim Matthews
  Fetch Softworks

  Posted 19 years ago #

• emjbork Member

  The transcript below is typical. As you can see it cycles through the files that have already been transferred. The last thing the transcript shows before it hangs is "150 Opening ASCII mode data connection for". I then have to stop the transfer in order for Fetch to start responding again. You can see the 10 more lines it writes after the "Stop" is clicked.

  The total number of files transferred for this test was 1,858 in 399 folders. It takes about 4 minutes to cycle through all the previously transferred files (a handfull are retransferred?). It hung 3 times in different places before I was able to get all the way through.

  I have one more question ... if I make mulitple mirrors of the same set of files, will they all recognize if the same file was transferred already?

  ex. mirror 1 contains folder 1 which contains folders a, b, c
  mirror 2 contains folder a
  mirror 3 contains folder b
  mirror 4 contains folder c

  If I run mirrors 2 - 4 seperately ... can I run mirror 1 to make one final check that all the files transferred correctly?
  __________

  257 "/test/Artfiles 00-00300/00086-000" is current directory.
  PORT 10,243,6,158,70,234
  200 PORT command successful.
  LIST
  150 Opening ASCII mode data connection for /bin/ls.
  ---------- 1 owner group 395392 Jul 28 18:32 00086-000 (4-01)
  ---------- 1 owner group 395392 Jul 28 18:32 00086-000 (9-02)
  ---------- 1 owner group 79104 Jul 28 18:32 00086-000 (9-02).pdf
  226 Transfer complete.
  MDTM 00086-000 (9-02).pdf
  500 'MDTM 00086-000 (9-02).pdf': command not understood
  MDTM 00086-000 (9-02)
  500 'MDTM 00086-000 (9-02)': command not understood
  MDTM 00086-000 (4-01)
  500 'MDTM 00086-000 (4-01)': command not understood
  CWD /test/Artfiles 00-00300
  250 CWD command successful.
  CWD 00085-000
  250 CWD command successful.
  PWD
  257 "/test/Artfiles 00-00300/00085-000" is current directory.
  PORT 10,243,6,158,228,26
  200 PORT command successful.
  LIST
  150 Opening ASCII mode data connection for /bin/ls.
  ---------- 1 owner group 491136 Jul 28 18:32 00085-000
  ---------- 1 owner group 19712 Jul 28 18:32 00085-000.pdf
  226 Transfer complete.
  MDTM 00085-000.pdf
  500 'MDTM 00085-000.pdf': command not understood
  MDTM 00085-000
  500 'MDTM 00085-000': command not understood
  CWD /test/Artfiles 00-00300
  250 CWD command successful.
  CWD 00084-000
  250 CWD command successful.
  PWD
  257 "/test/Artfiles 00-00300/00084-000" is current directory.
  PORT 10,243,6,158,131,143
  200 PORT command successful.
  LIST
  150 Opening ASCII mode data connection for /bin/ls.
  ---------- 1 owner group 285824 Jul 28 18:32 00084-000
  ---------- 1 owner group 7680 Jul 28 18:32 00084-000.pdf
  226 Transfer complete.
  MDTM 00084-000.pdf
  500 'MDTM 00084-000.pdf': command not understood
  MDTM 00084-000
  500 'MDTM 00084-000': command not understood
  CWD /test/Artfiles 00-00300
  250 CWD command successful.
  CWD 00083-000
  250 CWD command successful.
  PWD
  257 "/test/Artfiles 00-00300/00083-000" is current directory.
  PORT 10,243,6,158,46,211
  200 PORT command successful.
  LIST
  150 Opening ASCII mode data connection for /bin/ls.
  ---------- 1 owner group 210560 Jul 28 18:32 00083-000(9-03)
  ---------- 1 owner group 27392 Jul 28 18:32 00083-000(9-03).pdf
  226 Transfer complete.
  MDTM 00083-000(9-03).pdf
  500 'MDTM 00083-000(9-03).pdf': command not understood
  MDTM 00083-000(9-03)
  500 'MDTM 00083-000(9-03)': command not understood
  CWD /test/Artfiles 00-00300
  250 CWD command successful.
  CWD 00081-000
  250 CWD command successful.
  PWD
  257 "/test/Artfiles 00-00300/00081-000" is current directory.
  PORT 10,243,6,158,63,157
  200 PORT command successful.
  LIST
  150 Opening ASCII mode data connection for /bin/ls.
  ---------- 1 owner group 436864 Jul 28 18:32 00081-000
  226 Transfer complete.
  MDTM 00081-000
  500 'MDTM 00081-000': command not understood
  CWD /test/Artfiles 00-00300
  250 CWD command successful.
  CWD 00279-000
  250 CWD command successful.
  PWD
  257 "/test/Artfiles 00-00300/00279-000" is current directory.
  PORT 10,243,6,158,93,25
  200 PORT command successful.
  LIST
  150 Opening ASCII mode data connection for /bin/ls.
  ABOR
  425 Can't open data connection.
  225 ABOR command successful.
  PORT 10,243,6,158,20,139
  200 PORT command successful.
  LIST
  150 Opening ASCII mode data connection for /bin/ls.
  ---------- 1 owner group 65408 Jul 28 18:32 00279-000
  ---------- 1 owner group 11776 Jul 28 18:32 00279-000.pdf
  226 Transfer complete.

  [This message has been edited by emjbork (edited 07-28-2004).]

  [This message has been edited by emjbork (edited 07-28-2004).]

  [This message has been edited by emjbork (edited 07-28-2004).]

  Posted 19 years ago #

• Jim Matthews Administrator

  Yes, you could run mirrors 2-4 and then run mirror 1 to confirm.

  You might try checking "Use passive mode transfers (PASV)" in the Firewall section of Fetch Preferences; some users have better reliability for large transfers with that option.

  Thanks,

  Jim Matthews
  Fetch Softworks

  Posted 19 years ago #

• emjbork Member

  Thanks for the suggestion, but turning PASV mode on doesn't seem to make any difference for me. Today was more frustrating than other days hanging about 5 times in a row after about 2 minutes into a recycle. Still always at "150 Opening ASCII mode data connection for /bin/ls" (different locations). I'll try again later on in the day to finish.

  I think my best plan of action is to break the job into small mirrors around 1,000 files each. This way, once it gets all the way through, I can forget about it and move on to the next block. It will just require more preparation.

  Posted 19 years ago #

• emjbork Member

  I found something interesting this afternoon when performing my retest. I have been performing all my tests with the transcript window open so I can watch the progress. The latest 3 test, I did not have this window open and I noticed that not only was the processing faster, but I have not experienced any "hang" states. I will not know until I perform more tests to see if my issue disappears, but I thought it was worth mentioning. Not sure why this would have an effect on transferring files, but funny things happen with 0's and 1's =;-)

  Posted 19 years ago #