Your best friend for file transfer.

Fetch application logoFetch

mirroring mirrors even though no modifications: images and html files. (8 posts)

  • Started 8 years ago by john12345
  • Latest reply 8 years ago from john12345
  • john12345 Member

    i'm using my fetch 4.0.3 to mirror a site back to my hard drive. the files on the site were uploaded from my hard drive and haven't been modified since (apart from one or two). it seems every single file is being downloaded therefore fetch deems all the files on the server different/modified. i've compared two versions (the "new" and "old") of same html file with each other -- an html file -- using hexdump and yes they are different: all the line breaks in the "old" version their hex values are 0x0a and all the ones in the "new" version are 0x0d.

    i've used the command line utility diff on two versions of the same png (the "old" and "new") and diff says they're the same.

    so what's going on here? how have the html files mysteriously all suddenly got 0x0d line breaks rather than the original 0x0a line breaks and why are seemingly identicle images deemed different therefore being downloaded?

    how can i stop not really modified files being deemed modified therefore being downloaded/morrored?

    thanks.

    Posted 8 years ago #

  • Scott McGuire Administrator

    Hi,

    Regarding the line endings:

    When you upload files using FTP in Fetch - using either Put or Mirror to upload them - Fetch translates the line endings of text files to match the line endings used on the server. Usually this is what you want to happen, because the server may not display or read text files properly if they have a line ending that is not what the server expects. (And HTML files are, of course, text files.)

    When you download text files, Fetch translates line endings to the kind of line ending specified in the Download pane of the Fetch Preferences.

    So, it sounds like your files started with a line ending that is different from the one specified in the Download pane of Fetch Preferences. One way to avoid ending up with different line endings is to change the "Text file line endings" setting in the Download pane to match what your files start with.

    Another option is to transfer text files using both the Raw Data upload format and the Binary download mode. Then no line ending translation will be performed.

    Mirror operations use the default download mode and upload format as specified in the Download and Upload panes in the Preferences; so, to always transfer files in Binary, set those preferences to Binary and Raw Data. ("Raw Data" is the same as "Binary," really.)

    Most web browsers will display HTML files correctly regardless of the line endings, but you would want to check to make sure your files do display correctly after you do this.

    Regarding the files being re-downloaded even though you haven't changed them:

    The Fetch Mirror command decides whether files have changed based on modification dates, among other criteria.

    Fetch does not preserve modification dates of files on upload. So, when you first upload your files, the copies of the files on the server will have a modification date that matches the time you uploaded them, not the last time you modified them on your Macintosh. That means the files on the server will have a newer modification date than the local files on your Macintosh.

    So, the first time you mirror the files back to your Macintosh, all the files will be copied, because even though they are otherwise identical, the files on the server have a newer modification date. However, once this is done, the files on the Macintosh will now have the same modification date as on the server, and from then on, only files that have changed on either end should be transferred when you mirror. (If you mirror the files back to your Macintosh twice in a row, on the second mirror, nothing should be transferred.)

    If this is not what you are experiencing, please let us know.

    We hope to support preserving modification dates of uploads in a future release of Fetch.

    I hope this helps explain what's going on.

    Thanks,

    Scott McGuire
    Fetch Softworks

    Posted 8 years ago #

  • john12345 Member

    hello,

    thank you for the reply.

    > When you upload files using FTP in Fetch - using either Put or Mirror to upload them - Fetch translates the line endings of text files to match the line endings used on the server. Usually this is what you want to happen,

    didn't realise fetch did that. i had chosen unix line ending in my text editor as i assumed that's what would be best for the server as it's an apache one.

    > When you download text files, Fetch translates line endings to the kind of line ending specified in the Download pane of the Fetch Preferences.

    right, yes i didn't know about that setting. it was set to Mac (CR), which is old mac not new mac so not unix so that's what was happening.

    i was starting to think the line end difference was probably a red herring so far as the downloading/mirroring issue went as i knew fetch doesn't check the contents of the files so i knew something didn't quite add up there.

    > ... So, when you first upload your files, the copies of the files on the server will have a modification date that matches the time you uploaded them, not the last time you modified them on your Macintosh. That means the files on the server will have a newer modification date than the local files on your Macintosh. ...

    i see -- yes that explains it exactly.

    > If this is not what you are experiencing, please let us know.

    i haven't checked that fully but i'm sure that's what's happening -- completely tallies with the situation and makes complete sense.

    > We hope to support preserving modification dates of uploads in a future release of Fetch.

    yes that would be useful.

    > I hope this helps explain what's going on.

    yes totally -- thanks very much.

    a side question vaguely related to the end of line changing: is there anyway to do an analguous thing as what fetch does with line endings to file names? what i mean is this: in the site on my drive i have a file titled "_.htaccess". if it didn't have the underscore in front of it it'd be invisible and it would probably cause problems trying to edit it when it starts with a . and i'd forget about it, but on the server it should be called .htaccess as i'm sure you know. is there anyway to do the same kind of flip backwards and forwards (as you described with the line ending as you upload and donwload) with the file name with fetch: as it's uploaded the _.htaccess file gets changed to .htaccess, and as it's downloaded it gets changed to _.htaccess? if that's not possible i reckon that'd be quite useful, so i'd like to suggest that feature.

    thanks.

    Posted 8 years ago #

  • john12345 Member

    or, does anyone know if there's a way to say "don't make the file .htaccess invisible in this particular folder" in os x somehow?

    thanks.

    Posted 8 years ago #

  • Scott McGuire Administrator

    > > When you upload files using FTP in Fetch - using either
    > > Put or Mirror to upload them - Fetch translates the
    > > line endings of text files to match the line endings
    > > used on the server.
    >
    > didn't realise fetch did that. i had chosen unix line
    > ending in my text editor as i assumed that's what would
    > be best for the server as it's an apache one.

    Right, this shouldn't be a problem then - you have Unix locally, and Unix on the server, so probably no line ending translation is taking place on upload.

    > > When you download text files, Fetch translates line
    > > endings to the kind of line ending specified in the
    > > Download pane of the Fetch Preferences.
    >
    > right, yes i didn't know about that setting. it was set
    > to Mac (CR), which is old mac not new mac so not unix so
    > that's what was happening.

    Yes. Fetch 4.0.3 defaults to Mac (CR) line endings; in Fetch 5 we changed the default to Unix (LF) line endings.

    > i was starting to think the line end difference was
    > probably a red herring so far as the
    > downloading/mirroring issue went as i knew fetch doesn't
    > check the contents of the files so i knew something
    > didn't quite add up there.

    Right. It was really the modification dates.

    > > I hope this helps explain what's going on.
    >
    > yes totally -- thanks very much.

    You're welcome.

    > is there anyway to do the same kind of flip backwards and
    > forwards (as you described with the line ending as you
    > upload and donwload) with the file name with fetch: as
    > it's uploaded the _.htaccess file gets changed to
    > .htaccess, and as it's downloaded it gets changed to
    > _.htaccess? if that's not possible i reckon that'd be
    > quite useful, so i'd like to suggest that feature.

    It's not possible to do that directly in Fetch now, however, you could probably accomplish it by running an AppleScript to rename the files after they've been uploaded. (Even easier than AppleScript would be the new Automator stuff in Fetch 5.1, but I realize you're running 10.2.8 and Automator requires 10.4.)

    > or, does anyone know if there's a way to say "don't make
    > the file .htaccess invisible in this particular folder"
    > in os x somehow?

    You can sort of do that. You can tell the Finder to show all the files it normally hides in all folders, but you'll see a lot more than just the .htaccess files, so this may not be what you want. To do this, open the Terminal application, and enter the line:

    defaults write com.apple.Finder AppleShowAllFiles true

    Then log out and log back in or relaunch the Finder, and then the Finder should show you all the files it normally hides.

    To turn it off, type:

    defaults write com.apple.Finder AppleShowAllFiles false

    in Terminal, and the log out and log in or restart the Finder again.

    Best,

    Scott McGuire
    Fetch Softworks

    Posted 8 years ago #

  • john12345 Member

    ok thanks very much for all that info about hidden files etc. much appreciated.

    the problem with the file modification dates not being maintained by fetch is a bigger/worse one than i realised -- it makes syncing folders both ways not feasible unless you don't mind the whole folder being up/downloaded each time you sync to the server or to your hd. not good.

    i'm working on a site that other people may modify so i was trying to work out some way to handle not oeverwriting each other's work and i thought that fetch's mirroring would help but i don't think it will much because of the file modifications problem.

    my idea was to sync server -> my hard drive. make two copies (so have three versions of the site on my drive), spend time modifying one of the three copies. finish work. sync again from server -> one of the site copies (not the one i'd been working on obviously). use diff to see if all the files are still the same (compare the two site version on my hd -- not the one i've been modifiying). if diff reports no changes, sync from the modified site version on my hard drive -> server, but this doesn't work. doesn't work as in all files are transfered when syncing not just the modified ones. what a shame. any ideas how to handle that in a more automatic way rather than having to remember what files i've modified and uploading them individually (which is pretty error prone to say the least)?

    thanks.

    Posted 8 years ago #

  • Scott McGuire Administrator

    Hi,

    I have just done the following in this order:

    Mirrored from folder "web" on server -> local folder
    Made duplicate of local folder with the Finder
    Made changes to files in duplicate folder
    Mirrored from duplicate folder -> folder "web" on server

    Which is, I believe, essentially what you did (the mirroring to a third folder, and diffing of other folders should not be relevant because ultimately you only changed files in the duplicate folder).

    Only the changed files were uploaded.

    The modificaiton dates are the key; either you must have worked on a folder that had not been mirrroed from the server first, or the modification dates of all the files in the folder you worked on had all been changed by something. Once you mirror from the server to your Mac, the whole folder should not be uploaded again when you mirror back from the Mac to the server.

    I'd look at the modification dates carefully to see at which step they changed.

    Thanks,

    Scott McGuire
    Fetch Softworks

    Posted 8 years ago #

  • john12345 Member

    hello,

    thanks for the reply.

    yes i've just done a small test and it completely corresponded with what you said so i'm not sure what happened previously with the full site. i must have made some sort of mistake although i can't think what. doesn't matter though so long as i don't make it again. so the modification date only gets inadvertently changed when saving from local to server, not the other way round. the mod dates are preserved when going from server to local.

    ok hopefully what you described and what happened in the test will happen with the proper site, which i guess it must.

    thanks very much for your help.

    Posted 8 years ago #

Reply

  • Or nickname, if you prefer.
  • This will be kept confidential.
  • This is to ensure that you’re a person, not a spambot.