File Offsets for SCP (patch revised yet again)

[rapier]

I actually agree but have a couple of reservations. The first is the 
user issue. Getting users to do the right thing, especially after 
they've been doing something else for years and years, is always a 
challenge. If SFTP can be made to act like SCP then it's going to be 
critical to just get rid of SCP entirely and do the ln -s trick.

The second is that the messaging window size is a limitation for the 
users I'm supporting (major research institutions usually connected via 
multiple gig or 10Ge connections). Anything that ends up impeding 
maximum performance is going to be problematic (that is why I developed 
the HPN-SSH patch). At some point it may make sense to dynamically grow 
the message window size (because expecting users to set it correctly is 
just not going to work (thats why we did the work on the auto-tuning 
receive buffers)). I may get the funding to do that but I just don't 
know at this point. So right now I'm going with the quick and dirty fix :)

Basically, network performance is where my focus is rather than SSH in 
and of itself.

Chris

Ben Lindstrom wrote:
> I knows it is on the list to teach sftp to act like scp (using sftp protocol) if ln -s sftp scp ... That may be a better investment.
> 
> - Ben
> 
> 
> On Dec 6, 2010, at 4:03 PM, rapier wrote:
> 
>> This version of the patch now does partial file transfers in both directions. -A specifies the start byte -L the length. As systems might have the patched scp running as a secondary server I have also include a method to provide a path to the desired scp executable using the -a option.
>>
>> Again, I don't know if this will be useful to anyone else. We're going to be using this to do parallelized transfers of large (250GB+) files (using the HPN patch set as well). We're basically trying to avoid the slow start penalties on very fast throughputs by splitting the transfer up.
>>
>> I know scp is no longer supported in anyway except to maintain the versioning. Unfortunately, a huge number of people still use scp in preference to scp. I'm an enabler. :\
>>
>> chris
>>
>>
>>
>> rapier wrote:
>>> Just a follow up patch. You now just specify the byte length you want to transfer and a little more error checking. Again, don't know if this would be useful to anyone but I thought I'd share just in case. Also, just to make sure this is clear, this only works on the source side.
>>> --- ../canonical-openssh5.6/scp.c    2010-07-01 23:37:33.000000000 -0400
>>> +++ ./scp.c    2010-11-22 19:48:46.000000000 -0500
>>> @@ -302,6 +302,8 @@ struct passwd *pwd;
>>> uid_t userid;
>>> int errs, remin, remout;
>>> int pflag, iamremote, iamrecursive, targetshouldbedirectory;
>>> +// user requested file offset and amount of data to xfer
>>> +double fd_offset, fd_length;
>>> #define    CMDNEEDS    64
>>> char cmd[CMDNEEDS];        /* must hold "rcp -r -p -d\0" */
>>> @@ -324,6 +326,9 @@ main(int argc, char **argv)
>>>     extern char *optarg;
>>>     extern int optind;
>>> +    fd_length = 0;
>>> +    fd_offset = 0;
>>> +
>>>     /* Ensure that fds 0, 1 and 2 are open or directed to /dev/null */
>>>     sanitise_stdfd();
>>> @@ -344,7 +349,7 @@ main(int argc, char **argv)
>>>     addargs(&args, "-oClearAllForwardings yes");
>>>     fflag = tflag = 0;
>>> -    while ((ch = getopt(argc, argv, "dfl:prtvBCc:i:P:q1246S:o:F:")) != -1)
>>> +    while ((ch = getopt(argc, argv, "dfl:prtvBCc:i:P:q1246S:o:F:A:L:")) != -1)
>>>         switch (ch) {
>>>         /* User-visible flags. */
>>>         case '1':
>>> @@ -407,6 +412,16 @@ main(int argc, char **argv)
>>>             setmode(0, O_BINARY);
>>> #endif
>>>             break;
>>> +        case 'A':
>>> +            fd_offset = strtod(optarg, &endp);
>>> +            if (fd_offset < 0 || *endp != '\0')
>>> +                                usage();
>>> +            break;
>>> +        case 'L':
>>> +            fd_length = strtod(optarg, &endp);
>>> +            if (fd_length < 0 || *endp != '\0')
>>> +                                usage();
>>> +            break;
>>>         default:
>>>             usage();
>>>         }
>>> @@ -680,6 +695,34 @@ syserr:            run_err("%s: %s", name, strerr
>>>             run_err("%s: %s", name, "Negative file size");
>>>             goto next;
>>>         }
>>> +
>>> +        // if the offset is greater than the file size
>>> +        // we can't do this
>>> +        if (fd_offset > stb.st_size) {
>>> +            run_err("Offset greater than file size");
>>> +            goto next;
>>> +        }
>>> +       +        // if the offset plus the requested length are greater
>>> +        // than the file size then we have to quit
>>> +        if (fd_length + fd_offset > stb.st_size) {
>>> +            run_err("Length greater than file size");
>>> +            goto next;
>>> +        }
>>> +
>>> +        // if the user only has an offset then we reduce
>>> +        // the file size by the offset. if they have a length
>>> +        // then we use stb.st_size and the length
>>> +        // to determine the actual stopping point
>>> +        if (fd_length != 0) {
>>> +            stb.st_size -= (stb.st_size - fd_length);
>>> +        } else {
>>> +            stb.st_size -= fd_offset;
>>> +        }
>>> +       +        // seek to the position as requested by user
>>> +        lseek (fd, fd_offset, SEEK_SET);
>>> +
>>>         unset_nonblock(fd);
>>>         switch (stb.st_mode & S_IFMT) {
>>>         case S_IFREG:
>>> _______________________________________________
>>> openssh-unix-dev mailing list
>>> openssh-unix-dev at mindrot.org
>>> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>> _______________________________________________
>> openssh-unix-dev mailing list
>> openssh-unix-dev at mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>