File Offsets for SCP (patch revised yet again)

[Ben Lindstrom]

I knows it is on the list to teach sftp to act like scp (using sftp protocol) if ln -s sftp scp ... That may be a better investment.

- Ben


On Dec 6, 2010, at 4:03 PM, rapier wrote:

> This version of the patch now does partial file transfers in both directions. -A specifies the start byte -L the length. As systems might have the patched scp running as a secondary server I have also include a method to provide a path to the desired scp executable using the -a option.
> 
> Again, I don't know if this will be useful to anyone else. We're going to be using this to do parallelized transfers of large (250GB+) files (using the HPN patch set as well). We're basically trying to avoid the slow start penalties on very fast throughputs by splitting the transfer up.
> 
> I know scp is no longer supported in anyway except to maintain the versioning. Unfortunately, a huge number of people still use scp in preference to scp. I'm an enabler. :\
> 
> chris
> 
> 
> 
> rapier wrote:
>> Just a follow up patch. You now just specify the byte length you want to transfer and a little more error checking. Again, don't know if this would be useful to anyone but I thought I'd share just in case. Also, just to make sure this is clear, this only works on the source side.
>> --- ../canonical-openssh5.6/scp.c    2010-07-01 23:37:33.000000000 -0400
>> +++ ./scp.c    2010-11-22 19:48:46.000000000 -0500
>> @@ -302,6 +302,8 @@ struct passwd *pwd;
>> uid_t userid;
>> int errs, remin, remout;
>> int pflag, iamremote, iamrecursive, targetshouldbedirectory;
>> +// user requested file offset and amount of data to xfer
>> +double fd_offset, fd_length;
>> #define    CMDNEEDS    64
>> char cmd[CMDNEEDS];        /* must hold "rcp -r -p -d\0" */
>> @@ -324,6 +326,9 @@ main(int argc, char **argv)
>>     extern char *optarg;
>>     extern int optind;
>> +    fd_length = 0;
>> +    fd_offset = 0;
>> +
>>     /* Ensure that fds 0, 1 and 2 are open or directed to /dev/null */
>>     sanitise_stdfd();
>> @@ -344,7 +349,7 @@ main(int argc, char **argv)
>>     addargs(&args, "-oClearAllForwardings yes");
>>     fflag = tflag = 0;
>> -    while ((ch = getopt(argc, argv, "dfl:prtvBCc:i:P:q1246S:o:F:")) != -1)
>> +    while ((ch = getopt(argc, argv, "dfl:prtvBCc:i:P:q1246S:o:F:A:L:")) != -1)
>>         switch (ch) {
>>         /* User-visible flags. */
>>         case '1':
>> @@ -407,6 +412,16 @@ main(int argc, char **argv)
>>             setmode(0, O_BINARY);
>> #endif
>>             break;
>> +        case 'A':
>> +            fd_offset = strtod(optarg, &endp);
>> +            if (fd_offset < 0 || *endp != '\0')
>> +                                usage();
>> +            break;
>> +        case 'L':
>> +            fd_length = strtod(optarg, &endp);
>> +            if (fd_length < 0 || *endp != '\0')
>> +                                usage();
>> +            break;
>>         default:
>>             usage();
>>         }
>> @@ -680,6 +695,34 @@ syserr:            run_err("%s: %s", name, strerr
>>             run_err("%s: %s", name, "Negative file size");
>>             goto next;
>>         }
>> +
>> +        // if the offset is greater than the file size
>> +        // we can't do this
>> +        if (fd_offset > stb.st_size) {
>> +            run_err("Offset greater than file size");
>> +            goto next;
>> +        }
>> +       +        // if the offset plus the requested length are greater
>> +        // than the file size then we have to quit
>> +        if (fd_length + fd_offset > stb.st_size) {
>> +            run_err("Length greater than file size");
>> +            goto next;
>> +        }
>> +
>> +        // if the user only has an offset then we reduce
>> +        // the file size by the offset. if they have a length
>> +        // then we use stb.st_size and the length
>> +        // to determine the actual stopping point
>> +        if (fd_length != 0) {
>> +            stb.st_size -= (stb.st_size - fd_length);
>> +        } else {
>> +            stb.st_size -= fd_offset;
>> +        }
>> +       +        // seek to the position as requested by user
>> +        lseek (fd, fd_offset, SEEK_SET);
>> +
>>         unset_nonblock(fd);
>>         switch (stb.st_mode & S_IFMT) {
>>         case S_IFREG:
>> _______________________________________________
>> openssh-unix-dev mailing list
>> openssh-unix-dev at mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev