Debugging SFTP question

[Graeme Wallace]

I've no idea how big the files are supposed to be - as they've always shown
up at around the 260Mb size. As far as the close being missing - i didnt
remove anything from the logs - and the code looks like it should log at
DEBUG3 - so i guess the client doesnt call close directly. However, even
the implicit close in the code from a client disconnect looks like it
should log the close on the handle......

regards,


Graeme


On Sun, May 5, 2013 at 8:12 PM, Darren Tucker <dtucker at zip.com.au> wrote:

> On Sun, May 05, 2013 at 01:39:06PM -0500, Graeme Wallace wrote:
> > Corruption occurs early in the file. Filesystem is ext4 on Ubuntu 12.04.
> > I've attached the transfer log at DEBUG3.
>
> I notice the log does not show the file being closed.  Is that really
> the case or is it just missing from the log?  In theory it could be an
> error when the file is flushed to disk, and that wouldn't be detected
> because of the missing close, but that seems pretty unlikely.
>
> Looking at the server log I see huge chunks of the file missing.  I ran
> the perl script below over the log and got:
>
> chunks written:  0-2097152 41910272-44105728 95027200-97222656
>    120356864-122552320 156434432-158629888 193921024-196116480
>    237043712-239239168 269942784-272138240
>
> the non-contiguous portions are suspiciously regular, and they
> correspond to the jumps in the request numbers (the latter is not
> necessarily a problem but the former probably is).  My guess is that the
> client is multithreaded, doing multiple concurrent writes and is mixing
> up the increments of both the request numbers and offsets.
>
> How big is the file supposed to be?  Based on the number of packets I'd
> guess about 17M, but the client seems to be creating a sparse file about
> 260M in size (compare the sizes reported by ls -l and du).
>
> while (<>) {
>         unless (/off (\d+) len (\d+)/) { next; }
>         $off = $1; $len = $2;
>         #print "off $off len $len\n";
>         if (!defined($start)) {
>                 $start = $off;
>                 $end = $len;
>         } elsif ($off == $end) {
>                 $end = $off + $len;
>         } else {
>                 print "non-contig write at $off $len\n";
>                 $chunks = "$chunks $start-$end";
>                 $start = $off;
>                 $end = $off + $len;
>         }
> }
> $chunks = "$chunks $start-$end";
>
> print "chunks written: $chunks\n"
>
> --
> Darren Tucker (dtucker at zip.com.au)
> GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
>     Good judgement comes with experience. Unfortunately, the experience
> usually comes from bad judgement.
>



-- 
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018