[Bug 3931] wildcard decimates directory listing performance (sftp ls)

[bugzilla-daemon at mindrot.org]

https://bugzilla.mindrot.org/show_bug.cgi?id=3931

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #1 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Ben Lewis from comment #0)
> For directories containing many (150+) files:
> sftp> ls dir
> is an order of magnitude faster than
> sftp> ls dir/*

>  This amount of difference seems inexplicable (both produce identical output).

It's completely explicable because the latter is asking it to do orders
of magnitude more work.  This one:

> sftp> ls dir

is "open 'dir' and return what in it.  This one:

> sftp> ls dir/*

is "expand the names of everything in dir/ and then stat every single
file in that list of files".  You can see the difference if you run
sftp with -vvv.

The shell is similar, BTW, it's just that it's much quicker and doesn't
have a network hop in the middle, so you usually won't notice.

$ for i in $(seq 0 10000); do touch /tmp/$i; done

$ strace -e trace=file -f sh -c "ls /tmp >/dev/null" 2>&1 | wc -l
30

$ strace -e trace=file -f sh -c "ls /tmp/* >/dev/null" 2>&1 | wc -l
20102

$ time sh -c "ls /tmp >/dev/null 2>&1"
real    0m0.013s
user    0m0.006s
sys     0m0.007s

$ time sh -c "ls /tmp/* >/dev/null 2>&1"
real    0m0.041s
user    0m0.010s
sys     0m0.031s

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.