X forwarding uses up some resource?

[Tom Horsley]

After many weeks of running tests on different systems, all
started from inside the same X session and using ssh -X
forwarding to make the display accessible from the systems
running the tests, X forwarding just sorta stops working.
The remote machines still get a localhost:N DISPLAY variable
established, but trying to actually run an X client
results in connection errors.

Does this sound familiar to anyone? Is there some resource
that could be "used up" on the host machine if some of the
test sessions terminate abnormally so that eventually there
isn't any left to make forwarding work?

It isn't the X session itself that has problems. If I
talk to it directly without using X forwarding, it works
fine. It is just the X forwarded connections that stop.

Rebooting the host system makes everything work fine again
for a few weeks, then the X forwarded connections stop
working again.

Details: Host system is fedora 13 x86_64 running
openssh-server-5.4p1-3.fc13.x86_64. The X display
the tests are run from is a VNC server.