[netflow-tools] flowd output queue filling up

[Jesse Kempf]

James Cornman wrote:
> Hi.
>
> I'm using Flowd 0.9 built on Linux (CentOS 5) and I'm using an adapted
> version of your sockclient.py to inject qualified info into MySQL on
> another server. flowd is using the 'logsock' logging mechanism
>
> After about 20 seconds of running, I get the following:
>
> output_flow_enqueue: output queue full
> output_flow_enqueue: output queue full
> process_flow: enqueue failed after flush
> privsep_master: child exited
> child exited with status 1
>
>   
<snip>
> I'm not exactly sure what the problem may be but I'm wondering what
> any limitations may be with overall processing power (Not sending a
> tremendous amount of flows at it) and i'm also wondering how
> 'blocking' my python script could be, with all the mysql inserts and
> what not.
>
> Anyone have any thoughts?
>   
What you're seeing is a bug in 0.9 which got fixed back in the fall -- 
flowd wasn't emptying its file output queue and would explode after a 
certain, rather small (several hundred, I think?) number of records was 
received. I have a setup that does 200 records/sec up to several 
thousand records/sec, and that shoves everything into a PostgreSQL 
database. Flowd is decidedly not the bottleneck.
Try the latest snapshot and see if that works for you.

Cheers,
-Jesse



------------------------------------------------------------------------
The information contained in this communication is intended
only for the use of the recipient(s) named above. It may
contain information that is privileged or confidential, and
may be protected by State and/or Federal Regulations. If
the reader of this message is not the intended recipient,
you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of
its contents, is strictly prohibited. If you have received
this communication in error, please return it to the sender
immediately and delete the original message and any copy
of it from your computer system. If you have any questions
concerning this message, please contact the sender.
------------------------------------------------------------------------