[netflow-tools] flowd output queue filling up

[James Cornman]

Sweet. Its working like a champ now. That was quick.

Thanks

On Mon, May 19, 2008 at 5:47 PM, Jesse Kempf <jkempf at davisvision.com> wrote:
> James Cornman wrote:
>>
>> Hi.
>>
>> I'm using Flowd 0.9 built on Linux (CentOS 5) and I'm using an adapted
>> version of your sockclient.py to inject qualified info into MySQL on
>> another server. flowd is using the 'logsock' logging mechanism
>>
>> After about 20 seconds of running, I get the following:
>>
>> output_flow_enqueue: output queue full
>> output_flow_enqueue: output queue full
>> process_flow: enqueue failed after flush
>> privsep_master: child exited
>> child exited with status 1
>>
>>
>
> <snip>
>>
>> I'm not exactly sure what the problem may be but I'm wondering what
>> any limitations may be with overall processing power (Not sending a
>> tremendous amount of flows at it) and i'm also wondering how
>> 'blocking' my python script could be, with all the mysql inserts and
>> what not.
>>
>> Anyone have any thoughts?
>>
>
> What you're seeing is a bug in 0.9 which got fixed back in the fall -- flowd
> wasn't emptying its file output queue and would explode after a certain,
> rather small (several hundred, I think?) number of records was received. I
> have a setup that does 200 records/sec up to several thousand records/sec,
> and that shoves everything into a PostgreSQL database. Flowd is decidedly
> not the bottleneck.
> Try the latest snapshot and see if that works for you.
>
> Cheers,
> -Jesse
>
>
>
> ------------------------------------------------------------------------
> The information contained in this communication is intended
> only for the use of the recipient(s) named above. It may
> contain information that is privileged or confidential, and
> may be protected by State and/or Federal Regulations. If
> the reader of this message is not the intended recipient,
> you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of
> its contents, is strictly prohibited. If you have received
> this communication in error, please return it to the sender
> immediately and delete the original message and any copy
> of it from your computer system. If you have any questions
> concerning this message, please contact the sender.
> ------------------------------------------------------------------------
>
>



-- 
James Cornman
Chief Technical Officer
Atlantic Metro Communications

e: james at atlanticmetro.net
w: http://www.atlanticmetro.net
v: 212-792-9950
f: 718-559-4862

CONFIDENTIALITY NOTICE: This communication and any documents, files or
previous e-mail messages attached to it, constitute an electronic
communication within the scope of the Electronic Communication Privacy
Act, 18 USCA 2510. This communication may contain non-public,
confidential, or legally privileged information intended for the sole
use of the designated recipient(s). The unlawful interception, use or
disclosure of such information is strictly prohibited under 18 USCA
2511 and any applicable laws. If you are not the intended recipient,
or have received this communication in error, please notify the sender
immediately by reply email at support at atlanticmetro.net or by
telephone at 212-792-9950 and delete all copies of this communication,
including attachments, without reading them or saving them to disk.