[netflow-tools] flowd output queue filling up

[James Cornman]

Hi.

I'm using Flowd 0.9 built on Linux (CentOS 5) and I'm using an adapted
version of your sockclient.py to inject qualified info into MySQL on
another server. flowd is using the 'logsock' logging mechanism

After about 20 seconds of running, I get the following:

output_flow_enqueue: output queue full
output_flow_enqueue: output queue full
process_flow: enqueue failed after flush
privsep_master: child exited
child exited with status 1


Here are a few samples right before this happens:

output_flow_enqueue: offset 523972 alloc 524288
process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122
proto 6 tcpflags 11 tos 00 agent [208.78.27.14] src
[88.238.206.102]:1187 dst [69.9.45.36]:80 gateway [69.9.45.36] packets
1 octets 40 in_if 268 out_if 256 sys_uptime_ms 6w1d14h39m12s.099
time_sec 2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start
6w1d14h38m20s.364 flow_finish 6w1d14h38m20s.364 src_AS 9121
src_masklen 17 dst_AS 29838 dst_masklen 28 engine_type 0 engine_id 0
seq 79937 source 0 crc32 00000000
output_flow_enqueue: offset 524088 alloc 524288
process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122
proto 6 tcpflags 18 tos 00 agent [208.78.27.14] src [69.9.40.103]:80
dst [82.225.26.147]:3925 gateway [66.216.8.41] packets 1 octets 1500
in_if 102 out_if 268 sys_uptime_ms 6w1d14h39m12s.099 time_sec
2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start
6w1d14h38m4s.241 flow_finish 6w1d14h38m4s.241 src_AS 29838 src_masklen
24 dst_AS 12322 dst_masklen 11 engine_type 0 engine_id 0 seq 79937
source 0 crc32 00000000
output_flow_enqueue: offset 524204 alloc 524288
process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122
proto 6 tcpflags 10 tos 00 agent [208.78.27.14] src [69.9.40.103]:80
dst [82.225.26.147]:3931 gateway [66.216.8.41] packets 2 octets 3000
in_if 102 out_if 268 sys_uptime_ms 6w1d14h39m12s.099 time_sec
2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start
6w1d14h38m6s.068 flow_finish 6w1d14h38m6s.500 src_AS 29838 src_masklen
24 dst_AS 12322 dst_masklen 11 engine_type 0 engine_id 0 seq 79937
source 0 crc32 00000000



I'm not exactly sure what the problem may be but I'm wondering what
any limitations may be with overall processing power (Not sending a
tremendous amount of flows at it) and i'm also wondering how
'blocking' my python script could be, with all the mysql inserts and
what not.

Anyone have any thoughts?


Thanks

-- 
James Cornman
Chief Technical Officer
Atlantic Metro Communications

e: james at atlanticmetro.net
w: http://www.atlanticmetro.net
v: 212-792-9950
f: 718-559-4862

CONFIDENTIALITY NOTICE: This communication and any documents, files or
previous e-mail messages attached to it, constitute an electronic
communication within the scope of the Electronic Communication Privacy
Act, 18 USCA 2510. This communication may contain non-public,
confidential, or legally privileged information intended for the sole
use of the designated recipient(s). The unlawful interception, use or
disclosure of such information is strictly prohibited under 18 USCA
2511 and any applicable laws. If you are not the intended recipient,
or have received this communication in error, please notify the sender
immediately by reply email at support at atlanticmetro.net or by
telephone at 212-792-9950 and delete all copies of this communication,
including attachments, without reading them or saving them to disk.