From djm at fuyu.mindrot.org Fri May 16 04:22:02 2008 From: djm at fuyu.mindrot.org (Damien Miller) Date: Fri, 16 May 2008 04:22:02 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: softflowd Message-ID: <20080515182202.6FF26A4F23@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: softflowd Changes by: djm at fuyu.mindrot.org 08/05/16 04:22:02 Modified files: . : ChangeLog softflowd.8 softflowd.h Log message: - (djm) Fix typo in manpage for PID file location; patch from ice AT extreme.hu - (djm) Make privsep directory compile-time configurable; patch from ice AT extreme.hu Diff commands: cvs -nQq rdiff -u -r1.95 -r1.96 softflowd/ChangeLog cvs -nQq rdiff -u -r1.18 -r1.19 softflowd/softflowd.8 cvs -nQq rdiff -u -r1.12 -r1.13 softflowd/softflowd.h CVSWeb: http://cvsweb.mindrot.org/index.cgi/softflowd/ChangeLog?r1=1.95;r2=1.96 http://cvsweb.mindrot.org/index.cgi/softflowd/softflowd.8?r1=1.18;r2=1.19 http://cvsweb.mindrot.org/index.cgi/softflowd/softflowd.h?r1=1.12;r2=1.13 Please note that there may be a delay before commits are available on the public CVSWeb site. From james at atlanticmetro.net Tue May 20 07:24:22 2008 From: james at atlanticmetro.net (James Cornman) Date: Mon, 19 May 2008 17:24:22 -0400 Subject: [netflow-tools] flowd output queue filling up Message-ID: <9e496e040805191424y3672f1aap45fa3789c12b99bb@mail.gmail.com> Hi. I'm using Flowd 0.9 built on Linux (CentOS 5) and I'm using an adapted version of your sockclient.py to inject qualified info into MySQL on another server. flowd is using the 'logsock' logging mechanism After about 20 seconds of running, I get the following: output_flow_enqueue: output queue full output_flow_enqueue: output queue full process_flow: enqueue failed after flush privsep_master: child exited child exited with status 1 Here are a few samples right before this happens: output_flow_enqueue: offset 523972 alloc 524288 process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122 proto 6 tcpflags 11 tos 00 agent [208.78.27.14] src [88.238.206.102]:1187 dst [69.9.45.36]:80 gateway [69.9.45.36] packets 1 octets 40 in_if 268 out_if 256 sys_uptime_ms 6w1d14h39m12s.099 time_sec 2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start 6w1d14h38m20s.364 flow_finish 6w1d14h38m20s.364 src_AS 9121 src_masklen 17 dst_AS 29838 dst_masklen 28 engine_type 0 engine_id 0 seq 79937 source 0 crc32 00000000 output_flow_enqueue: offset 524088 alloc 524288 process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122 proto 6 tcpflags 18 tos 00 agent [208.78.27.14] src [69.9.40.103]:80 dst [82.225.26.147]:3925 gateway [66.216.8.41] packets 1 octets 1500 in_if 102 out_if 268 sys_uptime_ms 6w1d14h39m12s.099 time_sec 2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start 6w1d14h38m4s.241 flow_finish 6w1d14h38m4s.241 src_AS 29838 src_masklen 24 dst_AS 12322 dst_masklen 11 engine_type 0 engine_id 0 seq 79937 source 0 crc32 00000000 output_flow_enqueue: offset 524204 alloc 524288 process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122 proto 6 tcpflags 10 tos 00 agent [208.78.27.14] src [69.9.40.103]:80 dst [82.225.26.147]:3931 gateway [66.216.8.41] packets 2 octets 3000 in_if 102 out_if 268 sys_uptime_ms 6w1d14h39m12s.099 time_sec 2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start 6w1d14h38m6s.068 flow_finish 6w1d14h38m6s.500 src_AS 29838 src_masklen 24 dst_AS 12322 dst_masklen 11 engine_type 0 engine_id 0 seq 79937 source 0 crc32 00000000 I'm not exactly sure what the problem may be but I'm wondering what any limitations may be with overall processing power (Not sending a tremendous amount of flows at it) and i'm also wondering how 'blocking' my python script could be, with all the mysql inserts and what not. Anyone have any thoughts? Thanks -- James Cornman Chief Technical Officer Atlantic Metro Communications e: james at atlanticmetro.net w: http://www.atlanticmetro.net v: 212-792-9950 f: 718-559-4862 CONFIDENTIALITY NOTICE: This communication and any documents, files or previous e-mail messages attached to it, constitute an electronic communication within the scope of the Electronic Communication Privacy Act, 18 USCA 2510. This communication may contain non-public, confidential, or legally privileged information intended for the sole use of the designated recipient(s). The unlawful interception, use or disclosure of such information is strictly prohibited under 18 USCA 2511 and any applicable laws. If you are not the intended recipient, or have received this communication in error, please notify the sender immediately by reply email at support at atlanticmetro.net or by telephone at 212-792-9950 and delete all copies of this communication, including attachments, without reading them or saving them to disk. From james at atlanticmetro.net Tue May 20 08:12:33 2008 From: james at atlanticmetro.net (James Cornman) Date: Mon, 19 May 2008 18:12:33 -0400 Subject: [netflow-tools] flowd output queue filling up In-Reply-To: <4831F578.1050308@davisvision.com> References: <9e496e040805191424y3672f1aap45fa3789c12b99bb@mail.gmail.com> <4831F578.1050308@davisvision.com> Message-ID: <9e496e040805191512t4441c3c1xe7e3ff07a0e1ea77@mail.gmail.com> Sweet. Its working like a champ now. That was quick. Thanks On Mon, May 19, 2008 at 5:47 PM, Jesse Kempf wrote: > James Cornman wrote: >> >> Hi. >> >> I'm using Flowd 0.9 built on Linux (CentOS 5) and I'm using an adapted >> version of your sockclient.py to inject qualified info into MySQL on >> another server. flowd is using the 'logsock' logging mechanism >> >> After about 20 seconds of running, I get the following: >> >> output_flow_enqueue: output queue full >> output_flow_enqueue: output queue full >> process_flow: enqueue failed after flush >> privsep_master: child exited >> child exited with status 1 >> >> > > >> >> I'm not exactly sure what the problem may be but I'm wondering what >> any limitations may be with overall processing power (Not sending a >> tremendous amount of flows at it) and i'm also wondering how >> 'blocking' my python script could be, with all the mysql inserts and >> what not. >> >> Anyone have any thoughts? >> > > What you're seeing is a bug in 0.9 which got fixed back in the fall -- flowd > wasn't emptying its file output queue and would explode after a certain, > rather small (several hundred, I think?) number of records was received. I > have a setup that does 200 records/sec up to several thousand records/sec, > and that shoves everything into a PostgreSQL database. Flowd is decidedly > not the bottleneck. > Try the latest snapshot and see if that works for you. > > Cheers, > -Jesse > > > > ------------------------------------------------------------------------ > The information contained in this communication is intended > only for the use of the recipient(s) named above. It may > contain information that is privileged or confidential, and > may be protected by State and/or Federal Regulations. If > the reader of this message is not the intended recipient, > you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of > its contents, is strictly prohibited. If you have received > this communication in error, please return it to the sender > immediately and delete the original message and any copy > of it from your computer system. If you have any questions > concerning this message, please contact the sender. > ------------------------------------------------------------------------ > > -- James Cornman Chief Technical Officer Atlantic Metro Communications e: james at atlanticmetro.net w: http://www.atlanticmetro.net v: 212-792-9950 f: 718-559-4862 CONFIDENTIALITY NOTICE: This communication and any documents, files or previous e-mail messages attached to it, constitute an electronic communication within the scope of the Electronic Communication Privacy Act, 18 USCA 2510. This communication may contain non-public, confidential, or legally privileged information intended for the sole use of the designated recipient(s). The unlawful interception, use or disclosure of such information is strictly prohibited under 18 USCA 2511 and any applicable laws. If you are not the intended recipient, or have received this communication in error, please notify the sender immediately by reply email at support at atlanticmetro.net or by telephone at 212-792-9950 and delete all copies of this communication, including attachments, without reading them or saving them to disk. From jkempf at davisvision.com Tue May 20 07:47:36 2008 From: jkempf at davisvision.com (Jesse Kempf) Date: Mon, 19 May 2008 17:47:36 -0400 Subject: [netflow-tools] flowd output queue filling up In-Reply-To: <9e496e040805191424y3672f1aap45fa3789c12b99bb@mail.gmail.com> References: <9e496e040805191424y3672f1aap45fa3789c12b99bb@mail.gmail.com> Message-ID: <4831F578.1050308@davisvision.com> James Cornman wrote: > Hi. > > I'm using Flowd 0.9 built on Linux (CentOS 5) and I'm using an adapted > version of your sockclient.py to inject qualified info into MySQL on > another server. flowd is using the 'logsock' logging mechanism > > After about 20 seconds of running, I get the following: > > output_flow_enqueue: output queue full > output_flow_enqueue: output queue full > process_flow: enqueue failed after flush > privsep_master: child exited > child exited with status 1 > > > I'm not exactly sure what the problem may be but I'm wondering what > any limitations may be with overall processing power (Not sending a > tremendous amount of flows at it) and i'm also wondering how > 'blocking' my python script could be, with all the mysql inserts and > what not. > > Anyone have any thoughts? > What you're seeing is a bug in 0.9 which got fixed back in the fall -- flowd wasn't emptying its file output queue and would explode after a certain, rather small (several hundred, I think?) number of records was received. I have a setup that does 200 records/sec up to several thousand records/sec, and that shoves everything into a PostgreSQL database. Flowd is decidedly not the bottleneck. Try the latest snapshot and see if that works for you. Cheers, -Jesse ------------------------------------------------------------------------ The information contained in this communication is intended only for the use of the recipient(s) named above. It may contain information that is privileged or confidential, and may be protected by State and/or Federal Regulations. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. ------------------------------------------------------------------------ From thoumin at ipanematech.com Thu May 22 17:53:02 2008 From: thoumin at ipanematech.com (Damien THOUMIN) Date: Thu, 22 May 2008 09:53:02 +0200 Subject: [netflow-tools] [Fwd: Re: softflow and vlan ???] Message-ID: <1211442782.12819.23.camel@ws-thoumin.ipanema.local> Hi everybody, I will be delighted if Softflow could be supported vlan (802.1q). It is a small modification which consist in declared the 802.1q frame in the libpcap. For the moment, I jump the vlan extension but it's just a workaround. Maybe, I can participate in the development. Thanks you for your help. Regards. Damien THOUMIN | Consultant ALTEN Telecom | thoumin at ipanematech.com Tel. +33 (0)1 55 52 19 70 | Fax +33 (0)1 55 52 15 01 Beyond the Network | www.ipanematech.com .......................................................................................................................... The information in this message may be confidential and legally privileged and is intended solely for the addressee. Access to this message by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or other action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. P Please consider the environment and don't print this e-mail unless you really need to. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20080522/e2c08221/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: logo for email signature.bmp Type: image/bmp Size: 14662 bytes Desc: not available Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20080522/e2c08221/attachment-0001.bin -------------- next part -------------- An embedded message was scrubbed... From: "Damien Miller" Subject: Re: softflow and vlan ??? Date: Tue, 20 May 2008 14:23:41 +1000 (EST) Size: 3779 Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20080522/e2c08221/attachment-0001.mht