From josh.m.sharpe at gmail.com Thu Mar 1 03:48:12 2007 From: josh.m.sharpe at gmail.com (Josh Sharpe) Date: Wed, 28 Feb 2007 11:48:12 -0500 Subject: [netflow-tools] (no subject) Message-ID: <6a7129610702280848v405f97e6y6552da0a82d6cda9@mail.gmail.com> I'm trying to get flow-tools built on ubuntu 6.10. I've already patched a lot of stuff I've found in the archives but I can't find anything in the archives about this error except "Install flex." Which I did, but didn't help. Thanx! $ make Making all in lib make[1]: Entering directory `/home/jsharpe/flow-tools-0.66/lib' make all-am make[2]: Entering directory `/home/jsharpe/flow-tools-0.66/lib' make[2]: Nothing to be done for `all-am'. make[2]: Leaving directory `/home/jsharpe/flow-tools-0.66/lib' make[1]: Leaving directory `/home/jsharpe/flow-tools-0.66/lib' Making all in src make[1]: Entering directory `/home/jsharpe/flow-tools-0.66/src' gcc -g -Wall -g -Wall -o flow-filter -L../lib flow-filter.o aclyacc.o acllex.o acl2.o -lft -lnsl -lz acllex.o: In function `yylex':/home/jsharpe/flow-tools-0.66/src/lex.yy.c:958: undefined reference to `yywrap' acllex.o: In function `input':/home/jsharpe/flow-tools-0.66/src/lex.yy.c:1307: undefined reference to `yywrap' collect2: ld returned 1 exit status make[1]: *** [flow-filter] Error 1 make[1]: Leaving directory `/home/jsharpe/flow-tools-0.66/src' make: *** [all-recursive] Error 1 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20070228/f02688a0/attachment.html From brian.lindauer at counterstorm.com Fri Mar 2 00:44:41 2007 From: brian.lindauer at counterstorm.com (Brian Lindauer) Date: Thu, 01 Mar 2007 07:44:41 -0600 Subject: [netflow-tools] (no subject) In-Reply-To: <6a7129610702280848v405f97e6y6552da0a82d6cda9@mail.gmail.com> References: <6a7129610702280848v405f97e6y6552da0a82d6cda9@mail.gmail.com> Message-ID: <1172756681.22078.3.camel@vaughan> You might also need to install the bison and/or byacc packages. Brian On Wed, 2007-02-28 at 11:48 -0500, Josh Sharpe wrote: > I'm trying to get flow-tools built on ubuntu 6.10. I've already > patched a lot of stuff I've found in the archives but I can't find > anything in the archives about this error except "Install flex." > Which I did, but didn't help. > > Thanx! > > $ make > Making all in lib > make[1]: Entering directory `/home/jsharpe/flow-tools-0.66/lib' > make all-am > make[2]: Entering directory `/home/jsharpe/flow-tools-0.66/lib' > make[2]: Nothing to be done for `all-am'. > make[2]: Leaving directory `/home/jsharpe/flow-tools-0.66/lib' > make[1]: Leaving directory `/home/jsharpe/flow-tools-0.66/lib' > Making all in src > make[1]: Entering directory `/home/jsharpe/flow-tools-0.66 /src' > gcc -g -Wall -g -Wall -o flow-filter -L../lib flow-filter.o > aclyacc.o acllex.o acl2.o -lft -lnsl -lz > acllex.o: In function > `yylex':/home/jsharpe/flow-tools-0.66/src/lex.yy.c:958: undefined > reference to `yywrap' > acllex.o: In function > `input':/home/jsharpe/flow-tools-0.66/src/lex.yy.c:1307: undefined > reference to `yywrap' > collect2: ld returned 1 exit status > make[1]: *** [flow-filter] Error 1 > make[1]: Leaving directory `/home/jsharpe/flow- tools-0.66/src' > make: *** [all-recursive] Error 1 > > _______________________________________________ > netflow-tools mailing list > netflow-tools at mindrot.org > http://lists.mindrot.org/mailman/listinfo/netflow-tools -- This email message is for the sole use of the intended recipient/s and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator. From memic at paniert.org Tue Mar 27 18:01:16 2007 From: memic at paniert.org (memic) Date: Tue, 27 Mar 2007 10:01:16 +0200 Subject: [netflow-tools] unable to export flows Message-ID: <4608CF4C.1000108@paniert.org> hi, i get those strange messages in my syslog Mar 26 16:37:24 bgp softflowd[10249]: Unable to export flows Mar 26 16:37:28 bgp last message repeated 3 times Mar 26 16:39:28 bgp last message repeated 4 times ideas where they come from? memic From cristi at net.utcluj.ro Tue Mar 27 18:03:23 2007 From: cristi at net.utcluj.ro (Cristian KLEIN) Date: Tue, 27 Mar 2007 11:03:23 +0300 Subject: [netflow-tools] unable to export flows In-Reply-To: <4608CF4C.1000108@paniert.org> References: <4608CF4C.1000108@paniert.org> Message-ID: <4608CFCB.3090107@net.utcluj.ro> memic wrote: > hi, > > i get those strange messages in my syslog > > Mar 26 16:37:24 bgp softflowd[10249]: Unable to export flows > Mar 26 16:37:28 bgp last message repeated 3 times > Mar 26 16:39:28 bgp last message repeated 4 times > > ideas where they come from? Is it possible that your firewall disallows outgoing UDP for your flows? Try enabling verbosity. From memic at paniert.org Tue Mar 27 20:17:32 2007 From: memic at paniert.org (memic) Date: Tue, 27 Mar 2007 12:17:32 +0200 Subject: [netflow-tools] unable to export flows In-Reply-To: <4608CFCB.3090107@net.utcluj.ro> References: <4608CF4C.1000108@paniert.org> <4608CFCB.3090107@net.utcluj.ro> Message-ID: <4608EF3C.5050805@paniert.org> no, the export is working, but i get those errors from time to time.. Cristian KLEIN wrote: > memic wrote: > >> hi, >> >> i get those strange messages in my syslog >> >> Mar 26 16:37:24 bgp softflowd[10249]: Unable to export flows >> Mar 26 16:37:28 bgp last message repeated 3 times >> Mar 26 16:39:28 bgp last message repeated 4 times >> >> ideas where they come from? >> > > Is it possible that your firewall disallows outgoing UDP for your flows? > Try enabling verbosity. > From guanqun.lu at gmail.com Wed Mar 28 16:36:06 2007 From: guanqun.lu at gmail.com (Guanqun Lu) Date: Wed, 28 Mar 2007 06:36:06 +0000 Subject: [netflow-tools] [softflowd]about softflowd TODO Message-ID: Hi, 1. What do you mean by "Use strtonum()"? You want to replace "atoi()" with it? 2. I'm currently doing some research that is mainly based on the performance of softflowd. But it seems that the softflowd can't stand up with the heavy flow. My colleague did some hack into the code. The diff of file softflowd.c is attached. After doing this, the performance does enhance a little. But still, the usage of CPU climbs up to 100% as soon as the pps increases to about 13,000. I'm glad to see that there's a performance part in TODO. Performance - Profile and see where the hot spots are It seems that it's a CPU intensive task. - Fast "new flow" test using a bloom filter You named 'bloom filter', maybe we can have a try. - See if we can reduce per-packet overhead more - Cost of expiry remove and re-add per packet - Stop run-time malloc (maybe) Why is it necessary? I'm wondering. Will the run-time malloc cost the performance? - Preallocate a pool of expiry events and flow entries - keep a queue, pick/push first from head -- Guanqun -------------- next part -------------- A non-text attachment was scrubbed... Name: loopnum.patch Type: application/octet-stream Size: 1503 bytes Desc: not available Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20070328/4f046103/attachment.obj From djm at mindrot.org Thu Mar 29 15:01:19 2007 From: djm at mindrot.org (Damien Miller) Date: Thu, 29 Mar 2007 15:01:19 +1000 (EST) Subject: [netflow-tools] [softflowd]about softflowd TODO In-Reply-To: References: Message-ID: On Wed, 28 Mar 2007, Guanqun Lu wrote: > Hi, > > 1. What do you mean by "Use strtonum()"? You want to replace "atoi()" with it? Yes, it is a more simple API and can make the code more readable. > 2. I'm currently doing some research that is mainly based on the > performance of softflowd. But it seems that the softflowd can't stand > up with the heavy flow. My colleague did some hack into the code. > The diff of file softflowd.c is attached. After doing this, the > performance does enhance a little. But still, the usage of CPU climbs > up to 100% as soon as the pps increases to about 13,000. The diff trades off the time spent in packet processing against the time spend it managing expiry events, etc. You might be able to play with the "expint" timeout to achieve the same effect (without a code change). > I'm glad to see that there's a performance part in TODO. > Performance > - Profile and see where the hot spots are > It seems that it's a CPU intensive task. > - Fast "new flow" test using a bloom filter > You named 'bloom filter', maybe we can have a try. It is an idea, it may improve things or it may add overhead. > - See if we can reduce per-packet overhead more > - Cost of expiry remove and re-add per packet > - Stop run-time malloc (maybe) > Why is it necessary? I'm wondering. Will the run-time malloc cost the > performance? Malloc is designed to be a good general purpose allocator for objects of various sizes. For softflowd, we need fast allocations of fixed size objects and we generally know how many (maximum) we need ahead of time. It should be able to avoid some of malloc's cost by preallocating the struct FLOW and struct EXPIRY. One thing that has a good likelihood of improving performance is to replace or modify the data structure used to store flows. At present it is a splay tree, which is fast when matching existing flows that receive a lot of traffic but slower otherwise (new flows or lots of quiet flows). Coming up with a good flow hash and replacing the splay tree with a hash table, or putting a hash table in front of splay trees is likely to help a lot. I have changed jobs (several times) since I first wrote softflowd and no longer have easy access to large quantities of real-world traffic to test it against. Because of this, I will have to depend more on the user community to improve softflowd's performance. Thanks, Damien From djm at mindrot.org Thu Mar 29 15:04:48 2007 From: djm at mindrot.org (Damien Miller) Date: Thu, 29 Mar 2007 15:04:48 +1000 (EST) Subject: [netflow-tools] unable to export flows In-Reply-To: <4608EF3C.5050805@paniert.org> References: <4608CF4C.1000108@paniert.org> <4608CFCB.3090107@net.utcluj.ro> <4608EF3C.5050805@paniert.org> Message-ID: On Tue, 27 Mar 2007, memic wrote: > no, the export is working, but i get those errors from time to time.. > > Cristian KLEIN wrote: > > memic wrote: > > > >> hi, > >> > >> i get those strange messages in my syslog > >> > >> Mar 26 16:37:24 bgp softflowd[10249]: Unable to export flows > >> Mar 26 16:37:28 bgp last message repeated 3 times > >> Mar 26 16:39:28 bgp last message repeated 4 times > >> > >> ideas where they come from? > >> > > > > Is it possible that your firewall disallows outgoing UDP for your flows? > > Try enabling verbosity. This message can occur in a couple of cases, the two that spring to mind are: 1. UDP socket send buffer is full. softflowd may benefit from an increase of the send buffer, similar to the change I made to flowd: http://cvsweb.mindrot.org/index.cgi/flowd/privsep.c.diff?r1=1.29;r2=1.30 (patches welcome) 2. ICMP unreachable errors from the export target host or intervening gateways -- there isn't much softflowd can do in this case. -d From djm at mindrot.org Thu Mar 29 15:08:23 2007 From: djm at mindrot.org (Damien Miller) Date: Thu, 29 Mar 2007 15:08:23 +1000 (EST) Subject: [netflow-tools] CISCO HDLC Pcap format In-Reply-To: <45DC7A0C.9050001@uam.es> References: <45DC7A0C.9050001@uam.es> Message-ID: On Wed, 21 Feb 2007, V?ctor L?pez ?lvarez wrote: > I have tried to change from Pcap format to Netflow using softflowd but > it say me that the format I use is not support. However, I can extract > information using tcpdump, so I wonder why softflowd say me that is an > unsupported datalink type. > > Anyone knows which is the reason? Because support for it has not been implemented. > Besides, how could I add the support for this header? Near the start of softflowd.c there is a definition of "struct DATALINK" and a table of supported datalink types "lt[]". struct DATALINK defines where in the layer-2 frame softflowd should look for the frame type, how to extract it and how to interpet it to determine whether a frame represents a IPv4 or IPv6 packet. Adding a new frame type to softflowd is as simple as adding an entry to this array. -d From guanqun.lu at gmail.com Thu Mar 29 21:23:31 2007 From: guanqun.lu at gmail.com (Guanqun Lu) Date: Thu, 29 Mar 2007 19:23:31 +0800 Subject: [netflow-tools] CISCO HDLC Pcap format In-Reply-To: References: <45DC7A0C.9050001@uam.es> Message-ID: On 3/29/07, Damien Miller wrote: > On Wed, 21 Feb 2007, V?ctor L?pez ?lvarez wrote: > > > I have tried to change from Pcap format to Netflow using softflowd but > > it say me that the format I use is not support. However, I can extract > > information using tcpdump, so I wonder why softflowd say me that is an > > unsupported datalink type. > > > > Anyone knows which is the reason? > > Because support for it has not been implemented. > > > Besides, how could I add the support for this header? > > Near the start of softflowd.c there is a definition of "struct DATALINK" > and a table of supported datalink types "lt[]". struct DATALINK defines > where in the layer-2 frame softflowd should look for the frame type, how > to extract it and how to interpet it to determine whether a frame represents > a IPv4 or IPv6 packet. > > Adding a new frame type to softflowd is as simple as adding an entry to > this array. I'm wondering whether it's easy to collect other packet information such as MPLS and VPN. > > -d > _______________________________________________ > netflow-tools mailing list > netflow-tools at mindrot.org > http://lists.mindrot.org/mailman/listinfo/netflow-tools > > Thanks, -- Guanqun From guanqun.lu at gmail.com Thu Mar 29 21:18:18 2007 From: guanqun.lu at gmail.com (Guanqun Lu) Date: Thu, 29 Mar 2007 19:18:18 +0800 Subject: [netflow-tools] [softflowd]about softflowd TODO In-Reply-To: References: Message-ID: On 3/29/07, Damien Miller wrote: > On Wed, 28 Mar 2007, Guanqun Lu wrote: > > > Hi, > > > > 1. What do you mean by "Use strtonum()"? You want to replace "atoi()" with it? > > Yes, it is a more simple API and can make the code more readable. > > > 2. I'm currently doing some research that is mainly based on the > > performance of softflowd. But it seems that the softflowd can't stand > > up with the heavy flow. My colleague did some hack into the code. > > The diff of file softflowd.c is attached. After doing this, the > > performance does enhance a little. But still, the usage of CPU climbs > > up to 100% as soon as the pps increases to about 13,000. > > The diff trades off the time spent in packet processing against the > time spend it managing expiry events, etc. You might be able to play > with the "expint" timeout to achieve the same effect (without a code > change). Yes, the program with the diff patch spends more time on packet processing. But it has several demerits: 1. if there is no flow out there, `softflowctl' can't exist. The pcap_dispatch waits for the incoming flow and blocks the program. 2. The increase of performance is limited. It doesn't show that the performance gain is linear with the loopnum in the patch. When the loopnum is bigger than 5, no obvious performance increase is seen. Thanks for your mentioning of `expint' timeout, I'll have a look at this option. > > > I'm glad to see that there's a performance part in TODO. > > Performance > > - Profile and see where the hot spots are > > It seems that it's a CPU intensive task. > > - Fast "new flow" test using a bloom filter > > You named 'bloom filter', maybe we can have a try. > > It is an idea, it may improve things or it may add overhead. > > > - See if we can reduce per-packet overhead more > > - Cost of expiry remove and re-add per packet > > - Stop run-time malloc (maybe) > > Why is it necessary? I'm wondering. Will the run-time malloc cost the > > performance? > > Malloc is designed to be a good general purpose allocator for objects of > various sizes. For softflowd, we need fast allocations of fixed size > objects and we generally know how many (maximum) we need ahead of time. > It should be able to avoid some of malloc's cost by preallocating the > struct FLOW and struct EXPIRY. > > One thing that has a good likelihood of improving performance is to > replace or modify the data structure used to store flows. At present > it is a splay tree, which is fast when matching existing flows that > receive a lot of traffic but slower otherwise (new flows or lots of > quiet flows). Coming up with a good flow hash and replacing the splay > tree with a hash table, or putting a hash table in front of splay trees > is likely to help a lot. As seen in our experiment, the memory usage of softflowd is very low. Therefore, I think maybe we can trade the memory for the performance, using some auxiliary information to increase the performance. > > I have changed jobs (several times) since I first wrote softflowd and > no longer have easy access to large quantities of real-world traffic > to test it against. Because of this, I will have to depend more on the > user community to improve softflowd's performance. It would be my pleasure if I could do something useful to improve the performance. > > Thanks, > Damien > -- Guanqun From djm at mindrot.org Thu Mar 29 22:47:39 2007 From: djm at mindrot.org (Damien Miller) Date: Thu, 29 Mar 2007 22:47:39 +1000 (EST) Subject: [netflow-tools] CISCO HDLC Pcap format In-Reply-To: References: <45DC7A0C.9050001@uam.es> Message-ID: On Thu, 29 Mar 2007, Guanqun Lu wrote: > > Adding a new frame type to softflowd is as simple as adding an entry to > > this array. > > I'm wondering whether it's easy to collect other packet information such as > MPLS > and VPN. If it is in the packet, and there exist NetFlow fields in which to export it, then softflowd can be modified to collect and report it. It doesn't support either of these at present, but MPLS wouldn't be too difficult to add I imagine. You would need to be more specific about what you mean by "VPN". If you mean IPsec, then there is not much more additional information available beyond what is already reported (endpoints and IP protocol) unless you give softflowd the keys to the phase-2 IPsec SAs, which I think is a pretty scary proposition. -d