From djm at mindrot.org Thu Nov 2 16:17:35 2006 From: djm at mindrot.org (Damien Miller) Date: Thu, 2 Nov 2006 16:17:35 +1100 (EST) Subject: [netflow-tools] softflowd and pflog In-Reply-To: <4541153B.8030002@net.utcluj.ro> References: <4541153B.8030002@net.utcluj.ro> Message-ID: On Thu, 26 Oct 2006, Cristian KLEIN wrote: > Hi list, > > I found it useful to log packets from a FreeBSD / OpenBSD pflog > interface. This way, you may fine-tune the traffic you want to export. Have you tried pfflowd[1]? It does something very similar to what you want. > This is very useful if you have a box which routes Gigabit LAN traffic > and does NAT to the Internet. If you want to log the Internet traffic > (before being NATed) you would have to put softflowd on the Gigabit > interface, which would be a huge waste of CPU cycles. Well, using pflog means that many fields in the flow will not be accurate, especially since pflog typically records only the first packet matching a state entry. You can use the "log (all)" modifier, but then you are back to having softflowd look at every packet. What advantages do you see in using pflog instead of pfsync (which is what pfflowd uses)? > In the following patch, I have hardcoded the pflog header size and the > location of the address family, to reduce dependency. I think it would be better to detect the presence of the net/if_pflog.h header in configure and use PFLOG_HDRLEN directly so softflowd will automatically pick up changes in that file. -d [1] http://www.mindrot.org/projects/flowd/ From djm at mindrot.org Thu Nov 2 16:19:36 2006 From: djm at mindrot.org (Damien Miller) Date: Thu, 2 Nov 2006 16:19:36 +1100 (EST) Subject: [netflow-tools] v9 -> v5 conversion In-Reply-To: <6DED202D454D3B4EB7D98A7439218D61070CF9EA@vahqex2.gfgsi.com> References: <6DED202D454D3B4EB7D98A7439218D61070CF9EA@vahqex2.gfgsi.com> Message-ID: On Thu, 14 Sep 2006, Wycoff, Matthew \(US SSA\) wrote: > Is there a good way to do conversion from netflow v9 to v5? Similar to > the redirect function of nfdump, but converting the flows as well. Not at present - I plan on adding this as a feature to flowd, probably by reusing the netflow sender code of softflowd but I haven't got around to it yet. (apologies for taking a while to get back to you) -d From djm at mindrot.org Thu Nov 2 18:14:42 2006 From: djm at mindrot.org (Damien Miller) Date: Thu, 2 Nov 2006 18:14:42 +1100 (EST) Subject: [netflow-tools] ANNOUNCE: softflowd-0.9.8 released Message-ID: Hi, After a year of slow tweaking and bugfixing, I'm happy to announce the release of softflowd 0.9.8. The release is available from: http://www.mindrot.org/projects/softflowd/ New features ------------ - Added "send-template" softflowctl command to resend a NetFlow 9 template record immediately (handy if you restart your flow collector) - Softflowd will now encode ICMP type and code into the flow datagram's source/destination ports, similar to some Cisco exporters. Work mostly by ssnodgra AT pheran.com - Added an option to ignore port and/or protocol information when tracking flows, allowing flows from the same IP addresses to be automatically coalesced, saving CPU load and export volume at the expense of flow detail - Support Linux "cooked socket" datalink type, from Tony Lewis gnutered AT yahoo.com.au - Report pcap stats in "softflowctl statistics" display - There is now a RPM spec file, courtesy ssnodgra AT pheran.com Bugfixes -------- - Expire quiescent flows in a timely manner when they hit maximum_lifetime; bug noticed and patch tested by andreas.brillisauer AT hetzner.de - Fix DLT_RAW support, from jhanna AT shaw.ca - Fix installation when target directoris do not exist; spotted by alshu AT tut.by - Additional paranoia and verbosity on malloc failures - Type fixes from rbreathe AT brookes.ac.uk - Fix time printing bug in debug mode - Fix reversed NetFlow v.9 first_switched and last_switched times - Fixed sequence number generation bugs. Reported by b.ghita AT jack.see.plymouth.ac.uk and mwlucas AT blackhelicopters.org Thanks to everyone who reported bugs, contributed or tested patches and answered questions on the mailing list. Apologies to anyone that I have missed. Thanks, Damien Miller From cristi at net.utcluj.ro Thu Nov 2 19:52:08 2006 From: cristi at net.utcluj.ro (Cristian KLEIN) Date: Thu, 02 Nov 2006 10:52:08 +0200 Subject: [netflow-tools] softflowd and pflog In-Reply-To: References: <4541153B.8030002@net.utcluj.ro> Message-ID: <4549B1B8.4020401@net.utcluj.ro> Damien Miller wrote: > On Thu, 26 Oct 2006, Cristian KLEIN wrote: > >> I found it useful to log packets from a FreeBSD / OpenBSD pflog >> interface. This way, you may fine-tune the traffic you want to export. > > Have you tried pfflowd[1]? It does something very similar to what > you want. I found that using pfsync for flows export might be in contradiction, for example, if somebody wants NAT states to be distributed, but does not want to export them. > Well, using pflog means that many fields in the flow will not be > accurate, especially since pflog typically records only the first > packet matching a state entry. You can use the "log (all)" modifier, > but then you are back to having softflowd look at every packet. Imagine the following scenario: you have em0 and em1, having private IPs, doing Gigabit routing, and fxp0 going out the Internet. Obviously, you will do NAT on fxp0. If you install softflowd on fxp0 you will only see the translated packets, which is not very useful. Installing softflowd on em0 and/or em1 will inject 1Gbps into a poor user-space daemon. I found that using "pass log from to !" and softflowd solves the problem. > What advantages do you see in using pflog instead of pfsync (which is > what pfflowd uses)? I couldn't state *the* reason why pflog is better that pfsync, but I have the following arguments for pflog: 1) One might not want to use states For example, I don't like to use keep-state in pf, because reaching the state limit makes my clients very unhappy. On the other hand, if softflowd exports flows too early (either because of small timeout values, or because the state limit has been reached), that is quite okey with me. 2) pfsync and flow export might be in contradiction (already explained above) 3) Easier verification I think that doing a tcpdump in pflog0 to find out which flows are being exported is easier than watching pfsync messages. However, the ultimate reason is that I found it useful and it was easy to implement, so, why not have it? > I think it would be better to detect the presence of the net/if_pflog.h > header in configure and use PFLOG_HDRLEN directly so softflowd will > automatically pick up changes in that file. I must admit, I didn't bother too much, because I saw that all other encapsulations are using hardcoded values too. However, I have the following dilema: should only PFLOG_HDRLEN be taken from net/if_pflog.h, or all values, including the position / size of the address family? From guyon.moree at gmail.com Mon Nov 6 00:58:44 2006 From: guyon.moree at gmail.com (=?ISO-8859-1?Q?Guyon_Mor=E9e?=) Date: Sun, 5 Nov 2006 14:58:44 +0100 Subject: [netflow-tools] FLOWD logrotation question Message-ID: <78f5441d0611050558q3f2f6382u3701313ff5c0f31c@mail.gmail.com> Hi all, First of all, I'm still a Unix beginner, so my question might actually be related to that. I have a script which is periodically renaming the flowdlog file and send a signal to flowd to restart the logging to to the original logfile. This works quite well, but the problem is that my script needs to run as root. Because I get Flowd's PID from the pidfile, it gets the pid of the parent process which also runs as root instead of the child process which runs as the default _flowd. This way I can not signal the process without it running as root. I hope my problem is clear and you might have a suggestion for me. Regards, -- Guyon Mor?e guyon.moree at gmail.com http://gumuz.looze.net From djm at mindrot.org Mon Nov 6 12:56:46 2006 From: djm at mindrot.org (Damien Miller) Date: Mon, 6 Nov 2006 12:56:46 +1100 (EST) Subject: [netflow-tools] FLOWD logrotation question In-Reply-To: <78f5441d0611050558q3f2f6382u3701313ff5c0f31c@mail.gmail.com> References: <78f5441d0611050558q3f2f6382u3701313ff5c0f31c@mail.gmail.com> Message-ID: On Sun, 5 Nov 2006, Guyon Mor?e wrote: > Because I get Flowd's PID from the pidfile, it gets the pid of the > parent process which also runs as root instead of the child process > which runs as the default _flowd. This way I can not signal the > process without it running as root. Yes, you need root privileges to signal flowd. Is this a problem? -d From guyon.moree at gmail.com Thu Nov 16 01:13:55 2006 From: guyon.moree at gmail.com (=?ISO-8859-1?Q?Guyon_Mor=E9e?=) Date: Wed, 15 Nov 2006 15:13:55 +0100 Subject: [netflow-tools] Logsock problem Message-ID: <78f5441d0611150613l14621376q9aa6b920c4eff756@mail.gmail.com> Hi, I'm having trouble running flowd to log to a socket. I get this error message: connect to logsock: No such file or directory The directory exists. I read in a previous email in this archive that the file should be create automatically, but it doesn't. any ideas? regards, -- Guyon Mor?e guyon.moree at gmail.com http://gumuz.looze.net From guyon.moree at gmail.com Fri Nov 17 22:50:02 2006 From: guyon.moree at gmail.com (=?ISO-8859-1?Q?Guyon_Mor=E9e?=) Date: Fri, 17 Nov 2006 12:50:02 +0100 Subject: [netflow-tools] Logsock problem In-Reply-To: <78f5441d0611150613l14621376q9aa6b920c4eff756@mail.gmail.com> References: <78f5441d0611150613l14621376q9aa6b920c4eff756@mail.gmail.com> Message-ID: <78f5441d0611170350t7c80ed5bu3492f9a64c24f853@mail.gmail.com> Any ideas guys or did I not provide you with enough information? On 11/15/06, Guyon Mor?e wrote: > Hi, > > I'm having trouble running flowd to log to a socket. > > I get this error message: connect to logsock: No such file or directory > > The directory exists. I read in a previous email in this archive that > the file should be create automatically, but it doesn't. > > any ideas? > > > regards, > > -- > Guyon Mor?e > guyon.moree at gmail.com > http://gumuz.looze.net > -- Guyon Mor?e guyon.moree at gmail.com http://gumuz.looze.net From brian.lindauer at counterstorm.com Sat Nov 18 09:01:33 2006 From: brian.lindauer at counterstorm.com (Brian Lindauer) Date: Fri, 17 Nov 2006 16:01:33 -0600 Subject: [netflow-tools] Logsock problem Message-ID: <1163800893.24954.113.camel@localhost.localdomain> Guyon Mor?e wrote: > I'm having trouble running flowd to log to a socket. > > I get this error message: connect to logsock: No such file or directory > > The directory exists. I read in a previous email in this archive that > the file should be create automatically, but it doesn't. > > any ideas? Try starting the reader first. I forget which, but only one of the reader and writer can create the socket. Brian -- This email message is for the sole use of the intended recipient/s and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator. From brian.lindauer at counterstorm.com Sat Nov 18 08:46:05 2006 From: brian.lindauer at counterstorm.com (Brian Lindauer) Date: Fri, 17 Nov 2006 15:46:05 -0600 Subject: [netflow-tools] Flowd shared object patch Message-ID: <1163799965.24954.110.camel@localhost.localdomain> Attached is a patch which will generate a libflowd.so in addition to libflowd.a. This makes including flowd functionality in a libtool shared module (where statically linked libraries are frowned upon) much easier. If you could include this or something equivalent in future releases, I'd appreciate it. Thanks, Brian -- This email message is for the sole use of the intended recipient/s and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator. -------------- next part -------------- A non-text attachment was scrubbed... Name: flowd.patch Type: text/x-patch Size: 2469 bytes Desc: not available Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20061117/23ab3e10/attachment.bin From brian.lindauer at counterstorm.com Wed Nov 22 09:32:19 2006 From: brian.lindauer at counterstorm.com (Brian Lindauer) Date: Tue, 21 Nov 2006 16:32:19 -0600 Subject: [netflow-tools] flow_start and flow_finish Message-ID: <1164148339.24954.133.camel@localhost.localdomain> I'm confused about the flow_start and flow_finish fields of store_flow_complete. The values that I'm getting don't seem to correspond to unix timestamps (even after running them through ntohl). The store_format_flow() function formats them as intervals, but that doesn't make sense either. Is this a bug in netflow parsing? process_flow: ACCEPT flow FLOW recv_time 2006-11-20T23:35:38.901867 proto 6 tcpflags 00 tos 80 agent [172.16.1.150] src [172.16.1.154]:40534 dst [172.16.2.251]:443 gateway [0.0.0.0] packets 8 octets 1193 in_if 110 out_if 90 sys_uptime_ms 3w1d7h43m27s.204 time_sec 2006-11-20T23:35:38 time_nanosec 0 netflow ver 7 flow_start 3w1d7h38m18s.067 flow_finish 3w1d7h38m19s.218 src_AS 0 src_masklen 0 dst_AS 0 dst_masklen 0 engine_type 0 engine_id 0 seq 1025263 source 0 crc32 00000000 Thanks, Brian -- This email message is for the sole use of the intended recipient/s and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.