From memic at paniert.org Thu Mar 13 01:00:13 2008 From: memic at paniert.org (memic) Date: Wed, 12 Mar 2008 15:00:13 +0100 Subject: [netflow-tools] flow-cat / flowd Message-ID: <47D7E1ED.5030908@paniert.org> hi, im collecting netflow from a obsd box with flowd version 0.9, if i try to see some stats with flow-tools i get the following.. /usr/src/flow-tools-0.68# flow-cat *bin flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(2008-02-17-06:00-flow.bin): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(2008-02-17-06:15-flow.bin): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(2008-02-17-06:30-flow.bin): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(2008-02-17-06:45-flow.bin): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(flow-log2rrd): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(flow-rpt2rrd): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(flow-rptfmt): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(Makefile.am): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(Makefile.in): Failed, ignoring file. flow-cat: ftiheader_read(): Warning, bad magic number flow-cat: ftiheader_read(Makefile): Failed, ignoring file. thx memic From memic at paniert.org Thu Mar 13 02:50:48 2008 From: memic at paniert.org (memic) Date: Wed, 12 Mar 2008 16:50:48 +0100 Subject: [netflow-tools] flow-cat / flowd In-Reply-To: References: <47D7E1ED.5030908@paniert.org> Message-ID: <47D7FBD8.1050006@paniert.org> hm, thats the point, using perl or python seems to be to slow form me.. Jesse Kempf schrieb: > > On Mar 12, 2008, at 10:00 AM, memic wrote: > >> hi, >> >> im collecting netflow from a obsd box with flowd version 0.9, >> >> if i try to see some stats with flow-tools i get the following.. > > Flowd uses its own cooked format for on-disk storage. There are perl > and python modules for parsing the file format. > > Cheers, > -Jesse > > > > ------------------------------------------------------------------------ > The information contained in this communication is intended > only for the use of the recipient(s) named above. It may > contain information that is privileged or confidential, and > may be protected by State and/or Federal Regulations. If > the reader of this message is not the intended recipient, > you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of > its contents, is strictly prohibited. If you have received > this communication in error, please return it to the sender > immediately and delete the original message and any copy > of it from your computer system. If you have any questions > concerning this message, please contact the sender. > ------------------------------------------------------------------------ > From jkempf at davisvision.com Thu Mar 13 04:15:19 2008 From: jkempf at davisvision.com (Jesse Kempf) Date: Wed, 12 Mar 2008 13:15:19 -0400 Subject: [netflow-tools] flow-cat / flowd In-Reply-To: <47D7FBD8.1050006@paniert.org> References: <47D7E1ED.5030908@paniert.org> <47D7FBD8.1050006@paniert.org> Message-ID: <47D80FA7.9060800@davisvision.com> memic wrote: > hm, thats the point, using perl or python seems to > be to slow form me.. > I'm not entirely clear on what you mean by that. Have you actually tried the perl or python wrappers? Cheers, -Jesse Kempf ------------------------------------------------------------------------ The information contained in this communication is intended only for the use of the recipient(s) named above. It may contain information that is privileged or confidential, and may be protected by State and/or Federal Regulations. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. ------------------------------------------------------------------------ From JSaxe at briworks.com Sat Mar 29 02:12:45 2008 From: JSaxe at briworks.com (Jeff Saxe) Date: Fri, 28 Mar 2008 11:12:45 -0400 Subject: [netflow-tools] flowd is terrific! What do the continuous syslog messages mean? Message-ID: <96CACD3D-918A-42FA-A23A-6F21E7C73391@briworks.com> My apologies for first sending this directly to the author instead of to a proper mailing list, which he took the time to set up. I am resending it to the list. Good day! I'm a network engineer, smart guy, and Perl hacker at a smallish ISP in Charlottesville, Virginia, USA, and I'm trying to use flowd to put together a clever little client billing system. I believe everything is working fine, but when I turn on several routers' and several interfaces' worth of NetFlow packets toward this (reasonably powerful) Linux box at the same time, I get syslog messages like... Mar 26 20:13:27 chance flowd[17102]: Received max number of packets (512) on fd 3 Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows Mar 26 20:13:27 chance flowd[17102]: Received max number of packets (512) on fd 3 Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows Does this mean that actual NetFlow data are being discarded because they are arriving too fast? I should warn you that I'm using a lot of flowd.conf "accept" rules (on the order of 1,300, and more coming next week), so is that a problem? I mainly want to know if all the data are being collected or not; if they are, I will recompile the code to just suppress this warning message, but if it's a real problem, I'd like to know if you have any suggestions for not dropping packets. I've attached the current flowd.conf in case you are interested, and if you want to see the Perl code or MySQL table structure behind the automated rule generation, I can show you that, too. ? If you don't have time to help a stranger with this for free, I understand; just please tell me if the code is throwing away flows or not. Thanks very much, sir! -- Jeff Saxe, Network Engineer Blue Ridge InternetWorks, Charlottesville, VA CCIE # 9376 434-817-0707 ext. 2024 (work) / 434-882-3508 (cell) / JSaxe at briworks.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20080328/2f6b6280/attachment-0002.html -------------- next part -------------- A non-text attachment was scrubbed... Name: flowd.conf Type: application/octet-stream Size: 263352 bytes Desc: not available Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20080328/2f6b6280/attachment-0001.obj -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20080328/2f6b6280/attachment-0003.html