From n6vale at yahoo.co.uk Sat Jul 4 22:07:05 2009 From: n6vale at yahoo.co.uk (Simon) Date: Sat, 4 Jul 2009 12:07:05 +0000 (GMT) Subject: [netflow-tools] convert pcap to netflow with softflowd Message-ID: <526267.14748.qm@web26303.mail.ukl.yahoo.com> Hi everyone, I am trying to use softflowd and nfcapd to turn a pcap into neflow data. I am starting nfcapd by running: nfcapd -p 12345 -l netflow/ and softflowd by running: softflowd -n localhost:12345 -r singleflow.pcap This is the pcap I am trying to convert No.???? Time??????? Source??????????????? Destination?????????? Protocol Info ???? 1 0.000000??? 192.168.1.73????????? 74.86.135.174???????? TCP ???? 2 0.149059??? 74.86.135.174???????? 192.168.1.73????????? TCP ???? 3 0.149170??? 192.168.1.73????????? 74.86.135.174???????? TCP ???? 4 0.149322??? 192.168.1.73????????? 74.86.135.174???????? HTTP ???? 5 0.200823??? 192.168.1.73????????? 74.86.135.174???????? TCP ???? 6 0.299411??? 74.86.135.174???????? 192.168.1.73????????? TCP ???? 7 0.319394??? 74.86.135.174???????? 192.168.1.73????????? HTTP ???? 8 0.319474??? 192.168.1.73????????? 74.86.135.174???????? TCP ???? 9 0.376528??? 74.86.135.174???????? 192.168.1.73????????? TCP ??? 10 0.376569??? 192.168.1.73????????? 74.86.135.174???????? TCP This shows the pcap contains a single flow. ?However when I run nfdump on the produced nfcap file, I get: - Date flow start????????? Duration Proto????? Src IP Addr:Port Dst IP Addr:Port?? Packets??? Bytes Flows 2009-08-23 03:33:22.720???? 0.377 TCP????? 74.86.135.174:80??? -> 192.168.1.73:38589??????? 4????? 445???? 1 2009-08-23 03:33:22.720???? 0.377 TCP?????? 192.168.1.73:38589 -> 74.86.135.174:80?????????? 6????? 846???? 1 Summary: total flows: 2, total bytes: 1291, total packets: 10, avg bps: 27395, avg pps: 26, avg bpp: 129 Time window: 2009-08-23 03:33:22 - 2009-08-23 03:33:23 Total flows processed: 2, Records skipped: 0, Bytes read: 116 Sys: 0.000s flows/second: 0.0??????? Wall: 0.000s flows/second: 5333.3 Summary: total flows: 2, total bytes: 1291, total packets: 10, avg bps: 27395, avg pps: 26, avg bpp: 129 Time window: 2009-08-23 03:33:22 - 2009-08-23 03:33:23 Total flows processed: 2, Records skipped: 0, Bytes read: 116 Sys: 0.000s flows/second: 0.0 ? ? ? ?Wall: 0.000s flows/second: 5333.3 Which shows there are two flows. ?Every time I try and convert a pcap to netflow I get double the number of flows I expect. ?Can anyone tell me why this is, and what I can do to correct it? Thank you all for you help. Regards, Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From koti.kelam at gmail.com Wed Jul 8 17:25:43 2009 From: koti.kelam at gmail.com (Koteswar - Pandu) Date: Wed, 8 Jul 2009 12:55:43 +0530 Subject: [netflow-tools] Simple netflow probe for linux Message-ID: Hi all I want a simple netflow probe in linux which will export v5 and v9 flows to the collector. Any daemon is available for this or kernel can be patched to do this?? Thanks in advance Koteswar From martin at airwire.ie Wed Jul 8 19:43:01 2009 From: martin at airwire.ie (Martin List-Petersen) Date: Wed, 08 Jul 2009 10:43:01 +0100 Subject: [netflow-tools] Simple netflow probe for linux In-Reply-To: References: Message-ID: <4A546A25.2080903@airwire.ie> Koteswar - Pandu wrote: > Hi all > > I want a simple netflow probe in linux which will export v5 and v9 > flows to the collector. Any daemon is available for this or kernel can > be patched to do this?? I can highly recommend pmacctd. It's simple to set up and will work as a netflow probe for you. Kind regards, Martin List-Petersen -- Airwire - Ag Nascadh Pobail an Iarthair http://www.airwire.ie Phone: 091-865 968 From djm at mindrot.org Thu Jul 9 17:50:52 2009 From: djm at mindrot.org (Damien Miller) Date: Thu, 9 Jul 2009 17:50:52 +1000 (EST) Subject: [netflow-tools] Simple netflow probe for linux In-Reply-To: References: Message-ID: On Wed, 8 Jul 2009, Koteswar - Pandu wrote: > Hi all > > I want a simple netflow probe in linux which will export v5 and v9 > flows to the collector. Any daemon is available for this or kernel can > be patched to do this?? Well this list partially exists to support softflowd: http://www.mindrot.org/projects/softflowd/ Softflowd is a software netflow probe that supports v5, v9 and IPv6. -d From djm at mindrot.org Thu Jul 9 17:53:18 2009 From: djm at mindrot.org (Damien Miller) Date: Thu, 9 Jul 2009 17:53:18 +1000 (EST) Subject: [netflow-tools] convert pcap to netflow with softflowd In-Reply-To: <526267.14748.qm@web26303.mail.ukl.yahoo.com> References: <526267.14748.qm@web26303.mail.ukl.yahoo.com> Message-ID: On Sat, 4 Jul 2009, Simon wrote: > Hi everyone, > > I am trying to use softflowd and nfcapd to turn a pcap into neflow data. > > I am starting nfcapd by running: nfcapd -p 12345 -l netflow/ > and softflowd by running: softflowd -n localhost:12345 -r singleflow.pcap > > This is the pcap I am trying to convert > > No. Time Source Destination Protocol > Info > 1 0.000000 192.168.1.73 74.86.135.174 TCP > 2 0.149059 74.86.135.174 192.168.1.73 TCP > 3 0.149170 192.168.1.73 74.86.135.174 TCP > 4 0.149322 192.168.1.73 74.86.135.174 HTTP > 5 0.200823 192.168.1.73 74.86.135.174 TCP > 6 0.299411 74.86.135.174 192.168.1.73 TCP > 7 0.319394 74.86.135.174 192.168.1.73 HTTP > 8 0.319474 192.168.1.73 74.86.135.174 TCP > 9 0.376528 74.86.135.174 192.168.1.73 TCP > 10 0.376569 192.168.1.73 74.86.135.174 TCP > > This shows the pcap contains a single flow. However when I run nfdump on > the produced nfcap file, I get: - Correct, it shows the pcap contains a single _TCP_ flow. [snip] > Which shows there are two flows. Every time I try and convert a pcap to > netflow I get double the number of flows I expect. Can anyone tell me why > this is, and what I can do to correct it? Netflow flow records are unidirectional, so each TCP flow will yield two netflow flows; one for each direction. -d From djm at mindrot.org Thu Jul 9 17:55:24 2009 From: djm at mindrot.org (Damien Miller) Date: Thu, 9 Jul 2009 17:55:24 +1000 (EST) Subject: [netflow-tools] Collector does not aggregate single flows In-Reply-To: References: Message-ID: On Tue, 16 Jun 2009, Suraj Nellikar (snellika) wrote: > > Hi, > > When I observe the logs at the flowd collector collecting netflow v9 > packets, I see that it is not aggregating the packets coming from the same > flow. Instead it is just storing it separately. Is there any way to > aggregate the packets into a single flow? It isn't flowd's job to aggrgate flow data, though you can do it yourself using the supplied perl/python APIs. flowd just records whatever flows your probe sends to it. Some probes do support aggregation, though they may use formats or record tags that are not supported by flowd. -d From djm at mindrot.org Thu Jul 9 17:57:05 2009 From: djm at mindrot.org (Damien Miller) Date: Thu, 9 Jul 2009 17:57:05 +1000 (EST) Subject: [netflow-tools] flowinsert.pl tool gives an error In-Reply-To: References: Message-ID: On Mon, 15 Jun 2009, Suraj Nellikar (snellika) wrote: > > Hi, > > I am trying to store the binary flows from logfile into a sqlite DB. When I > run ?./flowinsert.pl ../logfile? command, it gives the following error:- > > DBD::SQLite::db do failed: file is encrypted or is not a database at > ./flowinsert.pl > > If I am right, we have to pass the logfile (which has the flows in binary > format) into flowinsert.pl and it will store it in the DB, right? > > Could you let me know why there is this error? It has been some time since I used flowinsert.pl, but I expect the error indicates that you need to give the tool a pre-initialised sqlite database. Use the "flows.sql" schema to prepare the SQLite database. -d From raphaelruiz at gmail.com Fri Jul 10 01:11:28 2009 From: raphaelruiz at gmail.com (Raphael Ruiz) Date: Thu, 9 Jul 2009 12:11:28 -0300 Subject: [netflow-tools] Softflowd Message-ID: <1a6f1ce60907090811i793efc05jbd0f6a3078e3d754@mail.gmail.com> Hi every one! I has observed what the informations about speed of the link of the Internet from softflowd and NFSEN are less than what i see in the Cacti. I see too what softflowd has use average 98% of process of CPU . The sofflowd is running in the same machine what nfsen. What the hardware profile for capture all flows in the link of the 100 Mbits/s? From djm at mindrot.org Mon Jul 13 13:21:23 2009 From: djm at mindrot.org (Damien Miller) Date: Mon, 13 Jul 2009 13:21:23 +1000 (EST) Subject: [netflow-tools] Softflowd In-Reply-To: <1a6f1ce60907090811i793efc05jbd0f6a3078e3d754@mail.gmail.com> References: <1a6f1ce60907090811i793efc05jbd0f6a3078e3d754@mail.gmail.com> Message-ID: On Thu, 9 Jul 2009, Raphael Ruiz wrote: > Hi every one! > > I has observed what the informations about speed of the link of the > Internet from softflowd and NFSEN are less than what i see in the > Cacti. Some difference is inevitable - softflowd will only look at IP traffic and disregard any link-layer traffic. Also, reconstructing point-in-time traffic utilisation from flow data is basically impossible - so whatever nfsen displays as a utilisation chart would involve some estimation and guesswork. > I see too what softflowd has use average 98% of process of CPU . The > sofflowd is running in the same machine what nfsen. Yes, softflowd is quite CPU intensive especially if your traffic mix consists to lots of tiny flows (e.g. web/DNS traffic). > What the hardware profile for capture all flows in the link of the 100 Mbits/s? "It depends" -d From sean at tinfoilhat.ca Tue Jul 14 02:04:03 2009 From: sean at tinfoilhat.ca (Sean Cody) Date: Mon, 13 Jul 2009 11:04:03 -0500 Subject: [netflow-tools] Softflowd & flow-tools on multiple interfaces. In-Reply-To: References: <1a6f1ce60907090811i793efc05jbd0f6a3078e3d754@mail.gmail.com> Message-ID: I've deployed both softflowd and flow-tools to devices that I can't easily add a mirror port to. So I've got around 5 sensors per site (softflowd on 3 mirror interfaces and on 2 devices directly) and 1 collector and am saving them in completely different flow-tools log sets. A bit of reading lends me to the idea of using the interface field in the flow records to record which device the flow came from (and have online 1 set of flow logs). Is this possible or should I continue using the 1 softflowd per flow- capture setup? As well is there an easy way to tell if softflowd is missing flows (ala tcpdump discards)? -- Sean