From mworld at twbc.net  Wed Dec  7 04:58:17 2011
From: mworld at twbc.net (Otto)
Date: Wed, 07 Dec 2011 03:58:17 +1000
Subject: [netflow-tools] flowd-reader with stats
Message-ID: <4EDE57B9.2000201@twbc.net>

Hi. I added some code to flowd-reader. Is any one interested in being 
able to do this ? (inspired by flow-tools flow-stat). I found it helpful 
for debugging etc. It would be great to know how other people use flowd 
to gather statistics etc.

flowd-reader -s flowd-2011-12-07-030500

Flow Statistics

IPv4
Total Flows   : 3349
Total Bytes   : 32372372
Total Packets : 71252

IPv6
Total Flows   : 1017
Total Bytes   : 2014163
Total Packets : 7263

Regards,
Otto.


From mworld at twbc.net  Wed Dec  7 05:12:05 2011
From: mworld at twbc.net (Otto)
Date: Wed, 07 Dec 2011 04:12:05 +1000
Subject: [netflow-tools] compressed flowd files
Message-ID: <4EDE5AF5.2010508@twbc.net>

Hi. Earlier this year I built a system to process netflow data plans 
(flow-tools) using pthreads in c++. On a dual quad it can process ~6GB 
of compressed flow data in 3 minutes including processing the data plans 
and uploading the results back into an API.  Currently, I'm working on 
the same thing for flowd processing.

It would be cool to have a way to stream data from bz2 directly into one 
of the functions in the flowd code. Like :

int store_get_flow(int fd, struct store_flow_complete *f, char *ebuf, 
int elen)  But not sending it a file pointer. This way, I could have all 
the files compressed when my roll over script runs. So far, I've gone 
ahead without compression, but am unsure how to proceed, or if it's 
easily achievable. Any thoughts ?

Regards,
Otto.


From thorhs at basis.is  Wed Dec  7 05:38:09 2011
From: thorhs at basis.is (=?iso-8859-1?Q?=DE=F3rhallur_Sverrisson?=)
Date: Tue, 6 Dec 2011 18:38:09 +0000
Subject: [netflow-tools] flowd-reader with stats
In-Reply-To: <4EDE57B9.2000201@twbc.net>
References: <4EDE57B9.2000201@twbc.net>
Message-ID: <A4DB7E8F-5A7C-45B0-AB3D-B1199470E630@basis.is>

I for one am interested, I have been using awk and such which is slow.

Best regards,

Thorhallur

On 6.12.2011, at 17:59, "Otto" <mworld at twbc.net> wrote:

> Hi. I added some code to flowd-reader. Is any one interested in being able to do this ? (inspired by flow-tools flow-stat). I found it helpful for debugging etc. It would be great to know how other people use flowd to gather statistics etc.
> 
> flowd-reader -s flowd-2011-12-07-030500
> 
> Flow Statistics
> 
> IPv4
> Total Flows   : 3349
> Total Bytes   : 32372372
> Total Packets : 71252
> 
> IPv6
> Total Flows   : 1017
> Total Bytes   : 2014163
> Total Packets : 7263
> 
> Regards,
> Otto.
> 
> _______________________________________________
> netflow-tools mailing list
> netflow-tools at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/netflow-tools

From mworld at twbc.net  Wed Dec  7 14:19:00 2011
From: mworld at twbc.net (Otto)
Date: Wed, 07 Dec 2011 13:19:00 +1000
Subject: [netflow-tools] flowd-reader with stats
In-Reply-To: <A4DB7E8F-5A7C-45B0-AB3D-B1199470E630@basis.is>
References: <4EDE57B9.2000201@twbc.net>
 <A4DB7E8F-5A7C-45B0-AB3D-B1199470E630@basis.is>
Message-ID: <4EDEDB24.8020407@twbc.net>

Hi Thorhallur. I tested the attached patches in FreeBSD on version 
flowd-0.9.1. I'm interested if that is of any help in speeding things up 
for you.

cd flowd-0.9.1

Need to run configure first to build the make file, then patch it, 
build, install.

patch -p0 <flowd-reader.c.stats.patch
patch -p0 <Makefile.stats.patch

I tested it with a filter like this: (the target ip address is 
192.168.20.247 and I discard any local traffic)

# testFilterOut.fil
discard src !192.168.20.247/32
discard dst 192.168.0.0/19

 > cat /usr/flows/eth0/2011/2011-12/2011-12-07/* | ./flowd-reader -s -f 
/usr/flows/eth0/filters/testFilterOut.fil -

Flow Statistics

IPv4
Total Flows   : 262921
Total Bytes   : 3819349208
Total Packets : 9070763

IPv6
Total Flows   : 0
Total Bytes   : 0
Total Packets : 0


Regards,
Otto.

On 7/12/2011 4:38 AM, ??rhallur Sverrisson wrote:
> I for one am interested, I have been using awk and such which is slow.
>
> Best regards,
>
> Thorhallur
>
> On 6.12.2011, at 17:59, "Otto"<mworld at twbc.net>  wrote:
>
>> Hi. I added some code to flowd-reader. Is any one interested in being able to do this ? (inspired by flow-tools flow-stat). I found it helpful for debugging etc. It would be great to know how other people use flowd to gather statistics etc.
>>
>> flowd-reader -s flowd-2011-12-07-030500
>>
>> Flow Statistics
>>
>> IPv4
>> Total Flows   : 3349
>> Total Bytes   : 32372372
>> Total Packets : 71252
>>
>> IPv6
>> Total Flows   : 1017
>> Total Bytes   : 2014163
>> Total Packets : 7263
>>
>> Regards,
>> Otto.
>>
>> _______________________________________________
>> netflow-tools mailing list
>> netflow-tools at mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/netflow-tools

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: flowd-reader.c.stats.patch
URL: <http://lists.mindrot.org/pipermail/netflow-tools/attachments/20111207/080fdc9c/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile.stats.patch
URL: <http://lists.mindrot.org/pipermail/netflow-tools/attachments/20111207/080fdc9c/attachment-0001.ksh>

From mworld at twbc.net  Wed Dec  7 14:30:59 2011
From: mworld at twbc.net (Otto)
Date: Wed, 07 Dec 2011 13:30:59 +1000
Subject: [netflow-tools] flowd-reader with stats
In-Reply-To: <A4DB7E8F-5A7C-45B0-AB3D-B1199470E630@basis.is>
References: <4EDE57B9.2000201@twbc.net>
 <A4DB7E8F-5A7C-45B0-AB3D-B1199470E630@basis.is>
Message-ID: <4EDEDDF3.1010305@twbc.net>

Hi Thorhallur. How many flows are you processing at a time ? On our 
Sydney routers we process any where from 200 million to 1 billion flows 
a day (still flow-tools and IPv4 only at the moment). Although I wrote a 
custom program for doing that as a single thread would be slow. I am 
experimenting with flowd as a replacement to support IPv6 some time next 
year.

Regards,
Otto.

On 7/12/2011 4:38 AM, ??rhallur Sverrisson wrote:
> I for one am interested, I have been using awk and such which is slow.
>
> Best regards,
>
> Thorhallur
>
> On 6.12.2011, at 17:59, "Otto"<mworld at twbc.net>  wrote:
>
>> Hi. I added some code to flowd-reader. Is any one interested in being able to do this ? (inspired by flow-tools flow-stat). I found it helpful for debugging etc. It would be great to know how other people use flowd to gather statistics etc.
>>
>> flowd-reader -s flowd-2011-12-07-030500
>>
>> Flow Statistics
>>
>> IPv4
>> Total Flows   : 3349
>> Total Bytes   : 32372372
>> Total Packets : 71252
>>
>> IPv6
>> Total Flows   : 1017
>> Total Bytes   : 2014163
>> Total Packets : 7263
>>
>> Regards,
>> Otto.
>>
>> _______________________________________________
>> netflow-tools mailing list
>> netflow-tools at mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/netflow-tools


From mworld at twbc.net  Mon Dec 19 13:37:23 2011
From: mworld at twbc.net (Otto)
Date: Mon, 19 Dec 2011 12:37:23 +1000
Subject: [netflow-tools] quite ?
Message-ID: <4EEEA363.2090704@twbc.net>

Hi. This community seems rather quite ?

Regards,
Otto.

From list2009 at lunch.za.net  Mon Dec 19 19:03:02 2011
From: list2009 at lunch.za.net (Andrew McGill)
Date: Mon, 19 Dec 2011 10:03:02 +0200
Subject: [netflow-tools] quite ?
In-Reply-To: <4EEEA363.2090704@twbc.net>
References: <4EEEA363.2090704@twbc.net>
Message-ID: <20111219100302.02d42c1b@beans.lunch.za.net>

On Mon, 19 Dec 2011 12:37:23 +1000
Otto <mworld at twbc.net> wrote:

> Hi. This community seems rather quite ?
Quite true.  Report a bug.  Write flow-split-by-ip-list netflow splitter and save me the trouble.  Happy Christmas!

&:-)