From djm at mindrot.org Thu Jul 6 12:30:22 2006 From: djm at mindrot.org (Damien Miller) Date: Thu, 6 Jul 2006 12:30:22 +1000 (EST) Subject: [netflow-tools] rsync server has moved Message-ID: Hi, For anyone who is using it, the mindrot.org rsync server has changed home. It's canonical location is rsync.mindrot.org, so if you use that then you will be able to find it as long as it exists. -d From dd at emeraldcityeg.com Fri Jul 7 02:52:30 2006 From: dd at emeraldcityeg.com (Emerald City / Daniel Duerr) Date: Thu, 06 Jul 2006 09:52:30 -0700 Subject: [netflow-tools] freebsd 6.1, pflowd, and virtual IPs Message-ID: <44AD3FCE.8070400@emeraldcityeg.com> Hello, I've successfully deployed pflowd on my freebsd 6.1-STABLE w/PF router/firewall and have it feeding its netflows to my collector on another machine. This router/firewall does bidirectional 1:1 NAT for a bunch of dedicated servers at a colo facility, so its got a bunch of virtual IPs assigned to its outside interface in addition to its own IP. Everything is working fine between the flow generator & collector, except that all the traffic is being reported against the primary IP of the router/firewall, not the virtual IPs that the traffic were used on. Anyone know of a way to make it behave the way I'd like? I use this for traffic accounting purposes so it is critical that I have this level of detail... Thanks, Daniel From djm at mindrot.org Fri Jul 7 18:56:26 2006 From: djm at mindrot.org (Damien Miller) Date: Fri, 07 Jul 2006 18:56:26 +1000 Subject: [netflow-tools] [ANNOUNCE] pfflowd-0.7 Message-ID: <44AE21BA.2050607@mindrot.org> Hi, After a long hiatus, pfflowd-0.7 has been released. Thanks to work done by Mathieu Sauve-Frankel, this release supports pfsync version 3 as used by OpenBSD 3.9 and above (older versions are not supported). A useful feature in pfsync v.3 is 64-bit packet and byte counters for tracked state entries and, again thanks to Mathieu, pfflowd will now use these larger counters to better account for large flows. Because NetFlow v.1 and v.5 support only 32-bit counters, pfflowd will send multiple flow records when a counter exceeds 2^32. Please see the pfflowd(8) manual page for further details. Please also note that the default privilege drop user has changed from "nobody" to "_pfflowd". Cheers, Damien Miller From djm at mindrot.org Fri Jul 7 19:09:22 2006 From: djm at mindrot.org (Damien Miller) Date: Fri, 07 Jul 2006 19:09:22 +1000 Subject: [netflow-tools] [ANNOUNCE] pfflowd-0.7 In-Reply-To: <44AE21BA.2050607@mindrot.org> References: <44AE21BA.2050607@mindrot.org> Message-ID: <44AE24C2.4090405@mindrot.org> Damien Miller wrote: > Hi, > > After a long hiatus, pfflowd-0.7 has been released. Thanks to work done > by Mathieu Sauve-Frankel, this release supports pfsync version 3 as used > by OpenBSD 3.9 and above (older versions are not supported). FYI, I have just updated the -current OpenBSD port to match. This port should compile fine on OpenBSD 3.9 too, but please don't complain on the OpenBSD lists if it doesn't - complain to me :) -d From djm at mindrot.org Sun Jul 9 19:46:16 2006 From: djm at mindrot.org (Damien Miller) Date: Sun, 09 Jul 2006 19:46:16 +1000 Subject: [netflow-tools] freebsd 6.1, pflowd, and virtual IPs In-Reply-To: <44AD3FCE.8070400@emeraldcityeg.com> References: <44AD3FCE.8070400@emeraldcityeg.com> Message-ID: <44B0D068.1040308@mindrot.org> Emerald City / Daniel Duerr wrote: > Hello, > > I've successfully deployed pflowd on my freebsd 6.1-STABLE w/PF > router/firewall and have it feeding its netflows to my collector on > another machine. This router/firewall does bidirectional 1:1 NAT for a > bunch of dedicated servers at a colo facility, so its got a bunch of > virtual IPs assigned to its outside interface in addition to its own IP. > Everything is working fine between the flow generator & collector, > except that all the traffic is being reported against the primary IP of > the router/firewall, not the virtual IPs that the traffic were used on. > Anyone know of a way to make it behave the way I'd like? I use this > for traffic accounting purposes so it is critical that I have this level > of detail... pfflowd should report whatever is recorded in the pfsync records. Does a manual tcpdump of the pfsync interface show the correct addresses? -d From dd at emeraldcityeg.com Mon Jul 10 04:08:31 2006 From: dd at emeraldcityeg.com (Emerald City / Daniel Duerr) Date: Sun, 09 Jul 2006 11:08:31 -0700 Subject: [netflow-tools] freebsd 6.1, pflowd, and virtual IPs In-Reply-To: <44B0D068.1040308@mindrot.org> References: <44AD3FCE.8070400@emeraldcityeg.com> <44B0D068.1040308@mindrot.org> Message-ID: <44B1461F.7040908@emeraldcityeg.com> Hi, For some reason I keep getting core dumps when I try to "tcpdump -pni pfsync0" on my firewall. pfsync_enable="YES" is configured in my /etc/rc.conf file and pfflowd is definitely producing output to my netflow collector, its just for the single IP of the firewall itself. I ran a "pfctl -ss" to view the state table and I see a lot of lines like this: self tcp 192.168.1.140:443 <- x.x.x.40:443 <- y.y.y.y:50970 TIME_WAIT:TIME_WAIT Where the address on the left is the private IP of one of my web servers, x.x.x.40 is the public IP of that server (a virtual IP on the firewall), and y.y.y.y is the public IP of the remote connection. This particular example is pretty common for me -- a state tracking for a remote user who connected to my web server via https. My not being able to tcpdump the pfsync0 interface definitely concerns me, and may be part of a problem? Regardless, it appears my system is tracking some states to the virtual IPs. Hope this helps. Appreciate your advice, Damien. Cheers, Daniel Damien Miller wrote: > Emerald City / Daniel Duerr wrote: >> Hello, >> >> I've successfully deployed pflowd on my freebsd 6.1-STABLE w/PF >> router/firewall and have it feeding its netflows to my collector on >> another machine. This router/firewall does bidirectional 1:1 NAT for a >> bunch of dedicated servers at a colo facility, so its got a bunch of >> virtual IPs assigned to its outside interface in addition to its own IP. >> Everything is working fine between the flow generator & collector, >> except that all the traffic is being reported against the primary IP of >> the router/firewall, not the virtual IPs that the traffic were used on. >> Anyone know of a way to make it behave the way I'd like? I use this >> for traffic accounting purposes so it is critical that I have this level >> of detail... > > pfflowd should report whatever is recorded in the pfsync records. Does > a manual tcpdump of the pfsync interface show the correct addresses? > > -d > > -- Daniel Duerr | President | Emerald City Entertainment Group, LLC dd at emeraldcityeg.com | +1 (831) 621-1767 | www.emeraldcityeg.com From guyon.moree at gmail.com Tue Jul 18 23:55:03 2006 From: guyon.moree at gmail.com (=?ISO-8859-1?Q?Guyon_Mor=E9e?=) Date: Tue, 18 Jul 2006 15:55:03 +0200 Subject: [netflow-tools] C flowd log example Message-ID: <78f5441d0607180655v3d225577yed6b6aa07196ac2c@mail.gmail.com> Hi all, I'm looking for a a simple example on how to read the flowd logfile, using the C header included. Als I notice 2 header files, store.h and store-v2.h. Which one should i use and how? thanx, -- Guyon Mor?e guyon.moree at gmail.com http://gumuz.looze.net From djm at mindrot.org Wed Jul 19 11:39:49 2006 From: djm at mindrot.org (Damien Miller) Date: Wed, 19 Jul 2006 11:39:49 +1000 (EST) Subject: [netflow-tools] C flowd log example In-Reply-To: <78f5441d0607180655v3d225577yed6b6aa07196ac2c@mail.gmail.com> References: <78f5441d0607180655v3d225577yed6b6aa07196ac2c@mail.gmail.com> Message-ID: On Tue, 18 Jul 2006, Guyon Mor?e wrote: > Hi all, > > I'm looking for a a simple example on how to read the flowd logfile, > using the C header included. Als I notice 2 header files, store.h and > store-v2.h. > > Which one should i use and how? store.h is the one you should use for new applications. store-v2.h is the legacy log format. It defines a couple of APIs that you can use to read flows: - A FILE* oriented API: store_read_flow() and store_write_flow(). This is probably the easiest to use, as stdio will take care of buffering, etc. Note that these do not try to back out cleanly when a write error occurs, so they are better suited to reading than writing. - A file-descriptor oriented API: store_get_flow() and store_put_flow(). These don't do any buffering, but they will back out a failed write. - Direct serialisation and deserialisation of flow records to/from memory buffers: store_flow_deserialise(), store_flow_serialise(), store_flow_serialise_masked(), store_calc_flow_len(). You might want to use these if you need to store binary flow records though something other than a file descriptor or FILE*, such as sending or receiving flow records over a Unix domain socket, passing them via shared memory or storing them in a database. Most of these functions take a error buffer (and length) in which they store an error message on failure. flowd-reader.c is a pretty good example of how to use the file-descriptor API. -d From guyon.moree at gmail.com Wed Jul 19 19:41:13 2006 From: guyon.moree at gmail.com (=?ISO-8859-1?Q?Guyon_Mor=E9e?=) Date: Wed, 19 Jul 2006 11:41:13 +0200 Subject: [netflow-tools] C flowd log example In-Reply-To: <78f5441d0607190139x3323b37fx2edad6ed8a76b02e@mail.gmail.com> References: <78f5441d0607180655v3d225577yed6b6aa07196ac2c@mail.gmail.com> <78f5441d0607190139x3323b37fx2edad6ed8a76b02e@mail.gmail.com> Message-ID: <78f5441d0607190241h6758db79xa652bad1aa947bc2@mail.gmail.com> On 7/19/06, Guyon Mor?e wrote: > Thanx Damien, > > When I try to make flowd i get this error: > > [guyon at geek]~/flowread/flowd > $ make > gcc -g -O2 -fPIC -c flowd.c > flowd.c: In function `usage': > flowd.c:1381: error: `PROGVER' undeclared (first use in this function) > flowd.c:1381: error: (Each undeclared identifier is reported only once > flowd.c:1381: error: for each function it appears in.) > flowd.c:1386: error: `SYSCONFDIR' undeclared (first use in this function) > flowd.c:1386: error: syntax error before string constant > flowd.c: In function `main': > flowd.c:1396: error: `SYSCONFDIR' undeclared (first use in this function) > flowd.c:1396: error: syntax error before string constant > *** Error code 1 > > Stop in /home/guyon/flowread/flowd. > > I ran ./configure before this. > > Any idea whats going on? > > thanx again, > > > > > On 7/19/06, Damien Miller wrote: > > On Tue, 18 Jul 2006, Guyon Mor?e wrote: > > > > > Hi all, > > > > > > I'm looking for a a simple example on how to read the flowd logfile, > > > using the C header included. Als I notice 2 header files, store.h and > > > store-v2.h. > > > > > > Which one should i use and how? > > > > store.h is the one you should use for new applications. store-v2.h is > > the legacy log format. It defines a couple of APIs that you can use > > to read flows: > > > > - A FILE* oriented API: store_read_flow() and store_write_flow(). > > This is probably the easiest to use, as stdio will take care of > > buffering, etc. Note that these do not try to back out cleanly > > when a write error occurs, so they are better suited to reading > > than writing. > > > > - A file-descriptor oriented API: store_get_flow() and store_put_flow(). > > These don't do any buffering, but they will back out a failed write. > > > > - Direct serialisation and deserialisation of flow records to/from > > memory buffers: store_flow_deserialise(), store_flow_serialise(), > > store_flow_serialise_masked(), store_calc_flow_len(). You might want > > to use these if you need to store binary flow records though something > > other than a file descriptor or FILE*, such as sending or receiving > > flow records over a Unix domain socket, passing them via shared memory > > or storing them in a database. > > > > Most of these functions take a error buffer (and length) in which they > > store an error message on failure. > > > > flowd-reader.c is a pretty good example of how to use the file-descriptor > > API. > > > > -d > > > > > > > > > > > > > > > -- > Guyon Mor?e > guyon.moree at gmail.com > http://gumuz.looze.net > -- Guyon Mor?e guyon.moree at gmail.com http://gumuz.looze.net From guyon.moree at gmail.com Sat Jul 22 00:28:57 2006 From: guyon.moree at gmail.com (=?ISO-8859-1?Q?Guyon_Mor=E9e?=) Date: Fri, 21 Jul 2006 16:28:57 +0200 Subject: [netflow-tools] flowd python extension on Win32 Message-ID: <78f5441d0607210728g15a24a86sc81fc8b85468dd06@mail.gmail.com> Hi all, Is it possible to compile the flowd python module on windows? Has anyone tried it and succeeded? If so, could someone explain em how to do it? thanx, -- Guyon Mor?e guyon.moree at gmail.com http://gumuz.looze.net From djm at mindrot.org Mon Jul 24 16:23:54 2006 From: djm at mindrot.org (Damien Miller) Date: Mon, 24 Jul 2006 16:23:54 +1000 Subject: [netflow-tools] flowd python extension on Win32 In-Reply-To: <78f5441d0607210728g15a24a86sc81fc8b85468dd06@mail.gmail.com> References: <78f5441d0607210728g15a24a86sc81fc8b85468dd06@mail.gmail.com> Message-ID: <44C4677A.5080700@mindrot.org> Guyon Mor?e wrote: > Hi all, > > Is it possible to compile the flowd python module on windows? Has > anyone tried it and succeeded? If so, could someone explain em how to > do it? I haven't tried, but I expect that it would need a couple of minor tweaks to build properly. I don't have a windows system with development tools set up, but I plan on downloading the gratis "Visual Studio Express" compiler and having a go soon. -d From djm at mindrot.org Wed Jul 26 20:19:39 2006 From: djm at mindrot.org (Damien Miller) Date: Wed, 26 Jul 2006 20:19:39 +1000 Subject: [netflow-tools] flowd python extension on Win32 In-Reply-To: <44C4677A.5080700@mindrot.org> References: <78f5441d0607210728g15a24a86sc81fc8b85468dd06@mail.gmail.com> <44C4677A.5080700@mindrot.org> Message-ID: <44C741BB.10401@mindrot.org> Damien Miller wrote: > I haven't tried, but I expect that it would need a couple of minor > tweaks to build properly. I don't have a windows system with development > tools set up, but I plan on downloading the gratis "Visual Studio > Express" compiler and having a go soon. I have tried to cajole Visual Studio Express C++ to build Python packages, but haven't had any success. Does anyone know the magic combination? I won't have any real time to spend on it for a couple of weeks at least, but if someone has solved it then I might be make a binary bundle of the Python module for Windows. -d