From djm at mindrot.org  Tue Oct  2 14:38:36 2007
From: djm at mindrot.org (Damien Miller)
Date: Tue, 2 Oct 2007 14:38:36 +1000 (EST)
Subject: [netflow-tools] CVS Snapshot Breakage
In-Reply-To: <6CEB7AA2-C7B2-41E5-8799-F6705CA366D9@cs.rpi.edu>
References: <6CEB7AA2-C7B2-41E5-8799-F6705CA366D9@cs.rpi.edu>
Message-ID: <Pine.BSO.4.64.0710021434290.28817@fuyu.mindrot.org>

On Sat, 29 Sep 2007, Jesse Kempf wrote:

> Hi,
> At the end of the email, I've attached the full session.
> In short, compiling the CVS snapshot on FreeBSD 6.2/i386 fails with:
> 
> waffle% make
> gcc -g -O2 -fPIC -c flowd.c
> flowd.c: In function `usage':
> flowd.c:1381: error: `PROGVER' undeclared (first use in this function)

Either your Makefile or your make command is broken. I just tested
http://www.mindrot.org/flowd_snap/flowd-SNAP-20071002.tar.gz on
OpenBSD (which uses BSD make) and Linux (GNU make) and it compiled fine.

You Makefile should contain definitions for these items:

djm at demiurge:~/t/flowd$ grep -E '^(PROGVER|C(PP)?FLAGS)' Makefile
PROGVER=0.9
CFLAGS=-g -O2 -fPIC -D_GNU_SOURCE
CPPFLAGS=-I$(srcdir)  $(PATHFLAGS) $(PROGFLAGS) -DHAVE_CONFIG_H

Perhaps the FreeBSD implicit make rules neglect CPPFLAGS?

-d

From djm at mindrot.org  Tue Oct  2 14:43:26 2007
From: djm at mindrot.org (Damien Miller)
Date: Tue, 2 Oct 2007 14:43:26 +1000 (EST)
Subject: [netflow-tools] Netflow Library
In-Reply-To: <4B99C25B-3EAE-42EA-8C21-9A7FD143E1C7@cs.rpi.edu>
References: <4B99C25B-3EAE-42EA-8C21-9A7FD143E1C7@cs.rpi.edu>
Message-ID: <Pine.BSO.4.64.0710021438440.28817@fuyu.mindrot.org>

On Sat, 29 Sep 2007, Jesse Kempf wrote:

> Hi,
> So far I've moved all the parsers into their own file, and have wired  
> up flowd, at least, to use the library.
> Since I fell asleep trying to read the IPFIX RFC, how much work would  
> need to be done to support IPFIX itself in the parser library?

It would probably not be too much work, as IPFIX is somewhat like 
NetFlow v.9. I haven't looked at the drafts for quite a while though,
so it may have diverged. 

An important task in supporting IPFIX is to identify which subset of
fields to select, add them to store.[ch] and define mappings from the
wire representations to the store fields.

> It might not be a bad idea to combine the parser library from flowd  
> with the encoding library for softflowd to provide an alternative to  
> libfixbuf or whatever CMU is calling their IPFIX library.

Yeah, I have been meaning to unify flowd, softflowd and pfflowd for
a while. The ideal would be to make it easy for {pf,soft}flowd to be
able to log directly to disk (in store.c format), have flowd 
operate as a "translating relay" (e.g. from NetFlow v.9 or IPFIX to
NetFlow v.5 for legacy apps), and be able to replay logs as a stream
of NetFlow packets.

-d


From kempfj2 at cs.rpi.edu  Thu Oct  4 03:57:42 2007
From: kempfj2 at cs.rpi.edu (Jesse Kempf)
Date: Wed, 3 Oct 2007 13:57:42 -0400
Subject: [netflow-tools] CVS Snapshot Breakage
In-Reply-To: <Pine.BSO.4.64.0710021434290.28817@fuyu.mindrot.org>
References: <6CEB7AA2-C7B2-41E5-8799-F6705CA366D9@cs.rpi.edu>
	<Pine.BSO.4.64.0710021434290.28817@fuyu.mindrot.org>
Message-ID: <20071003135742.c7d83623.kempfj2@cs.rpi.edu>

On Tue, 2 Oct 2007 14:38:36 +1000 (EST)
Damien Miller <djm at mindrot.org> wrote:

> On Sat, 29 Sep 2007, Jesse Kempf wrote:
> 
> > Hi,
> > At the end of the email, I've attached the full session.
> > In short, compiling the CVS snapshot on FreeBSD 6.2/i386 fails with:
> > 
> > waffle% make
> > gcc -g -O2 -fPIC -c flowd.c
> > flowd.c: In function `usage':
> > flowd.c:1381: error: `PROGVER' undeclared (first use in this function)
> 
> Either your Makefile or your make command is broken. I just tested
> http://www.mindrot.org/flowd_snap/flowd-SNAP-20071002.tar.gz on
> OpenBSD (which uses BSD make) and Linux (GNU make) and it compiled fine.

Having found that it works with gmake, I checked the FreeBSD ports makefile. I suppose that explains why ports/net-mgmt/flowd/Makefile has "USE_GMAKE=yes" in it.

Whee.

Cheers,
-Jesse

From hugo.rebello at dhl.com  Thu Oct  4 22:04:07 2007
From: hugo.rebello at dhl.com (Hugo Rebello)
Date: Thu, 04 Oct 2007 09:04:07 -0300
Subject: [netflow-tools] Some doubts
Message-ID: <4704D6B7.4030104@dhl.com>

Hello Guys,

I need some help.

There is no problem to perform the command "flowd -f
/usr/local/etc/flowd.conf" its seems working, however the log file
/var/log/flowd carry on empty.

Anybody knows what's happing ?

Thank you.

Cheers,
Hugo




From phatbuckett at gmail.com  Fri Oct  5 11:03:47 2007
From: phatbuckett at gmail.com (Darren Spruell)
Date: Thu, 4 Oct 2007 18:03:47 -0700
Subject: [netflow-tools] Some doubts
In-Reply-To: <4704D6B7.4030104@dhl.com>
References: <4704D6B7.4030104@dhl.com>
Message-ID: <839aec700710041803i715f627pe2c1d542ddd1a10a@mail.gmail.com>

On 10/4/07, Hugo Rebello <hugo.rebello at dhl.com> wrote:
> Hello Guys,
>
> I need some help.
>
> There is no problem to perform the command "flowd -f
> /usr/local/etc/flowd.conf" its seems working, however the log file
> /var/log/flowd carry on empty.

"flowd is a small NetFlow collector daemon..."

Are you sending netflow data to your flowd from a device?

DS

From hugo.rebello at dhl.com  Fri Oct  5 22:19:20 2007
From: hugo.rebello at dhl.com (Hugo Rebello)
Date: Fri, 05 Oct 2007 09:19:20 -0300
Subject: [netflow-tools] Some doubts
In-Reply-To: <839aec700710041803i715f627pe2c1d542ddd1a10a@mail.gmail.com>
References: <4704D6B7.4030104@dhl.com>
	<839aec700710041803i715f627pe2c1d542ddd1a10a@mail.gmail.com>
Message-ID: <47062BC8.5000506@dhl.com>

Yes I am, but I can't see the logs growing.

Cheers,
Hugo





Darren Spruell escreveu:
> On 10/4/07, Hugo Rebello <hugo.rebello at dhl.com> wrote:
>   
>> Hello Guys,
>>
>> I need some help.
>>
>> There is no problem to perform the command "flowd -f
>> /usr/local/etc/flowd.conf" its seems working, however the log file
>> /var/log/flowd carry on empty.
>>     
>
> "flowd is a small NetFlow collector daemon..."
>
> Are you sending netflow data to your flowd from a device?
>
> DS
>
>   

From djm at mindrot.org  Sun Oct  7 00:36:19 2007
From: djm at mindrot.org (Damien Miller)
Date: Sun, 7 Oct 2007 00:36:19 +1000 (EST)
Subject: [netflow-tools] Some doubts
In-Reply-To: <4704D6B7.4030104@dhl.com>
References: <4704D6B7.4030104@dhl.com>
Message-ID: <Pine.BSO.4.64.0710070035200.11245@fuyu.mindrot.org>

On Thu, 4 Oct 2007, Hugo Rebello wrote:

> Hello Guys,
> 
> I need some help.
> 
> There is no problem to perform the command "flowd -f
> /usr/local/etc/flowd.conf" its seems working, however the log file
> /var/log/flowd carry on empty.
> 
> Anybody knows what's happing ?

Try running in debug mode (add -d to the command line)

-d

From kempfj2 at cs.rpi.edu  Wed Oct 10 02:43:09 2007
From: kempfj2 at cs.rpi.edu (Jesse Kempf)
Date: Tue, 9 Oct 2007 12:43:09 -0400
Subject: [netflow-tools] Patch for socket-only logging
Message-ID: <20071009124309.1f8344ee.kempfj2@cs.rpi.edu>

I've inlined the diff at the bottom of this email.

This fixes two issues:
1: When no log file is open, no flows should be put in the disk output
queue.
2: (logsock_num_errors % 10) == 0 will be true when there are no
errors, causing spurious warnings to be generated.

Cheers,
-Jesse Kempf

Index: utils/flowd-dsock-oqueue/flowd.c
===================================================================
--- utils/flowd-dsock-oqueue/flowd.c (revision 62)
+++ utils/flowd-dsock-oqueue/flowd.c (revision 112)
@@ -355,5 +355,5 @@
 		logerrx("%s: exiting on %s", __func__, ebuf);
 
-	if (output_flow_enqueue(fbuf, flen,
+	if (log_fd != -1 && output_flow_enqueue(fbuf, flen,
 	    conf->opts & FLOWD_OPT_VERBOSE) == -1) {
 		output_flow_flush(log_fd, conf->opts &
FLOWD_OPT_VERBOSE); @@ -366,5 +366,5 @@
 	/* Track failures to send on log socket so we can reopen it */
 	if (log_socket != -1 && send(log_socket, fbuf, flen, 0) == -1)
{
-		if ((logsock_num_errors % 10) == 0) {
+		if ((logsock_num_errors > 0) && (logsock_num_errors %
10) == 0) { logit(LOG_WARNING, "log socket send: %s "
 			    "(num errors %d)", strerror(errno),

From djm at fuyu.mindrot.org  Wed Oct 10 12:54:53 2007
From: djm at fuyu.mindrot.org (Damien Miller)
Date: Wed, 10 Oct 2007 12:54:53 +1000 (EST)
Subject: [netflow-tools] CVS: fuyu.mindrot.org: flowd
Message-ID: <20071010025453.C21203C6B3@fuyu.mindrot.org>

CVSROOT:        /var/cvs
Module name:    flowd
Changes by:     djm at fuyu.mindrot.org 07/10/10 12:54:53

Modified files:
    .               : ChangeLog flowd.c

Log message:
 - (djm) Make local socket-only logging work, and fix spurious warnings
   from socket logging. Patch from kempfj2 AT cs.rpi.edu

Diff commands:
cvs -nQq rdiff -u -r1.167 -r1.168 flowd/ChangeLog
cvs -nQq rdiff -u -r1.75 -r1.76 flowd/flowd.c

CVSWeb:
http://cvsweb.mindrot.org/index.cgi/flowd/ChangeLog?r1=1.167;r2=1.168
http://cvsweb.mindrot.org/index.cgi/flowd/flowd.c?r1=1.75;r2=1.76

Please note that there may be a delay before commits are available
on the public CVSWeb site.

From djm at fuyu.mindrot.org  Wed Oct 10 12:55:11 2007
From: djm at fuyu.mindrot.org (Damien Miller)
Date: Wed, 10 Oct 2007 12:55:11 +1000 (EST)
Subject: [netflow-tools] CVS: fuyu.mindrot.org: flowd
Message-ID: <20071010025511.5E9063C6B3@fuyu.mindrot.org>

CVSROOT:        /var/cvs
Module name:    flowd
Changes by:     djm at fuyu.mindrot.org 07/10/10 12:55:11

Modified files:
    .               : ChangeLog
    tools           : flowinsert.pl

Log message:
 - (djm) Unbreak flowinsert.pl

Diff commands:
cvs -nQq rdiff -u -r1.168 -r1.169 flowd/ChangeLog
cvs -nQq rdiff -u -r1.2 -r1.3 flowd/tools/flowinsert.pl

CVSWeb:
http://cvsweb.mindrot.org/index.cgi/flowd/ChangeLog?r1=1.168;r2=1.169
http://cvsweb.mindrot.org/index.cgi/flowd/tools/flowinsert.pl?r1=1.2;r2=1.3

Please note that there may be a delay before commits are available
on the public CVSWeb site.

From djm at mindrot.org  Wed Oct 10 12:56:08 2007
From: djm at mindrot.org (Damien Miller)
Date: Wed, 10 Oct 2007 12:56:08 +1000 (EST)
Subject: [netflow-tools] Patch for socket-only logging
In-Reply-To: <20071009124309.1f8344ee.kempfj2@cs.rpi.edu>
References: <20071009124309.1f8344ee.kempfj2@cs.rpi.edu>
Message-ID: <Pine.BSO.4.64.0710101255240.19497@fuyu.mindrot.org>

On Tue, 9 Oct 2007, Jesse Kempf wrote:

> I've inlined the diff at the bottom of this email.
> 
> This fixes two issues:
> 1: When no log file is open, no flows should be put in the disk output
> queue.
> 2: (logsock_num_errors % 10) == 0 will be true when there are no
> errors, causing spurious warnings to be generated.

Your mailer ate you diff a little, but it was simple enough to apply by 
hand.

Thanks,
Damien

From shane.gaumond at matrox.com  Thu Oct 11 06:26:12 2007
From: shane.gaumond at matrox.com (Shane Gaumond)
Date: Wed, 10 Oct 2007 16:26:12 -0400
Subject: [netflow-tools] inaccurate traffic rates...
In-Reply-To: <mailman.65.1159129849.32009.netflow-tools@mindrot.org>
References: <mailman.65.1159129849.32009.netflow-tools@mindrot.org>
Message-ID: <1192047972.5409.164.camel@Hendrix.matrox.com>


Hello all..

I may have missed something in the new install of NfSen version:
1.3b-20070720.

Is it possible that sampling is enabled by default.  I have a graph
showing me approx 2.4 Gb/s peaks which i can confirm is correct via SNMP
counters.  The statistics read out below the graph is reading a rate of
4.7 Gb/s (double the amount).  When i dselect the SUM button i get 177.9
GB which is also double what it should be.  The same can be send for all
my profiles except the live profile.  All of my profiles are shadow
profiles.  I have created a real profile, I'll collect info overnight
and verify if the problem is present in a real profile as well.

The thing that i find strange is that if a sampling rate of 2:1 is
somehow enabled the graph should reflect it.  If i did something wrong
let me know.  Thanks for the help and thanks for a great piece of
software.


Shane Gaumond
sgaumond at matrox.com



From djm at mindrot.org  Thu Oct 11 16:13:07 2007
From: djm at mindrot.org (Damien Miller)
Date: Thu, 11 Oct 2007 16:13:07 +1000 (EST)
Subject: [netflow-tools] inaccurate traffic rates...
In-Reply-To: <1192047972.5409.164.camel@Hendrix.matrox.com>
References: <mailman.65.1159129849.32009.netflow-tools@mindrot.org>
	<1192047972.5409.164.camel@Hendrix.matrox.com>
Message-ID: <Pine.BSO.4.64.0710111612190.19497@fuyu.mindrot.org>

On Wed, 10 Oct 2007, Shane Gaumond wrote:

> 
> Hello all..
> 
> I may have missed something in the new install of NfSen version:
> 1.3b-20070720.

NfSen isn't maintained here, just flowd[1], softflowd[2] and pfflowd[3].

-d

[1] http://www.mindrot.org/projects/flowd
[2] http://www.mindrot.org/projects/softflowd
[3] http://www.mindrot.org/projects/pfflowd

From kempfj2 at cs.rpi.edu  Wed Oct 24 05:37:04 2007
From: kempfj2 at cs.rpi.edu (Jesse Kempf)
Date: Tue, 23 Oct 2007 15:37:04 -0400
Subject: [netflow-tools] Patch to allow explicit specification of SO_SNDBUF
	and SO_RCVBUF
Message-ID: <20071023153704.38327c8a.kempfj2@cs.rpi.edu>


I've inlined the diff at the bottom of the email.

In short, this patch makes it possible to specify the receive buffer size for each ``listen on'' directive, and the send buffer size for ``logsock''. 

There's some receive buffer guessing for listening sockets already, but I figured it would be more helpful (and more precise) to let the administrator explicitly tune the values. Well, that and I just had need to take a crack at tuning these values.

Cheers,
-Jesse Kempf

Index: flowd-bufsiz/parse.y
===================================================================
--- flowd-bufsiz/parse.y	(revision 173)
+++ flowd-bufsiz/parse.y	(revision 177)
@@ -102,7 +102,7 @@
 
 %}
 
-%token	LISTEN ON JOIN GROUP LOGFILE LOGSOCK STORE PIDFILE FLOW SOURCE
+%token	LISTEN ON JOIN GROUP LOGFILE LOGSOCK BUFSIZE STORE PIDFILE FLOW SOURCE
 %token	ALL TAG ACCEPT DISCARD QUICK AGENT SRC DST PORT PROTO TOS ANY
 %token	TCP_FLAGS EQUALS MASK INET INET6 DAYS AFTER BEFORE DATE
 %token  IN_IFNDX OUT_IFNDX
@@ -385,8 +385,22 @@
 			la->fd = -1;
 			la->addr = $3.addr;
 			la->port = $3.port;
+			la->bufsiz = 0;
 			TAILQ_INSERT_TAIL(&conf->listen_addrs, la, entry);
 		}
+		| LISTEN ON address_port BUFSIZE number {
+			struct listen_addr	*la;
+
+			if ((la = calloc(1, sizeof(*la))) == NULL)
+				logerrx("listen_on: calloc");
+
+			la->fd = -1;
+			la->addr = $3.addr;
+			la->port = $3.port;
+			la->bufsiz = $5;
+			TAILQ_INSERT_TAIL(&conf->listen_addrs, la, entry);
+
+		}
 		| FLOW SOURCE prefix_or_any	{
 			struct allowed_device	*ad;
 
@@ -414,6 +428,10 @@
 		| LOGSOCK string		{
 			conf->log_socket = $2;
 		}
+		| LOGSOCK string BUFSIZE number {
+			conf->log_socket = $2;
+			conf->log_socket_bufsiz = $4;
+		}
 		| PIDFILE string		{
 			conf->pid_file = $2;
 		}
@@ -902,6 +920,7 @@
 		{ "all",		ALL},
 		{ "any",		ANY},
 		{ "before",		BEFORE},
+		{ "bufsize",		BUFSIZE},
 		{ "date",		DATE},
 		{ "days",		DAYS},
 		{ "discard",		DISCARD},
Index: flowd-bufsiz/flowd.c
===================================================================
--- flowd-bufsiz/flowd.c	(revision 173)
+++ flowd-bufsiz/flowd.c	(revision 177)
@@ -1365,7 +1365,7 @@
 	struct listen_addr *la;
 
 	TAILQ_FOREACH(la, &conf->listen_addrs, entry) {
-		if ((la->fd = open_listener(&la->addr, la->port,
+		if ((la->fd = open_listener(&la->addr, la->port, la->bufsiz,
 		    &conf->join_groups)) == -1) {
 			logerrx("Listener setup of [%s]:%d failed",
 			    addr_ntop_buf(&la->addr), la->port);
Index: flowd-bufsiz/flowd.h
===================================================================
--- flowd-bufsiz/flowd.h	(revision 173)
+++ flowd-bufsiz/flowd.h	(revision 177)
@@ -55,6 +55,7 @@
 	struct xaddr			addr;
 	u_int16_t			port;
 	int				fd;
+	size_t				bufsiz;
 	TAILQ_ENTRY(listen_addr)	entry;
 };
 TAILQ_HEAD(listen_addrs, listen_addr);
@@ -72,6 +73,7 @@
 struct flowd_config {
 	char			*log_file;
 	char			*log_socket;
+	size_t			log_socket_bufsiz;
 	char			*pid_file;
 	u_int32_t		store_mask;
 	u_int32_t		opts;
Index: flowd-bufsiz/flowd.conf.5.in
===================================================================
--- flowd-bufsiz/flowd.conf.5.in	(revision 173)
+++ flowd-bufsiz/flowd.conf.5.in	(revision 177)
@@ -135,7 +135,20 @@
 listen on [::]:12345
 .Ed
 .Pp
-This option is mandatory, there is no default value.
+
+This option accepts the modifier
+.Pa bufsize
+to allow the specification (in bytes) of the receive buffer for this socket.
+.Pp
+For example,
+.Bd -literal -offset indent
+listen on 0.0.0.0:12345 bufsize 9876
+.Ed
+.Pp
+The
+.Cm listen on
+directive is mandatory. There is no default value.
+
 .It Ar logfile
 Specifies the file in which the received flow records are stored.
 The full path to the file must be specified in quotes.
@@ -161,7 +174,18 @@
 logsock "/var/log/flowd.sock"
 .Ed
 .Pp
-There is no default value for this option and it it mandatory 
+This option accepts the modifier
+.Pa bufsize
+to allow the specification (in bytes) of the send buffer for this socket.
+.Pp
+For example,
+.Bd -literal -offset indent
+logsock "/var/log/flowd.sock" bufsize 1234
+.Ed
+.Pp
+There is no default value for
+.Cm logfile
+and it is mandatory 
 to specify at least one of the
 .Cm logfile
 and
Index: flowd-bufsiz/privsep.c
===================================================================
--- flowd-bufsiz/privsep.c	(revision 173)
+++ flowd-bufsiz/privsep.c	(revision 177)
@@ -119,7 +119,7 @@
 }
 
 int
-open_listener(struct xaddr *addr, u_int16_t port, struct join_groups *groups)
+open_listener(struct xaddr *addr, u_int16_t port, size_t bufsiz, struct join_groups *groups)
 {
 	int fd, fl, i, orig;
 	struct sockaddr_storage ss;
@@ -167,10 +167,23 @@
 	logit(LOG_DEBUG, "Listener for [%s]:%d fd = %d", addr_ntop_buf(addr),
 	    port, fd);
 
-	/* Crank up socket receive buffer size to cope with bursts of flows */
+	/* Crank up socket receive buffer size to cope with bursts of flows
+	 * If the config doesn't contain an explicit buffer size, we
+	 * fall back to guessing.
+	 */
+
 	slen = sizeof(fl);
 	if (getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &orig, &slen) == 0) {
-		for (i = 3; i >= 1; i--) {
+		if (bufsiz > 0) {
+			fl = bufsiz;
+			if (setsockopt(fd, SOL_SOCKET, SO_RCVBUF,
+				&bufsiz, sizeof(bufsiz)) == 0) {
+				logit(LOG_DEBUG, "Increased socket receive "
+					"buffer from %d to %d", orig, fl);
+			} else if (i == 1)
+				logerr("%s: setsockopt(SO_RCVBUF)", __func__);
+		} else{
+		    for (i = 3; i >= 1; i--) {
 			fl = (1024 * 64) << i;
 			if (setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &fl,
 			    sizeof(fl)) == 0) {
@@ -179,6 +192,7 @@
 				break;
 			} else if (i == 1)
 				logitm(LOG_DEBUG, "setsockopt(SO_RCVBUF)");
+		    }
 		}
 	}
 
@@ -300,6 +314,13 @@
 	newconf.log_file = privsep_read_string(fd, 1);
 	newconf.log_socket = privsep_read_string(fd, 1);
 
+	if (atomicio(read, fd, &newconf.log_socket_bufsiz,
+	    sizeof(newconf.log_socket_bufsiz)) !=
+	    sizeof(newconf.log_socket_bufsiz)) {
+		logitm(LOG_ERR, "%s: read(conf.log_socket_bufsiz)", __func__);
+		return (-1);
+	}
+
 	if ((newconf.pid_file = privsep_read_string(fd, 0)) == NULL) {
 		logit(LOG_ERR, "%s: Couldn't read conf.pid_file", __func__);
 		return (-1);
@@ -431,6 +452,13 @@
 		logit(LOG_ERR, "%s: Couldn't write conf.log_socket", __func__);
 		return (-1);
 	}
+	if (atomicio(vwrite, fd, &conf->log_socket_bufsiz,
+	    sizeof(conf->log_socket_bufsiz)) !=
+	    sizeof(conf->log_socket_bufsiz)) {
+		logitm(LOG_ERR, "%s: write(conf.log_socket_bufsiz)", __func__);
+		return (-1);
+	}
+
 	if (privsep_write_string(fd, conf->pid_file, 0) == -1) {
 		logit(LOG_ERR, "%s: Couldn't write conf.pid_file", __func__);
 		return (-1);
@@ -581,7 +609,7 @@
 	FILE *cfg;
 	struct passwd *pw = NULL;
 	struct flowd_config newconf = {
-		NULL, NULL, NULL, 0, 0,
+		NULL, NULL, 0, NULL, 0, 0,
 		TAILQ_HEAD_INITIALIZER(newconf.listen_addrs),
 		TAILQ_HEAD_INITIALIZER(newconf.filter_list),
 		TAILQ_HEAD_INITIALIZER(newconf.allowed_devices),
@@ -786,7 +814,7 @@
 static int
 answer_open_socket(struct flowd_config *conf, int client_fd)
 {
-	int fd;
+	int fd, slen, orig;
 	struct sockaddr_un to;
 	socklen_t tolen;
 
@@ -816,6 +844,20 @@
 		return (-1);
 	}
 
+	slen = sizeof(orig);
+	if (conf->log_socket_bufsiz > 0 &&
+	    getsockopt(fd, SOL_SOCKET, SO_SNDBUF, &orig, &slen) == 0) {
+		
+		if (setsockopt(fd, SOL_SOCKET, SO_SNDBUF, 
+		    &conf->log_socket_bufsiz,
+		    sizeof(conf->log_socket_bufsiz)) == 0) {
+			logit(LOG_DEBUG, "Increased log_socket send "
+			    "buffer from %d to %d", orig,
+			    conf->log_socket_bufsiz);
+		} else
+			logerr("%s: setsockopt(SO_SNDBUF)", __func__);
+	}
+
 	if (send_fd(client_fd, fd) == -1)
 		return (-1);
 
@@ -850,7 +892,7 @@
 	}
 
 	TAILQ_FOREACH(la, &newconf.listen_addrs, entry) {
-		if ((la->fd = open_listener(&la->addr, la->port,
+		if ((la->fd = open_listener(&la->addr, la->port, la->bufsiz,
 		    &conf->join_groups)) == -1) {
 			logit(LOG_ERR, "Listener setup of [%s]:%d failed",
 			    addr_ntop_buf(&la->addr), la->port);
Index: flowd-bufsiz/privsep.h
===================================================================
--- flowd-bufsiz/privsep.h	(revision 173)
+++ flowd-bufsiz/privsep.h	(revision 177)
@@ -27,7 +27,7 @@
 void privsep_init(struct flowd_config *, int *, const char *);
 int client_open_log(int);
 int client_open_socket(int);
-int open_listener(struct xaddr *, u_int16_t, struct join_groups *);
+int open_listener(struct xaddr *, u_int16_t, size_t, struct join_groups *);
 int read_config(const char *, struct flowd_config *);
 int client_reconfigure(int, struct flowd_config *);
 

From gijs at e-commercepark.com  Wed Oct 24 10:08:45 2007
From: gijs at e-commercepark.com (Gijs Molenaar)
Date: Tue, 23 Oct 2007 20:08:45 -0400
Subject: [netflow-tools] softflowd non-IP problem
Message-ID: <005701c815d2$0b5c4150$6600000a@ecp.noc>

Hello,

I'm trying to set up a softflowd configuration but I encountered a problem.

We have 2 identical switches which I configured with port mirroring. Both
mirroring ports sent all the data to 2 network interfaces on a FreeBSD 6.2
machine. On the FreeBSD machine I'm running 2 instances of softflowd. 

Commands:
/usr/local/sbin/softflowd -i em2 -n 127.0.0.1:8818 -c IDCCORE1.sock
/usr/local/sbin/softflowd -i em3 -n 127.0.0.1:8828 -c IDCCORE2.sock

The problem is, one of the softflow processes is rejecting IP packages, but
the other one isn't.

# softflowctl -c IDCCORE1.sock statistics
softflowd[1552]: Accumulated statistics:
Number of active flows: 0
Packets processed: 0
Fragments: 0
Ignored packets: 14537066 (14537066 non-IP, 0 too short)      <-----------
Flows expired: 0 (0 forced)
Flows exported: 0 in 0 packets (0 failures)
Packets received by libpcap: 14537387
Packets dropped by libpcap: 0
Packets dropped by interface: 3217012700


# softflowctl -c IDCCORE2.sock statistics
softflowd[1550]: Accumulated statistics:
Number of active flows: 12
Packets processed: 115
Fragments: 0
Ignored packets: 1455 (1455 non-IP, 0 too short)
Flows expired: 0 (0 forced)
Flows exported: 0 in 0 packets (0 failures)
Packets received by libpcap: 1692
Packets dropped by libpcap: 0
Packets dropped by interface: 3217012700

IDCCORE1 receives a lot more data (50Mb/s) than IDCORE2 (1Kb/s). When I look
at the interfaces em2 and em3 with tcpdump I see normal TCP, ICMP and BGP
data... I'm using version 0.9.8 of softflowd, build from the freebsd ports
repository. Running debug mode (-d) doesn't really help me either. I also
tried to run only one instance of softflowd, but this doesn't help either. 

Does anyone have an idea or a suggestion? Thanks!

 - gijs


From djm at mindrot.org  Wed Oct 24 10:50:11 2007
From: djm at mindrot.org (Damien Miller)
Date: Wed, 24 Oct 2007 10:50:11 +1000 (EST)
Subject: [netflow-tools] softflowd non-IP problem
In-Reply-To: <005701c815d2$0b5c4150$6600000a@ecp.noc>
References: <005701c815d2$0b5c4150$6600000a@ecp.noc>
Message-ID: <Pine.BSO.4.64.0710241045100.13542@fuyu.mindrot.org>

On Tue, 23 Oct 2007, Gijs Molenaar wrote:

> Hello,
> 
> I'm trying to set up a softflowd configuration but I encountered a problem.
> 
> We have 2 identical switches which I configured with port mirroring. Both
> mirroring ports sent all the data to 2 network interfaces on a FreeBSD 6.2
> machine. On the FreeBSD machine I'm running 2 instances of softflowd. 
> 
> Commands:
> /usr/local/sbin/softflowd -i em2 -n 127.0.0.1:8818 -c IDCCORE1.sock

Is it always this softflowd instance that doesn't work? I.e. is it only
packets coming in on em2 that are treated as non-IP?

My first guess is that your switch is giving you VLAN tagged packets on
this interface. I guess you might be able to tell by doing a 
"tcpdump -evvv" on them.

-d

From djm at fuyu.mindrot.org  Wed Oct 24 11:04:11 2007
From: djm at fuyu.mindrot.org (Damien Miller)
Date: Wed, 24 Oct 2007 11:04:11 +1000 (EST)
Subject: [netflow-tools] CVS: fuyu.mindrot.org: flowd
Message-ID: <20071024010411.306103C6B3@fuyu.mindrot.org>

CVSROOT:        /var/cvs
Module name:    flowd
Changes by:     djm at fuyu.mindrot.org 07/10/24 11:04:11

Modified files:
    .               : ChangeLog flowd.c flowd.conf.5.in flowd.h parse.y privsep.c privsep.h

Log message:
 - (djm) Support explicit specification of "listen on" and "logsock"
   socket buffer sizes. Patch from kempfj2 AT cs.rpi.edu

Diff commands:
cvs -nQq rdiff -u -r1.169 -r1.170 flowd/ChangeLog
cvs -nQq rdiff -u -r1.76 -r1.77 flowd/flowd.c
cvs -nQq rdiff -u -r1.14 -r1.15 flowd/flowd.conf.5.in
cvs -nQq rdiff -u -r1.18 -r1.19 flowd/flowd.h
cvs -nQq rdiff -u -r1.35 -r1.36 flowd/parse.y
cvs -nQq rdiff -u -r1.31 -r1.32 flowd/privsep.c
cvs -nQq rdiff -u -r1.8 -r1.9 flowd/privsep.h

CVSWeb:
http://cvsweb.mindrot.org/index.cgi/flowd/ChangeLog?r1=1.169;r2=1.170
http://cvsweb.mindrot.org/index.cgi/flowd/flowd.c?r1=1.76;r2=1.77
http://cvsweb.mindrot.org/index.cgi/flowd/flowd.conf.5.in?r1=1.14;r2=1.15
http://cvsweb.mindrot.org/index.cgi/flowd/flowd.h?r1=1.18;r2=1.19
http://cvsweb.mindrot.org/index.cgi/flowd/parse.y?r1=1.35;r2=1.36
http://cvsweb.mindrot.org/index.cgi/flowd/privsep.c?r1=1.31;r2=1.32
http://cvsweb.mindrot.org/index.cgi/flowd/privsep.h?r1=1.8;r2=1.9

Please note that there may be a delay before commits are available
on the public CVSWeb site.

From djm at mindrot.org  Wed Oct 24 11:05:53 2007
From: djm at mindrot.org (Damien Miller)
Date: Wed, 24 Oct 2007 11:05:53 +1000 (EST)
Subject: [netflow-tools] Patch to allow explicit specification of
 SO_SNDBUF and SO_RCVBUF
In-Reply-To: <20071023153704.38327c8a.kempfj2@cs.rpi.edu>
References: <20071023153704.38327c8a.kempfj2@cs.rpi.edu>
Message-ID: <Pine.BSO.4.64.0710241104150.13542@fuyu.mindrot.org>

On Tue, 23 Oct 2007, Jesse Kempf wrote:

> 
> I've inlined the diff at the bottom of the email.
>
> In short, this patch makes it possible to specify the receive buffer
> size for each ``listen on'' directive, and the send buffer size for
> ``logsock''.
>
> There's some receive buffer guessing for listening sockets already,
> but I figured it would be more helpful (and more precise) to let the
> administrator explicitly tune the values. Well, that and I just had
> need to take a crack at tuning these values.

Applied (with some trivial tweaks) - thanks!

It would be good if you could specify a scale suffix to the bufsize number,
e.g. "listen on 0.0.0.0:1234 bufsize 512k", if you are interested in doing
more work :)

-d