[netflow-tools] problems with pfflowd that don't happen with softflowd
Michael W. Lucas
mwlucas at blackhelicopters.org
Sun May 1 03:13:01 EST 2005
On Sat, Apr 30, 2005 at 11:02:18PM +1000, Damien Miller wrote:
> Michael W. Lucas wrote:
> >>That is possible: softflowd's timeouts are pretty conservative,
> >>especially for TCP - 30 minutes post-FIN. You can tune these on the
> >>commandline though :)
> >
> >
> >Ah, you learn something every day.
> >
> >That 30 minutes can be important when you want to, say, grovel through
> >netflow data looking for port scans and worms and whatnot. :-)
>
> gah, it it not 30 minutes, it is really 5 minutes post-FIN. I probably
> started out writing 300 seconds, but botched changing the units. My
> apologies.
OK, no sweat. That's better to work with, anyway.
I will do the below during my next maintenance window (tonight).
>
> >>When you run tcpdump, did you try the "-T cnfp" to get it to parse the
> >>NetFlow packets? What collector are you using?
> >
> >I'm using flow-tools' flow-capture.
> >
> >Under softflowd, -T cnfp on the collector shows:
> >
> >08:08:01.118622 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007
> >uptime, 1114862881.120821000, #0, 29 recs
> >08:08:01.118770 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007
> >uptime, 1114862881.120821000, #0, 30 recs
> >08:08:01.118918 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007
> >uptime, 1114862881.120821000, #0, 29 recs
> >08:08:01.118923 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007
> >uptime, 1114862881.120821000, #0, 29 recs
>
> OK, that looks like a softflowd bug there - it is not updating the
> sequence number correctly. I can see one bug there (fix attached), but
> I don't see how it could give rise to all the sequence numbers being
> zero like this...
>
> >with pfflowd, we see:
> >
> >08:10:20.540514 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime,
> >1114863020.542900000, #0, 12 recs
> >08:10:20.540519 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime,
> >1114863020.542953000, #12, 12 recs
> >08:10:20.540662 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime,
> >1114863020.542971000, #24, 12 recs
> >08:10:20.540668 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime,
> >1114863020.542987000, #36, 11 recs
> >
> >The only difference I see is that with softflowd, the # doesn't
> >increment.
>
> There might be more to it. Could you try capturing with
> "tcpdump -s1500 -vvvTcnfp"? It will show some more details.
>
> >I'm willing to accept that there's something in flow-capture that's
> >choking on something with pfflowd. Still, it seems that there will be
> >more people on this list using flow-capture than there will be on the
> >flow-capture list using pfflowd and softflowd. :-)
> >
> >I'm also using the same flow-capture binary to collect netflow from a
> >Cisco router, so I somehow think that there's some detail with pfflowd
> >that I just don't have right.
>
> pfflowd doesn't fill in all the NetFlow fields - it just doesn't get
> enough information for the pfsync interface to do it. IIRC some programs
> may not like getting a 0 interface index.
>
> You could try the second attached patch, which is a very crude hack that
> might fool it (just hardcode a couple of interface numbers).
>
> -d
> Index: netflow1.c
> ===================================================================
> RCS file: /var/cvs/softflowd/netflow1.c,v
> retrieving revision 1.1
> diff -u -p -r1.1 netflow1.c
> --- netflow1.c 13 Sep 2004 02:25:09 -0000 1.1
> +++ netflow1.c 30 Apr 2005 12:54:20 -0000
> @@ -65,7 +65,7 @@ struct NF1_FLOW {
> */
> int
> send_netflow_v1(struct FLOW **flows, int num_flows, int nfsock,
> - u_int64_t flows_exported, struct timeval *system_boot_time,
> + u_int64_t *flows_exported, struct timeval *system_boot_time,
> int verbose_flag)
> {
> struct timeval now;
> @@ -90,6 +90,7 @@ send_netflow_v1(struct FLOW **flows, int
> &err, &errsz); /* Clear ICMP errors */
> if (send(nfsock, packet, (size_t)offset, 0) == -1)
> return (-1);
> + *flows_exported += j;
> j = 0;
> num_packets++;
> }
> @@ -162,5 +163,6 @@ send_netflow_v1(struct FLOW **flows, int
> num_packets++;
> }
>
> + *flows_exported += j;
> return (num_packets);
> }
> Index: netflow5.c
> ===================================================================
> RCS file: /var/cvs/softflowd/netflow5.c,v
> retrieving revision 1.2
> diff -u -p -r1.2 netflow5.c
> --- netflow5.c 29 Sep 2004 03:57:10 -0000 1.2
> +++ netflow5.c 30 Apr 2005 12:54:20 -0000
> @@ -63,7 +63,7 @@ struct NF5_FLOW {
> */
> int
> send_netflow_v5(struct FLOW **flows, int num_flows, int nfsock,
> - u_int64_t flows_exported, struct timeval *system_boot_time,
> + u_int64_t *flows_exported, struct timeval *system_boot_time,
> int verbose_flag)
> {
> struct timeval now;
> @@ -78,7 +78,7 @@ send_netflow_v5(struct FLOW **flows, int
> uptime_ms = timeval_sub_ms(&now, system_boot_time);
>
> hdr = (struct NF5_HEADER *)packet;
> - for(num_packets = offset = j = i = 0; i < num_flows; i++) {
> + for (num_packets = offset = j = i = 0; i < num_flows; i++) {
> if (j >= NF5_MAXFLOWS - 1) {
> if (verbose_flag)
> logit(LOG_DEBUG, "Sending flow packet len = %d", offset);
> @@ -88,6 +88,7 @@ send_netflow_v5(struct FLOW **flows, int
> &err, &errsz); /* Clear ICMP errors */
> if (send(nfsock, packet, (size_t)offset, 0) == -1)
> return (-1);
> + *flows_exported += j;
> j = 0;
> num_packets++;
> }
> @@ -98,7 +99,7 @@ send_netflow_v5(struct FLOW **flows, int
> hdr->uptime_ms = htonl(uptime_ms);
> hdr->time_sec = htonl(now.tv_sec);
> hdr->time_nanosec = htonl(now.tv_usec * 1000);
> - hdr->flow_sequence = htonl(flows_exported);
> + hdr->flow_sequence = htonl(*flows_exported);
> /* Other fields are left zero */
> offset = sizeof(*hdr);
> }
> @@ -163,6 +164,7 @@ send_netflow_v5(struct FLOW **flows, int
> num_packets++;
> }
>
> + *flows_exported += j;
> return (num_packets);
> }
>
> Index: netflow9.c
> ===================================================================
> RCS file: /var/cvs/softflowd/netflow9.c,v
> retrieving revision 1.6
> diff -u -p -r1.6 netflow9.c
> --- netflow9.c 10 Jan 2005 01:02:34 -0000 1.6
> +++ netflow9.c 30 Apr 2005 12:54:20 -0000
> @@ -248,7 +248,7 @@ nf_flow_to_flowset(const struct FLOW *fl
> */
> int
> send_netflow_v9(struct FLOW **flows, int num_flows, int nfsock,
> - u_int64_t flows_exported, struct timeval *system_boot_time,
> + u_int64_t *flows_exported, struct timeval *system_boot_time,
> int verbose_flag)
> {
> struct NF9_HEADER *nf9;
> @@ -275,7 +275,7 @@ send_netflow_v9(struct FLOW **flows, int
> nf9->flows = 0; /* Filled as we go, htons at end */
> nf9->uptime_ms = htonl(timeval_sub_ms(&now, system_boot_time));
> nf9->time_sec = htonl(time(NULL));
> - nf9->package_sequence = htonl(flows_exported);
> + nf9->package_sequence = htonl(*flows_exported + j);
> nf9->source_id = 0;
> offset = sizeof(*nf9);
>
> @@ -365,5 +365,6 @@ send_netflow_v9(struct FLOW **flows, int
> j += i;
> }
>
> + *flows_exported += j;
> return (num_packets);
> }
> Index: softflowd.c
> ===================================================================
> RCS file: /var/cvs/softflowd/softflowd.c,v
> retrieving revision 1.80
> diff -u -p -r1.80 softflowd.c
> --- softflowd.c 10 Jan 2005 01:50:07 -0000 1.80
> +++ softflowd.c 30 Apr 2005 12:54:20 -0000
> @@ -92,7 +92,7 @@ static const struct DATALINK lt[] = {
> };
>
> /* Netflow send functions */
> -typedef int (netflow_send_func_t)(struct FLOW **, int, int, u_int64_t,
> +typedef int (netflow_send_func_t)(struct FLOW **, int, int, u_int64_t *,
> struct timeval *, int);
> struct NETFLOW_SENDER {
> int version;
> @@ -774,12 +774,11 @@ check_expired(struct FLOWTRACK *ft, stru
> if (num_expired > 0) {
> if (target != NULL && target->fd != -1) {
> r = target->dialect->func(expired_flows, num_expired,
> - target->fd, ft->flows_exported,
> + target->fd, &ft->flows_exported,
> &ft->system_boot_time, verbose_flag);
> if (verbose_flag)
> logit(LOG_DEBUG, "sent %d netflow packets", r);
> if (r > 0) {
> - ft->flows_exported += num_expired * 2;
> ft->packets_sent += r;
> /* XXX what if r < num_expired * 2 ? */
> } else {
> Index: softflowd.h
> ===================================================================
> RCS file: /var/cvs/softflowd/softflowd.h,v
> retrieving revision 1.7
> diff -u -p -r1.7 softflowd.h
> --- softflowd.h 30 Sep 2004 04:12:36 -0000 1.7
> +++ softflowd.h 30 Apr 2005 12:54:20 -0000
> @@ -189,13 +189,13 @@ u_int32_t timeval_sub_ms(const struct ti
>
> /* Prototypes for functions to send NetFlow packets, from netflow*.c */
> int send_netflow_v1(struct FLOW **flows, int num_flows, int nfsock,
> - u_int64_t flows_exported, struct timeval *system_boot_time,
> + u_int64_t *flows_exported, struct timeval *system_boot_time,
> int verbose_flag);
> int send_netflow_v5(struct FLOW **flows, int num_flows, int nfsock,
> - u_int64_t flows_exported, struct timeval *system_boot_time,
> + u_int64_t *flows_exported, struct timeval *system_boot_time,
> int verbose_flag);
> int send_netflow_v9(struct FLOW **flows, int num_flows, int nfsock,
> - u_int64_t flows_exported, struct timeval *system_boot_time,
> + u_int64_t *flows_exported, struct timeval *system_boot_time,
> int verbose_flag);
>
> #endif /* _SOFTFLOWD_H */
> ? pfflowd
> Index: pfflowd.c
> ===================================================================
> RCS file: /var/cvs/pfflowd/pfflowd.c,v
> retrieving revision 1.18
> diff -u -p -r1.18 pfflowd.c
> --- pfflowd.c 6 Sep 2004 12:25:57 -0000 1.18
> +++ pfflowd.c 30 Apr 2005 13:00:41 -0000
> @@ -319,6 +319,9 @@ send_netflow_v1(const struct _PFSYNC_STA
> flw->flow_finish = htonl(uptime_ms);
> flw->protocol = st[i].proto;
> flw->tcp_flags = 0;
> + /* XXX: hack */
> + flw->if_index_in = htons(1);
> + flw->if_index_out = htons(2);
> offset += sizeof(*flw);
> j++;
> hdr->flows++;
> @@ -335,6 +338,9 @@ send_netflow_v1(const struct _PFSYNC_STA
> flw->flow_finish = htonl(uptime_ms);
> flw->protocol = st[i].proto;
> flw->tcp_flags = 0;
> + /* XXX: hack */
> + flw->if_index_in = htons(2);
> + flw->if_index_out = htons(1);
> offset += sizeof(*flw);
> j++;
> hdr->flows++;
> @@ -491,6 +497,9 @@ send_netflow_v5(const struct _PFSYNC_STA
> flw->flow_finish = htonl(uptime_ms);
> flw->tcp_flags = 0;
> flw->protocol = st[i].proto;
> + /* XXX: hack */
> + flw->if_index_in = htons(1);
> + flw->if_index_out = htons(2);
> offset += sizeof(*flw);
> j++;
> hdr->flows++;
> @@ -507,6 +516,9 @@ send_netflow_v5(const struct _PFSYNC_STA
> flw->flow_finish = htonl(uptime_ms);
> flw->tcp_flags = 0;
> flw->protocol = st[i].proto;
> + /* XXX: hack */
> + flw->if_index_in = htons(2);
> + flw->if_index_out = htons(1);
> offset += sizeof(*flw);
> j++;
> hdr->flows++;
--
Michael W. Lucas mwlucas at FreeBSD.org, mwlucas at BlackHelicopters.org
http://www.BlackHelicopters.org/~mwlucas/
Latest book: Cisco Routers for the Desperate
http://www.CiscoRoutersForTheDesperate.com
More information about the netflow-tools
mailing list