[netflow-tools] Cisco ASA OS 9 flowd errors

Fri Feb 21 23:48:26 EST 2014

John,

One "simple" patch for ASA 9  would be to (a) only recognize flow-update events and (b) create two unidirectional flows from each bidirectional flow -- swap the IP/port/ifIndex and include only the appropriate byte counter.

Not only is it simple, but the resulting data would be fully compatible with existing scripts and tools that work on unidirectional flows.

-Craig

________________________________
From: John Marrett [johnf at zioncluster.ca]
Sent: Friday, February 21, 2014 5:17 AM
To: Craig Weinhold
Cc: netflow-tools at mindrot.org
Subject: Re: [netflow-tools] Cisco ASA OS 9 flowd errors

I should have known that Craig would know the subject matter well, you're really throwing down with are you up to the challenge!

Can't believe I didn't think to check the netflow docs as I did last time. You've linked us to some very interesting information.

My little patch for ASA 8 Netflow gives a very basic interpretation, it's enough to allow you to use flow-tools to perform very basic reporting, but doesn't provide any temporal information as there is only a single packet on close. As such it's unsuited to use with the most powerful open source flow reporting system ( http://wvnetflow.sourceforge.net/ ).

ASA 9 seems much more interesting as a full implementation would allow an ASA to function as a full featured collector. It's unfortunate that none of the fields align with standard fields.

My reading also seems to suggest that I would need to configure the collector in a specific fashion; this annotation in the NSEL documentation:

"Different events in the life of a flow may be issued in separate NetFlow packets and may arrive out-of-order at the collector. For example, the packet containing a flow teardown event may reach the collector before the packet containing a flow creation event. As a result, it is important that collector applications use the Event Time field to correlate events. " [1] and "a configurable CLI parameter is provided to delay sending of the flow-create event. If the timer fires, the flow-create event is sent. However, if the flow is torn down before the timer expires, only the flow-teardown event is sent; no flow-create event is sent. " [2]

As I read this it would probably be best to ensure that the refresh-interval is the same as, or one second less than the delay flow-create timer. [3]

I'm not certain how different this behaviour is from regular netflow as it's been several years since I went this deep into netflow packet collection and processing and my last visit was with much less ambitious goals. If you look at my patch [4] you'll see that only fairly small changes were required to make things work; I wonder if they changed things in such a way that the "standard" netflow v9 field identifiers used on other platforms no longer match the ASA 9 ones, even though their field  content is the same. If this is the case then it could actually be pretty easy to address.

If anyone else is interested in this subject please feel free to contact me on or off list. I can provide all kinds of assistance captures and potentially even (remote) hardware access. I will probably start in on this to see how challenging it actually is, if it's not as bad as I fear it is then I may be able to get something done fairly quickly. It's not entirely clear to me why the fields

All of this said, I think it's clear that I AM up to the challenge!

Whether my availability and scheduling are is an open question however :(

I've got a lot on my plate at home and work; in the environment where I have these ASAs running as collectors I now have new options (N7k Core Switches) which give me other substantially easier options to address the issue as well.

[1] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1029397
[2] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1028239
[3] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/configuration/general/asa_91_general_config/monitor_nsel.html#wp1301461
[4] http://zioncluster.ca/netflow/asa_patch_2.diff

-JohnF

On Thu, Feb 20, 2014 at 11:17 PM, Craig Weinhold <Craig.Weinhold at cdw.com<mailto:Craig.Weinhold at cdw.com>> wrote:
NSEL from ASA is quite different from traditional NetFlow, and it changed with ASA 9. Each flow is a bidirectional entity with separate byte/packet counters in each direction (previously there was one total byte counter). There are also new event types -- flow alert and flow update.

ASA 8.x http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/netflow/netflow.html

ASA 9.x  http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html

I'd love to see a complete implementation of NSEL in flowd, but that requires changing the output format. You up to the challenge?

-Craig

________________________________
From: netflow-tools [netflow-tools-bounces+craig.weinhold=cdw.com at mindrot.org<mailto:cdw.com at mindrot.org>] on behalf of John Marrett [johnf at zioncluster.ca<mailto:johnf at zioncluster.ca>]
Sent: Thursday, February 20, 2014 8:29 PM
To: netflow-tools at mindrot.org<mailto:netflow-tools at mindrot.org>
Subject: [netflow-tools] Cisco ASA OS 9 flowd errors

I'm running a version of flowd 0.9.1 with my ASA patches applied ( http://zioncluster.ca/netflow/asa_patch_2.diff ).

I've recently realized that when running against flows from ASA running versions of 9.1(4) (and probably earlier releases in 9) I'm seeing error messages and no data is recorded to disk.

When templates are received I see the following:

NetFlow v.9 template set from 1.1.1.1/0x0<http://1.1.1.1/0x0> with len 1368:
 Contains template 0x00000000/0x0100 with 21 records (offset 8):
forced deletion of template 0x0100 from peer 1.1.1.1/0x00000000<http://1.1.1.1/0x00000000>
 Contains template 0x00000000/0x0101 with 21 records (offset 96):
forced deletion of template 0x0101 from peer 1.1.1.1/0x00000000<http://1.1.1.1/0x00000000>
 Contains template 0x00000000/0x0102 with 21 records (offset 184):
forced deletion of template 0x0102 from peer 1.1.1.1/0x00000000<http://1.1.1.1/0x00000000>
[...]

Even after receipt of the template I see the following:

netflow v.9 packet (len 1412) 17 recs, source 0x00000000
netflow v.9 data flowset (len 104) source 0x00000000
netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0100<http://1.1.1.1/0x00000000/0x0100>
netflow v.9 data flowset (len 68) source 0x00000000
netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0107<http://1.1.1.1/0x00000000/0x0107>
netflow v.9 data flowset (len 104) source 0x00000000
[...]

When I compare it with another host running an older version I see different log information.

NetFlow v.9 template set from 1.1.1.2/0x0<http://1.1.1.2/0x0> with len 992:
 Contains template 0x00000000/0x0100 with 21 records (offset 8):
 Contains template 0x00000000/0x0101 with 21 records (offset 96):
 Contains template 0x00000000/0x0102 with 17 records (offset 184):
 Contains template 0x00000000/0x0103 with 17 records (offset 256):
 Contains template 0x00000000/0x0104 with 18 records (offset 328):
 Contains template 0x00000000/0x0105 with 14 records (offset 404):
 Contains template 0x00000000/0x0106 with 14 records (offset 464):
 Contains template 0x00000000/0x0107 with 18 records (offset 524):
 Contains template 0x00000000/0x0108 with 14 records (offset 600):
forced deletion of template 0x0108 from peer 1.1.1.2/0x00000000<http://1.1.1.2/0x00000000>
 Contains template 0x00000000/0x0109 with 22 records (offset 660):
forced deletion of template 0x0109 from peer 1.1.1.2/0x00000000<http://1.1.1.2/0x00000000>
 Contains template 0x00000000/0x010a with 22 records (offset 752):
forced deletion of template 0x010a from peer 1.1.1.2/0x00000000<http://1.1.1.2/0x00000000>
 Contains template 0x00000000/0x010b with 18 records (offset 844):
forced deletion of template 0x010b from peer 1.1.1.2/0x00000000<http://1.1.1.2/0x00000000>
 Contains template 0x00000000/0x010c with 18 records (offset 920):

I note that with the newer release of the ASA code that none of the template records are accepted, with the older version only a few of them are force deleted.

Does anyone have any idea what may be happening here?

I am ready to provide samples off list and perform any debugging requested. If it's possible to receive and parse the template and post it publicly so we can compare the two versions I'd be more than happy too.

I'm eager to solve the problem and ready to do whatever it takes to address it.

Thanks in advance,

-JohnF

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mindrot.org/pipermail/netflow-tools/attachments/20140221/84fc3bc6/attachment-0001.html>