[netflow-tools] Cisco ASA OS 9 flowd errors

Fri Feb 21 22:17:08 EST 2014

I should have known that Craig would know the subject matter well, you're
really throwing down with are you up to the challenge!

Can't believe I didn't think to check the netflow docs as I did last time.
You've linked us to some very interesting information.

My little patch for ASA 8 Netflow gives a very basic interpretation, it's
enough to allow you to use flow-tools to perform very basic reporting, but
doesn't provide any temporal information as there is only a single packet
on close. As such it's unsuited to use with the most powerful open source
flow reporting system ( http://wvnetflow.sourceforge.net/ ).

ASA 9 seems much more interesting as a full implementation would allow an
ASA to function as a full featured collector. It's unfortunate that none of
the fields align with standard fields.

My reading also seems to suggest that I would need to configure the
collector in a specific fashion; this annotation in the NSEL documentation:

"Different events in the life of a flow may be issued in separate NetFlow
packets and may arrive out-of-order at the collector. For example, the
packet containing a flow teardown event may reach the collector before the
packet containing a flow creation event. As a result, it is important that
collector applications use the Event Time field to correlate events. " [1]
and "a configurable CLI parameter is provided to delay sending of the
flow-create event. If the timer fires, the flow-create event is sent.
However, if the flow is torn down before the timer expires, *only* the
flow-teardown event is sent; no flow-create event is sent. " [2]

As I read this it would probably be best to ensure that the
refresh-interval is the same as, or one second less than the delay
flow-create timer. [3]

I'm not certain how different this behaviour is from regular netflow as
it's been several years since I went this deep into netflow packet
collection and processing and my last visit was with much less ambitious
goals. If you look at my patch [4] you'll see that only fairly small
changes were required to make things work; I wonder if they changed things
in such a way that the "standard" netflow v9 field identifiers used on
other platforms no longer match the ASA 9 ones, even though their field
content is the same. If this is the case then it could actually be pretty
easy to address.

If anyone else is interested in this subject please feel free to contact me
on or off list. I can provide all kinds of assistance captures and
potentially even (remote) hardware access. I will probably start in on this
to see how challenging it actually is, if it's not as bad as I fear it is
then I may be able to get something done fairly quickly. It's not entirely
clear to me why the fields

All of this said, I think it's clear that I AM up to the challenge!

Whether my availability and scheduling are is an open question however :(

I've got a lot on my plate at home and work; in the environment where I
have these ASAs running as collectors I now have new options (N7k Core
Switches) which give me other substantially easier options to address the
issue as well.

[1]
http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1029397
[2]
http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1028239
[3]
http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/configuration/general/asa_91_general_config/monitor_nsel.html#wp1301461
[4] http://zioncluster.ca/netflow/asa_patch_2.diff

-JohnF

On Thu, Feb 20, 2014 at 11:17 PM, Craig Weinhold <Craig.Weinhold at cdw.com>wrote:

>  NSEL from ASA is quite different from traditional NetFlow, and it
> changed with ASA 9. Each flow is a bidirectional entity with separate
> byte/packet counters in each direction (previously there was one total byte
> counter). There are also new event types -- flow alert and flow update.
>
> ASA 8.x
> http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/netflow/netflow.html
>
> ASA 9.x
> http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html
>
> I'd love to see a complete implementation of NSEL in flowd, but that
> requires changing the output format. You up to the challenge?
>
> -Craig
>
>
>  ------------------------------
> *From:* netflow-tools [netflow-tools-bounces+craig.weinhold=
> cdw.com at mindrot.org] on behalf of John Marrett [johnf at zioncluster.ca]
> *Sent:* Thursday, February 20, 2014 8:29 PM
> *To:* netflow-tools at mindrot.org
> *Subject:* [netflow-tools] Cisco ASA OS 9 flowd errors
>
>    I'm running a version of flowd 0.9.1 with my ASA patches applied (
> http://zioncluster.ca/netflow/asa_patch_2.diff ).
>
>  I've recently realized that when running against flows from ASA running
> versions of 9.1(4) (and probably earlier releases in 9) I'm seeing error
> messages and no data is recorded to disk.
>
>  When templates are received I see the following:
>
> NetFlow v.9 template set from 1.1.1.1/0x0 with len 1368:
>  Contains template 0x00000000/0x0100 with 21 records (offset 8):
> forced deletion of template 0x0100 from peer 1.1.1.1/0x00000000
>  Contains template 0x00000000/0x0101 with 21 records (offset 96):
> forced deletion of template 0x0101 from peer 1.1.1.1/0x00000000
>  Contains template 0x00000000/0x0102 with 21 records (offset 184):
> forced deletion of template 0x0102 from peer 1.1.1.1/0x00000000
> [...]
>
> Even after receipt of the template I see the following:
>
> netflow v.9 packet (len 1412) 17 recs, source 0x00000000
> netflow v.9 data flowset (len 104) source 0x00000000
> netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0100
> netflow v.9 data flowset (len 68) source 0x00000000
> netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0107
> netflow v.9 data flowset (len 104) source 0x00000000
> [...]
>
>  When I compare it with another host running an older version I see
> different log information.
>
> NetFlow v.9 template set from 1.1.1.2/0x0 with len 992:
>  Contains template 0x00000000/0x0100 with 21 records (offset 8):
>  Contains template 0x00000000/0x0101 with 21 records (offset 96):
>  Contains template 0x00000000/0x0102 with 17 records (offset 184):
>  Contains template 0x00000000/0x0103 with 17 records (offset 256):
>  Contains template 0x00000000/0x0104 with 18 records (offset 328):
>  Contains template 0x00000000/0x0105 with 14 records (offset 404):
>  Contains template 0x00000000/0x0106 with 14 records (offset 464):
>  Contains template 0x00000000/0x0107 with 18 records (offset 524):
>  Contains template 0x00000000/0x0108 with 14 records (offset 600):
> forced deletion of template 0x0108 from peer 1.1.1.2/0x00000000
>  Contains template 0x00000000/0x0109 with 22 records (offset 660):
> forced deletion of template 0x0109 from peer 1.1.1.2/0x00000000
>  Contains template 0x00000000/0x010a with 22 records (offset 752):
> forced deletion of template 0x010a from peer 1.1.1.2/0x00000000
>  Contains template 0x00000000/0x010b with 18 records (offset 844):
> forced deletion of template 0x010b from peer 1.1.1.2/0x00000000
>  Contains template 0x00000000/0x010c with 18 records (offset 920):
>
>  I note that with the newer release of the ASA code that none of the
> template records are accepted, with the older version only a few of them
> are force deleted.
>
> Does anyone have any idea what may be happening here?
>
>  I am ready to provide samples off list and perform any debugging
> requested. If it's possible to receive and parse the template and post it
> publicly so we can compare the two versions I'd be more than happy too.
>
>  I'm eager to solve the problem and ready to do whatever it takes to
> address it.
>
>  Thanks in advance,
>
>  -JohnF
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mindrot.org/pipermail/netflow-tools/attachments/20140221/df2f3b7d/attachment.html>