From johnf at zioncluster.ca Fri Feb 21 13:29:41 2014 From: johnf at zioncluster.ca (John Marrett) Date: Thu, 20 Feb 2014 21:29:41 -0500 Subject: [netflow-tools] Cisco ASA OS 9 flowd errors Message-ID: I'm running a version of flowd 0.9.1 with my ASA patches applied ( http://zioncluster.ca/netflow/asa_patch_2.diff ). I've recently realized that when running against flows from ASA running versions of 9.1(4) (and probably earlier releases in 9) I'm seeing error messages and no data is recorded to disk. When templates are received I see the following: NetFlow v.9 template set from 1.1.1.1/0x0 with len 1368: Contains template 0x00000000/0x0100 with 21 records (offset 8): forced deletion of template 0x0100 from peer 1.1.1.1/0x00000000 Contains template 0x00000000/0x0101 with 21 records (offset 96): forced deletion of template 0x0101 from peer 1.1.1.1/0x00000000 Contains template 0x00000000/0x0102 with 21 records (offset 184): forced deletion of template 0x0102 from peer 1.1.1.1/0x00000000 [...] Even after receipt of the template I see the following: netflow v.9 packet (len 1412) 17 recs, source 0x00000000 netflow v.9 data flowset (len 104) source 0x00000000 netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0100 netflow v.9 data flowset (len 68) source 0x00000000 netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0107 netflow v.9 data flowset (len 104) source 0x00000000 [...] When I compare it with another host running an older version I see different log information. NetFlow v.9 template set from 1.1.1.2/0x0 with len 992: Contains template 0x00000000/0x0100 with 21 records (offset 8): Contains template 0x00000000/0x0101 with 21 records (offset 96): Contains template 0x00000000/0x0102 with 17 records (offset 184): Contains template 0x00000000/0x0103 with 17 records (offset 256): Contains template 0x00000000/0x0104 with 18 records (offset 328): Contains template 0x00000000/0x0105 with 14 records (offset 404): Contains template 0x00000000/0x0106 with 14 records (offset 464): Contains template 0x00000000/0x0107 with 18 records (offset 524): Contains template 0x00000000/0x0108 with 14 records (offset 600): forced deletion of template 0x0108 from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x0109 with 22 records (offset 660): forced deletion of template 0x0109 from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010a with 22 records (offset 752): forced deletion of template 0x010a from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010b with 18 records (offset 844): forced deletion of template 0x010b from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010c with 18 records (offset 920): I note that with the newer release of the ASA code that none of the template records are accepted, with the older version only a few of them are force deleted. Does anyone have any idea what may be happening here? I am ready to provide samples off list and perform any debugging requested. If it's possible to receive and parse the template and post it publicly so we can compare the two versions I'd be more than happy too. I'm eager to solve the problem and ready to do whatever it takes to address it. Thanks in advance, -JohnF -------------- next part -------------- An HTML attachment was scrubbed... URL: From Craig.Weinhold at cdw.com Fri Feb 21 15:17:55 2014 From: Craig.Weinhold at cdw.com (Craig Weinhold) Date: Fri, 21 Feb 2014 04:17:55 +0000 Subject: [netflow-tools] Cisco ASA OS 9 flowd errors In-Reply-To: References: Message-ID: <480523C07F950444BE371E85152C87C0121AC2AD@EXMBD6VH.corp.cdw.com> NSEL from ASA is quite different from traditional NetFlow, and it changed with ASA 9. Each flow is a bidirectional entity with separate byte/packet counters in each direction (previously there was one total byte counter). There are also new event types -- flow alert and flow update. ASA 8.x http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/netflow/netflow.html ASA 9.x http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html I'd love to see a complete implementation of NSEL in flowd, but that requires changing the output format. You up to the challenge? -Craig ________________________________ From: netflow-tools [netflow-tools-bounces+craig.weinhold=cdw.com at mindrot.org] on behalf of John Marrett [johnf at zioncluster.ca] Sent: Thursday, February 20, 2014 8:29 PM To: netflow-tools at mindrot.org Subject: [netflow-tools] Cisco ASA OS 9 flowd errors I'm running a version of flowd 0.9.1 with my ASA patches applied ( http://zioncluster.ca/netflow/asa_patch_2.diff ). I've recently realized that when running against flows from ASA running versions of 9.1(4) (and probably earlier releases in 9) I'm seeing error messages and no data is recorded to disk. When templates are received I see the following: NetFlow v.9 template set from 1.1.1.1/0x0 with len 1368: Contains template 0x00000000/0x0100 with 21 records (offset 8): forced deletion of template 0x0100 from peer 1.1.1.1/0x00000000 Contains template 0x00000000/0x0101 with 21 records (offset 96): forced deletion of template 0x0101 from peer 1.1.1.1/0x00000000 Contains template 0x00000000/0x0102 with 21 records (offset 184): forced deletion of template 0x0102 from peer 1.1.1.1/0x00000000 [...] Even after receipt of the template I see the following: netflow v.9 packet (len 1412) 17 recs, source 0x00000000 netflow v.9 data flowset (len 104) source 0x00000000 netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0100 netflow v.9 data flowset (len 68) source 0x00000000 netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0107 netflow v.9 data flowset (len 104) source 0x00000000 [...] When I compare it with another host running an older version I see different log information. NetFlow v.9 template set from 1.1.1.2/0x0 with len 992: Contains template 0x00000000/0x0100 with 21 records (offset 8): Contains template 0x00000000/0x0101 with 21 records (offset 96): Contains template 0x00000000/0x0102 with 17 records (offset 184): Contains template 0x00000000/0x0103 with 17 records (offset 256): Contains template 0x00000000/0x0104 with 18 records (offset 328): Contains template 0x00000000/0x0105 with 14 records (offset 404): Contains template 0x00000000/0x0106 with 14 records (offset 464): Contains template 0x00000000/0x0107 with 18 records (offset 524): Contains template 0x00000000/0x0108 with 14 records (offset 600): forced deletion of template 0x0108 from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x0109 with 22 records (offset 660): forced deletion of template 0x0109 from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010a with 22 records (offset 752): forced deletion of template 0x010a from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010b with 18 records (offset 844): forced deletion of template 0x010b from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010c with 18 records (offset 920): I note that with the newer release of the ASA code that none of the template records are accepted, with the older version only a few of them are force deleted. Does anyone have any idea what may be happening here? I am ready to provide samples off list and perform any debugging requested. If it's possible to receive and parse the template and post it publicly so we can compare the two versions I'd be more than happy too. I'm eager to solve the problem and ready to do whatever it takes to address it. Thanks in advance, -JohnF -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnf at zioncluster.ca Fri Feb 21 22:17:08 2014 From: johnf at zioncluster.ca (John Marrett) Date: Fri, 21 Feb 2014 06:17:08 -0500 Subject: [netflow-tools] Cisco ASA OS 9 flowd errors In-Reply-To: <480523C07F950444BE371E85152C87C0121AC2AD@EXMBD6VH.corp.cdw.com> References: <480523C07F950444BE371E85152C87C0121AC2AD@EXMBD6VH.corp.cdw.com> Message-ID: I should have known that Craig would know the subject matter well, you're really throwing down with are you up to the challenge! Can't believe I didn't think to check the netflow docs as I did last time. You've linked us to some very interesting information. My little patch for ASA 8 Netflow gives a very basic interpretation, it's enough to allow you to use flow-tools to perform very basic reporting, but doesn't provide any temporal information as there is only a single packet on close. As such it's unsuited to use with the most powerful open source flow reporting system ( http://wvnetflow.sourceforge.net/ ). ASA 9 seems much more interesting as a full implementation would allow an ASA to function as a full featured collector. It's unfortunate that none of the fields align with standard fields. My reading also seems to suggest that I would need to configure the collector in a specific fashion; this annotation in the NSEL documentation: "Different events in the life of a flow may be issued in separate NetFlow packets and may arrive out-of-order at the collector. For example, the packet containing a flow teardown event may reach the collector before the packet containing a flow creation event. As a result, it is important that collector applications use the Event Time field to correlate events. " [1] and "a configurable CLI parameter is provided to delay sending of the flow-create event. If the timer fires, the flow-create event is sent. However, if the flow is torn down before the timer expires, *only* the flow-teardown event is sent; no flow-create event is sent. " [2] As I read this it would probably be best to ensure that the refresh-interval is the same as, or one second less than the delay flow-create timer. [3] I'm not certain how different this behaviour is from regular netflow as it's been several years since I went this deep into netflow packet collection and processing and my last visit was with much less ambitious goals. If you look at my patch [4] you'll see that only fairly small changes were required to make things work; I wonder if they changed things in such a way that the "standard" netflow v9 field identifiers used on other platforms no longer match the ASA 9 ones, even though their field content is the same. If this is the case then it could actually be pretty easy to address. If anyone else is interested in this subject please feel free to contact me on or off list. I can provide all kinds of assistance captures and potentially even (remote) hardware access. I will probably start in on this to see how challenging it actually is, if it's not as bad as I fear it is then I may be able to get something done fairly quickly. It's not entirely clear to me why the fields All of this said, I think it's clear that I AM up to the challenge! Whether my availability and scheduling are is an open question however :( I've got a lot on my plate at home and work; in the environment where I have these ASAs running as collectors I now have new options (N7k Core Switches) which give me other substantially easier options to address the issue as well. [1] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1029397 [2] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1028239 [3] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/configuration/general/asa_91_general_config/monitor_nsel.html#wp1301461 [4] http://zioncluster.ca/netflow/asa_patch_2.diff -JohnF On Thu, Feb 20, 2014 at 11:17 PM, Craig Weinhold wrote: > NSEL from ASA is quite different from traditional NetFlow, and it > changed with ASA 9. Each flow is a bidirectional entity with separate > byte/packet counters in each direction (previously there was one total byte > counter). There are also new event types -- flow alert and flow update. > > ASA 8.x > http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/netflow/netflow.html > > ASA 9.x > http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html > > I'd love to see a complete implementation of NSEL in flowd, but that > requires changing the output format. You up to the challenge? > > -Craig > > > ------------------------------ > *From:* netflow-tools [netflow-tools-bounces+craig.weinhold= > cdw.com at mindrot.org] on behalf of John Marrett [johnf at zioncluster.ca] > *Sent:* Thursday, February 20, 2014 8:29 PM > *To:* netflow-tools at mindrot.org > *Subject:* [netflow-tools] Cisco ASA OS 9 flowd errors > > I'm running a version of flowd 0.9.1 with my ASA patches applied ( > http://zioncluster.ca/netflow/asa_patch_2.diff ). > > I've recently realized that when running against flows from ASA running > versions of 9.1(4) (and probably earlier releases in 9) I'm seeing error > messages and no data is recorded to disk. > > When templates are received I see the following: > > NetFlow v.9 template set from 1.1.1.1/0x0 with len 1368: > Contains template 0x00000000/0x0100 with 21 records (offset 8): > forced deletion of template 0x0100 from peer 1.1.1.1/0x00000000 > Contains template 0x00000000/0x0101 with 21 records (offset 96): > forced deletion of template 0x0101 from peer 1.1.1.1/0x00000000 > Contains template 0x00000000/0x0102 with 21 records (offset 184): > forced deletion of template 0x0102 from peer 1.1.1.1/0x00000000 > [...] > > Even after receipt of the template I see the following: > > netflow v.9 packet (len 1412) 17 recs, source 0x00000000 > netflow v.9 data flowset (len 104) source 0x00000000 > netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0100 > netflow v.9 data flowset (len 68) source 0x00000000 > netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0107 > netflow v.9 data flowset (len 104) source 0x00000000 > [...] > > When I compare it with another host running an older version I see > different log information. > > NetFlow v.9 template set from 1.1.1.2/0x0 with len 992: > Contains template 0x00000000/0x0100 with 21 records (offset 8): > Contains template 0x00000000/0x0101 with 21 records (offset 96): > Contains template 0x00000000/0x0102 with 17 records (offset 184): > Contains template 0x00000000/0x0103 with 17 records (offset 256): > Contains template 0x00000000/0x0104 with 18 records (offset 328): > Contains template 0x00000000/0x0105 with 14 records (offset 404): > Contains template 0x00000000/0x0106 with 14 records (offset 464): > Contains template 0x00000000/0x0107 with 18 records (offset 524): > Contains template 0x00000000/0x0108 with 14 records (offset 600): > forced deletion of template 0x0108 from peer 1.1.1.2/0x00000000 > Contains template 0x00000000/0x0109 with 22 records (offset 660): > forced deletion of template 0x0109 from peer 1.1.1.2/0x00000000 > Contains template 0x00000000/0x010a with 22 records (offset 752): > forced deletion of template 0x010a from peer 1.1.1.2/0x00000000 > Contains template 0x00000000/0x010b with 18 records (offset 844): > forced deletion of template 0x010b from peer 1.1.1.2/0x00000000 > Contains template 0x00000000/0x010c with 18 records (offset 920): > > I note that with the newer release of the ASA code that none of the > template records are accepted, with the older version only a few of them > are force deleted. > > Does anyone have any idea what may be happening here? > > I am ready to provide samples off list and perform any debugging > requested. If it's possible to receive and parse the template and post it > publicly so we can compare the two versions I'd be more than happy too. > > I'm eager to solve the problem and ready to do whatever it takes to > address it. > > Thanks in advance, > > -JohnF > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Craig.Weinhold at cdw.com Fri Feb 21 23:48:26 2014 From: Craig.Weinhold at cdw.com (Craig Weinhold) Date: Fri, 21 Feb 2014 12:48:26 +0000 Subject: [netflow-tools] Cisco ASA OS 9 flowd errors In-Reply-To: References: <480523C07F950444BE371E85152C87C0121AC2AD@EXMBD6VH.corp.cdw.com>, Message-ID: <480523C07F950444BE371E85152C87C0121AC2F6@EXMBD6VH.corp.cdw.com> John, One "simple" patch for ASA 9 would be to (a) only recognize flow-update events and (b) create two unidirectional flows from each bidirectional flow -- swap the IP/port/ifIndex and include only the appropriate byte counter. Not only is it simple, but the resulting data would be fully compatible with existing scripts and tools that work on unidirectional flows. -Craig ________________________________ From: John Marrett [johnf at zioncluster.ca] Sent: Friday, February 21, 2014 5:17 AM To: Craig Weinhold Cc: netflow-tools at mindrot.org Subject: Re: [netflow-tools] Cisco ASA OS 9 flowd errors I should have known that Craig would know the subject matter well, you're really throwing down with are you up to the challenge! Can't believe I didn't think to check the netflow docs as I did last time. You've linked us to some very interesting information. My little patch for ASA 8 Netflow gives a very basic interpretation, it's enough to allow you to use flow-tools to perform very basic reporting, but doesn't provide any temporal information as there is only a single packet on close. As such it's unsuited to use with the most powerful open source flow reporting system ( http://wvnetflow.sourceforge.net/ ). ASA 9 seems much more interesting as a full implementation would allow an ASA to function as a full featured collector. It's unfortunate that none of the fields align with standard fields. My reading also seems to suggest that I would need to configure the collector in a specific fashion; this annotation in the NSEL documentation: "Different events in the life of a flow may be issued in separate NetFlow packets and may arrive out-of-order at the collector. For example, the packet containing a flow teardown event may reach the collector before the packet containing a flow creation event. As a result, it is important that collector applications use the Event Time field to correlate events. " [1] and "a configurable CLI parameter is provided to delay sending of the flow-create event. If the timer fires, the flow-create event is sent. However, if the flow is torn down before the timer expires, only the flow-teardown event is sent; no flow-create event is sent. " [2] As I read this it would probably be best to ensure that the refresh-interval is the same as, or one second less than the delay flow-create timer. [3] I'm not certain how different this behaviour is from regular netflow as it's been several years since I went this deep into netflow packet collection and processing and my last visit was with much less ambitious goals. If you look at my patch [4] you'll see that only fairly small changes were required to make things work; I wonder if they changed things in such a way that the "standard" netflow v9 field identifiers used on other platforms no longer match the ASA 9 ones, even though their field content is the same. If this is the case then it could actually be pretty easy to address. If anyone else is interested in this subject please feel free to contact me on or off list. I can provide all kinds of assistance captures and potentially even (remote) hardware access. I will probably start in on this to see how challenging it actually is, if it's not as bad as I fear it is then I may be able to get something done fairly quickly. It's not entirely clear to me why the fields All of this said, I think it's clear that I AM up to the challenge! Whether my availability and scheduling are is an open question however :( I've got a lot on my plate at home and work; in the environment where I have these ASAs running as collectors I now have new options (N7k Core Switches) which give me other substantially easier options to address the issue as well. [1] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1029397 [2] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1028239 [3] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/configuration/general/asa_91_general_config/monitor_nsel.html#wp1301461 [4] http://zioncluster.ca/netflow/asa_patch_2.diff -JohnF On Thu, Feb 20, 2014 at 11:17 PM, Craig Weinhold > wrote: NSEL from ASA is quite different from traditional NetFlow, and it changed with ASA 9. Each flow is a bidirectional entity with separate byte/packet counters in each direction (previously there was one total byte counter). There are also new event types -- flow alert and flow update. ASA 8.x http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/netflow/netflow.html ASA 9.x http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html I'd love to see a complete implementation of NSEL in flowd, but that requires changing the output format. You up to the challenge? -Craig ________________________________ From: netflow-tools [netflow-tools-bounces+craig.weinhold=cdw.com at mindrot.org] on behalf of John Marrett [johnf at zioncluster.ca] Sent: Thursday, February 20, 2014 8:29 PM To: netflow-tools at mindrot.org Subject: [netflow-tools] Cisco ASA OS 9 flowd errors I'm running a version of flowd 0.9.1 with my ASA patches applied ( http://zioncluster.ca/netflow/asa_patch_2.diff ). I've recently realized that when running against flows from ASA running versions of 9.1(4) (and probably earlier releases in 9) I'm seeing error messages and no data is recorded to disk. When templates are received I see the following: NetFlow v.9 template set from 1.1.1.1/0x0 with len 1368: Contains template 0x00000000/0x0100 with 21 records (offset 8): forced deletion of template 0x0100 from peer 1.1.1.1/0x00000000 Contains template 0x00000000/0x0101 with 21 records (offset 96): forced deletion of template 0x0101 from peer 1.1.1.1/0x00000000 Contains template 0x00000000/0x0102 with 21 records (offset 184): forced deletion of template 0x0102 from peer 1.1.1.1/0x00000000 [...] Even after receipt of the template I see the following: netflow v.9 packet (len 1412) 17 recs, source 0x00000000 netflow v.9 data flowset (len 104) source 0x00000000 netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0100 netflow v.9 data flowset (len 68) source 0x00000000 netflow v.9 data flowset without template 1.1.1.1/0x00000000/0x0107 netflow v.9 data flowset (len 104) source 0x00000000 [...] When I compare it with another host running an older version I see different log information. NetFlow v.9 template set from 1.1.1.2/0x0 with len 992: Contains template 0x00000000/0x0100 with 21 records (offset 8): Contains template 0x00000000/0x0101 with 21 records (offset 96): Contains template 0x00000000/0x0102 with 17 records (offset 184): Contains template 0x00000000/0x0103 with 17 records (offset 256): Contains template 0x00000000/0x0104 with 18 records (offset 328): Contains template 0x00000000/0x0105 with 14 records (offset 404): Contains template 0x00000000/0x0106 with 14 records (offset 464): Contains template 0x00000000/0x0107 with 18 records (offset 524): Contains template 0x00000000/0x0108 with 14 records (offset 600): forced deletion of template 0x0108 from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x0109 with 22 records (offset 660): forced deletion of template 0x0109 from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010a with 22 records (offset 752): forced deletion of template 0x010a from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010b with 18 records (offset 844): forced deletion of template 0x010b from peer 1.1.1.2/0x00000000 Contains template 0x00000000/0x010c with 18 records (offset 920): I note that with the newer release of the ASA code that none of the template records are accepted, with the older version only a few of them are force deleted. Does anyone have any idea what may be happening here? I am ready to provide samples off list and perform any debugging requested. If it's possible to receive and parse the template and post it publicly so we can compare the two versions I'd be more than happy too. I'm eager to solve the problem and ready to do whatever it takes to address it. Thanks in advance, -JohnF -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnf at zioncluster.ca Sat Feb 22 13:38:27 2014 From: johnf at zioncluster.ca (John Marrett) Date: Fri, 21 Feb 2014 21:38:27 -0500 Subject: [netflow-tools] Cisco ASA OS 9 flowd errors In-Reply-To: <480523C07F950444BE371E85152C87C0121AC2F6@EXMBD6VH.corp.cdw.com> References: <480523C07F950444BE371E85152C87C0121AC2AD@EXMBD6VH.corp.cdw.com> <480523C07F950444BE371E85152C87C0121AC2F6@EXMBD6VH.corp.cdw.com> Message-ID: I'm somewhat pleased to announce the first version of my patch for ASA 9 support [1] . Unfortunately it is far from complete. In fact, it's only marginally usable. The initial problems were caused by the ASA 9 templates massively exceeded the value of DEFAULT_MAX_TEMPLATES, I have increased it to 1024 and it can now process the full template load. I think there is some confusion between DEFAULT_MAX_TEMPLATES templates, which appears to be intended to be a counter of the number of templates, however seems to actually be the maximum number of fields. There is also a value for DEFAULT_MAX_TEMPLATE_LEN which appears to be intended to be a counter of the number of template fields, possibly per template. The first template from the ASA in version 9 contains a large number of fields it can't be processed and it starts aborting immediately reporting the "forced deletion of template 0x0100 from peer" error. Unfortunately this is where the first ASA 9 patch begins and also ends. It will report all flows as 0 packet, 0 bytes. My next update should implement processing of update fields as Craig has proposed. It will work based on only processing update events [1] and by handling the two new ASA packet counters. Hopefully more to come this weekend. [1] http://zioncluster.ca/netflow/asa-9-patch-1.diff [2] http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/system/netflow/netflow.html#wp1028202 -JohnF -------------- next part -------------- An HTML attachment was scrubbed... URL: