WEBVTT

1
00:00:04.108 --> 00:00:10.108
Okay, so I won't say I'm great at running meetings, but let's, um.

2
00:00:10.108 --> 00:00:14.009
Let's let, let's just do a quick instruction round of, um.

3
00:00:14.009 --> 00:00:18.298
Introductions just so folks know who's here I'll just call down the list. Uh, Dan.

4
00:00:18.298 --> 00:00:22.289
They're an old network architect.

5
00:00:22.289 --> 00:00:26.879
Dan Peterson Peterson wash.

6
00:00:26.879 --> 00:00:31.199
Yeah, same here. Same here with you. It'd be system.

7
00:00:31.199 --> 00:00:37.469
Jason is narrow system.

8
00:00:37.469 --> 00:00:41.609
Jeannie, any school.

9
00:00:41.609 --> 00:00:49.560
Do it network services Mark Mark girls uh, Madison.

10
00:00:49.560 --> 00:00:57.570
Hello? Hello?

11
00:00:57.570 --> 00:01:03.090
Network engineer across a campaign.

12
00:01:04.530 --> 00:01:08.760
Yep, jumping networking Claire. Tim.

13
00:01:08.760 --> 00:01:11.760
Tim, what's this.

14
00:01:11.760 --> 00:01:18.000
Tom Jackie? Yeah, Tom, Jackie, you to be stout and down and violence with me as well.

15
00:01:19.079 --> 00:01:24.329
And then, well, we'll venture from us from Cisco.

16
00:01:24.329 --> 00:01:27.780
All right, so I'm just gonna dive right in.

17
00:01:27.780 --> 00:01:33.719
What I'll do is, I'm just going to share my slide deck. Uh, I don't intend this to be death by PowerPoint.

18
00:01:33.719 --> 00:01:39.930
I do want it to be kind of free flowing if we take an hour and a half. Great. If we don't take it an hour and a half. Right? Um.

19
00:01:39.930 --> 00:01:54.299
See, if I get this started. Okay.

20
00:01:54.299 --> 00:02:02.459
Power point yes.

21
00:02:02.459 --> 00:02:06.090
Great. Thanks. So, um.

22
00:02:06.090 --> 00:02:18.564
I guess this is my loose agenda I, uh, would love to be interrupted at any time if there's questions or if people don't like the agenda or want to take it somewhere else. This is kind of very loosely what I was gonna talk about. And then we're going to go from here.

23
00:02:19.014 --> 00:02:30.085
So, I mean, to start with the very 1st thing, which is like, why have I been talking about this recently? All of a sudden? Well, 1st of all I, I'll say that we've been pretty busy with 20.

24
00:02:30.360 --> 00:02:34.530
20 through 2022, which that's starting to come to close.

25
00:02:34.530 --> 00:02:38.215
Um, this really was just a single event.

26
00:02:38.365 --> 00:02:49.764
Uh, it wasn't like, for example, event that caused an issue, it was more of a, a concern that, you know, what are we gonna do if we have a network event during something that is quote, unquote, important.

27
00:02:50.275 --> 00:03:00.865
Uh, WH, WH, what tools do we have in our toolbox in our, um, basket to, to work on those sorts of problems? Um, so I started this, um, started poking looking at.

28
00:03:01.139 --> 00:03:06.150
How we can handle the nice dial a service events in the system that work again.

29
00:03:06.150 --> 00:03:11.699
All the efforts I'll say that I've done so far, have been focused on cost effectiveness. Um.

30
00:03:11.699 --> 00:03:17.550
What you mean by that is using things that we already have today from, in terms of hardware.

31
00:03:17.550 --> 00:03:25.409
And using data that we either have, or can collect, and trying to cobble something together, that is effective without having a lot of, um.

32
00:03:25.409 --> 00:03:29.759
Negative impact, so that's kind of the history.

33
00:03:29.759 --> 00:03:35.069
I'll say of some of these things in this bullet point, I mentioned, um, so, you know.

34
00:03:35.069 --> 00:03:38.159
You'd be scared host the president um.

35
00:03:38.159 --> 00:03:42.569
Says, hey, what can we do to protect this network? I said, well.

36
00:03:42.569 --> 00:03:47.280
We have some longstanding application and things in place.

37
00:03:47.280 --> 00:03:56.430
That we've had in place for 5 or 6 years. At this point, I think at the time material, as a destination campus wasn't enrolled in, uh.

38
00:03:56.430 --> 00:04:11.425
You'd be fragment protection we've been allowing campuses to roll in optionally to that some are enrolled in some are not some of them are have that traffic police down some of them have it just set to a lower level of so that

39
00:04:11.425 --> 00:04:14.544
they're discarded in terms of links fill so,

40
00:04:14.574 --> 00:04:14.784
uh,

41
00:04:14.784 --> 00:04:15.085
I think,

42
00:04:15.085 --> 00:04:15.264
uh,

43
00:04:15.294 --> 00:04:15.655
you know,

44
00:04:15.655 --> 00:04:15.895
W.

45
00:04:16.199 --> 00:04:27.059
We decided, I should say, I was told it was decided that you use fear was going to be opted into that. I got turned on a day or so before the event.

46
00:04:27.803 --> 00:04:41.754
Now, I thought about, what do I hear about from folks, the most about things that are events that might cause a negative impact on their campuses? I feel like the most common thing that we've been seeing with the last couple of years has been sent events, send floods.

47
00:04:42.238 --> 00:04:48.689
Uh, I'll just say, presumably overwhelming, uh, state tables on firewalls or downstream things.

48
00:04:48.689 --> 00:04:54.149
Um, so I call it that.

49
00:04:54.149 --> 00:05:01.439
I had happened to been looking at some events, uh, recently before the superior visit so.

50
00:05:01.439 --> 00:05:12.809
Um, I had a couple things already in play that I was looking at doing so we're gonna get into some of that stuff. Um, but, you know, acknowledging the fact that I didn't really have any tools for that.

51
00:05:12.809 --> 00:05:19.079
Um, Here's a graph, I'll say this, uh.

52
00:05:19.079 --> 00:05:22.259
Is from the beginning of this year.

53
00:05:22.259 --> 00:05:26.098
Through the mid March, which is presumably about when I pulled the graph.

54
00:05:26.098 --> 00:05:29.488
These are tcp, send packets that are leaving.

55
00:05:29.488 --> 00:05:33.658
The utility system, network devices up at superior heading towards their campus.

56
00:05:33.658 --> 00:05:42.809
And so you can see well, okay, this doesn't really do a very good job. Other than the last. It's about 5,000 packets per. 2nd.

57
00:05:42.809 --> 00:05:46.889
Send going out to spirit now I'll say even most of those end up.

58
00:05:46.889 --> 00:05:53.639
Being garbage, I'll say are undesirable traffic, but you'll see that. We have quite a few events. Um.

59
00:05:53.639 --> 00:05:58.769
For just a single campus over the course of a couple months of varying capacity.

60
00:05:58.769 --> 00:06:03.959
Now, this 1 popping out, you know, 1,100,000 packets per 2nd.

61
00:06:03.959 --> 00:06:07.139
Happening a couple times.

62
00:06:07.139 --> 00:06:13.168
So, what I started doing was specifically looking at.

63
00:06:13.168 --> 00:06:26.694
What's been happening this peer lately? I was looking at our net flow data, export it from our router system routers and I started seeing a pattern um, we don't have any tools that are automated looking at these flows to try to find these sorts of patterns.

64
00:06:26.694 --> 00:06:32.783
So, it was, you know, find look at look at the send rate, go find the flow files. We have flow files for about a month.

65
00:06:33.088 --> 00:06:41.338
How long bout we keep them looked at 1st, you know, backwards in time from now, back, you know, going towards the.

66
00:06:41.338 --> 00:06:44.819
When we no longer have profiles.

67
00:06:44.819 --> 00:06:51.809
Started to see was mostly a couple subnets that were doing these port scans of superior. So, uh.

68
00:06:52.949 --> 00:06:56.369
Basically worked up a process to say, well.

69
00:06:56.369 --> 00:07:10.014
If it's really just a couple subnets and then, you know, in this case, it's pretty hokey. Right? But, like, this is what I'm seeing. So that's that's pretty easy. I worked up a process to essentially limit the impact of those couple source. Something that's off on the Internet.

70
00:07:10.403 --> 00:07:14.184
This is what I keep calling, um, you know, uh, option a.

71
00:07:14.488 --> 00:07:18.329
Which is policing sentence by the source subnet.

72
00:07:18.329 --> 00:07:29.639
Um, I guess with the slides also saying is that I realized what, when we forced opted in, uh, superior to the fragment protection, the, the, the, uh.

73
00:07:29.639 --> 00:07:41.158
Admin exterior is, uh, pretty new to the job certainly doesn't have the, um, knowledge of what was going on in the system network and the 2016 timeframe, and realized I didn't really have anything even written.

74
00:07:41.158 --> 00:07:50.459
That described very well what system network was doing the packets that were coming into it what? Okay. Forcefully or what the options were. So.

75
00:07:50.459 --> 00:07:59.158
Spent some time and, um, did that as well I should point out that, um, this, this slide deck as you're seeing it.

76
00:07:59.158 --> 00:08:08.069
Is linked off of the stats page if you go to, um, stats dot net and look quick on, um.

77
00:08:08.069 --> 00:08:18.209
Other Docs in the upper right hand corner and in the private folder, I'm sorry it's not actually in the private folder has is currently just sitting in the main folder of other. You'll find the slide deck.

78
00:08:18.209 --> 00:08:22.288
In case you want to click on some of these URLs without typing.

79
00:08:22.288 --> 00:08:25.829
Uh, all those in, so, um.

80
00:08:27.353 --> 00:08:42.024
Go from here any questions, so far or any comments. So after, uh, getting something kind of in place for superior, you know, just just in time. I'll call it now. Nothing actually happened that next day.

81
00:08:42.293 --> 00:08:45.594
Uh, there'll be, uh, or the day. That the, uh, the presidential as it was.

82
00:08:46.168 --> 00:08:53.639
You know, a lot of you've heard me or seen me type something about this already. So, at this point, I'll say it was I was kind of unaware of that.

83
00:08:53.639 --> 00:09:04.524
How frequently these events were occurring from these source subnets and the fact that these were the same sort of subnets that were scanning other campuses. So, you know, this is an illusion to the fact. Well, started looking at.

84
00:09:04.524 --> 00:09:12.173
Well, if this is going to actually be useful for superior, maybe maybe I should look at how, how painful is it going to be to expand this to other campuses?

85
00:09:12.599 --> 00:09:20.908
So, um, you know, I had my process about what I was doing manually to look the stuff up in that flow. Pretty well, worked out.

86
00:09:20.908 --> 00:09:35.519
Kind of building a, um, plan in my mind is I was gonna have to if I was going to have some tool assistance to try to find these events without doing it mailing every time, kind of just documenting that process. And so I could decide whether or not any, it was worth automating.

87
00:09:35.519 --> 00:09:40.619
So, you know, I started looking at some other camps is getting, um.

88
00:09:40.619 --> 00:09:47.009
He sent floods found out, it was really limited source of 4 or 5 source subnets that were doing 90.

89
00:09:47.009 --> 00:09:59.639
9% of the activity started offering this option around to others. I know that many of the campuses have heard from me in the last month about this. Uh, so I won't belabor it too much at this time.

90
00:09:59.639 --> 00:10:09.359
Um, I know maybe 2 thirds of the folks that, uh, contacted decided it was okay to be enrolled. So, they're enrolled in that, uh.

91
00:10:09.359 --> 00:10:13.198
I recognize how weak this, um.

92
00:10:13.198 --> 00:10:20.519
Approaches because it is, uh, dealing with the known known problems. What I'm doing. I'll call it on a, uh.

93
00:10:20.519 --> 00:10:25.139
Quote, daily basis now to operate this is that you have it, um.

94
00:10:25.139 --> 00:10:28.828
A report that will tell me when I see a.

95
00:10:28.828 --> 00:10:37.019
Since, like going to a campus which campus it is, if it's a campus that has opted in to this service.

96
00:10:37.019 --> 00:10:41.428
And I see a large sin flood go out towards them. Uh, I.

97
00:10:41.428 --> 00:10:46.528
Go investigate it and see what the source was. I, uh, basically tried to see.

98
00:10:46.528 --> 00:10:50.969
What would be the collateral damage in my mind? What will be the collateral damage of.

99
00:10:50.969 --> 00:11:05.249
Clamping down on it a little bit and deciding whether or not to add it to the audit list uh, the audio list I have published, I think, when it was launched with superior, it had 4 something that's in it over the course of the last. I don't know.

100
00:11:05.249 --> 00:11:10.168
Many weeks, maybe it's been a month. I'm not sure that list has grown to a whopping of 6.

101
00:11:10.168 --> 00:11:13.198
So, uh, I haven't really been spending a lot of time.

102
00:11:15.053 --> 00:11:16.224
Operating it per se.

103
00:11:17.514 --> 00:11:18.323
So I did,

104
00:11:18.354 --> 00:11:18.923
um,

105
00:11:19.073 --> 00:11:21.953
start we have some thought into other things that we could,

106
00:11:21.953 --> 00:11:22.793
or could not do,

107
00:11:23.333 --> 00:11:23.604
uh,

108
00:11:23.634 --> 00:11:37.073
if we found it useful or interesting or needed in terms of perhaps having an ability to deal with certain events when we don't know what the source subnet is going to be ahead of time,

109
00:11:37.673 --> 00:11:38.364
which I'll call it is,

110
00:11:38.364 --> 00:11:38.693
you know,

111
00:11:38.724 --> 00:11:38.964
uh.

112
00:11:39.239 --> 00:11:44.489
Probably a more normal situation. So, let me see the next slide here.

113
00:11:49.828 --> 00:11:56.668
So, I've had a couple of conversations, uh, with folks out, you know, between when i1st sent my email out and this call.

114
00:11:56.668 --> 00:12:00.658
A lot of back and forth to the, uh, Dan Peterson.

115
00:12:00.658 --> 00:12:04.048
Um, started basically writing up yet, another document of.

116
00:12:04.048 --> 00:12:08.129
Options and documentations. Maybe questions we might have.

117
00:12:08.573 --> 00:12:22.283
For ourselves about things that we either want answered, or we wonder if we answered them, if we would be able to improve an outcome outcome for campus I should say, to should have said this very early on is my interest is not stopping every packet.

118
00:12:22.313 --> 00:12:33.653
That is unwanted from coming to your campuses. This is why you have firewalls. My main interest is making sure that your firewalls don't fall over. And if you are capable of doing that on your own, uh.

119
00:12:34.764 --> 00:12:46.974
More power to you, and I don't want to be in your way. So I should just reiterate as my. My goal is really not to just stop every packet. Uh, our, our goal is 4 bits to you for you to decide whether you want them or not.

120
00:12:47.063 --> 00:12:54.653
So, that, that hasn't, I should say that my, my view of what our role is, as the system network, my personal view hasn't changed from that perspective.

121
00:12:55.558 --> 00:12:58.708
It's more along the lines to trying to make sure it keeps this stay on line.

122
00:13:01.408 --> 00:13:04.859
So, as I promise not that's by PowerPoint.

123
00:13:04.859 --> 00:13:19.678
This is where we kind of come into a free form discussion about where we want to go from here. Um, I have some suggested topics that we could decide to talk about. They come, uh, the 1st thing is, like I said, there may be folks on this call that.

124
00:13:19.678 --> 00:13:31.254
Would like more information either interactively or not interactively about what we were doing before March 2022. uh, and I feel like we Ha, I have some documentation about that. So we could, we could glance through that.

125
00:13:31.254 --> 00:13:36.203
Or, if there are some questions about that, uh, I'd be happy to talk about that.

126
00:13:41.519 --> 00:13:49.828
Don't be shy. Michael was Tom from unclear just to.

127
00:13:49.828 --> 00:13:56.129
So, when you, let's say, you, you noticed a, you know, slash 24 or that would have been.

128
00:13:56.129 --> 00:14:01.259
You know, hitting superior per se. Did you ever look to see if that same block?

129
00:14:01.259 --> 00:14:15.119
Had impacted other campuses, or if it was, is it typically just targeted drive by kind of stuff and it's not really a correlation or a time window that it's just a matter of time they hit somebody else or is it pretty unique to 1 event?

130
00:14:15.119 --> 00:14:24.119
So, if you look at the list of so, list is something that we currently have in the naughty bucket, I would say that.

131
00:14:24.119 --> 00:14:27.509
If you look at where they originate from.

132
00:14:27.509 --> 00:14:30.778
Originating autonomous system.

133
00:14:30.778 --> 00:14:34.918
Or, if you look at the who has blocked as associated to the net block.

134
00:14:34.918 --> 00:14:39.479
Many of them come out of a single.

135
00:14:39.479 --> 00:14:48.208
Not all of them, but most of them and many of them have a, who is tag of a.

136
00:14:48.208 --> 00:14:55.649
Suppose it research, uh, security organization called our I don't know how to say it's our, a cyber re, cyber.

137
00:14:55.649 --> 00:15:00.509
Not quite sure who has has a opt out email.

138
00:15:00.509 --> 00:15:15.149
That you can supposedly email to have your addresses removed from the scanning service. Um, I was giving a terrible analogy before most people called. My analogy is this if someone comes rings my doorbell at midnight.

139
00:15:15.149 --> 00:15:22.889
I don't think I should have to opt out of that. I think that these methods, if this is a legitimate scanning service, um.

140
00:15:22.889 --> 00:15:36.448
Could stand to be improved because what we're often seeing is 5 to 600,000 packets per 2nd, sends coming from these events when they occur. They are usually, uh, 5 minutes or less often, 2 minutes or less.

141
00:15:36.448 --> 00:15:45.448
So, um, yeah, does that answer? Is that correct? Yeah. Yeah, that doesn't that doesn't sound like a normal even a.

142
00:15:45.448 --> 00:15:50.759
You know, even a modest Recon of any kind or assessment that looks yeah.

143
00:15:50.759 --> 00:16:04.288
If they were scared, I don't want to get into this. It's not my forte if they were scanning the Internet for legitimate purposes, looking for open ports or whatnot for a security perspective. There's certainly ways. I think they could do it still be timely and not be as impactful.

144
00:16:04.288 --> 00:16:10.619
Um, really vetted that that who is there? I mean, it could be just garbage anyway. Right? I mean.

145
00:16:10.619 --> 00:16:13.889
Very well could be I mean, agreed.

146
00:16:15.178 --> 00:16:22.048
Well, I'm guessing that, uh, what is happening is they're just hosting a number of researchers.

147
00:16:22.048 --> 00:16:26.759
And so there's different activities being done by different people.

148
00:16:26.759 --> 00:16:30.869
Probably out of their control and some just don't know how to play. Nice.

149
00:16:30.869 --> 00:16:36.119
You know, that's interesting. Dan, I never really thought about that perspective, but let me tell a cool story.

150
00:16:36.119 --> 00:16:40.349
You W, mass in campus we have the single cloud lab.

151
00:16:40.943 --> 00:16:53.423
And I don't know, the history of the whole thing, it's 1 of these, uh, grant funded research projects, but it does involve researchers from other institutions, having access to compute, uh, on, that ultimately is public. I.

152
00:16:53.423 --> 00:17:06.294
P, space connected to Madison and time time again. Uh, I'll say we haven't had anything really bad in a while researcher will spin up a project with, uh, a colonel with default, um, credentials.

153
00:17:06.568 --> 00:17:10.769
And you can imagine about how long it takes for something with default credentials to be found on the Internet.

154
00:17:11.969 --> 00:17:17.429
So, even we have seen this issue where we'll have a project that is researched based.

155
00:17:17.429 --> 00:17:21.598
Legitimately, uh, often doing things that are on savory.

156
00:17:23.729 --> 00:17:31.048
I wouldn't say anything like what we're seeing here, which I'll say is if I had to make a guess this kind of behavior has been coming out of these net blocks for.

157
00:17:31.048 --> 00:17:34.199
Many many months, so.

158
00:17:35.308 --> 00:17:39.118
I have made no, no attempt to, um.

159
00:17:39.118 --> 00:17:44.969
Escalate a desire to not have these packets sent to us based on the newest data.

160
00:17:46.769 --> 00:17:55.019
I think that's a game of whack a mole that may not prove to be maybe maybe may prove to be useful in this case. But I also feel like we have a solution for this case.

161
00:17:55.019 --> 00:17:58.318
And certainly, I wouldn't want to be doing that for.

162
00:17:58.318 --> 00:18:06.269
Everyone that decides to cost us on the Internet. If these folks were actually receptive, uh, the next set of folks may not be so receptive.

163
00:18:06.269 --> 00:18:11.669
Yeah, I mean, it's the same use case for just some compromise box or whatever.

164
00:18:11.669 --> 00:18:15.419
It's the same same issue.

165
00:18:23.128 --> 00:18:28.828
Were there any more questions specific about what, uh, has been done with syntac? Um.

166
00:18:28.828 --> 00:18:39.923
I keep calling attack, which implies, uh, my attitude towards these packets coming to us, but at the same flood event, and the production that we have put in place so far or made as an option.

167
00:18:41.064 --> 00:18:49.314
If you, if you haven't been contacted by me about this, and you haven't explicitly given me your permission to police these packets down, I'm not doing that for you.

168
00:18:51.058 --> 00:18:59.699
To be clear, you might want to look at your individual campus graphs that Michael has created.

169
00:18:59.699 --> 00:19:09.628
Regarding these, and 1 of the questions I also had for Michael is, um, you know, how many of these graphs are just sin versus which is.

170
00:19:09.628 --> 00:19:13.318
You know, the typical, um.

171
00:19:13.318 --> 00:19:26.699
Inbound packet for the 1st packet in your handshake. Yep. And also, I think there's a potential for seeing maybe some where you have a reflexive type or reflective type of deck.

172
00:19:28.378 --> 00:19:34.499
So, that was 1 of the conversations, or that came up with ideas that came up after my 1st email in March, was that.

173
00:19:34.499 --> 00:19:38.368
The juniper counters that we had deployed, and we've had them deployed for.

174
00:19:38.368 --> 00:19:41.548
Almost Cisco has rolled out.

175
00:19:41.548 --> 00:19:45.239
I have had a counter for each type of flag.

176
00:19:45.239 --> 00:19:51.209
So, for example, a counter for Santa counters within, you know, counter for or whatnot.

177
00:19:51.209 --> 00:19:54.959
And the the question came up was well.

178
00:19:54.959 --> 00:19:59.219
Send the send counter, uh, ACL or the, the, um.

179
00:19:59.219 --> 00:20:04.798
The graphs that we have for that data, is it matching sends and matching cause, you know, is it.

180
00:20:04.798 --> 00:20:10.648
What's going on? Well, it's matching anytime there's a sin in the flag no matter what else is in there. So, um.

181
00:20:10.648 --> 00:20:13.798
I have sense instantiated, um.

182
00:20:14.818 --> 00:20:22.739
I think, uh, send not act I'd have to go back and look and I, I just took a note. Dan. You had pointed out about, um, maybe sharing a.

183
00:20:22.739 --> 00:20:27.269
Uh, easier way to find the sin.

184
00:20:27.269 --> 00:20:31.618
Statistics for a given campus, I can certainly do that.

185
00:20:31.618 --> 00:20:45.749
All I can put together a little, I don't know if There'll be 13, customized links, but I can certainly write some pros about how to how to find that, uh, hopefully in an easy way for your campus, based on the tools. But, yeah, I, I've.

186
00:20:45.749 --> 00:20:48.808
I am definitely not counting every permutation.

187
00:20:48.808 --> 00:20:56.489
Uh, that may be interesting at this point, I'm obviously open to looking to what we think is interesting, but I do believe that.

188
00:20:56.489 --> 00:21:10.048
Send not act is also what are the counters that I have right now so it's kind of using that because we 1st launched this, uh, policing based on the negative or the naughty subnets that I was finding it was matching on.

189
00:21:10.048 --> 00:21:21.358
Whether there was a sin flag in the packet or not, but I have sense based on looking at these particular events have moved out so that it will only impact, uh, send not.

190
00:21:22.163 --> 00:21:22.463
So,

191
00:21:35.933 --> 00:21:37.854
I guess this would also be good time.

192
00:21:37.854 --> 00:21:41.784
Just ask since we haven't really had a forum in quite some time.

193
00:21:42.449 --> 00:21:57.358
I do hear from campuses once in a while about what the heck was that and when we look at flows, it's often, uh, things like this couple of those happened since between March. And now. So, uh, there was some, you know.

194
00:21:57.358 --> 00:22:04.919
Offer of protection feel like the, the mob here. It's like, oh, yeah. Look, I found this and now I, I can protect you.

195
00:22:06.209 --> 00:22:13.648
How how our campus is feeling like the routinely being affected by, uh, network events our outages that they, they don't.

196
00:22:15.538 --> 00:22:19.618
Do they can't quite explain or what is the current feel for.

197
00:22:19.618 --> 00:22:22.769
What folks are dealing with I don't know if anyone wants to share.

198
00:22:39.148 --> 00:22:42.419
My on Friday, um.

199
00:22:42.419 --> 00:22:54.269
Yeah, it's not from a clear, uh, I mean, I guess I, I feel like we've been pretty lucky then I don't, I guess we haven't really I can't say we've had a lot of service impacting events but, I mean, I'll see things, you know, I get the kind of just.

200
00:22:54.269 --> 00:23:01.108
6 hour reporting, I send kind of a picture of what's going on and I can see things that will occur, but it doesn't typically be.

201
00:23:01.108 --> 00:23:04.108
Typically service impacting, I should say, but.

202
00:23:06.598 --> 00:23:16.288
Yeah, from Green Bay, I was gonna say the same thing. We haven't really noticed also with Travis he was kind of keeping an eye on this stuff.

203
00:23:16.288 --> 00:23:27.028
For us now that he's left, we'll have to revisit. I'm not sure if I don't think he signed up for this and we should talk about that later, Mike, but we probably want to be involved in this.

204
00:23:27.028 --> 00:23:40.259
Or add it to list, I should say, kinda liking this to back when we were having voters creating floods.

205
00:23:40.259 --> 00:23:43.288
To to the tax and.

206
00:23:43.288 --> 00:23:47.189
Back then, when I noticed that occurring.

207
00:23:47.189 --> 00:23:51.989
I didn't, you know, I was having a few problems here and there, but never really.

208
00:23:51.989 --> 00:23:58.348
Put much mind to it, or attributed to other things until I really started building a system.

209
00:23:58.348 --> 00:24:04.888
That would detect this sort of thing and then I realized that it really was creating an impact.

210
00:24:04.888 --> 00:24:17.219
And then was able to do something to mitigate it but I think part of it is just 1st, realizing when these events occur so that you can then correlate it to something that might have happened or.

211
00:24:17.219 --> 00:24:22.019
Having a way to do that and maybe the graphs are good enough.

212
00:24:22.019 --> 00:24:25.019
I've said since set up, um.

213
00:24:25.019 --> 00:24:28.169
The Palo Alto with the, um.

214
00:24:28.169 --> 00:24:36.088
A protection profile that has an alert sent when it.

215
00:24:36.088 --> 00:24:41.608
Exceeds a certain rate of tcp sends so that I can start looking at, um.

216
00:24:41.608 --> 00:24:52.318
These things so far, I haven't noticed anything that really has caused any impact. And I think a lot of that has to do with the size of the firewall that we have in place. Now.

217
00:24:52.318 --> 00:24:57.269
But, yeah, in in the distance pass, I was having problems.

218
00:24:57.269 --> 00:25:00.838
The was causing so much.

219
00:25:00.838 --> 00:25:06.598
Load that it would knock down a, or even cause split brain conditions in a.

220
00:25:06.598 --> 00:25:11.578
And fail overs of the firewall itself. I haven't had that.

221
00:25:11.578 --> 00:25:15.118
Um, really much more.

222
00:25:24.808 --> 00:25:32.818
Are you alerting to, uh, console, uh, or just do an email alerts or what's your method of alert for those Palo Alto? His own protection.

223
00:25:32.818 --> 00:25:43.618
Notifications so, for right now, I just have it emailing me which I'm actually getting quite a bit. Um, and I've been monkeying around trying to tune it to.

224
00:25:43.618 --> 00:25:51.028
Not be so aggressive at sending me an email. I've set the action, um.

225
00:25:51.028 --> 00:25:55.648
Powers pretty high or our thresholds pretty high.

226
00:25:55.648 --> 00:26:00.538
And actually, for the tcp center, I have it so high that it's disabled.

227
00:26:00.538 --> 00:26:04.798
But I'm, I guess thinking about, okay.

228
00:26:04.798 --> 00:26:09.689
At some point, maybe setting a level at which it would activate and do something.

229
00:26:11.068 --> 00:26:20.548
Can you, uh, do you have, uh, 1 pair of Palo Altos at your border or do you have multiple parents? Can you give me a little more information about how that's, um, set up.

230
00:26:20.548 --> 00:26:27.388
So, we have 2 pairs on the main campus, uh, 1 pair at the border.

231
00:26:27.388 --> 00:26:36.088
Which also handles a lot of other kind of restricted subnets, wireless and various other things that it acts as a router.

232
00:26:36.088 --> 00:26:40.648
And then we have another pair that we have explicitly in front of our data center.

233
00:26:40.648 --> 00:26:47.548
And handles all the data center subnets and then at branch campuses, we have individual firewalls.

234
00:26:49.229 --> 00:26:58.439
Got it, thank you in our situation we have the branch campus is set up as a split tunnel.

235
00:26:58.439 --> 00:27:02.338
So, it hands off Internet on traffic directly to system.

236
00:27:05.459 --> 00:27:13.078
That's yeah, I think that's pretty unusual. Uh, Dan, you might be the only 1 doing that at this point like most campuses either layer 2 or 3.

237
00:27:13.344 --> 00:27:23.453
Get back to home the home campus. Yeah, we're also using the zone protections on the edge firewalls, but I do have those set to block.

238
00:27:23.453 --> 00:27:30.443
So, if there's anything that does any kind of address sweeping, or I, I send rates, they end up in a basically a blacklist for an hour.

239
00:27:31.403 --> 00:27:45.384
So, um, that doesn't obviously doesn't stop the traffic, but it does, you know, it's still produces the flow, but even even then they're, they're probably not hitting a rule, it's gonna create any state. So it's not usually impacting to the firewall anyway.

240
00:27:46.344 --> 00:27:49.703
Um, but it does mitigate them from actually hitting something that is open.

241
00:27:49.979 --> 00:27:57.088
Over time is anybody using the, um.

242
00:27:57.088 --> 00:28:04.259
Sam cookies approach and either the policy or in the.

243
00:28:04.259 --> 00:28:07.648
Prediction profile, um.

244
00:28:07.648 --> 00:28:12.239
I find that a pretty neat way to protect interior.

245
00:28:12.239 --> 00:28:25.614
Post from attack I'm not certain what we're doing at Madison.

246
00:28:25.824 --> 00:28:28.763
We don't have, um, 1 thing. We don't have, um.

247
00:28:29.068 --> 00:28:32.939
Edge, uh, like a campus boarder firewall pair.

248
00:28:32.939 --> 00:28:39.028
We have a bunch of pairs carving up campus and it's mostly departments between each other.

249
00:28:39.028 --> 00:28:42.509
Uh, you know, setting up and distributed, um.

250
00:28:42.509 --> 00:28:47.608
Uh, administration of all those firewalls, um, are.

251
00:28:47.608 --> 00:28:57.118
We've got a couple of folks that are the primary responsible folks for the firewall. There are much more versed in the specifics of how we're doing our own protection and things like that. So, I can't really address that question.

252
00:28:59.848 --> 00:29:04.469
Well, 1 of the cool things about the tcp cookie approach is.

253
00:29:04.469 --> 00:29:08.308
Why don't you cross the threshold of attack?

254
00:29:08.308 --> 00:29:14.548
For tcp send packets, it basically starts practicing all.

255
00:29:14.548 --> 00:29:22.288
Handshakes for, and what it does is it doesn't forward the sin Noah.

256
00:29:22.288 --> 00:29:26.038
To the server until after it's.

257
00:29:26.038 --> 00:29:29.489
It basically crafts its own.

258
00:29:29.489 --> 00:29:32.669
And wait for the, the.

259
00:29:32.669 --> 00:29:35.729
Corresponding, um.

260
00:29:35.729 --> 00:29:41.939
Before handing that off to the server so it essentially protects the interior.

261
00:29:41.939 --> 00:29:45.538
Servers from resource exhaustion.

262
00:29:45.538 --> 00:29:51.148
During a send flood attack that might be directed towards them.

263
00:29:51.148 --> 00:29:59.429
Um, so, you know, when you're in carrier services, and even other external things continue continue to operate, because.

264
00:29:59.429 --> 00:30:03.388
A normal 3 week handshake can occur.

265
00:30:03.388 --> 00:30:09.449
Um, it's only the 3 way handshakes that do not get, um.

266
00:30:09.449 --> 00:30:13.919
You know, processed, um, or or.

267
00:30:13.919 --> 00:30:17.878
Satisfied are not forwarded to the server.

268
00:30:17.878 --> 00:30:26.638
Yeah, it sounds like this option is probably best for a situation where you have a central that's targeted towards a single or a small number of destinations.

269
00:30:30.449 --> 00:30:44.459
Yeah, cause, I mean, this, you'd mentioned a, a threshold. This isn't this isn't me asking questions to learn you mentioned threshold. So, for example, what we're seeing now, as the types of events are usually scans across the 16.

270
00:30:44.459 --> 00:30:48.509
That isn't really something that tcp sent cookies is useful for.

271
00:30:48.509 --> 00:30:53.818
Question mark yeah. I don't know. I've not used it. Um.

272
00:30:53.818 --> 00:30:58.469
Um, there is some Apple documentation out there on it.

273
00:30:58.469 --> 00:31:04.798
The the question or comment, I guess I would have out of this would be, you know, what.

274
00:31:04.798 --> 00:31:09.058
I have a I have an assumption mostly based on what.

275
00:31:09.058 --> 00:31:17.368
When folks contact me and say that they had a network issue and if I go look up flow, Datas, and I can point to, um, you know, a traffic incident.

276
00:31:17.368 --> 00:31:23.519
It's often a related to Resourcing Josh on the firewall. So it.

277
00:31:23.519 --> 00:31:33.659
What makes me wonder, you know, this is, is this actually harmful in some ways? Uh, if you were to try to proxy for too many, um, downstream clients.

278
00:31:33.659 --> 00:31:37.439
I don't know if, you know, 1 of the things I thought about, uh.

279
00:31:37.614 --> 00:31:46.013
Through this effort is how common is it fit for folks uh, other folks on they don't have to be protecting and I know we're, we're already.

280
00:31:46.013 --> 00:31:56.213
So, maybe we're special and I understand that is a vast amount of space, but I'm talking about before a comment is it for folks to have sixteens and have firewalls protecting an entire 16.

281
00:31:56.489 --> 00:32:04.618
Uh, not anyone for it's not not something for someone to to answer per se but, you know, it does make me wonder.

282
00:32:04.618 --> 00:32:09.959
If we have added difficulty based on the how larger surface, uh, vector is for.

283
00:32:14.398 --> 00:32:21.328
So, on the Palo Alto is the, the package buffer protection really is more the option for.

284
00:32:21.328 --> 00:32:26.878
Protecting the firewall itself from exhaustion and that's kind of a different.

285
00:32:26.878 --> 00:32:30.719
Head of, um, settings.

286
00:32:30.719 --> 00:32:35.939
Um, I haven't noticed ours triggering on that at all.

287
00:32:35.939 --> 00:32:39.028
Um, at least not in.

288
00:32:39.028 --> 00:32:42.479
The recent past.

289
00:32:45.628 --> 00:32:50.638
And some of the, um, features that Dan's talking about in the Palo Alto.

290
00:32:50.638 --> 00:32:54.058
I have these URLs in the slide deck, so.

291
00:32:55.739 --> 00:33:00.959
I guess if there's interest in going to look at those, uh, either on your own, or on this call.

292
00:33:00.959 --> 00:33:07.409
Just let me know, I guess another thing I kind of wanted to bring up about.

293
00:33:07.409 --> 00:33:11.338
Is if you're logging them.

294
00:33:11.338 --> 00:33:15.209
And you're getting a Doss of some sort of.

295
00:33:15.209 --> 00:33:27.509
Your long entries can be substantially larger than the packets that are are attacking if they're particularly in the same situation because the same packets are very small.

296
00:33:27.509 --> 00:33:34.769
It's not a volumetric bandwidth attack. It's a volume metric packet attack.

297
00:33:34.769 --> 00:33:38.459
Trying to kill your packet processing rate.

298
00:33:38.459 --> 00:33:42.479
And the logs that you generate can actually.

299
00:33:42.479 --> 00:33:48.689
Larger than the attack and potentially take down interior, logging systems or.

300
00:33:48.689 --> 00:33:54.058
You know, maybe exhaust some resources on the firewall, trying to generate these logs.

301
00:33:54.058 --> 00:33:58.469
We had taken the approach to not log.

302
00:33:58.469 --> 00:34:06.298
Incoming packets to our resident into our wireless because there's nothing in those networks.

303
00:34:06.298 --> 00:34:11.429
That we allow inbound sessions to be created to anyways.

304
00:34:11.429 --> 00:34:17.759
Simply to cut down on the amount of blogs that we've have to process, or or we're looking at.

305
00:34:17.759 --> 00:34:22.559
And also the, the potential that the firewall can be just, you know.

306
00:34:22.559 --> 00:34:25.739
Funded by generating traffic logs.

307
00:34:30.449 --> 00:34:40.289
I know that you got me mass and isn't in a similar condition with, um.

308
00:34:40.289 --> 00:34:44.518
Some security mechanisms that they have at the border of their network basically.

309
00:34:44.518 --> 00:34:50.579
And sees packets before they will reach the campus border routers. So, in this case, um.

310
00:34:50.579 --> 00:34:58.559
Madison has opted in to this specific send flood on audio listing. I'm talking about it had, um.

311
00:34:58.559 --> 00:35:02.248
That, uh, on campus that actually was unusual.

312
00:35:02.248 --> 00:35:06.298
From the subnets, but it actually went on for about 3 or 4 hours.

313
00:35:06.298 --> 00:35:14.128
And it basically, uh, did exactly, uh, what you're saying, Dan, our goal is that essentially overwhelmed with their incident.

314
00:35:14.128 --> 00:35:18.568
Management system, because they kept logging every, it is a new flow.

315
00:35:19.679 --> 00:35:28.318
That was a case where, uh, that system does not sit in the forwarding path.

316
00:35:28.318 --> 00:35:32.969
So so it wasn't that that particular thing was it impactful?

317
00:35:32.969 --> 00:35:38.969
Forwarding, um, we spent.

318
00:35:38.969 --> 00:35:47.369
A good amount of time talking about firewalls and I think we should continue to do that. If that's what, uh, folks are interested in. I guess I do want to ask, uh.

319
00:35:47.369 --> 00:35:53.219
Of is is it is there value to do folks see value on this call?

320
00:35:53.219 --> 00:35:59.668
Or otherwise talking about other things that the system network may, or may not be able to do.

321
00:36:00.838 --> 00:36:07.438
Regarding some floods, or our folks pretty much happy with the way things currently are.

322
00:36:17.128 --> 00:36:20.639
Do you want to talk it all about my idea for sort of.

323
00:36:20.639 --> 00:36:24.478
Targeted blocking of all incoming SIM not.

324
00:36:27.054 --> 00:36:41.753
Sure, um, while I go look for the document or our conversation, do you want to do a little bit of an intro to make? Sure I also have the right topic. I, I know a lot of the things that we discussed. I've I've put down into print, I want to make sure I'm getting the right 1.

325
00:36:43.559 --> 00:36:48.268
Sure, um, how much like, uh, embargo we don't log our.

326
00:36:48.268 --> 00:36:54.268
Incoming denies cause it's like 5,000 log entries per 2nd of trash.

327
00:36:54.268 --> 00:36:58.559
And she had mentioned the wireless resonate and.

328
00:36:58.559 --> 00:37:09.148
Much like them, we don't have any allow rules at our edge for allowing traffic to those subnets and system actually has our list of I. T addresses our. I. T. subnets.

329
00:37:09.148 --> 00:37:18.958
That our resident and wireless, so 1 idea I had had was with all of this rate limiting and stuff is.

330
00:37:18.958 --> 00:37:25.498
If we can trigger on packets, which are sin, not act therefore, they're a incoming undesired.

331
00:37:25.498 --> 00:37:29.159
It's impact that it's never going to go anywhere to specific subnets.

332
00:37:29.159 --> 00:37:32.548
Potentially, those can be really limited down to 0.

333
00:37:32.548 --> 00:37:37.318
So, you never even see them at the end, you don't have to try and mitigate anything.

334
00:37:37.318 --> 00:37:42.478
And the advantage to that in my mind, is it not only blocks denial of service.

335
00:37:42.478 --> 00:37:49.949
Uh, from a specific slash 24, but also distributed denial service, because it doesn't matter where they're coming from.

336
00:37:49.949 --> 00:37:54.119
Coming towards, you know, likely an X box or some other.

337
00:37:54.119 --> 00:38:01.108
Student target, that's doing something that someone else on the Internet doesn't like, and is launching a against them.

338
00:38:01.108 --> 00:38:04.528
And we just would never see it because it's blocked entirely at the edge.

339
00:38:04.528 --> 00:38:07.798
For the edge firewall and it wouldn't affect our firewalls.

340
00:38:07.798 --> 00:38:12.179
Because it's handled by something that's strictly looking at a packet.

341
00:38:12.179 --> 00:38:17.579
Header and throwing it away and not trying to create sessions and do the things more advanced things that.

342
00:38:17.579 --> 00:38:31.855
What else can I give up on trying to share an application which I'm sharing my screen so I apologize if, uh, people get a little bit of whiplash. What I was trying to do is I was trying to find a PDF.

343
00:38:31.885 --> 00:38:37.795
I have has this in here. I think this is the wrong 1. I think this 1 that describes what we're doing. Um.

344
00:38:39.269 --> 00:38:45.780
Another 1, this is the 1.

345
00:38:47.280 --> 00:38:51.059
So, uh, this idea that Dan had, which is, um.

346
00:38:51.059 --> 00:38:58.829
Kind of the reason that we recognize that looking at some and not act was probably more valuable than just looking at. Um.

347
00:38:58.829 --> 00:39:13.465
I think, uh, the question, the couple of questions I had about that. I'll say that they are currently unresolved, which means that. I, I personally know. I haven't put any effort into it. Dan.

348
00:39:13.465 --> 00:39:20.875
I don't know if you've had so many time to think about it or if anyone else has some concepts about this but as we're talking about the value of packets, not showing up.

349
00:39:21.119 --> 00:39:32.670
To a firewall, uh, some of the questions I had were what, when, when you have a situation with firewall, and I recognize this could be vendor model configuration, specific.

350
00:39:32.670 --> 00:39:40.619
Including not necessarily the order of your rule sets, but various protection mechanisms that firewall may have like, so when or buffer or whatnot.

351
00:39:40.619 --> 00:39:47.099
So, if you have a packet that's coming in, and in the case of Dan's talking about, he knows he's going to throw it away based on his rules that.

352
00:39:47.099 --> 00:39:58.019
Um, obviously the packet not showing up to the firewall is going to be the least impactful for the, for that particular device right? Doesn't see the packet. It's got no work to do.

353
00:39:58.644 --> 00:40:02.335
Pushing up the tossing a little way to a device further upstream.

354
00:40:02.875 --> 00:40:13.885
I will admit that I still have my to do lists frankly, because I should know this by now and I, I don't, uh, 1 thing that we're not doing from the system network and we do have the tools to do it.

355
00:40:13.885 --> 00:40:20.005
So, it's a matter of doing it is, I haven't, for example, ran a 100 gig traffic generator through, uh.

356
00:40:21.355 --> 00:40:27.474
Our tool chain are ACL rules as we change them to see how they impact throughput and latency.

357
00:40:27.864 --> 00:40:36.684
So, that's that's something that's on my to do list, but I only bringing it up in this order because recognizing that 1 of the things again, I feel like we're looking at is.

358
00:40:37.465 --> 00:40:43.704
I'm not personally not doing a ton of interested about stopping things, unless we feel it's going to have an impact on the firewall.

359
00:40:44.005 --> 00:40:44.485
So,

360
00:40:44.844 --> 00:40:45.534
1 of the,

361
00:40:45.565 --> 00:40:45.954
so,

362
00:40:45.985 --> 00:40:47.125
getting back to the firewall,

363
00:40:47.125 --> 00:40:47.755
the question is,

364
00:40:47.755 --> 00:40:47.905
well,

365
00:40:47.905 --> 00:40:58.764
what is the impact of a packet being dropped by a firewall based on a rule set versus the impact of packet being passed through the firewall and creating state if the latter,

366
00:40:58.764 --> 00:41:00.715
which is presumably more resources,

367
00:41:00.744 --> 00:41:01.224
uh,

368
00:41:01.224 --> 00:41:01.855
expensive,

369
00:41:01.855 --> 00:41:02.094
right.

370
00:41:02.094 --> 00:41:10.885
To to open a session table and send a packet through and watch the session. It seems that minimizing that or having protections around that.

371
00:41:11.550 --> 00:41:19.170
Probably the goal, do we have a good understanding of how painful it is? Or have we tested these things to know?

372
00:41:19.170 --> 00:41:31.800
How many packets per 2nd or flows? Could could we take for something? I just hit some normal rule set is basically a question of what's the cost of something hitting the rule set versus opening, you know, like a tcp, uh, state machine.

373
00:41:31.800 --> 00:41:35.789
Does anyone have any experience with that, or any comments on that?

374
00:41:39.570 --> 00:41:43.170
I know that's that was sort of a question that we'd had with. Um, yeah.

375
00:41:43.170 --> 00:41:47.760
Specifically the Palo Alto, but I know the other vendors would be, uh, probably similar.

376
00:41:47.760 --> 00:41:52.889
For example, follow up with the default login if you have login enabled is.

377
00:41:52.889 --> 00:41:59.610
Session end, so to me, that would mean that it's going to hang on to that sin.

378
00:41:59.610 --> 00:42:04.260
Or the 32nd timer, or whatever it's got for a sin.

379
00:42:04.260 --> 00:42:07.920
Until it would log in at the end. Now, if you disable logging.

380
00:42:07.920 --> 00:42:14.159
Does that mean that it throws that half open session away immediately? Or does it ever never actually get.

381
00:42:14.159 --> 00:42:20.579
Throwing it in a table anywhere and it's loved immediately because it's a denied that, that I'm not sure.

382
00:42:20.579 --> 00:42:24.570
I know we'll create a session regardless if it's a deny rule or in a while.

383
00:42:24.570 --> 00:42:29.639
And it might be other options for that, but has had any more.

384
00:42:29.639 --> 00:42:33.150
Interaction potentially with Palo Alto if anyone else's too.

385
00:42:33.150 --> 00:42:38.489
What sort of happens behind the scenes in the deny rule if there's any impact to it at all.

386
00:42:38.489 --> 00:42:42.239
It's an initial packet it creates a session.

387
00:42:42.239 --> 00:42:46.679
Goes to the management plan before we switched into hardware. So that's sort of a.

388
00:42:46.679 --> 00:42:50.190
The most impactful thing you can do is create new sessions.

389
00:42:52.019 --> 00:42:58.679
Well, I do know that sometimes when you look in the session table, there are sessions in discard mode.

390
00:42:58.679 --> 00:43:04.530
So, it can set up sessions where it's discarding packets.

391
00:43:04.530 --> 00:43:08.940
I think the idea being that yet.

392
00:43:08.940 --> 00:43:16.650
And more efficiently discard those rather than having to go through the, uh, policies to match.

393
00:43:16.650 --> 00:43:20.699
New packets coming in that match that tool.

394
00:43:20.699 --> 00:43:27.900
But at the cost of creating session entries, and, um, you know, maybe bursting your session table.

395
00:43:27.900 --> 00:43:31.530
I'm not sure what the circumstances are for creating.

396
00:43:31.530 --> 00:43:35.159
These discard sessions versus, um.

397
00:43:35.159 --> 00:43:39.840
Just discarding the packet because it didn't match the rule in the 1st place.

398
00:43:45.269 --> 00:43:50.730
I think another part of that is, like, we, you know, we're generating net flow from there too. So whether it's.

399
00:43:50.730 --> 00:43:56.550
Dropped or not, it still creates a flow for us. So it's, you know, it's still gonna have obviously some kind of state.

400
00:43:56.550 --> 00:44:03.179
Um, but from my experience, yeah, if it's if it's not creating the session per se.

401
00:44:03.179 --> 00:44:12.570
It's definitely not nearly as impacting and resources. You're creating that flow. Incoming from the land side is what you're fine.

402
00:44:12.570 --> 00:44:22.949
Correct okay. Even though it's not, even though it's not, you know, generating a session, it's in a drop rule and we're logging it but.

403
00:44:22.949 --> 00:44:26.969
What's a few 1000 records? A 2nd.

404
00:44:26.969 --> 00:44:31.650
Looks important.

405
00:44:31.650 --> 00:44:42.420
You know, there's been times when I've been troubleshooting something and find out the sessions are in the discard state. For instance.

406
00:44:42.420 --> 00:44:46.440
I know that it happens when you hit a threat, a signature.

407
00:44:46.440 --> 00:44:50.789
So, for instance, 1, terrible example was.

408
00:44:50.789 --> 00:44:57.420
There is a thread signature out there for various things.

409
00:44:57.420 --> 00:45:06.929
Well, what happened was a resonant user maybe triggered a threat, but that request was.

410
00:45:06.929 --> 00:45:13.230
Forwarded by a really, and it created a discard flow.

411
00:45:13.230 --> 00:45:17.730
And then any devices in that subnet.

412
00:45:17.730 --> 00:45:25.590
That we're really, we're being, we're hitting that discard flow and so was essentially disabled in that. Subnet.

413
00:45:25.590 --> 00:45:33.300
And it persisted because the timer never expired on the discard flow because it was continually getting packets.

414
00:45:33.300 --> 00:45:37.380
So that was 1 of the rules I had to disable.

415
00:45:37.380 --> 00:45:42.059
Is do not do processing on package.

416
00:45:47.519 --> 00:45:57.570
I hadn't thought about net flow is, uh, or most folks doing that flow on their, uh, border devices. Uh, I guess if they're firewalls I know there's probably there's gonna be a different, um.

417
00:45:57.570 --> 00:46:06.389
Spots there, but we're doing it for stealthwatch.

418
00:46:09.719 --> 00:46:14.940
That's for the Palo Alto here.

419
00:46:19.860 --> 00:46:23.369
So, I would say regarding the the.

420
00:46:23.369 --> 00:46:27.960
Thing that Dan Peterson I was talking about Dan. I would have to admit I have to go. Look, I.

421
00:46:27.960 --> 00:46:32.309
Think the status 1 right now for you with this is I put in a counter.

422
00:46:32.309 --> 00:46:38.550
I have to go look at that. Um, this is the concept of, I would say it's just blocking, uh, send.

423
00:46:38.550 --> 00:46:44.550
Not act from known firewall rules on your campus essentially.

424
00:46:44.550 --> 00:46:52.650
That you would otherwise be discarding on the firewall that that's a possibility. I would say the downside of system doing that. Uh.

425
00:46:52.650 --> 00:47:06.960
If there is a downside, it would probably be any performance impacts on it. I will say that where I had mostly been applying these rules are at Internet Ingress. So, for example, if, uh.

426
00:47:06.960 --> 00:47:15.659
Green Bay decides to decides to go off the Green Bay or something like that. Uh, a lot of the things I put into place are not built.

427
00:47:15.659 --> 00:47:17.034
To protect that,

428
00:47:17.065 --> 00:47:17.335
uh,

429
00:47:17.364 --> 00:47:18.054
I'll just say,

430
00:47:18.085 --> 00:47:21.235
I so far had this assumption that if it really isn't an internal issue,

431
00:47:21.684 --> 00:47:21.864
uh,

432
00:47:21.925 --> 00:47:25.704
it'll be easier for us to deal with because we control both ends of that,

433
00:47:25.735 --> 00:47:26.094
um,

434
00:47:26.335 --> 00:47:27.144
that would be something,

435
00:47:27.175 --> 00:47:27.445
you know,

436
00:47:27.445 --> 00:47:29.005
it'd be useful to get feedback on,

437
00:47:29.005 --> 00:47:30.264
if that's folks here,

438
00:47:30.264 --> 00:47:34.135
that's not the correct approach or the correct thought process there.

439
00:47:34.764 --> 00:47:38.394
So, you know, the rules that I'm talking about, uh, you know, the.

440
00:47:38.670 --> 00:47:42.809
Blocking snack for whatever space, uh, folks are interested in.

441
00:47:42.809 --> 00:47:46.769
How do you applying that on the Internet edge? With the other rules? Uh.

442
00:47:46.769 --> 00:47:59.519
For I would say, I do a lot more of that. I probably do owe it to everyone on this call and everyone says not to make sure we do some pack testing to make sure that, uh, I understand the performance implications and continue to making that Ingress.

443
00:47:59.519 --> 00:48:02.820
Stateless longer and longer.

444
00:48:02.820 --> 00:48:08.940
Um, so it's, it's something for me to do the value of putting those rules there on the edge too. And, um.

445
00:48:08.940 --> 00:48:13.619
I think I've said this, but, you know, not to belabor it, but when.

446
00:48:13.619 --> 00:48:26.730
System has a quite a wide variety in the in the, the number of incoming interfaces from the Internet. They range from a call very large peering interfaces through, uh, rips or other uh, uh.

447
00:48:26.730 --> 00:48:32.280
What's the word I'm looking for? Settlement? Free peering the things we don't pay for.

448
00:48:32.280 --> 00:48:37.230
Permit, uh, transit, uh, smaller peers. We have 30 some so the advantage of.

449
00:48:37.230 --> 00:48:44.760
What we built so far is that if there's an event on 1 of those interfaces, but not the other 33, the other 33 are not impacted.

450
00:48:44.760 --> 00:48:57.864
So, uh, for example, if I were to apply, uh, uh, a rule now, the very specific thing that Dan's talking about here to and Peterson is, I think, realistically a police down to 0, uh, for certain things.

451
00:48:58.195 --> 00:49:09.954
Um, and if you're gonna police down to 0, it kinda doesn't really matter where you put the rule but if we're looking at a number of greater than 0, there is a distinct advantage of putting it further upstream closer to the Internet edge the handle cases.

452
00:49:09.954 --> 00:49:11.215
I guess I'm spoofing or if,

453
00:49:11.215 --> 00:49:11.394
like,

454
00:49:11.394 --> 00:49:13.135
an attack is coming in a single interface,

455
00:49:13.135 --> 00:49:14.844
you want to try to limit collateral damage,

456
00:49:14.875 --> 00:49:17.514
especially for dealing with stateless,

457
00:49:17.574 --> 00:49:17.934
um,

458
00:49:18.474 --> 00:49:20.155
processing packets where,

459
00:49:20.184 --> 00:49:20.635
you know,

460
00:49:20.934 --> 00:49:21.445
there is no,

461
00:49:21.474 --> 00:49:23.425
there is no evil or not evil,

462
00:49:23.454 --> 00:49:25.344
but in them to decide what to do with.

463
00:49:30.329 --> 00:49:35.880
Hello.

464
00:49:37.500 --> 00:49:42.780
So, essentially, what you're talking about is either blacklisting or white listing.

465
00:49:42.780 --> 00:49:49.440
Unused or I should say, Andrew space that shouldn't be hosting a service.

466
00:49:49.440 --> 00:50:02.635
Yeah, D***, Peterson. Well, I think his original thought was well, let's talk about knowing deny rules on the firewall that we don't want to PSN come into anyway. It did lead me to to question or wonder and some of that.

467
00:50:02.635 --> 00:50:09.565
A lot of this again, I I feel is, uh, would be ideal if we had an understanding of what the the performance impacts are.

468
00:50:09.565 --> 00:50:24.414
So, for example, the concept of talking about NetFlow and lots of folks who are doing NetFlow in their border, well, clearly, there's, there's still gonna be a, an issue or a performance impact for bringing a packet in. Even if you drop it even if it's quick. Right? For example, like 1 of the questions I had as well, you know, what?

469
00:50:24.414 --> 00:50:35.184
If there's this card role route on the firewall so the packet doesn't have a destination does it go through the same level of processing? Well, we may not know that answer, but presumably through running NetFlow on the border.

470
00:50:35.429 --> 00:50:42.780
Uh, incoming from the land interface, the answer is going to be. Yeah, of course it's going to have an impact. Um, it.

471
00:50:42.780 --> 00:50:51.090
1 question I have is, is there is there, uh, ability to, or are there problems about running, uh, net flow and egress in the landslide?

472
00:50:51.090 --> 00:50:59.010
To instead of running an aggressive on the land side, we only got half the traffic.

473
00:50:59.010 --> 00:51:04.619
Okay, I've been running a NetFlow on all interfaces and the firewalls.

474
00:51:06.239 --> 00:51:10.289
Right, yeah.

475
00:51:10.289 --> 00:51:15.750
And it's that's a system network typically does too. We run Ingress on the juniper. We.

476
00:51:15.750 --> 00:51:20.070
Typically, don't run egress. We have the ability to do it, but I usually do it as well.

477
00:51:22.920 --> 00:51:27.235
Yeah, I'm not even sure the stealth watch, but the stealth watch stuff.

478
00:51:27.264 --> 00:51:38.275
So, like, with the NetFlow extensions, you'll get, you know, if there's any translations going on that data is in there, but I don't think it really manifests itself or they're not correlating flows.

479
00:51:38.579 --> 00:51:45.389
If it's been translated to either, so I don't know what the value is maybe to having in both directions, but.

480
00:51:45.389 --> 00:51:48.989
That might be why I only see it in 1 direction.

481
00:51:48.989 --> 00:51:58.500
I'll ask an uncomfortable question, but, like, if these packets get dropped and they're visible to stealthwatch, but they come to the system network. Does anyone care.

482
00:52:11.579 --> 00:52:20.400
The only thing that could come up was if it isn't and hitting a threshold that would cause either stealthwatch or something else to take action. But.

483
00:52:21.840 --> 00:52:28.110
I know that you're under attack. I don't I don't think it would also seem that stealth watch would be.

484
00:52:28.110 --> 00:52:42.869
I don't want to say it's capable, but presumably, it's not unreasonable for us to forward the net flow data that we collect the stealth watch. But obviously, if you, if the campus collects it, I collect it. I, we both send it there. There is gonna be duplications. So.

485
00:52:47.940 --> 00:52:51.000
Yeah, I don't know we collect, um, um.

486
00:52:51.000 --> 00:52:55.409
Hello, at multiple different levels so there's a lot of duplication anyhow.

487
00:52:56.460 --> 00:53:01.409
Although I don't send it all to stealth watch. Just yet I do have some internal.

488
00:53:03.355 --> 00:53:03.925
Systems,

489
00:53:13.914 --> 00:53:15.235
so I guess I'd like to hear,

490
00:53:15.235 --> 00:53:15.505
you know,

491
00:53:15.625 --> 00:53:16.914
if other folks here,

492
00:53:16.914 --> 00:53:18.534
this digest it and think about it.

493
00:53:18.840 --> 00:53:23.070
There seems to be any interest for utility in us. Um.

494
00:53:23.070 --> 00:53:28.829
Dropping packets, or policing packets based on the unallocated space or space it.

495
00:53:28.829 --> 00:53:35.159
Just know not to get snack, or should not be getting based on the firewall rules. 1 of the things I point out here is that.

496
00:53:35.159 --> 00:53:43.405
I do worry about, are we are we going to get something out of it? That's good enough to put in a bureaucratic process right?

497
00:53:43.525 --> 00:53:56.815
Uh, what I mean, by that is, you'll notice that I haven't specifically said that I'm going to build your automation to control these lists per se. Um, now, for something that is dropping all packets to unallocated space.

498
00:53:57.144 --> 00:53:58.405
I kind of hint that maybe,

499
00:53:58.465 --> 00:53:58.735
you know,

500
00:53:58.735 --> 00:54:00.534
a campus setting in a remote trigger,

501
00:54:00.565 --> 00:54:02.454
trigger black hole route um,

502
00:54:02.934 --> 00:54:08.755
would be a plausible option because at least it puts you in administrative control that the concept,

503
00:54:08.755 --> 00:54:09.385
though of,

504
00:54:09.385 --> 00:54:10.135
for example,

505
00:54:10.585 --> 00:54:14.605
blocking sitting out acts based on destination addresses.

506
00:54:14.605 --> 00:54:16.284
On campus, I'm.

507
00:54:16.559 --> 00:54:26.039
Dan, Peter or Dan dark maybe you can say this or point out to this. I'm just not sure if the current flow spec speck is, um.

508
00:54:27.090 --> 00:54:32.159
Good enough to send and send that back a flow spec role. Do you know is that is that something you're doing?

509
00:54:32.159 --> 00:54:37.260
It is in the syntax, um, but it, um.

510
00:54:37.260 --> 00:54:41.489
Isn't implemented by X BGP completely.

511
00:54:41.489 --> 00:54:45.329
Now, that doesn't saying that, you know, it.

512
00:54:45.329 --> 00:54:49.530
Is it possible to change it to to do that or use some other.

513
00:54:49.530 --> 00:54:55.289
The GP engine, I guess the question is whether or not juniper would honor.

514
00:54:55.289 --> 00:55:00.780
Tcp flags being used and crosstalk.

515
00:55:00.780 --> 00:55:07.619
And also, let's see, there was 1 other thing I wanted to say about full spec, um.

516
00:55:07.619 --> 00:55:14.760
So there is a rate limit capability in there. I'm not sure if, um, post specter, if juniper would be.

517
00:55:14.760 --> 00:55:18.809
Honoring that, um.

518
00:55:18.809 --> 00:55:23.429
There was 1 other little thing. Maybe I'll remember in a moment here.

519
00:55:24.929 --> 00:55:31.349
I can't say authoritative. I, I would be surprised if they don't. I I know I've seen a bunch of those in the.

520
00:55:31.349 --> 00:55:42.480
Uh, juniper, in addition, obviously taking them over HP you can, uh, instantiate local flow spec rules. So, most of the things in the farmer then clause I know I've seen this command line completes.

521
00:55:42.480 --> 00:55:50.940
I just there's things that I haven't tried. Another thing you can do with Postback is if the flow matches, you can set a community.

522
00:56:01.980 --> 00:56:07.260
I, I'm trying to understand that cause we're talking about, uh, um.

523
00:56:07.260 --> 00:56:12.360
2nd role matches a packet. So where would the community be set?

524
00:56:12.360 --> 00:56:16.469
Are you applying that? Um, when it's speech P learned, you can.

525
00:56:16.469 --> 00:56:26.159
Do something on the rule based on the community, right? Yeah, that that latter part is known. I would just say that, because I know like.

526
00:56:26.159 --> 00:56:29.639
Working with Dan, we, uh.

527
00:56:29.639 --> 00:56:44.340
We have at least 2 communities. I think it is. I haven't looked at this for a long time that I know 1 of them was essentially if you send it in with a certain community, it'll go to all assistant routers versus just the board of routers. And the purpose of that was, you know.

528
00:56:44.340 --> 00:56:50.039
Uh, to acknowledge the fact so, like, when we 1st rolled out 1st, back on system network.

529
00:56:50.039 --> 00:57:01.409
Basically, you're putting, uh, an ACL on aggressive every interface before your current ACL rules. It's kinda how it gets applied. And then, uh, since that 1st, uh.

530
00:57:01.409 --> 00:57:08.699
Deployment, uh, juniper has, uh, included a feature so that, for example, you could choose which, um.

531
00:57:08.699 --> 00:57:12.150
Interfaces flow spec is enabled or disabled on.

532
00:57:12.150 --> 00:57:26.844
So, I've, I've since put in a call it, you know, tool assist automation so that our flow spec rules, actually, only run right now on Ingress on untrusted interfaces. So, for example, I don't run flow spec on the backbone interfaces.

533
00:57:26.994 --> 00:57:34.284
There's a presumption that, uh, that's best. Handled on Ingress to utility system from an untrusted interface.

534
00:57:34.559 --> 00:57:41.099
And untrusted in this case, really just means that it's not a utility system managed router on the far end.

535
00:57:42.389 --> 00:57:56.369
I'm not sure, but I think the, the idea behind setting a community on the action of a full spec role is so that then it would be treated, you know, sort of like you do with route.

536
00:57:56.369 --> 00:58:03.000
And that you could then implement an action at the juniper level to take if.

537
00:58:03.000 --> 00:58:06.690
You know, it match that community.

538
00:58:08.099 --> 00:58:14.070
I'll have to look into it.

539
00:58:14.070 --> 00:58:18.630
I guess the big question is how impactful is close back on.

540
00:58:18.630 --> 00:58:21.869
You know, if you've got a lot of flow, stack rules.

541
00:58:21.869 --> 00:58:28.349
You know how impactful it is? Yep. It's 1 of the things that, uh, I can certainly test.

542
00:58:28.349 --> 00:58:31.739
In the lab, um.

543
00:58:31.739 --> 00:58:35.909
The segue to that actually unintentional is kind of this.

544
00:58:35.909 --> 00:58:41.730
Comment I had down at the bottom. There's no, there's something for folks to talk about just that a lot of the, um.

545
00:58:41.730 --> 00:58:45.000
Efforts that have been done so far had been per campus.

546
00:58:45.000 --> 00:58:54.239
We have a limited number of campuses, which makes it conceivable at least, it feels like it makes it conceivable, uh, testing, uh, aside.

547
00:58:54.239 --> 00:58:59.849
To offer basically a campus to kind of pick and choose their own adventure about what they want.

548
00:58:59.849 --> 00:59:07.289
Now, whether or not that scales, for example, to each campus, having their own set of 50 rules.

549
00:59:07.289 --> 00:59:14.130
Uh, on the border routers and, you know, some of the each 1 of those roles being applied to, you know, maybe 10 or so interfaces.

550
00:59:14.130 --> 00:59:21.480
Uh, on our edge routers, I don't know how reasonable that is. It's something I have to test. Um.

551
00:59:21.480 --> 00:59:31.735
So, I guess something to consider is I based on what I've heard on the call so far, I suspect the answer to this is, uh, probably not interested at this point.

552
00:59:31.735 --> 00:59:39.744
But I, you know, once once I get some testing of, uh, performance impacts to some of these things, I think I'll have a better, uh, idea about whether, or not.

553
00:59:40.260 --> 00:59:48.389
Some options given the hardware that we have, unless we look at, um, augmenting the service somehow with additional hardware.

554
00:59:49.739 --> 00:59:53.969
Are there certain things that are, it's only going to be plausible to do them if they are.

555
00:59:53.969 --> 00:59:54.449
Affecting,

556
00:59:54.775 --> 00:59:55.074
you know,

557
00:59:55.105 --> 00:59:56.425
everyone who has opted in,

558
00:59:57.054 --> 00:59:57.534
for example,

559
00:59:57.534 --> 00:59:58.375
instead of not seeing,

560
00:59:58.375 --> 00:59:58.675
like,

561
00:59:58.704 --> 01:00:00.175
force forcibly saying,

562
01:00:00.175 --> 01:00:04.135
every campus must be beholden to having traffic from so,

563
01:00:04.135 --> 01:00:04.644
and so,

564
01:00:04.675 --> 01:00:04.974
you know,

565
01:00:05.005 --> 01:00:08.545
set down to 10 packets per 2nd but for example,

566
01:00:08.545 --> 01:00:12.085
maybe if there's a single rule that covers all the people that have opted in,

567
01:00:12.085 --> 01:00:14.304
we would lose the granularity of.

568
01:00:14.610 --> 01:00:22.920
Um, having statistics for campus, so that's on my list of things. I also reasons why I want to do that. Uh.

569
01:00:22.920 --> 01:00:28.500
With testing the live with, uh, various filter sets to try to understand where the pain points are.

570
01:00:30.719 --> 01:00:36.119
1 of the things I had considered doing, and actually I wrote most of the code for ready is.

571
01:00:36.119 --> 01:00:43.800
Every time in the detailed list as, uh, basically close spec, rules to drop.

572
01:00:43.800 --> 01:00:46.980
That list is only 20, um.

573
01:00:46.980 --> 01:00:58.469
Subnets slash 24 s and looking at our our current load right now on our firewall that represents 20 or 42%.

574
01:00:58.469 --> 01:01:03.510
Of the discard, or the denying rules being hit currently.

575
01:01:05.429 --> 01:01:08.880
So, it's a fairly big chunk of.

576
01:01:08.880 --> 01:01:12.869
Traffic not, you know.

577
01:01:12.869 --> 01:01:16.469
Just from session build perspective.

578
01:01:29.070 --> 01:01:33.929
And that's probably reduced by about 10% since I did advertise a few rules this morning.

579
01:01:38.880 --> 01:01:44.550
The rules you advertise the flow spec this morning where things outside D shield sounds like, is what you're, you're applying.

580
01:01:44.550 --> 01:01:48.269
No, actually, 1 of them was a D shield, uh, subnet.

581
01:01:51.599 --> 01:01:59.159
Oh, I see. Yeah, no, you're not. Of course, I guess, haven't looked at your role to see if it's a deny role instead of a police role.

582
01:02:01.409 --> 01:02:08.940
Yeah, I've only been doing deny Dan. How long do those stay.

583
01:02:08.940 --> 01:02:14.969
In that posture then, are you up, do you remove things from this list or is it just.

584
01:02:14.969 --> 01:02:28.559
Accumulating things well, I currently have been just manually maintaining this in the configuration. Um, and I don't typically remove things. I've only really dabbled with this.

585
01:02:28.559 --> 01:02:34.679
I did, like, advertised some of these, like, many years ago, and they've been in the list for.

586
01:02:34.679 --> 01:02:40.800
Wait some time I think we've had a list of about 65 to 6 rules.

587
01:02:40.800 --> 01:02:51.900
For many years, and then just this morning, I removed 1 rule because I looked at the graphs so no traffic for it in a year or more.

588
01:02:51.900 --> 01:02:58.650
And then edit to more rules and it's pretty nice um, because of the way that, um.

589
01:02:58.650 --> 01:03:02.489
Michael's got the graphics set up. It's very easy to see.

590
01:03:02.489 --> 01:03:07.530
These rules in action and what, you know.

591
01:03:07.530 --> 01:03:12.030
Is the quantity hitting them? Um, over time.

592
01:03:12.030 --> 01:03:17.940
And, you know, I'm basically being able to see all the rules that a particular campus is generating.

593
01:03:17.940 --> 01:03:22.289
So 1 of the things I have on my to do, listen.

594
01:03:24.929 --> 01:03:33.000
You know, it's the concept of whether there's gonna be a performance impact for a.

595
01:03:33.000 --> 01:03:39.000
Um, juniper entry, or they call them firewall filters, but in an ACL entry that matches.

596
01:03:39.835 --> 01:03:50.934
You know, you could I, the juniper ACL entry is, er, language is expressive enough that if you had a 1000 source subnets, you wanted to impact you can put all those things in a single from clause. Right? You can have a single rule.

597
01:03:51.445 --> 01:03:55.375
That says, you know, if it's from a certain number of subnets, um.

598
01:03:56.070 --> 01:04:00.570
The flow spec rules. Can you put multiple match?

599
01:04:00.570 --> 01:04:05.489
Or conditions in, like, a from, or is it usually, uh, it's a single source per rule.

600
01:04:06.659 --> 01:04:12.809
I think you can get multiple. Um, I'm not certain about that. I, I guess I'd have to check.

601
01:04:12.809 --> 01:04:27.210
So, there could be a yeah, I mean, uh, to be around the Bush, it's 1 of the things test is, like, there could be a negative performance impact. For example, you know, Dan, you said 20. so, I don't mean to go, you know, on the far end. But, like, if every campus did their own 20.

602
01:04:27.385 --> 01:04:27.744
Right,

603
01:04:28.434 --> 01:04:30.025
and maybe those campus,

604
01:04:30.054 --> 01:04:33.534
maybe maybe this was 20 were the same maybe they were slightly different,

605
01:04:34.074 --> 01:04:34.315
uh,

606
01:04:34.344 --> 01:04:37.315
that could be hundreds of ACL rules,

607
01:04:37.614 --> 01:04:37.795
uh,

608
01:04:37.824 --> 01:04:38.545
versus,

609
01:04:38.574 --> 01:04:38.965
you know,

610
01:04:39.054 --> 01:04:41.335
either maintained by hand or automation,

611
01:04:41.335 --> 01:04:45.565
or some kind of assistance something that's a single term that matches.

612
01:04:45.565 --> 01:04:52.434
You know, an ACL with 40 things in it. So that's part of the stuff that we need to test to understand how plausible is.

613
01:04:54.269 --> 01:04:57.480
Right and I don't know, I mean, we could all agree.

614
01:04:57.480 --> 01:05:00.719
For instance, to opt into.

615
01:05:00.719 --> 01:05:03.750
Did you show lists universally and.

616
01:05:03.750 --> 01:05:13.469
Just discard that universally. If people felt that that was useful. Yeah. I also would say that, like.

617
01:05:13.469 --> 01:05:24.210
Before we talk about universal as well I think it could also be effective. Like, again, if we have a situation where everyone that decides to opt in is in right?

618
01:05:24.210 --> 01:05:33.780
It doesn't necessarily mean that we have to we have to match every destination address on system. It could be that the campuses have adopted in there still may be, uh, again.

619
01:05:33.780 --> 01:05:45.175
A performance benefit of doing it that way like, if some wants to be like, no, I don't want to be part of this, it also seems plausible, because obviously, with the prospect reel, you can match setting.

620
01:05:45.414 --> 01:05:49.434
Let's say we could do this as slow spec you can set source and destination. So.

621
01:05:52.230 --> 01:06:01.710
So, just for curiosity, the 2 goals that I created this morning was currently matching 700 packets per 2nd.

622
01:06:01.710 --> 01:06:10.860
The other 1 stopped, it was matching 2000 packets per 2nd up until about.

623
01:06:10.860 --> 01:06:13.920
Oh, maybe 10 o'clock and then it stopped.

624
01:06:13.920 --> 01:06:19.559
I assume it's 1 of these roles. Um, hopefully sharing my screen here.

625
01:06:20.760 --> 01:06:25.500
Yeah, so the last the 2 of that, the destination port.

626
01:06:48.414 --> 01:06:50.394
And 1 of those is, you know, um.

627
01:06:50.760 --> 01:06:54.630
Volume dot com or whatever the.

628
01:06:54.630 --> 01:06:58.260
Mm, hmm.

629
01:07:01.289 --> 01:07:08.070
These are presumably mostly 40 by packets is what we were what you were seeing I would say I'm guessing.

630
01:07:08.070 --> 01:07:12.360
Cause the bit rate is pretty well, it's pretty darn low.

631
01:07:12.360 --> 01:07:17.610
Right so this is more, you know, looking at the packets per 2nd, rather than the bites.

632
01:07:23.760 --> 01:07:26.820
Oh, yeah, you're right. And I accidentally click that it's done.

633
01:07:30.539 --> 01:07:34.650
I'm going to fail to give you a live demonstration. This is why you don't do live demonstrations.

634
01:07:36.750 --> 01:07:51.059
Oh, cause I'm on the list. Okay. should've been a sec.

635
01:07:54.599 --> 01:08:06.690
Yeah, yeah, so when we talk about, you know, your normal send rate being Dan, I don't know what you normally say. I think it's around.

636
01:08:06.690 --> 01:08:10.440
My guess is, is around 5002nd coming in.

637
01:08:11.699 --> 01:08:20.220
Yeah, to be able to know all 40% of it. Now again, you know, if the tree falls in the woods, there's no 1 there to hear. It doesn't matter.

638
01:08:26.335 --> 01:08:38.574
Significantly reduces the size of my logs. Yeah, no, I, I'm laughing, but I understand that. I understand the value of that as well. It's a, it's it's almost solving a different problem in a way. Right?

639
01:08:38.904 --> 01:08:48.954
Like, um, I, I, what what the things that keep me up at night is wondering what I'm gonna do, if someone decides to send, you know, 2Million packets per 2nd to 1 of you guys, and it falls over. Um, which is good. Good.

640
01:08:49.800 --> 01:08:51.145
Its own classic problems,

641
01:08:54.175 --> 01:08:55.704
especially in the will never,

642
01:08:55.734 --> 01:08:56.125
uh,

643
01:08:56.154 --> 01:08:57.024
presumably,

644
01:08:57.414 --> 01:08:57.654
uh,

645
01:08:57.685 --> 01:08:59.364
have something that is gonna be,

646
01:08:59.484 --> 01:08:59.755
you know,

647
01:08:59.935 --> 01:09:03.024
if there's an incident that is targeted towards 1 of our campuses,

648
01:09:03.024 --> 01:09:03.685
specifically,

649
01:09:03.685 --> 01:09:04.135
whether it's,

650
01:09:04.135 --> 01:09:04.404
you know,

651
01:09:04.675 --> 01:09:05.425
gaming or.

652
01:09:05.729 --> 01:09:09.359
Or, you know, more nefarious than that. It's never going to hit the shield.

653
01:09:09.359 --> 01:09:19.170
So, what are the things if, if you build this beautiful machine? However, it ends up working how it ends up being affected. What are the key.

654
01:09:19.170 --> 01:09:24.420
Key pieces that is missing I would guess I would say is what level of automation.

655
01:09:24.420 --> 01:09:28.649
All right, do we think we want or should we have.

656
01:09:28.649 --> 01:09:32.550
To more reactively respond to things like this.

657
01:09:32.550 --> 01:09:43.319
I don't know if that's supposed to be a segue for you to talk about fast, fast enough not or not Dan or? No. So I guess 1 of the things I was going to build into this was an exclusion.

658
01:09:43.319 --> 01:09:47.069
Which would include ourselves for 1 thing.

659
01:09:47.069 --> 01:09:50.939
Just in case we hit, you know, where the source of.

660
01:09:50.939 --> 01:09:55.199
Most top 20 attackers on the Internet.

661
01:09:55.199 --> 01:09:58.829
You wouldn't want to block yourselves um.

662
01:09:58.829 --> 01:10:09.060
But, um, yeah, I don't know, that's not mine. I haven't used it for tcp based. Detects. It certainly can do that. You can also generate.

663
01:10:09.060 --> 01:10:23.699
In response to those attacks, I'm not sure that the type of tax attacks that we're seeing well, I suppose it would it would work in the case that we've seen from the IV volume people.

664
01:10:23.699 --> 01:10:27.869
Yeah, I haven't played around with it and I haven't really.

665
01:10:27.869 --> 01:10:35.640
Done much with fast on recently. I mean, we haven't had a lot of floods.

666
01:10:35.640 --> 01:10:39.119
In the recent past.

667
01:10:39.119 --> 01:10:44.399
It should be mentioned that they do have, um.

668
01:10:44.399 --> 01:10:49.979
You keep peace for version and called advanced.

669
01:10:49.979 --> 01:10:53.460
That has more capabilities.

670
01:10:53.460 --> 01:10:57.479
But, um, in general, um.

671
01:10:57.479 --> 01:11:01.800
There's 2 modes you can create.

672
01:11:01.800 --> 01:11:08.640
Um, real time black holes, which basically block you from accessing anything on the Internet, which.

673
01:11:08.640 --> 01:11:13.890
You know, can be used to limit collateral damage, but it's kind of a heavy hammer.

674
01:11:13.890 --> 01:11:18.510
Or you could generate flow spec, rules that might block a particular.

675
01:11:18.510 --> 01:11:26.520
Attacker, um, because you can then include source information and not just destination information.

676
01:11:26.520 --> 01:11:31.739
And I hadn't tried that I haven't done any automation with flow spec.

677
01:11:31.739 --> 01:11:35.100
Yet with FAS net mine, but.

678
01:11:35.100 --> 01:11:38.880
I don't know, I'm not sure that it's a.

679
01:11:38.880 --> 01:11:45.329
Where's all effort to implement this? Just yet if people are being attacked, um.

680
01:11:45.329 --> 01:11:49.920
Send floods, unless unless it gets pretty bad.

681
01:11:49.920 --> 01:11:53.609
Then maybe it would be, it would only work. Of course, if.

682
01:11:53.609 --> 01:11:59.489
If those attacks were from a limited list of because you wouldn't you want to be generating a huge.

683
01:11:59.489 --> 01:12:09.239
Multiple of spec rules you could certainly generate close rules to blocking coming.

684
01:12:10.289 --> 01:12:13.439
Your interior addresses, uh.

685
01:12:13.439 --> 01:12:18.210
Yeah, when you talk about that sort of a.

686
01:12:18.210 --> 01:12:23.189
That work a pattern that thankfully haven't really seen much of recently, which would be.

687
01:12:23.364 --> 01:12:38.005
Almost, you know, unique spoofed or spoofed or not a source address, port, hitting, you know, a unique source or source, um, unique destination, address, import on campus. Uh, things get pretty difficult unless you have.

688
01:12:38.609 --> 01:12:43.289
I'm certainly not gonna be able to mitigate that very well in the router roughly.

689
01:12:43.289 --> 01:12:49.800
Um, some of the, um, all tricks that I have thought about that might, uh.

690
01:12:49.800 --> 01:12:58.800
Be useful in that scenario I wouldn't call them necessarily ideal, but, um, you know, I wouldn't say they necessarily come into this concept of.

691
01:12:58.800 --> 01:13:03.029
There's some text I have here that basically came to. Well, what if we basically had.

692
01:13:03.029 --> 01:13:06.630
And inclusion policy of what not to police.

693
01:13:06.630 --> 01:13:17.100
Um, but more specifically, you know, this, this approach that I mentioned about, how we have, uh, unique policing, uh, abilities coming in.

694
01:13:17.100 --> 01:13:22.074
So, like, we go back to, like, when Dan Peterson was saying, hey, what, if we had a rule that set to 0 for something?

695
01:13:22.494 --> 01:13:35.154
Uh, well, if we had a rule that wanted to set, you know, something greater than 0 for destinations, uh, let's say, let's just pick on Hopscotch since, you know, so I'm thinking of, you know, I'm not advocating this is a great solution.

696
01:13:35.335 --> 01:13:43.074
But, you know, in a situation where you don't know the source, and you don't tell the destination, what what tools do you have your disposable? Well, you know.

697
01:13:43.350 --> 01:13:46.649
Saying something like I could limit since coming in.

698
01:13:46.649 --> 01:13:50.340
To a particular you W, down to 5,000.

699
01:13:50.725 --> 01:14:03.654
Now, I don't know, for example, that 5,000 per 2nd would include, um, matching, you know, some exclusion list that says, well, here's, you know, 1000 or 10,000 or, you know, something that's that we don't want to this to be, uh, applied to, um.

700
01:14:07.619 --> 01:14:18.270
Again, a lot of performance testing would be needed to be done in there to know if it's even a conceivable thing but, you know, to try to brainstorm, you know, the, the amount of things that we can do to try to limit, um.

701
01:14:18.270 --> 01:14:23.460
It collateral damage for something when we don't know what we're going to see until it happens.

702
01:14:24.479 --> 01:14:30.449
But that's a situation where, you know, if what what we've been seeing a lot of lately, I think, is that.

703
01:14:30.449 --> 01:14:36.720
The events that are coming in, and it's with do with D show things are really coming in.

704
01:14:36.720 --> 01:14:46.944
The, uh, the regional, the rep's peering service so these policing events that are hitting are really only hitting on 1 of the 33 interfaces or whatever that we have going to the Internet.

705
01:14:46.944 --> 01:14:59.335
Now, reality is, if if I, uh, take all the sends coming into, like that's coming in, from the what's that peering service, or or rips, uh, indiscriminately, you know, limit them to 5,000 per?

706
01:14:59.335 --> 01:15:09.835
2nd well, there's probably also important things coming in there too. It's not just garbage, right? That comes from the hearing service, but it seems like it's something that could be plausible used. Uh.

707
01:15:12.085 --> 01:15:24.505
You know, I don't want to say, in a crisis because, like, if these events that I'm, I'm seeing are typically 5 minutes and done, I think it would be like, if we start seeing repeated types of events that we have difficulty, um, reacting to quickly enough.

708
01:15:24.505 --> 01:15:28.824
You know, that's 1 tool and possible tool in the toolbox for things that we already have.

709
01:15:29.250 --> 01:15:33.060
Um, another thing that I have, um.

710
01:15:33.060 --> 01:15:43.140
Talked about, or sent that stuff about was this concept that the juniper have a way to auto substantiate. Uh, please, sir buckets.

711
01:15:43.140 --> 01:15:55.229
So, like, you know, I say that probably, I don't have really the time probably left on this call because I should probably do the last round. If we have things we're going to talk about. But the concept that, like, uh.

712
01:15:55.229 --> 01:16:01.319
We do have an ability on a p. E. campus for putting in a police. There's.

713
01:16:01.319 --> 01:16:16.260
Or, like 20 slash 24 or something like that. So, and that's a situation where I guess you are placing based on the destination and you might recognize that, you know, if it really depends on the traffic pattern, honestly, right? If it's if it's a, it's a.

714
01:16:16.375 --> 01:16:30.265
If it's a flood across the entire subnets are retired 16 like, we were commonly seen now, this approach may not be very helpful if it's if it's an event where it's targeted towards, like a single residential user or something like that.

715
01:16:30.265 --> 01:16:41.364
And as, uh, I think, uh, Dan Peterson was alluding to well, if we really are seeing, that's what we're seeing a lot of send floods towards, like, a wireless or a dorm. And that's what's causing us grief.

716
01:16:41.694 --> 01:16:55.854
You know, maybe the, the concept of having that blocked or severely rate limited on, uh, Internet Ingress maybe that maybe that is the next good goal to look at, in terms of coverage and what we will most often.

717
01:16:55.854 --> 01:17:00.835
See yeah. There's a lot a lot of ideas out there about things that we could do. I would say.

718
01:17:01.140 --> 01:17:08.489
You know, if I don't want to spend any more effort on this than we feel like we need to, I think there's value in.

719
01:17:08.489 --> 01:17:21.444
Brainstorming exploring so that we have a collection of tools in the toolbox to deploy if we find ourselves in the situation we may have the concept some concept of well, what is the, what is the least worst option to do here?

720
01:17:21.444 --> 01:17:25.944
So, I don't think I don't think it's efforts in documentation or necessarily wasted even if we decide.

721
01:17:26.250 --> 01:17:32.310
You know, either individually or collectively to not do anything further. Really right now.

722
01:17:34.500 --> 01:17:41.729
There is only 10 minutes left on the call. I'm happy to go for. As long as people are interested in. But I want to be respectful of that. Are there topics that.

723
01:17:41.729 --> 01:17:46.500
Folks feel like they, they want to discuss on this call that we haven't hit upon.

724
01:17:50.970 --> 01:17:55.409
1 thing, I guess I'll just say that we haven't really talked about at all. Um.

725
01:17:55.409 --> 01:18:04.289
Was the, um, if there is any interest in this bottom thing about, uh, cloud scrubbing, uh, and I would say that.

726
01:18:05.310 --> 01:18:13.739
There's, there's 2 ways to think about this 1 of this like, you know, you system may owe it to itself to investigate this because the.

727
01:18:13.739 --> 01:18:16.800
So, what are the types of events that I definitely can't deal with?

728
01:18:16.800 --> 01:18:19.829
Uh, with my junior process, if by inbounds get clogged.

729
01:18:19.829 --> 01:18:24.149
So, um.

730
01:18:24.149 --> 01:18:30.359
Yeah, me again, this is Mike. Um, so, yeah, I was gonna say, I saw that on your slide before and.

731
01:18:30.359 --> 01:18:36.449
That is something I'm interested in looking into. I think Pat's working on that also.

732
01:18:39.720 --> 01:18:51.960
I'm of a of the, the firm mindset that many of the types of events we see. It's not a good option for. But I do also recognize that the cloud scrubbing.

733
01:18:51.960 --> 01:18:57.359
Can definitely be viewed as an insurance policy for something that we are unable to deal with ourselves.

734
01:18:57.359 --> 01:19:04.109
Right and I know it can be very expensive, but I'd like to at least look into it so mm. Hmm.

735
01:19:04.109 --> 01:19:15.149
No, no, unfortunately it's quite a bit of time to stand up. In some ways. It would be nice to have some experience to actually have tested it.

736
01:19:18.390 --> 01:19:23.760
What was that pan? It takes a bit to stand it up.

737
01:19:23.760 --> 01:19:30.569
So, if there was an event that we were dealing with, it would have been actually something in place.

738
01:19:30.569 --> 01:19:40.529
Or something that we've played with. Yeah, I mean, we're talking about the horror story ones, right? So.

739
01:19:42.600 --> 01:19:48.510
You know, if we don't have some options and have to scramble at that point in time is.

740
01:19:48.510 --> 01:19:52.529
You know, it's going to be not very fun.

741
01:19:52.529 --> 01:19:58.470
I don't know if any of these clouds covers have a, you know, let's try it.

742
01:19:58.470 --> 01:20:02.850
Trial type thing, just to kind of proof of concept type of thing.

743
01:20:02.850 --> 01:20:07.710
To allow you to get a feel for how it would be done and just have, you know.

744
01:20:07.710 --> 01:20:10.710
Instrumentation in place to actually have done it.

745
01:20:10.710 --> 01:20:17.550
Or what other big 10 schools are doing, or.

746
01:20:17.550 --> 01:20:21.989
You know what, you know, what the doing or other.

747
01:20:21.989 --> 01:20:32.340
Examples Pat are you on the line? I am I right up the conversation? Yeah. Um.

748
01:20:32.340 --> 01:20:37.800
So, uh, yeah, there's, there's a, uh, a number of services available.

749
01:20:37.800 --> 01:20:41.670
Um, yeah, we've been looking at it from sort of.

750
01:20:41.670 --> 01:20:46.260
More of the big 1, um, than anything.

751
01:20:46.260 --> 01:20:50.399
That we just can't block that could potentially take.

752
01:20:50.399 --> 01:20:54.210
You know, golf, the entire network out of service.

753
01:20:54.210 --> 01:20:59.640
Which is roughly 1 to 5% of the attacks that are out there.

754
01:20:59.640 --> 01:21:07.260
Definitely was hoping to chat more about this at the system all hands meeting, um, in May.

755
01:21:07.260 --> 01:21:11.100
And kind of give an update on where we're at.

756
01:21:11.100 --> 01:21:15.960
With some initial R and D on this.

757
01:21:15.960 --> 01:21:20.340
And kind of get thoughts about.

758
01:21:20.340 --> 01:21:23.550
I guess get feedback about.

759
01:21:23.550 --> 01:21:28.770
Uh, what others within the system are thinking about this.

760
01:21:28.770 --> 01:21:32.970
Um, so, yeah, I.

761
01:21:32.970 --> 01:21:38.250
I know from other, I guess, Mike, your comment about, you know, other big, 10 universities.

762
01:21:38.250 --> 01:21:46.949
It's kind of all over the board, like, in a lot of cases, or in a couple of cases, like Michigan and Ohio.

763
01:21:46.949 --> 01:21:56.729
Their state network is doing, uh, the scrubbing service at the borders. So that's so, that's kind of what we are thinking.

764
01:21:56.729 --> 01:22:04.260
Our state network your meaning like, for Michigan merit or do you mean Eric? Meredith or not okay. Okay.

765
01:22:04.260 --> 01:22:09.390
Yeah, Minnesota that's right. We're that step child. That's not allowed to.

766
01:22:09.390 --> 01:22:15.960
To work with our R. E. institution. Yeah. Minnesota on the other hand.

767
01:22:15.960 --> 01:22:19.140
Has, uh, appliance based.

768
01:22:19.140 --> 01:22:24.510
Uh, services from what used to be networks.

769
01:22:24.510 --> 01:22:34.350
Uh, which is now net Scout, and they, um, they are architected more for up to 40 gigs.

770
01:22:34.350 --> 01:22:39.000
Which sort of handles the small to medium size items.

771
01:22:39.000 --> 01:22:43.649
But basically, would their system would fall over on the.

772
01:22:43.649 --> 01:22:49.890
Largest attacks basically so that's why I say there's sort of a hodgepodge going around.

773
01:22:49.890 --> 01:22:53.430
Was it using also at at some point in time.

774
01:22:53.430 --> 01:23:01.529
Yeah, no harbor networks came out of and all right, well, that's why. Yeah, so that's why.

775
01:23:01.529 --> 01:23:07.859
And there's also Internet to, uh, really, it's really rebranding service.

776
01:23:07.859 --> 01:23:14.520
Um, there's several schools, um, at 1 point, it was the University of Iowa.

777
01:23:14.520 --> 01:23:20.340
University of Illinois, and I believe Northwestern University.

778
01:23:20.340 --> 01:23:24.390
Using that service, um.

779
01:23:24.390 --> 01:23:29.550
And so that's why I say there's a, there's a number of different options available.

780
01:23:29.550 --> 01:23:36.390
And, you know, it's sort of, we need to understand what we're protecting.

781
01:23:36.390 --> 01:23:42.149
Um, and we have some ideas on that that we kind of want to confirm.

782
01:23:42.149 --> 01:23:46.680
What's the level? What's what types of service levels? Like how.

783
01:23:46.680 --> 01:23:51.359
How fast might as such a service spin up? Um.

784
01:23:51.359 --> 01:24:01.590
Uh, as well as, um, I guess other aspects of the service, you know, support costs, uh, nice. Yeah.

785
01:24:01.590 --> 01:24:05.850
And so that's why we're early in that process right now.

786
01:24:05.850 --> 01:24:12.539
Um, but as Michael's alluded to, there's there's only so much we can do.

787
01:24:12.539 --> 01:24:16.920
Uh, filtering, uh, our policing.

788
01:24:16.920 --> 01:24:21.810
Uh, uh, things on our routing infrastructure, but.

789
01:24:21.810 --> 01:24:29.039
Quite frankly, it's, uh, the work that's been done, you know, several years ago, I think we had a tax that took out.

790
01:24:29.039 --> 01:24:34.800
If I remember correctly, Whitewater gosh, maybe even platform.

791
01:24:34.800 --> 01:24:40.979
Uh, correct me there folks I remember it was 3 larger campuses.

792
01:24:40.979 --> 01:24:47.189
1010 to 30 gigabit per 2nd fragment of tax? Yeah, we're more common a couple of years ago.

793
01:24:47.189 --> 01:24:51.659
That we've seen was in the mid thirties towards O, Claire.

794
01:24:51.659 --> 01:24:55.079
Yeah, and so those are, um.

795
01:24:55.079 --> 01:25:07.439
Yeah, and so that's certainly on our mind. Um, the network delivered the bits up to people's door steps, but that's that's not necessarily what.

796
01:25:07.439 --> 01:25:21.750
Everyone wants out of the networks, so right. So the, the efforts that we put in back in 2016 to, uh, have I feel like, was fairly effective, uh, policing, uh.

797
01:25:21.750 --> 01:25:28.229
For the most part, put it in to the impact, uh, of for that, for folks that sign in to that service.

798
01:25:28.229 --> 01:25:35.220
Yeah, so Michael, what what happened in the attack we have to I mean, they're running their own, right?

799
01:25:35.220 --> 01:25:42.329
Uh, my memory of back, then that most of those events I'd have to look it up to see what I wrote down, but.

800
01:25:42.329 --> 01:25:50.279
Even back then most of those events for almost all of them are less than 30 minutes, and most of more or less than 5. so what my suspicion is that we probably had.

801
01:25:50.279 --> 01:26:03.630
5 minutes of really not a good time for all. Claire, uh, in terms of packet loss, uh, 10 gigabit per 2nd hand off to them. I don't think that they would have been, uh, pleased or down.

802
01:26:03.630 --> 01:26:07.829
Okay, so what was it was it Cornell that had the bad 1.

803
01:26:07.829 --> 01:26:15.149
Rockers records. That's right. Directors. Yeah. Luckily, situation I believe is.

804
01:26:15.149 --> 01:26:23.130
Their, their incoming links were overwhelmed, which is possible to have in the system network, but we are.

805
01:26:23.130 --> 01:26:27.989
Protected a bit based on the University of the number of links we have and their capacity.

806
01:26:29.609 --> 01:26:35.970
Yeah, it never hurts to remove the residue and scum off the fiber with someone else and.

807
01:26:42.720 --> 01:26:49.109
We're we're coming right up on 4 o'clock. Uh, so, so if anyone wants to stick around, I'll stick around for a bit. Um.

808
01:26:49.109 --> 01:27:01.260
I would say everyone has an invitation obviously if you want to talk to, uh, uh, system network myself, or others about this, if you have questions that you wanted to ask, or you come up with something, just feel free to reach out.

809
01:27:01.260 --> 01:27:05.579
Thanks can I ask 1 more question since we are here?

810
01:27:05.579 --> 01:27:13.859
Do you guys hear from your cio's about this? Is there any discussion internally or is it just from us where we're.

811
01:27:13.859 --> 01:27:17.520
Asking you guys about this, do you hear about it? Internally? Questions?

812
01:27:21.180 --> 01:27:24.869
I need the campuses in a word. No.

813
01:27:24.869 --> 01:27:29.340
No, yeah, no, no, no, not even at Madison. Really?

814
01:27:29.340 --> 01:27:34.020
Well, no, you're your CEO is interested obviously yeah. At.

815
01:27:34.020 --> 01:27:38.909
Yeah, okay. Thank you.

816
01:27:41.880 --> 01:27:45.449
Oh, um, kind of I'll find a different topic you.

817
01:27:45.449 --> 01:27:51.539
People are interested maybe need to stick around a little bit. There is 1 tool that I kind of brewed up that.

818
01:27:51.539 --> 01:27:58.140
I find useful for Palo Alto, just a script that I use to.

819
01:27:58.140 --> 01:28:03.359
Monitor, uh, traffic flows, um, basically see the logs.

820
01:28:05.909 --> 01:28:10.800
Floor is yours if you'd like it go ahead and if you have something you want to share or.

821
01:28:10.800 --> 01:28:15.029
Feel free to, uh, do you want me to keep, uh, keep the recording going for now?

822
01:28:17.939 --> 01:28:23.699
You know, I can keep the recording going, um, let's see if I can figure it out. Sure. My screen.

823
01:28:31.770 --> 01:28:35.880
All right, can people see that.

824
01:28:35.880 --> 01:28:43.199
I see your screen, so you're seeing, like a shell screen with, um, yep.

825
01:28:43.199 --> 01:28:48.539
Okay, pretty small though. Yeah, and we can probably change that.

826
01:28:48.539 --> 01:28:53.909
Hmm.

827
01:28:53.909 --> 01:29:03.869
I forget.

828
01:29:15.840 --> 01:29:20.550
It's right there. Dan, uh, on the right the change. Oh, okay.

829
01:29:23.939 --> 01:29:28.829
Jason.

830
01:29:28.829 --> 01:29:33.479
Yeah.

831
01:29:33.479 --> 01:29:37.859
I can see you fine. All right so it's.

832
01:29:37.859 --> 01:29:41.729
A parsing, it's a Perl script that person's firewall.

833
01:29:41.729 --> 01:29:46.529
Traffic logs from the Palo Alto, so.

834
01:29:46.529 --> 01:29:49.770
For instance, if you look at, um.

835
01:29:51.510 --> 01:29:55.800
The firewall logs that come out of the Palo Alto.

836
01:29:55.800 --> 01:29:59.369
They kind of look like this, um.

837
01:29:59.369 --> 01:30:03.359
But I wrote a little tool here, um.

838
01:30:03.359 --> 01:30:07.680
That allows you to specify a number of filters.

839
01:30:07.680 --> 01:30:14.039
In different output formats to, uh, either pass through a log file.

840
01:30:14.039 --> 01:30:17.039
Um, and it will pick up hello?

841
01:30:17.039 --> 01:30:26.220
Um, traffic records, so you can have this mixed in with your regular syslog traffic and it'll pick out just known Palo Alto.

842
01:30:26.220 --> 01:30:31.500
Formatted records and either create a summary at the end.

843
01:30:31.500 --> 01:30:35.039
Or you can have it produce summaries.

844
01:30:35.039 --> 01:30:40.649
Um, any periodic, you know, every so many seconds.

845
01:30:40.649 --> 01:30:46.079
Um, and plus you can follow your log, so you can monitor things in real time.

846
01:30:46.079 --> 01:30:49.710
And I found it very useful to to watch traffic. Um.

847
01:30:49.710 --> 01:30:53.430
So, for instance, what we were seeing here is, I was watching.

848
01:30:53.430 --> 01:31:05.850
For a particular firewall called guard cart firewall 3, just the deny traffic from what we call with man. I never changed the name of instantiated.

849
01:31:05.850 --> 01:31:10.020
And I'll put traffic every 2 seconds.

850
01:31:10.020 --> 01:31:16.500
And also only output source information.

851
01:31:16.500 --> 01:31:19.560
And so you can see here.

852
01:31:19.560 --> 01:31:25.590
You know, what rules we're hitting, what type of traffic? You know, what the action was.

853
01:31:25.590 --> 01:31:32.760
Protocols involved the top iop addresses sourcing that traffic. Um.

854
01:31:32.760 --> 01:31:39.210
Countries it's coming from there's also destination information too. If you.

855
01:31:39.210 --> 01:31:42.390
The, it'll show you both.

856
01:31:42.390 --> 01:31:46.319
Source and destination information.

857
01:31:48.149 --> 01:31:51.359
Um.

858
01:31:51.359 --> 01:31:56.878
And it can also spit off the log entries, um, real time, you know.

859
01:31:56.878 --> 01:32:00.599
And also in a formatted fashion.

860
01:32:01.798 --> 01:32:10.649
Um, so that you can view things real time, more useful. Say, for instance, when you specify a particular, um.

861
01:32:10.649 --> 01:32:16.139
You know, source or destination ID so maybe he said, um.

862
01:32:16.139 --> 01:32:20.488
Destination or yeah, destination equals.

863
01:32:20.488 --> 01:32:25.588
Particular ID number.

864
01:32:25.588 --> 01:32:28.769
Um, of course.

865
01:32:28.769 --> 01:32:33.538
I'd want to get rid of the deny stuff here.

866
01:32:33.538 --> 01:32:38.578
Then you can kind of see traffic occurring in real time.

867
01:32:41.639 --> 01:32:44.849
It'd be easier just to see it without the.

868
01:32:44.849 --> 01:32:53.009
Summary information here, so.

869
01:32:53.009 --> 01:32:59.939
If it's something people are interested in using it's a great way to to see what's going on.

870
01:32:59.939 --> 01:33:05.609
Um, look at high traffic flow of session builds.

871
01:33:05.609 --> 01:33:10.588
Rates and things like that, and just recently threw it out in GitHub. Um.

872
01:33:11.849 --> 01:33:17.069
Under Dan portal. Um, so if people want to try like.

873
01:33:17.069 --> 01:33:24.569
Didn't know, is that I might have added a bug to it. It does occasionally die here. So I might have to look at it again.

874
01:33:24.569 --> 01:33:29.009
And fix that, but anyways, I don't know if it was something.

875
01:33:29.009 --> 01:33:33.298
People would be interested in.

876
01:33:33.298 --> 01:33:47.998
Thanks for sharing Dan. I'm curious about your use case. Do you use this complimentary too? Just, you know, typical going through the monitor tab and, you know, filtering and doing things that way or you find that you're getting a different summarization this way.

877
01:33:47.998 --> 01:33:54.809
And sort of trying to get information in that way I find it a lot easier for me to use. Um.

878
01:33:54.809 --> 01:34:00.059
Then the monitor tab, and I feel that I get information more real time and.

879
01:34:00.059 --> 01:34:03.779
And also much more specific information with this.

880
01:34:03.779 --> 01:34:07.019
Because I can limit it to.

881
01:34:07.019 --> 01:34:10.798
Particular interface in particular ways I wanted to view it.

882
01:34:10.798 --> 01:34:14.609
Things like that. Um, okay.

883
01:34:14.609 --> 01:34:18.179
So, yeah, it's, um.

884
01:34:19.408 --> 01:34:26.248
Just a tool that I found useful. It it, it also is a great tool to hump through your logs for.

885
01:34:26.248 --> 01:34:39.479
Something that occurred in the past, you know, maybe you want to see everything for particular aicp addresses and stuff and and although the, there's a great role search tool in the Palo Alto.

886
01:34:39.479 --> 01:34:43.588
It, um, you really.

887
01:34:43.588 --> 01:34:51.658
I only keep so many log entries in the panorama or Palo Alto. It tends to be very slow parsing a lot of that.

888
01:34:51.658 --> 01:34:56.128
This rips through the logs pretty quickly.

889
01:34:56.128 --> 01:35:01.168
And you get some information that you can then process with other tools.

890
01:35:01.168 --> 01:35:07.529
Sometimes I even think it through itself, because you can pick blog entries into this.

891
01:35:07.529 --> 01:35:13.529
And then I might do 1 filter operation 1 way, and then maybe present it in another way.

892
01:35:13.529 --> 01:35:19.198
In the 2nd pass or something like that, I'm assuming you're logging at session and.

893
01:35:19.198 --> 01:35:24.988
Yes, so the firewall does log only at session.

894
01:35:24.988 --> 01:35:31.139
The way we have it set up thanks for sharing this. It's, uh, it's interesting and.

895
01:35:31.139 --> 01:35:35.009
We, we have a bit of a problem of volume, um.

896
01:35:35.009 --> 01:35:39.448
With traffic logs firewalls, I mean, this.

897
01:35:39.448 --> 01:35:46.559
Massive directories of things to parse through where, you know, spelunking through the logs themselves has become.

898
01:35:46.559 --> 01:35:54.269
Something most folks don't want to do so we tend to use the monitor tab more direct on the firewall. So I'm.

899
01:35:54.269 --> 01:35:57.628
Curious of how this all play out with some of what we've got.

900
01:35:57.628 --> 01:36:05.488
Interested to try it out. Hey, Mark, how do you do you have to share this stuff with your security group or how does that work?

901
01:36:07.349 --> 01:36:10.708
We're feeding it to them. We have, I don't remember if we're doing.

902
01:36:10.708 --> 01:36:16.529
I need to do they're either grabbing it from our logging server.

903
01:36:16.529 --> 01:36:20.759
Or we're doing it to 2 destinations. I don't recall which.

904
01:36:20.759 --> 01:36:25.588
So, are they, they're monitoring and also, so when you're.

905
01:36:25.588 --> 01:36:29.998
Um, with their own tools, or what? Yes, their own tools.

906
01:36:29.998 --> 01:36:34.259
Okay, I got the sock and they got I mean, we're.

907
01:36:34.259 --> 01:36:40.708
I think a lot of folks on this call are probably like a, you know, army of 1 or army of a few.

908
01:36:40.708 --> 01:36:43.798
For your campuses, we end up having this.

909
01:36:43.798 --> 01:36:48.538
Situation where there's a lot of special specializations and things like that. So.

910
01:36:48.538 --> 01:36:51.748
Separate cybersecurity separate.

911
01:36:51.748 --> 01:36:58.979
That's not necessarily the same same group for folks who do fire walling many of us get involved with them and do some, you know, Ops and.

912
01:36:58.979 --> 01:37:03.389
Secondary level support, but, you know.

913
01:37:03.389 --> 01:37:06.448
Got it down a couple of folks that are really expert in it and.

914
01:37:06.448 --> 01:37:10.019
Things of that nature got it thanks.

915
01:37:10.019 --> 01:37:19.828
Yeah, the other thing that I've leveraged quite a bit is the open dynamic block lists, um, that are out there that are free.

916
01:37:19.828 --> 01:37:23.548
Um, this site creates a.

917
01:37:23.548 --> 01:37:28.738
Palo Alto, consumable versions that you can then just, uh.

918
01:37:29.939 --> 01:37:34.408
Set your parallel to the import as the external.

919
01:37:34.408 --> 01:37:38.609
Blacklist or external dynamic list feature.

920
01:37:38.609 --> 01:37:43.198
And then basically set of rules to take action.

921
01:37:43.198 --> 01:37:47.128
And you can see that, um, you know.

922
01:37:47.128 --> 01:37:51.899
There are a number of these roles are getting hit, like the D shield 1.

923
01:37:51.899 --> 01:37:57.538
Emerging threats 1, Alto 1. well, I just went off the bottom. It here it is.

924
01:37:57.538 --> 01:38:01.469
That one's, uh, Palo Alto provided 1.

925
01:38:01.469 --> 01:38:05.248
Um, anyways.

926
01:38:05.248 --> 01:38:08.908
Find a lot of traffic gets filtered by that method.

927
01:38:10.889 --> 01:38:18.509
Hello.

928
01:38:18.509 --> 01:38:23.429
Thanks.

929
01:38:31.859 --> 01:38:35.548
Okay.

930
01:38:37.349 --> 01:38:40.918
Okay.

931
01:38:40.918 --> 01:38:44.248
Hello.

932
01:38:44.248 --> 01:38:49.559
Yeah, thanks for sharing your, uh.

933
01:38:49.559 --> 01:38:53.099
You're able to get hub and I will make sure that I put that in, uh.

934
01:38:54.208 --> 01:39:06.689
The notes for this as well. So, uh, it'd be addressing. I've never actually recorded a Webex. It'll be interesting to see what pops out the other side. Um, but once that's done, if it looks like a shareable, I will link it back out to folks.

935
01:39:06.689 --> 01:39:13.427
Thanks, Mike.