We recently sat down with Neil Wyler (a.k.a. Grifter in the hacker community), a principal threat hunter for NetWitness, about his other job: managing the network operations center (NOC) for Black Hat. With the U.S. show coming up fast, we wanted to learn more about Neil’s background and history with Black Hat, and how his role and the NOC itself have changed over the years.
NetWitness: How did you get involved with Black Hat?
Neil: I originally got involved with Black Hat 20 years ago through my involvement with DEF CON. I was an admin on the DEF CON forums and I was posting online about the cost to go to some of these industry events. I had just got out of the military and didn’t have the money to attend. Someone else on the forum suggested to me that if I wanted to attend these shows, I should volunteer to help out. So I got in touch with the Black Hat organizers and expected that I’d be going to the Black Hat USA show in Las Vegas, but I ended up being asked to help at their Windows show in Seattle a few weeks later.
I was part of the team that set up the network and it was a great time. Black Hat’s next show was in Amsterdam, so I didn’t go out there for that one. But there was a mistake and the equipment never made it off the loading dock in Seattle, so the team in Amsterdam had nothing – no routers, no switches, nothing. After that incident, they got back in touch with me and asked if I’d like to organize and run the network at the Black Hat USA show. They told me that I could bring a team, and they’d pay me, so this was turning into a paid gig! I accepted the offer and brought some friends as the team – and it was not enough people.
We did really well at that first show. The network was stable just about the whole time. We only went down for 15 minutes, and that was because one of the instructors had a zero-day for some of the equipment we were using and demonstrated it for his class. Before my time managing the network, it would be down about 50 percent of the time, and it just wasn’t reliable. After we went from 50 percent downtime to being offline for just 15 minutes, the folks at Black Hat asked if I wanted to stay in this role for all their shows. That was 20 years ago, and I’ve been doing it ever since. All because that equipment never made it off the loading dock in Seattle!
NetWitness: How has the NOC (network operations center) evolved over those 20 years?
Neil: The original goal of the NOC was to keep the network up and stable. There were always rumors that the network at Black Hat was one of the most hostile networks in the world – with people attacking each other and testing out the techniques they’re learning in the trainings – so our first goal was to keep the network up.
Over time though, we wanted to see what was actually happening on the network. After all, this is a hacker conference and there’s always some shenanigans going on with some of the attendees. We wanted to see things like if the web hacking class was attacking the internet, or if the offensive hacking class was attacking the defensive hacking class. We had our suspicions, but we just couldn’t see it.
The first threat detection product we had in the NOC didn’t give us the visibility and dashboards we thought we were going to have. By that point, I was working full-time for EMC, who had acquired RSA, and we were able to make some internal connections for getting the NetWitness Platform put into the NOC.
As soon as we set it up, we had dashboards right out of the box, and as it started ingesting traffic, we started getting metadata. It was exactly the data we were curious about. It allowed us to carve through the traffic, meaning that when we saw things that were potentially malicious, we could see where and when it was actually happening (like in a lab session), and make sure it wouldn’t bleed over and affect other attendees.
NetWitness: What sort of activity are you watching for in the NOC now?
Neil: Our first priority: maintain network stability. We watch for people trying to knock the network offline or signs that someone is overloading a certain part of the network.
Our next priority is of course security. We watch for activity that looks like an attack. If we see it, we try to determine if it’s a demo or just happening in one classroom, or an attack being directed at a specific machine. Then we’ll know if it’s part of a planned Black Hat program activity and we add it to an allow list. We can also see where things are actually happening on the floor plan, so if it’s something we need to go and investigate, we’ll go to that classroom or booth, tell them we’re from the NOC, and make sure that what we’re seeing is expected.
One interesting thing we weren’t expecting to see: people coming to the conference already “owned.” They already have malware on their systems, they’re already compromised. And they don’t even know it. Since we have the best-of-the-best equipment across the environment and are monitoring everything, we can see beaconing out to a command-and-control infrastructure or what appears to be crypto mining taking place. When we see that, we’ll walk down to that classroom where we’re seeing it, find the person who is compromised, pull them aside and let them know.
It’s really hard to do that now since the show has grown to tens of thousands of attendees, so now we rely on orchestration to help us meet that need at scale. When we see something that appears to be command-and-control, any clear text credentials, any crypto mining, whatever, we have a playbook in place so that the next time that person tries to browse to anything on the network, they receive a message from the NOC team alerting them to the potential problem.
Lastly, our role is educational. Everyone who attends Black Hat is part of a security organization and the wider security industry, and it’s interesting to see if those people are “eat their own dog food.” Throughout the show, we share information about what we saw on the network during the event. We’re basically trying to give something back to the community and help inform the people that contribute to it.
NetWitness: What are some of the trends you expect to see in the NOC data this year?
Neil: The biggest thing we’re expecting to see is less data than we’re used to. We’ll probably have around 30% of the usual number of attendees. All the trainings will be done virtually as well. The amount of traffic will be less, but it won’t change the overall behavior.
We have weird things happen every year, frankly. For example, I was walking down the hall in Mandalay Bay a few years ago, and there was a guy maybe 50 feet ahead of me walking by training classes and throwing USB drives into the rooms. I grabbed one myself, took it back to the NOC, and tested it; of course it had malware on it.
Sometimes people will bring cheap devices with antennae designed to disrupt the network, and since we have such good wireless equipment, we know where the devices are. We’ll go and find them — they’ll be hidden in a planter or something — so we’re always dealing with some sort of shenanigans or scavenger hunt. I’m sure there will be something that pops up this year too.
NetWitness: Have you ever had any major issues in the NOC?
Neil: Sure. I should preface this by saying that when we set up the NOC, we bring in all our own equipment. The only thing we use that’s already there are the wires and cable in the walls. We bring in all our own switches, routers, everything. And we do this first and foremost because I’m a control freak! But also, we need to be 100% sure that our equipment is up to the task.
In the NOC, we invite other companies who offer solutions we’ve tested and trust to come in and run their equipment as part of the larger team. There was one year when the personnel from one of these partners pushed out a change in the configuration that was meant to make the network faster and more stable, and it ended up taking down the network for about 90 minutes. As a result of that – and the fact that I had to go around and apologize to about 70 trainers and instructors – we no longer allow our partners to change their configurations unchecked while they’re in the NOC. If a configuration needs to be changed or an update pushed, one of the NOC leads has to actually push the button.
It was a painful experience, but we learned from it, and since we controlled everything, we knew precisely what happened and how to avoid it again in the future.
NetWitness: How can vendors get involved in the Black Hat NOC?
Neil: We are actually very careful with how we operate the NOC. None of the companies who provide gear for the NOC are sponsors. There are no sponsors, there are only partners. We don’t accept any payment to use any vendor’s equipment. And we’ve been offered VERY large sums by some vendors to get us to use their products.
Bart Stump, Steve Fink, and I assess different technologies and see how it would fit into our configuration. If we find something we really like, we’ll approach that company and see if they’d like to partner with us and provide equipment for the NOC. We’re really spoiled because they say yes every time we ask.
NetWitness: This has been great, Neil. Thanks so much for giving us a peek behind the curtain into the Black Hat NOC.
Neil: No problem. See you at Black Hat!
# # #
Be sure to tune into Neil and Bart’s end-of-show report that recaps all the Black Hat network activity the team saw in the NOC.