Gary Schwartz: Hey, everybody. Welcome to today's LinkedIn Live. It's episode three of Keeping the Pace with NetRise. I'm Senior Vice President of Marketing. My name is Gary Schwartz. And today, our team is going to talk about the blast radius problem, why software supply chain visibility isn't optional. And I'm going to introduce or I'm going to let our guys introduce themselves. Michael, why don't you go ahead and kick it off?

Michael Scott: Yeah, I'm Michael Scott. I'm the CTO and co-founder of NetRise and basically in charge of all things technology and product at the company.

Gary Schwartz: Colin?

Colin Lernihan: Senior Director of Product Management, Colin Lernihan. Been here for about ten months now and been working with Michael since then to build this vision into a reality. So super excited to show it to you guys.

Gary Schwartz: Now Ishan.

Ishan Sethi: Cool. My name is Ishan Sethi, also a product manager here at NetRise. I've been working with Colin and Michael over these past ten months as well. Super excited to bring new features to NetRise.

Gary Schwartz: Thanks, Ishan. So it's been a really exciting period, as the guys have talked about, the time to be developing this product. And we announced and launched NetRise Provenance at the RSAC conference a couple of weeks ago in San Francisco. And I'll just say that the initial response from the market was absolutely outstanding. There were a lot of really, really good conversations on the show floor. And NetRise Provenance was actually highlighted by CRN as one of the top twenty cool product launches at RSA, something that we're really proud of and really looking forward to understanding, you know, what the problem is that we're solving and why there's urgency really for this today. So, Michael, can you start us off and tell, you know, why did you create this and what's the problem that we're trying to solve?

Michael Scott: Yeah, thanks, Gary. So this is actually the fruition of years long planning and sort of preparation to get to this point. So really what kicked this all off is when the xz-utils compromise occurred a few years back. Just at a high level, there was a dependency that a maintainer had compromised, and this dependency was very prevalent, let's say, within the software supply chain ecosystem. And it became very clear at that time that it was a difficult problem to understand not just exactly which versions of the package were compromised or anything like that, but really where did that package end up and was it compiled into other binaries, which packages that are downstream from the affected repository were actually sourcing from the source code that had the initial problem. And so we started off on this journey to be able to answer that question immediately and you know frankly it's been a very timely release let's say i believe the axios breach happened on the same day that we did the launch at rsa which was totally coincidental um but you know there's a lot of uncertainty in the market i would say right now around these attacks with good reason because the nature of the software supply chain today is that you know it's very easy to write code especially given the explosion of software that's coming from ai. And also just the fact that the software supply chain ecosystem has been built out to be as frictionless as possible. So you could potentially have a piece of software that solves some problem written and deployed and distributed in a matter of hours. And it's very difficult to trace back exactly where that software will end up with some sort of visibility like what we provide with Provenance.

Gary Schwartz: Why is the problem so hard? Why is this such a difficult thing to solve when XE utils, for example, happen?

Michael Scott: Yeah, so there's a lot of different problems that are adding to the overall problem. So, you know, there's disparate ecosystems and different languages that distribute packages. You know, for example, Node.js has something like NPM. Python has PyPI. Obviously binaries are a whole different animal, which is something that we also cover at NetRise where dependencies are super difficult to trace back . um and you know overall the way that these packages are distributed there's no sort of standardized method that requires that you include certain metadata as part of the distribution package so just as an example you know there's many packages that are distributed on the software supply chain that don't even tell you the upstream source code that the package is derived from so you need to create a system that allows you to reliably both from the breadth and depth perspective to be able to understand exactly what the connections are between the different layers of the software supply chain. You take that a step further and there are also many contributors. There are many organizations that also contribute to repositories as well. There's dependency relationships between those packages that are resolved whenever packages and dependencies are installed. So there's sort of this disparate sprawl of different layers of the software supply chain. And you have on top of that, the fact that installation of dependencies is temporarily based. So, you know, if I do an install of a package today, I'm not going to get the same answer about which packages will be installed if I run that install tomorrow. And so you need to have sort of an all encompassing wide view of everything from contributors to organizations to different types of packages across different types of languages, their dependency relationships. And ultimately, you want to be able to trace back where those dependencies end up. And that could be anything from a laptop or a desktop or a server to a container to a VM, whatever the case may be. And so if you don't have full visibility, you're not going to get the full picture.

Gary Schwartz: Well, I mean, that's an interesting one, talking about the laptop, because that sounds to me like it's not just a developer problem anymore, right? People are using AI tools that execute code even if we don't know about it on our laptops. And does that increase the attack surface?

Michael Scott: Absolutely. You know, traditionally, we've talked about securing developers' laptops and, you know, typical user endpoints as two separate things. And what is being shown today with the prevalence of AI tools is that they are one in the same and the market is going to move in that direction of needing tools that can secure both. And just to make that a little more concrete, if I install cloud code on my system and I go to cloud code and I tell it to build a dashboard for me or to give me a report about something or to scrape stuff from the web and serve it in some way, if you look at what's actually happening on the back end of that, the AI is actually installing dependencies and running Python and other languages, and you just don't see it most of the time, even if you're using the GUI. And so understanding that in order to facilitate solving the requests that are made, you basically have to treat every laptop that there is an AI user on as a developer at the same time. And it's going to continue to move in that direction, I would say.

Gary Schwartz: That means like I, as a marketing person, I'm not a developer. You still have to treat my laptop as though I'm a developer.

Michael Scott: Absolutely. Every laptop is a build pipeline now. That's how we're viewing it as NetRise. And I think that that's going to prove to be true over time.

Gary Schwartz: That's insane. And so I know one of the most important things that we're going to talk about is helping people understand the blast radius when incidents occur. Maybe this is a good segue for you to talk about that.

Michael Scott: Absolutely. I'm going to go ahead and start sharing my screen. OK. Here we go. Is that pulled up?

Gary Schwartz: Yes, it is.

Michael Scott: Alrighty, so let me pull this into slideshow mode. So this is just sort of highlighting the issue at a higher level. This is an example of some of the problems that exist specifically around the light LLM compromise that occurred recently. You see a lot of stuff in the news and open source reporting where people are talking about light LLM and maybe the direct dependence of light LLM. But the fact of the matter is there is a very deep dependency web of things that are reliant upon light llm and that the further out from light llm that you go the more things are affected and it's actually true as well that you don't even have to run these dependencies in many cases in order to be compromised because there are scripts that run whenever the the dependencies are actually installed. So just because there's an indirect link to LIDL LM, you could be compromised in real time as you pull some dependencies in that are directly or indirectly relying upon an effective version of LIDL LM. And a lot of folks right now are talking about the packages. Really, where this ends up affecting people, yes, the packages are important. But also, what are the deployables at the end of the day that are going to end up in the enterprise that are really going to cause the problem? So your containers, your VMs, your Kubernetes workloads, firmware builds. And by the way, that's something that has not been talked about at all when it comes to this. Firmware that ends up on hardware, on devices that are running in networks, are also affected by this. You might have a device that's running in your enterprise network and you'll never know it, but it could have been affected by something like the Axios breach. Here at NetRise, we've actually shown that that is the case, that is something folks have to worry about specifically. And so this is sort of just indicative of the problem here. You go from one package that is light LLM that everybody's talking about. You take one step out, you end up with twelve hundred direct dependents and another step out, you're at thirty eight thousand. And this goes even deeper than that. Yet most companies that are helping to solve the problem are only talking about the left two layers here.

Gary Schwartz: So does that mean that I might not know that I'm pulling in compromised libraries?

Michael Scott: Absolutely. Yeah. So let me just give you a specific example. I did a little bit of research a few weeks back. I'm sure everybody's aware. Maybe it was a week ago. I don't even know. A week is like a year these days with AI . That the source code for cloud code was actually leaked and I just did a little bit of looking in that and it was shown that axios is one of the dependencies of cloud code. So I'm not saying cloud code is affected by the compromise. The point is, it could have been compromised, had a build pipeline run at a time when the affected package was available on the distribution pipeline. So absolutely, yes. If you're pulling in apps, even if that app is not the thing that you think is affected, it could be directly or indirectly affected by compromises. I'll say this as well. There was a lot of repositories that were dependent upon Axios as an example. And the way that these sorts of supply chain breaches work is that when you start with one repository that's affected, you take a step out, just like these dependencies or dependency relationships exploding, you can potentially get a lot more credentials and other repositories that are pointing to this layer of things that are affected, and you can really expand out and affect a lot of repositories really quickly just based on how tools like GitHub works. So there's a lot of credentials potentially that have not been used yet as part of this compromise. And so you have to be very vigilant and keep an eye on these things and block them if you can in real time.

Gary Schwartz: Well, so I mean, if I understand what you said correctly, you said that there's sort of a window, right? And so the compromise may be resolved. But that doesn't mean I'm clear, because I may have folded in during that time, while it was compromised without knowing it. So what can people do about that?

Michael Scott: Well, this is where I'll start to talk about the demo here, specifically about the provenance product. What you really need is to have a tool that gives you full visibility into the full dependency graph, all layers of it, up to and including containers and everything else, so that you can really, in real time, start to understand what is affected. And so just here in the UI for provenance, I'll give you a few examples. Let's start with light LLM here. I'll pull up the repository. I may need to refresh my session, so there we go. You can see there's a lot of information that gets pulled up here. First and foremost, you see contributors, everybody who's contributed to the LightLLM repository. You can see interesting little indicators here about different accounts who have had information that has been dumped as part of password breaches, as an example. Another thing that you see is all of the downstream packages that are dependent upon this source code. You'll see the number there, that's eight hundred and thirty-four different package names that are dependent upon this source code repository and many versions for each of those. I'll guarantee you, ninety-nine percent of these packages, nobody has even mentioned in open-source reporting. Not saying they're affected, but it's something that needs to be understood as part of the dependency relationship graph . um also you know geographic information is important to some companies you know there are certain rules and standards that you have to follow if say you're like an automotive company you need to know exactly where your contributors contributors live so that you can make sure that you stay within compliance of those rules that you have to follow as an automobile manufacturer. And also we have some of the stuff around repository health. Is this a repository that's super important in the supply chain and is sort of a choke point that if it were to be compromised, how bad would it be? And also is good hygiene being followed for this repository so that I know that I should or should not use it? And so just to take this single example a little bit further, if I open up the light LLM Python packages and I click on one of these versions, it'll bring me over to that specific version of the package. And I just want to point out one thing that's very interesting. There are nineteen thousand four hundred and nine dependents on this version of this package in the software supply chain. And there's a good chance that with future compromises like this, just the act of indirectly installing a package will end up with you being compromised. And if you couple that with the fact that every laptop is now essentially a developer machine, you really start to see how important it is to understand exactly what's going on. And so, you know, just to take this to the canonical example here with XZutils where this all started. If I type in the advisory for it, luckily with XZutils, it didn't make it to the stable distribution of the package distribution layer. So there are no directly affected packages anymore other than the package itself. But you can see if it had made it there, there was ninety two thousand packages that were indirectly affected by this compromise, any one of which could have resulted in a compromise for you. And there's a lot of good information here as well about the repositories that Gia Tan, the malicious contributor, had also contributed to, many of which were not even discussed. If you want to take a look at his email address and look at his direct contributions, you can get some of those versions of the actual packages where Geotan's code ended up, et cetera. And so you could see this giant graph of the software supply chain allows you to pivot around and to be able to understand what the impact is. And really, that's where you start to talk about prevention. In order to prevent these attacks from affecting you, you need to be able to block builds. You need to be able to prevent dependencies from being installed, et cetera, in real time as these different attacks are occurring. And so we also have a CLI tool for this where you put it into your build pipeline. We have a GitHub action for this as well. And just to give you an example of what this looks like, if you want to have a policy for preventing software supply chain compromises, you can make it where if any of the advisories that we create in real time with all the blast radius information included is flagged, your build will come back with the information. It'll fail, first of all, but also come back with the information that you need in order to fix the problem. So I'll stop there. There's a lot of information that I just fire-hosed everyone with, but... Yeah, that's kind of the state of things right now.

Gary Schwartz: So that's I mean, there's a heck of a lot to unpack there. So, you know, just to kind of summarize a little bit, so you've shown how we can discover where the impact is and then what one can do is is about the know the developer use case um and maybe Colin Lernihan uh Ishan Sethi can you guys talk about like if there's people responsible for third-party risk and we're starting to think about all of our um again our laptops and all different sorts of devices that can be impacted by this what can people do on that side?

Colin Lernihan: So on the engineering side, I mean, the CLI tool that we have with Prominence allows you to set policies. So if your organization wants to set blocking policies off of geo risk or the presence of an advisory or single bus factors or a percentage of the... Folks contributing may have had breaches. There's any number and any combination of different policies that you can set with our policy engine, and you can actually set policies that your organization is comfortable with and make sure that, one, you're not pulling anything that's been impacted by an advisory, but also adheres to whatever policies you and your organization need to adhere to, whether that be making sure contributors from certain countries aren't able to contribute or whether that be pulling in repos that aren't obscure and that aren't well-maintained.

Michael Scott: Absolutely. And as another piece of this, I hadn't mentioned this before, but we do have a system that is in real time keeping up with the software supply chain ecosystem. So as new packages are getting published, we are in real time analyzing them, figuring out if they're bad. And we're calculating the blast radius on that so that you can stop things in real time so that things do not affect you like the next Axios or the light LLM breach.

Ishan Sethi: Yeah, I wanted to ask a question on that actually, because it seems like flight LLM in those examples that you were showing, it was more of a reactive approach, but we are capturing a lot of signals like geographic information, contributors, like whether or not they're malicious. I wanted to ask, how can we use all these signals to kind of prevent an attack like that from affecting you before it happens? I know you just touched on it, but I was wondering if we could explore that a little bit more.

Michael Scott: Yeah, so there's a few different things. Obviously, the breaking of the builds in real time as malicious things are found are a big piece of this. So what that looks like is in real time when we do see something bad come up, we calculate the blast radius, we save that, we create an advisory for it. And then because you have advisories set to break your builds in real time, you'll be able to prevent things from that perspective. Also, there's a proactive element to this as well where we do a component criticality index on the back end where we look at the choke points to understand which packages deserve more scrutiny just based on the fact that if this thing were to be compromised, what would happen in the software supply chain? Providing a little bit of extra scrutiny on those packages, an additional layer of analysis is super important and something that we do as well. And lastly, what's bad to me is not necessarily bad to someone else. Maybe I'm an automobile manufacturer. Maybe I care about certain types of indicators related to software supply chain that others do not. And being able to very granularly define these policies that are important to you as an organization is also something that these policies provide. So everything that you just saw in the UI, every bit of data that we showed is something that you can layer on top of your organization as it pertains to you and your builds and your systems so that you ensure that those systems are not affected at whatever level you want to set the policy for. So that's kind of where the state of things are today.

Gary Schwartz: Hey, there was a question from Nate, I see, that was asking, from the automotive example, would it be possible to apply a U.S. trade agreement to act compliance?

Michael Scott: um if you're talking about geographical location excluding individuals from locations we use many different clues for geolocation uh some examples would be looking at email addresses websites and even the commit times and looking using lms to help derive probable locations so we give confidence intervals and all that good stuff so you could block off of all the signals that we have. We'll leave that up to you to decide which signals, which confidence intervals you'd want to put blocking policies on top of. But we give you all the raw material so you can make decisions on that.

Gary Schwartz: And then we allow them then to set the policy so everyone can kind of customize that CLI that you showed by the policies that they want to set. Is that correct?

Michael Scott: Absolutely.

Gary Schwartz: Awesome. Thanks. Keep the questions coming, people. That was a great one. And so on the predictive thing, right? Xeutils, that was kind of discovered by accident, right? Are we saying that this gives us a better way? I'm not saying that we might necessarily would have found that one, but to be more predictive?

Michael Scott: Yeah, and this is really where AI comes in. We have a multipass system that is looking at the firehose of packages that are coming in in real time. We do sort of a broad pass on those packages to see what looks interesting or suspicious or malicious. And then we do a deeper pass after we get that initial pass done on sort of the whole picture. Is this package actually malicious? Dig into it forensically. Look into all the clues . tie this back to the overall ecosystem? Is there some sort of campaign that is ongoing that this ties to so that you can get sort of the full picture? And so we're doing this at scale, we're doing it in real time, and AI has really sort of opened up the aperture for what is possible when it comes to this. So absolutely.

Gary Schwartz: Okay, that's awesome. I don't see any more questions right now coming from the audience. What do people need to implement this?

Michael Scott: So, you know, reach out to our sales team. The CLI tool is available on GitHub, so everybody can go look at it and see what sort of knobs you can turn with your policies and everything like that. But yeah, I would just say reach out to NetRise, reach out to the sales team, and we'd love to chat.

Gary Schwartz: OK, that is awesome. Yeah, as I said earlier, we're super excited about this, I think, and the urgency. Mike, can you just really quickly talk to the urgency? I think since we launched several weeks ago, you mentioned the access. How many incidents are we finding daily?

Michael Scott: It's really bad. I'll say this. So we have a slack channel that all the alerts come into. And I was telling people over the weekend that I just have anxiety even being in that channel, both from the perspective of the sheer number of things that are coming in that are malicious, that are dependencies that will affect the supply chain in a significant way, but also like sort of the ankle biter issues that people think are not a problem, but actually are. So just to give you an example, we find a lot of hard-coded credentials and tokens that are baked into the packages as well, and we're detecting and tracking those at the same time. On its surface, that doesn't seem like a big deal. But if that contributor or organization who is publishing these packages has a token that's available and say they have another package, that is actually popular and is prevalent in the software supply chain, you potentially have the ability to push to that organization's repository now. We're doing this, we're tracking this information, it is a guarantee that attackers are doing the same thing. And so the thing that keeps me up at night is the web of dependencies and the software supply chain and the fact that if we get to a certain point, it's very difficult to unwind all the problems. So we wanted to start proactively. We wanted this tool to come out and start to solve problems quickly. So it's very timely, once again, that this is available now. That's kind of the state right now.

Gary Schwartz: Yeah. And it scales when it's an automated agent rather than a human, right?

Michael Scott: Yeah, absolutely.

Gary Schwartz: Okay, cool. I'll appreciate all the questions that have come in. And thank you, you know, Michael, Colin Lernihan, Ishan Sethi for sharing today. I mean, we are, as we say, we are super excited. I want to thank all the folks who have been with us during the session today. You know, be sure to follow us on LinkedIn. Come see us at netrise.io. And we're here, as Michael said, it causes all of us a lot of anxiety to see what the state of software supply chain is today. We're really working to solve a serious problem because this could potentially disrupt the way software is created and the way open source was built on trust. And this is really about understanding and I think trying to recapture trust in that world. Is that a fair summary?

Michael Scott: Absolutely. We'll walk back all the issues that we're having right now at a time when it's super important. So absolutely.

Gary Schwartz: Okay. Well, I think that's it from all of us, Michael, Colin Lernihan, and Ishan Sethi. Thanks all of you for incredible content today. And we'll see all of you who've been with us today soon on the next episode of Keeping the Pace with NetRise. Awesome. Thanks, everybody.

Outro: Thank you. Want to learn more about NetRise? Visit us at netrise.io, follow us on LinkedIn, subscribe to our YouTube channel, or join our mailing list.