Paranoids Vulnerability Research: Ivanti Issues Security Alert

TL;DR: 

• On June 20th 2023, Ivanti released a hotfix for its endpoint management tool meant to stopgap a critical remote code execution (RCE) vulnerability in its endpoint management product.  
• In March, the Paranoids’ Vulnerability Research Team originally notified Ivanti of that critical vulnerability.    
• Unpatched, the weakness allows an attacker with direct access to distribute malicious software through Ivanti’s patch management solution — Ivanti Endpoint Manager.  

Part of our purpose, as the Paranoids, is to benefit our peers and people across the Internet as we all grapple with a digital wilderness. Vendor disclosure is a natural and important part of that mission.   

In that vein, near the beginning of the year, the Paranoids’ Vulnerability Research Team (VRT) discovered a devastating vulnerability in Ivanti’s Endpoint Management (EPM) product as part of a vendor assessment.  

The vulnerability allowed for remote code execution — giving a bad actor a method to distribute malicious software through a tool that sends out security updates.  

And, as part of the research process, we confirmed the feasibility of this by developing an end-to-end exploit that showcases how malware can be distributed to managed endpoints (demo).     

In March, after going through an internal review process, the Paranoids’ VRT contacted the Utah-based security vendor. Soon after, Ivanti issued a hotfix for the RCE to affected customers. The small code change was meant to correct the issue outside a regular release cycle.

In July, Ivanti issued a service update, containing the RCE patch alongside other fixes and regular release cycle changes. 

Not all customers should be immediately concerned. 

In this scenario, an attacker looking to exploit this vulnerability would first need to gain internal network access in cases where the EPM server is not directly exposed to the internet. Think VPN access or a secondary vulnerability such as Server Side Request Forgery. 

Regardless, it’s important to patch because of the severity of the software weaknesses. Versions 2022 SU3 and prior are vulnerable. The issues are fixed in 2022 SU4, and a hotfix for CVE-2023-28323 is available for 2021 versions. 

Introduction

The Paranoids’ Vulnerability Research Team (VRT) identified a series of vulnerabilities in a popular enterprise endpoint management suite called Ivanti Endpoint Manager (also known as Ivanti EPM) which can result in remote code execution, file disclosure, and privilege escalation:

• CVE-2023-28323: Unsafe Deserialization Leading to RCE
• CVE-2023-28324: Insufficient Client Validation Leading to Privilege Escalation
• CVE-2023-38343: XXE Leading to File Disclosure and SSRF
• CVE-2023-38344: Authenticated Arbitrary File Read

A hotfix was released on June 20th 2023, providing an early stopgap for CVE-2023-28323. If you’re running 2022 SU4, you have the required fixes for the above vulnerabilities.

If you’re running 2021 SU4, you’ll need to apply the hotfix or upgrade. If you’ve not yet updated your installations, you should stop reading now and do that immediately. You can find Ivanti’s advisory and update information here.

If following best practice by not exposing your management server to the internet, these vulnerabilities will require some level of internal network access or a strong SSRF primitive to exercise. However, there are currently a handful of internet facing installations at risk. 

On the Agenda

Today we’ll break down the root cause of CVE-2023-28323 and identify how it can be exploited for Remote Code Execution. Then, we’ll demonstrate how an attacker with in-depth knowledge of the target product can escalate the impact of this type of bug and move laterally into any managed endpoint to take over an enterprise network. To finish things off, we’ll demonstrate all this in action with a demo of the end-to-end exploit that we developed in-house for a Red Team operation.

Target Selection

Our VRT performs security research on products used by Yahoo and by the greater industry. By exploring the products that we take advantage of in-depth, we can develop a more informed view of the level of risk taken on by integrating them into our networks.  We’re primarily focused on vulnerabilities that grant a strong foothold into strategic targets, such as vulnerabilities in products responsible for configuration management or software distribution.  Ivanti Endpoint Manager fit the bill, and became the target of our research efforts in recent months. 

Ivanti Endpoint Manager is an enterprise product that enables companies to simplify and centralize patch management, software distribution, and configuration across their fleet of desktop and mobile devices. It provides both a robust desktop application (Ivanti Management Console) and a web app that enable point-and-click handling of software distribution, inventory, provisioning, reporting, and compliance features. It even includes a tool for live remote control of endpoints through RCViewer, featuring file upload/download, remote desktop viewing, and command execution.

The architecture consists of one or more “Core” servers, where administrators can perform management actions, and “Agents” which are installed on the devices to be managed. 

Vulnerability Discovery

CVE-2023-28323 is a deserialization of untrusted input leading to remote code execution. The issue exists in the ProcessEPMAuthToken method of the AuthHelper class, which sits inside of the assembly located at C:\Program Files\LANDesk\ManagementSuite\remotecontrol\Auth\bin\RemoteControlAuth.dll on a machine EPM is installed on. 

ProcessEPMAuthToken, as you might guess by the name, is responsible for extracting and validating signature information from a user supplied auth token, which is contained in the EPMToken field of the Auth parameter. The EPMToken appears to be a JWT in typical format  with 3 parts: the header, the body, and the signature. Each portion is Base64 encoded and separated by dots. The ProcessEPMAuthToken Base64 decodes the body portion, and looks for a magic byte marker (“|” in ascii) that indicates signature information that is embedded as a plain string. However, if this marker isn't found, it falls back to performing a call to BinaryFormatter.Deserialize on the bytes in the body. 

This is a straightforward example of insecure deserialization in C# .NET.  BinaryFormatter, as described by Microsoft, is a serialization helper that "Serializes and deserializes an object, or an entire graph of connected objects, in binary format". Microsoft has helpfully published an article describing the risks and security vulnerabilities that arise when leveraging BinaryFormatter.Deserialize in your code.

In fact, that article says “The BinaryFormatter.Deserialize method is never safe when used with untrusted input.” In this case, the auth token is controlled by an attacker, and is most definitely untrusted.

Reaching the Sink

With that dangerous sink identified, let’s figure out how to reach it with user input in order to exploit it. By searching for usages of ProcessEPMAuthToken in the same assembly, we see that it’s invoked within DoAuthentication_Integrated_SpecifiedDevice, which itself is called from within the method Post. 

It’s invoked from the AuthController class’ Post method, which has the HttpPost attribute. This appears to be a handler for POST requests, but POST requests sent where? 

This looks like a standard ASP.NET Web Api, so let’s consult the docs. The docs tell us:

In an ASP.NET application, configure Web API by calling GlobalConfiguration.Configure in the Application_Start method. The Configure method takes a delegate with a single parameter of type HttpConfiguration. Perform all of your configuration inside the delegate.

We can quickly locate the Application_Start method, and clearly see the call to GlobalConfiguration.Configure, which uses WebApiConfig.Register as the delegate. Within WebApiConfig.Register is exactly what we’ve been looking for: a route template! This shows us how the application expects to route requests based on the supplied path.

Based on this, we know that the route format for this application is /api/{controller name}/{id}, with the id parameter being optional. The last piece of routing information we’d need is the application base. We can just consult IIS for this. Perusing the configured sites within the IIS Manager, we quickly see one called RemoteControlAuth with the alias RemoteControlAuth, mapped to the physical path that contains our target assembly. This is the application base for request routing. 

With all this information in hand, we know that to hit AuthController.Post we’ll be sending a POST request to /RemoteControlAuth/api/auth that contains an Auth parameter as JSON in the request body.

Recalling the input handling seen in the AuthHelper.Post method, we know that we’ll have to set a few fields carefully to reach the vulnerable function. 

The source type must not be “ids” to reach DoAuthenticationISM, and additionally the source type must not be “ism” while logintype is 1 to reach DoAuthentication_Integrated_SpecifiedDevice. Both of these target methods flow down into the dangerous ProcessEPMAuthToken with a fully controlled Auth parameter.

So now we know how to get fully controlled input into the dangerous BinaryFormatter.Deserialize call, via a request that looks like this:

Identifying a Usable Gadget

The terrific ysoserial.net project provides us with ready to use configurable gadget chains, targeting a number of vulnerable .NET formatters. In this case, we can use the DataSet chain. 

Recall from the ProcessEPMAuthToken code that the data being deserialized will be in the body section of the JWT. The header and signature in this case do not matter, the code only needs to be able to parse out our payload successfully. I’ll stick a dummy value as the header and omit the signature here. This results in a request that looks like the following:

Launching this request at the EPM server results in successful command execution on the target host as the NT AUTHORITY\NETWORK SERVICE user. 

Escalating Impact

We’ve proven the ability to remotely execute programs on the target, but ideally we’d like to give ourselves the ability to move laterally into managed endpoints. Let’s first work on getting a persistent webshell deployed on the target, then move toward building out a way to deploy software using EPM.

Getting a WebShell

We’re going to adjust our gadget to result in a call that writes content to a file on disk. This will let us write out an ASP.NET webshell, from which we can begin to load and run arbitrary code. During deserialization of the TextFormattingRunProperties object, a call to XamlReader.Parse is triggered. 

RunPropertiesTemplate.xml

Rather than entering a call to Process.Start, we trigger a call to File.WriteAllBytes. To pass in the content of the 2nd parameter (a byte array) that will be written to the file on disk, we make a reference to another object in our ResourceDictionary named data. Our encoded webshell will be decoded into a byte array type through the chosen FactoryMethod of Convert.FromBase64String.  We’ll have our gadget generator populate the placeholders DESTDATAB64 (which will contain the webshell) and DESTPATH (which will contain the path to write to). 

Next, let’s write a little C# helper that will populate those above parameters, and serialize the payload for us. To keep the example small, I’ve extracted and slightly modified the relevant gadget construction code from ysoserial.net. Please refer to the original source if you’re interested in exploring the TextFormattingRunProperties gadget generator further. 

Main.cs

Finally, we need the actual webshell! We’ll want something flexible that lets us run arbitrary code on the target, not just execute commands. We’ll have it load .NET assemblies delivered through POST requests. The assembly will be sent in the mod parameter, and loaded in-memory with the Assembly.Load API. We can pass in arguments to the module through a second POST parameter, modargs. EPM leverages the Json.NET package, so we can load this from disk and use it to ease argument and result passing back and forth to the attacker. 

An important side note: Assembly.Load can fail when it finds transitive dependencies that sit outside of the current AppDomain. This is very likely to happen when our modules try to load some of the EPM product’s assemblies from disk, as those assemblies will also in turn try to load some assemblies. The loader won’t know how to locate them properly! To fix this, we can simply add an assembly load event handler, which lets us step in and load the assembly ourselves. You can see how this is handled within the InitLoaderShim function (which installs the event handler, LoadFromDepDir). 

webshell.aspx

Let’s rerun our original exploit with the payload adjusted to contain the base64 output of our gadget generator. Making a test request to execute the ping operation confirms that the webshell landed successfully!

Flexible Post-Exploitation

Here’s what an example module will look like, which we can compile and send over the wire to be loaded and run:

Module.cs

On The Path To Lateral Movement

Now that we have a way to upload and run our own post-exploitation modules on-target, let’s move on to writing modules that will let us leverage EPM’s functionality to move laterally into managed endpoints. 

The typical flow in EPM for deploying software onto a managed endpoint ASAP is as follows:

1. Create a Distribution Package

• A DistributionPackage contains information about the software to deploy on the endpoint. There are a few different flavors of packages including but not limited to: Exe, Powershell, Batch Script, etc. 
• It contains the Primary File location (a url pointing to the EXE/PowerShell script/Batch script)
• It specifies the user to run as (NT Authority\SYSTEM by default)

2. Create a Scheduled Task

• A scheduled task can be used to install a distribution package on endpoints that are assigned to it
• A few task types are available:

1. PSP (Policy Supported Push, a combo push-pull task)
2. Policy (the next time the agent checks in and receives its tasks during a Policy Sync, the package can be installed)
3. Push (the EPM server will remote into the endpoint and immediately trigger a Policy Sync, causing the package to be installed)

3. (Optionally) Immediately start the scheduled task

• A task can be scheduled for availability at a later date, or it can be set to “start now”

While this functionality is often performed using the desktop GUI client, it can also be accomplished via the Distribution REST API. An authorized client can create and modify packages and task information. 

Helpfully, there’s some documentation that details how you can access these APIs by adding a new client to the IdentityServer3.Core.Models.Client.json file with access to all scopes. With our code execution, we could potentially add a new client and use this API to accomplish our objectives. We run into an issue however: we’re running code as the IIS worker user, which doesn’t have write access to that file!

While we could read that file and leak the secret for a client that has the permissions we need, that’s dependent on the target having added one themselves already. So we need to find a different way to create distribution packages and scheduled tasks. 

The server side code already interacts with the database directly to store and manipulate package and tasking information. What if we loaded and reused existing product code to do that work for us? We can have our post-exploitation modules load types and methods from the product’s .NET assemblies to do this.

Creating a Task

Let’s take a look under the hood to see how the code for DistributionApi works. Since the target code is developed with C# .NET, it should decompile pretty cleanly using freely available tooling. For the following sections, I’ll be using JetBrains dotPeek.  

The IIS Manager shows us that the code for DistributionApi is located at C:\Program Files\LANDesk\ManagementSuite\distributionapi in the bin folder. 

Our prime suspect is DistributionApi.dll. Upon loading this .NET assembly in dotPeek, we can quickly locate the backing Controller code for the Package API, which is invoked upon a POST request to /DistributionApi/api/v1/Package:

Scrolling down a bit we can see how the user input is deconstructed and used to populate types and call methods from other .NET assemblies in the product. 

The main operating code appears to be the call to PackageFactory.CreateExePackage, which accepts a name, description, “primary file” (a url pointing to an exe accessible via http[s]), and an owner id (a user id who will own this package entry). Some other metadata fields are (optionally) populated, followed by a call to Commit() to save those field changes. Let’s replicate this functionality as a post-ex module that can be run through our webshell.

We’ll compile our .NET assembly targeting version 4.8, to match the version available and utilized by the target. We can leverage functionality from System.Reflection to load the types and methods we need, then invoke them. This example code performs the following steps to mimic the real product code:

1. Load the .NET assembly that holds the types 
2. Load the PackageFactory type from the .NET assembly
3. Construct an object of the PackageFactory type
4. Resolve the CreateExePackage method for the PackageFactory type
5. Invoke the CreateExePackage method on our PackageFactory object, passing in the required parameters
6. Save the ID of the created package for later, so it can be paired to a scheduled task

Testing the Module

Let’s try out our module that creates an EXE Distribution Package. I’ve written a simple wrapper tool called foist that handles making the POST request to the webshell with the .NET assembly and module arguments in the body parameters.

Package creation through code reuse is successful.

We can see from the demo video that our post-exploitation module successfully creates the Distribution Package, which we can confirm by navigating to the My Packages section within the the Distribution section and refreshing after our module runs.

The Rest of the Owl 

The final steps of programmatically creating a scheduled task and scheduling it are left as an exercise for the reader. Locating the responsible code, loading it, and reusing it follows a similar workflow to the example presented for Distribution Package creation. 

To demonstrate that the whole flow is feasible, we have a demo of a working end-to-end exploit. What you’ll see in the video is the following steps in order:

1. Exploit the deserialization vulnerability and drop the webshell to persist access with the install command
2. Find an owner for our packages and tasks. We dump the list of admins and choose the user with id 3.
3. Create a distribution package, with the source url pointed at a web host we control which is hosting our malware
4. Create a scheduled task associated with our distribution package
5. Dump the hostnames from the inventory to identify a target to assign our task to. This is the target we will move laterally to
6. Assign our scheduled task to the victim machine
7. Start the task, and wait
8. Pop! Our malware is deployed, runs, and writes out the name of the user it is executing as to a file on disk. 

A demonstration of the full end-to-end exploit that accomplishes lateral movement 

In closing, be sure to patch your Ivanti EPM installations immediately if you’ve not already done so by this point. We’d like to extend thanks to Ivanti for patiently working with us to get the submitted vulnerabilities remediated. 

About the Author

Blaine Herro is a member of the Paranoids' Vulnerability Research Team and Yahoo’s Red Team. He conducts research to uncover vulnerabilities in products used by Yahoo and the greater industry, as well as develops tooling and capability for use in full scale Red Team operations.