January 18, 2018

Handcrafted Gadgets

Introduction

In Q4 2017 I was pentesting a customer. Shortly before, I had studied json attacks when I stumbled over an internet-facing B2B-portal-type-of-product written in Java they were using (I cannot disclose more details due to responsible disclosure). After a while, I found that one of the server responses sent a serialized Java object, so I downloaded the source code and found a way to make the server deserialize untrusted input. Unfortunately, there was no appropriate gadget available. However, they are using groovy-2.4.5 so when I saw [1] end of december on twitter, I knew I could pwn the target if I succeeded to write a gadget for groovy-2.4.5. This led to this blog post which is based on work by Sam Thomas [2], Wouter Coekaerts [3] and Alvaro Muñoz (pwntester) [4].

Be careful when you fix your readObject() implementation...

We'll start by exploring a popular mistake some developers made during the first mitigation attempts, after the first custom gadgets surfaced after the initial discovery of a vulnerability. Let's check out an example, the Jdk7u21 gadget. A brief recap of what it does: It makes use of a hashcode collision that occurs when a specially crafted instance of java.util.LinkedHashSet is deserialized (you need a string with hashcode 0 for this). It uses a java.lang.reflect.Proxy to create a proxied instance of the interface javax.xml.transform.Templates, with sun.reflect.annotation.AnnotationInvocationHandler as InvocationHandler. Ultimately, in an attempt to determine equality of the provided 2 objects the invocation handler calls all argument-less methods of the provided TemplatesImpl class which yields code execution through the malicious byte code inside the TemplatesImpl instance. For further details, check out what the methods AnnotationInvocationHandler.equalsImpl() and TemplatesImpl.newTransletInstance() do (and check out the links related to this gadget).
The following diagram, taken from [5], depicts a graphical overview of the architecture of the gadget.

LinkedHashSet
| |
| +--> Proxy (Templates)
| |
| +--> AnnotationInvocationHandler
| |
| +--> HashMap
| | |
| | +----> String ("f5a5a608")
| |
+-----> TemplatesImpl <------+
|
+-------> byte[] (malicious class definition)
view raw jdk7u21-architecture.txt hosted with ❤ by GitHub
So far, so well known.
In recent Java runtimes, there are in total 3 fixes inside AnnotationInvocationHandler which break this gadget (see epilogue). But let's start with the first and most obvious bug. The code below is from AnnotationInvocationHandler in Java version 1.7.0_21:

private void readObject(ObjectInputStream paramObjectInputStream)
throws IOException, ClassNotFoundException
{
paramObjectInputStream.defaultReadObject();
AnnotationType localAnnotationType = null;
try
{
localAnnotationType = AnnotationType.getInstance(this.type);
}
catch (IllegalArgumentException localIllegalArgumentException)
{
return; // <=== Problem!!! this.type is not an annotation!
}
Map localMap = localAnnotationType.memberTypes();
for (Map.Entry localEntry : this.memberValues.entrySet())
{
String str = (String)localEntry.getKey();
Class localClass = (Class)localMap.get(str);
if (localClass != null)
{
Object localObject = localEntry.getValue();
if ((!localClass.isInstance(localObject)) && (!(localObject instanceof ExceptionProxy))) {
localEntry.setValue(new AnnotationTypeMismatchExceptionProxy(localObject.getClass() + "[" + localObject + "]").setMember((Method)localAnnotationType.members().get(str)));
}
}
}
}
view raw AnnoInvo-readObject-jdk7u21.txt hosted with ❤ by GitHub
There is a try/catch around an attempt to get the proxied annotation type. But the proxied interface javax.xml.transform.Templates is not an annotation. This constitutes a clear case of potentially dangerous input that would need to be dealt with. However, instead of throwing an exception there is only a return statement inside the catch-branch. Fortunately for the attacker, the instance of the class is already fit for purpose and does not need the rest of the readObject() method in order to be able to do its malicious work. So the "return" is problematic and would have to be replaced by a throw new Exception of some sort.
Let's check how this method looks like in Java runtime 1.7.0_80:

private void readObject(ObjectInputStream paramObjectInputStream)
throws IOException, ClassNotFoundException
{
paramObjectInputStream.defaultReadObject();
AnnotationType localAnnotationType = null;
try
{
localAnnotationType = AnnotationType.getInstance(this.type);
}
catch (IllegalArgumentException localIllegalArgumentException)
{
throw new InvalidObjectException("Non-annotation type in annotation serial stream");
}
Map localMap = localAnnotationType.memberTypes();
for (Map.Entry localEntry : this.memberValues.entrySet())
{
String str = (String)localEntry.getKey();
Class localClass = (Class)localMap.get(str);
if (localClass != null)
{
Object localObject = localEntry.getValue();
if ((!localClass.isInstance(localObject)) &amp;&amp; (!(localObject instanceof ExceptionProxy))) {
localEntry.setValue(new AnnotationTypeMismatchExceptionProxy(localObject.getClass() + "[" + localObject + "]").setMember((Method)localAnnotationType.members().get(str)));
}
}
}
}
view raw AnnoInvo-readObject-jdk7u80.txt hosted with ❤ by GitHub
Ok, so problem fixed? Well, yes and no. On the one hand, the use of the exception in the catch-clause will break the gadget which currently ships with ysoserial. On the other hand, this fix is a perfect example of the popular mistake I'm talking about. Wouter Coekaerts (see [3]) came up with an idea how to bypass such "fixes" and Alvaro Muñoz (see [4]) provided a gadget for JRE8u20 which utilizes this technique (in case you're wondering why there is no gadget for jdk1.7.0_80: 2 out of the total 3 fixes mentioned above are already incorporated into this version of the class. Even though it is possible to bypass fix number one, fix number two would definitely stop the attack).
Let's check out how this bypass works in detail.

A little theory

Let's recap what the Java (De-)Serialization does and what the readObject() method is good for. Let's take the example of java.util.HashMap. An instance of it contains data (key/value pairs) and structural information (something derived from the data) that allows logarithmic access times to your data. When serializing an instance of java.util.HashMap it would not be wise to fully serialize its internal representation. Instead it is completely sufficient to only serialize the data that is required to reconstruct its original state: Metadata (loadfactor, size, ...) followed by the key/value pairs as flat list. Let's have a look at the code:

private void readObject(java.io.ObjectInputStream s)
throws IOException, ClassNotFoundException {
// Read in the threshold (ignored), loadfactor, and any hidden stuff
s.defaultReadObject();
reinitialize();
if (loadFactor &lt;= 0 || Float.isNaN(loadFactor))
throw new InvalidObjectException("Illegal load factor: " +
loadFactor);
//
// ... some more stuff ...
//
// Read the keys and values, and put the mappings in the HashMap
for (int i = 0; i &lt; mappings; i++) {
@SuppressWarnings("unchecked")
K key = (K) s.readObject();
@SuppressWarnings("unchecked")
V value = (V) s.readObject();
putVal(hash(key), key, value, false, false);
}
}
view raw HashMap-readObject-excr.txt hosted with ❤ by GitHub
As you can see, the method starts with a call to defaultReadObject. After that, the instance attributes loadFactor and threshold are initialized and can be used. The key/value pairs are located at the end of the serialized stream. Since the key/value pairs are contained as an unstructured flat list in the stream calling putVal(key,value) basically restores the internal structure, what allows to efficiently use them later on.
In general, it is fair to assume that many readObject() methods look like this:

private void readObject (ObjectInputStream is)
throws IOException, ClassNotFoundException {
/*
* Read all non-transient stuff, e.g.
* non-redundant variables, primitive
* types etc.
*
*/
is.defaultReadObject ();
/*
* take care of transient
* attributes, redundant
* stuff, etc.
*
*/
...
// some custom code
}
view raw generic-readObject.txt hosted with ❤ by GitHub
Coming back to AnnotationInvocationHandler, we can see that its method readObject follows this pattern. Since the problem was located in the custom code section of the method, the fix was also applied there. In both versions, ObjectInputStream.defaultReadObject() is the first instruction. Now let's discuss why this is a problem and how the bypass works.

Handcrafted Gadgets

At work we frequently use ysoserial gadgets. I suppose that many readers are probably familiar with the ysoserial payloads and how these are created. A lot of Java reflection, a couple of fancy helper classes doing stuff like setting Fields, creating Proxy and Constructor instances. With "Handcrafted Gadgets" I meant gadgets of a different kind. Gadgets which cannot be created in the fashion ysoserial does (which is: create an instance of a Java object and serialize it). The gadgets I'm talking about are created by compiling a serialization stream manually, token by token. The result is something that can be deserialized but does not represent a legal Java class instance. If you would like to see an example, check out Alvaro's JRE8_20 gadget [4]. But let me not get ahead of myself, let's take a step back and focus on the problem I mentioned at the end of the last paragraph. The problem is that if the developer does not take care when fixing the readObject method, there might be a way to bypass that fix. The JRE8_20 gadget is an example of such a bypass. The original idea was, as already mentioned in the introduction, first described by Wouter Coekaerts [2]. It can be summarized as follows:

Idea

The fundamental insight is the fact that many classes are at least partly functional when the default attributes have been instantiated and propagated by the ObjectInputStream.defaultReadObject() method call. This is the case for AnnotationInvocationHandler (in older Java versions, more recent versions don't call this method anymore). The attacker does not need the readObject to successfully terminate, an object instance where the method ObjectInputStream.defaultReadObject() has executed is perfectly okay. However, it is definitely not okay from an attacker's perspective if readObject throws an exception, since, eventually this will break deserialization of the gadget completely. The second very important detail is the fact that if it is possible to suppress somehow the InvalidObjectException (to stick with the AnnotationInvocationHandler example) then it is possible to access the instance of AnnotationInvocationHandler later through references. During the deserialization process ObjectInputStream keeps a cache of various sorts of objects. When AnnotationInvocationHandler.readObject is called an instance of the object is available in that cache.
This brings the number of necessary steps to write the gadget down to two. Firstly, store the AnnotationInvocationHandler in the cache by somehow wrapping it such that the exception is suppressed. Secondly, build the original gadget, but replace the AnnotationInvocationHandler in it by a reference to the object located in the cache.

Now let's step through the detailed technical explanation.
  1. References
  2. The wrapper class: BeanContextSupport
  3. The cache

References

If one thinks about object serialization and the fact that you can nest objects recursively it is clear that something like references must exist. Think about the following construct:
class A implements Serializable
{
B b;
};
class B implements Serializable
{
C c;
}
class C implements Serializable
{
A a;
}
public static void main (String [] args)
{
A a = new A();
a.b = new B();
a.b.c = new C();
a.b.c.a = a;
new ObjectOutputStream (new FileOutputStream ("somefile")).writeObject (a);
}
view raw A-serializable.txt hosted with ❤ by GitHub
Here, the attribute a of class instance c points to an existing instance already serialized before and the serialized stream must reflect this somehow. When you look at a serialized binary stream you can immediately see the references: The hex representation usually looks like this: 
71 00 7E AB CD
where AB CD is a short value which represents the array index of the referenced object in the cache. You can easily spot references in the byte stream since hex 71 is "q" and hex 7E is "~":


The wrapper class: BeanContextSupport

Wouter Coekaerts found the class java.beans.beancontext.BeanContextSupport. At some point during deserialization it does the following:
public final void readChildren(ObjectInputStream ois) throws IOException, ClassNotFoundException {
int count = serializable;
while (count-- &gt; 0) {
Object child = null;
BeanContextSupport.BCSChild bscc = null;
try {
child = ois.readObject();
bscc = (BeanContextSupport.BCSChild)ois.readObject();
} catch (IOException ioe) {
continue;
} catch (ClassNotFoundException cnfe) {
continue;
}
}
}
view raw BeanContextSupport-readChildren.txt hosted with ❤ by GitHub
continue in the catch-branch, exactly what we need. So if we can build a serialized stream with an AnnotationInvocationHandler as first child of an instance of BeanContextSupport during deserialization we will end up in the catch (IOException ioe) branch and deserialization will continue.
Let's test this out. I will build a serialized stream with an illegal AnnotationInvocationHandler in it ("illegal" means that the type attribute is not an annotation) and we will see that the stream deserializes properly without throwing an exception. Here is what the structure of this stream will look like:
+-------------------------------------------+
| HashMap |
| |
| +----------------------+ |
| | BeanContextSupport | |
| | | |
| | children | |
| +-------+--------------+ |
| | |
| | |
| +---v-------------------------+ |
| | AnnotationInvocationHandler | |
| +-----------------------------+ |
| |
| +-------------------+ |
| | java.lang.String | |
| | | |
| | whatever | |
| +-------------------+ |
| |
+-------------------------------------------+
view raw diagram-1.txt hosted with ❤ by GitHub
Once done, the deserialized object is a HashMap with one key/value pair, key is an instance of BeanContextSupport, value is "whatever".
Click here to see the code on github.com
You need to build Alvaro's project [6] to get the jar file necessary for building this: 
kai@CodeVM:~/eworkspace/deser$ javac -cp /home/kai/JRE8u20_RCE_Gadget/target/JRE8Exploit-1.0-SNAPSHOT.jar BCSSerializationTest.java
kai@CodeVM:~/eworkspace/deser$ java -cp .:/home/kai/JRE8u20_RCE_Gadget/target/JRE8Exploit-1.0-SNAPSHOT.jar BCSSerializationTest > 4blogpost
Writing java.lang.Class at offset 1048
Done writing java.lang.Class at offset 1094
Writing java.util.HashMap at offset 1094
Done writing java.util.HashMap at offset 1172
Adjusting reference from: 6 to: 8
Adjusting reference from: 6 to: 8
Adjusting reference from: 8 to: 10
Adjusting reference from: 9 to: 11
Adjusting reference from: 6 to: 8
Adjusting reference from: 14 to: 16
Adjusting reference from: 14 to: 16
Adjusting reference from: 14 to: 16
Adjusting reference from: 14 to: 16
Adjusting reference from: 17 to: 19
Adjusting reference from: 17 to: 19
kai@CodeVM:~/eworkspace/deser$

A little program that deserializes the created file and prints out the resulting object shows us this: 
kai@CodeVM:~/eworkspace/deser$ java -cp ./bin de.cw.deser.Main deserialize 4blogpost
{java.beans.beancontext.BeanContextSupport@723279cf=whatever}

This concludes the first part, we successfully wrapped an instance of AnnotationInvocationHandler inside another class such that deserialization completes successfully.

The cache

Now we need to make that instance accessible. First we need to get hold of the cache. In order to do this, we need to debug. We set a breakpoint at the highlighted line in java.util.HashMap:
private void readObject(java.io.ObjectInputStream s)
throws IOException, ClassNotFoundException {
// Read in the threshold (ignored), loadfactor, and any hidden stuff
s.defaultReadObject();
reinitialize();
if (loadFactor &lt;= 0 || Float.isNaN(loadFactor))
throw new InvalidObjectException("Illegal load factor: " +
loadFactor);
//
// ... some more stuff ...
//
// Read the keys and values, and put the mappings in the HashMap
for (int i = 0; i &lt; mappings; i++) {
@SuppressWarnings("unchecked")
K key = (K) s.readObject();
@SuppressWarnings("unchecked") // !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
V value = (V) s.readObject(); // <== B R E A K P O I N T H E R E !!!!!
putVal(hash(key), key, value, false, false);
}
}
view raw HashMap-readObject-excr2.txt hosted with ❤ by GitHub
Then start the deserializer program and step into readObject:


When we open it we can see that number 24 is what we were looking for.


Here is one more interesting thing: If you deserialize with an older patch level of the Java Runtime, the object is initialized as can be seen in the sceenshot below:


If you use a more recent patch level like Java 1.7.0_151 you will see that the attributes memberValues and type are null. This is the effect of the third improvement in the class I've been talking about before. More recent versions don't call defaultReadObject at all, anymore. Instead, they first check if type is an annotation type and only after that they populate the default fields.
Let's do one more little exercise. In the program above in line 150, change 

        TC_STRING,
        "whatever",

to 
        TC_REFERENCE,
        baseWireHandle + 24,

and run the program again: 
kai@CodeVM:~/eworkspace/deser$ java -cp ./bin de.cw.deser.Main deserialize 4blogpost2
{java.beans.beancontext.BeanContextSupport@723279cf=sun.reflect.annotation.AnnotationInvocationHandler@10f87f48}

As you can see, the entry in the handles table can easily be referenced.
Now we'll leave the Jdk7u21 gadget and AnnotationInvocationHandler and build a gadget for groovy 2.4.5 using the techniques outlined above.

A deserialization gadget for groovy-2.4.5

Based on an idea of Sam Thomas (see [2]).
The original gadget for version 2.3.9 looks like this:
AnnotationInvocationHandler
+
|
+--> Proxy(java.util.Map)
+
|
+---> ConvertedClosure
+
|
+---> MethodClosure
view raw groovy-2.3.9-gadget.txt hosted with ❤ by GitHub
Trigger is readObject of our beloved AnnotationInvocationHandler, it will call entrySet of the memberValues hash map, which is a proxy class with invocation handler of type org.codehaus.groovy.runtime.ConvertedClosure. Now every invocation of ConvertedClosure will be delegated to doCall of the nested instance of MethodClosure which is a wrapper of the call to the groovy function execute. The OS command that will be executed is provided as member attribute to MethodClosure.
After the original gadget for version 2.3.9 showed up MethodClosure was fixed by adding a method readResolve to the class org.codehaus.groovy.runtime.MethodClosure:
private Object readResolve()
{
if (ALLOW_RESOLVE) {
return this;
}
throw new UnsupportedOperationException();
}
view raw MethodeClosure-readResolve.txt hosted with ❤ by GitHub
If the global constant ALLOW_RESOLVE is not set to true an UnsupportedOperationException is supposed to break the deserialization. Basically, this means that an instance of MethodClosure cannot be deserialized anymore unless one explicitely enables it. Let's quickly analyze MethodClosure: The class does not have a readObject method and readResolve is called after the default built-in deserialization. So when readResolve throws the exception the situation is almost identical to the one explained in the above paragraphs: An instance of MethodClosure is already in the handle table. But there is one important difference: AnnotationInvocationHandler throws an InvalidObjectException which is a child of IOException whereas readResolve throws an UnsupportedOperationException, which is a child of RuntimeException. BeanContextSupport, however, only catches IOException and ClassCastException. So the identical approach as outlined above would not work: The exception would not be caught. Fortunately, in late 2016 Sam Thomas found the class sun.security.krb5.KRBError which in its readObject method transforms every type of exception into IOException:
private void readObject(ObjectInputStream paramObjectInputStream)
throws IOException, ClassNotFoundException
{
try
{
init(new DerValue((byte[])paramObjectInputStream.readObject()));
parseEData(this.eData);
}
catch (Exception localException)
{
throw new IOException(localException);
}
}
view raw KRBError-readObject.txt hosted with ❤ by GitHub
This means if we put KRBError in between BeanContextSupport and MethodClosure the UnsupportedOperationException will be translated into IOException which is ultimately caught inside the readChildren method of BeanContextSupport. So our wrapper construct looks like this:
+-----------------------+
| BeanContextSupport |
| |
| children |
+------+----------------+
|
|
+--v---------------------+
| KRBError |
+-----+------------------+
|
|
+--v--------------------+
| MethodClosure |
+-----------------------+
view raw wrapper-construct.txt hosted with ❤ by GitHub
Some readers might be confused by the fact that you can nest an object of type MethodClosure inside a KRBError. Looking at the code and interface of the latter, there is no indication that this is possible. But it is important to keep in mind that what we are concerned with here are not Java objects! We are dealing with a byte stream that is deserialized. If you look again at the readObject method of KRBError you can see that this class calls ObjectInputStream.readObject() right away. So here, every serialized Java object will do fine. Only the cast to byte array will throw a ClassCastException, but remember: An exception will be thrown already before that and this is perfectly fine with the design of our exploit.
Now it is time to put the pieces together. The complete exploit consists of a hash map with one key/value pair, the BeanContextSupport is the key, the groovy gadget is the value. [1] suggests putting the BeanContextSupport inside the AnnotationInvocationHandler but it has certain advantages for debugging to use the hash map. Final structure looks like this:
HashMap
+
|
+----> BeanContextSupport
| +
| |
| +--> KRBError
| +
| |
| +---> MethodClosure <----------------------+
| |
| |
+----> AnnotationInvocationHandler |
+ |
| |
+--> Proxy(java.util.Map) |
+ |
| |
| |
+---> ConvertedClosure |
+ |
| |
+---> Reference (MethodClosure) +--+
view raw gadget-structure.txt hosted with ❤ by GitHub
The final exploit can be found on github.com.

Epilogue

I had mentioned 3 improvements in AnnotationInvocationHandler but I only provided one code snippet. For the sake of completeness, here are the two:
The second fix in jdk1.7.0_80 which already breaks the jdk gadget is a check in equalsImpl:
private Boolean equalsImpl(Object paramObject)
{
if (paramObject == this) {
return Boolean.valueOf(true);
}
if (!this.type.isInstance(paramObject)) {
return Boolean.valueOf(false);
}
for (Method localMethod : getMemberMethods()) // -------------------------------------+
{ |
... // somewhere here TemplatesImpl.newTransformer is supposed to be invoked ... |
} |
} |
|
private Method[] getMemberMethods() <----------------------------------------------------+
{
if (this.memberMethods == null) {
this.memberMethods = ((Method[])AccessController.doPrivileged(new PrivilegedAction()
{
public Method[] run()
{
Method[] arrayOfMethod = AnnotationInvocationHandler.this.type.getDeclaredMethods();
AnnotationInvocationHandler.this.validateAnnotationMethods(arrayOfMethod); // <== will check if method really an annotation method
AccessibleObject.setAccessible(arrayOfMethod, true);
return arrayOfMethod;
}
}));
}
return this.memberMethods;
}
view raw AnnoInvo-jdk7u80.txt hosted with ❤ by GitHub
The highlighted check will filter out the methods getOutputProperties and newTransformer of TemplatesImpl because they are not considered annotation methods, and getMemberMethods returns an empty array so the methods of TemplatesImpl are never called and nothing happens. The third fix which you can find for example in version 1.7.0_151 finally fixes readObject:
private void readObject(ObjectInputStream paramObjectInputStream)
throws IOException, ClassNotFoundException
{
ObjectInputStream.GetField localGetField = paramObjectInputStream.readFields();
Class localClass1 = (Class)localGetField.get("type", null);
Map localMap1 = (Map)localGetField.get("memberValues", null);
AnnotationType localAnnotationType = null;
try
{
localAnnotationType = AnnotationType.getInstance(localClass1);
}
catch (IllegalArgumentException localIllegalArgumentException)
{
throw new InvalidObjectException("Non-annotation type in annotation serial stream");
}
Map localMap2 = localAnnotationType.memberTypes();
LinkedHashMap localLinkedHashMap = new LinkedHashMap();
for (Map.Entry localEntry : localMap1.entrySet())
{
String str = (String)localEntry.getKey();
Object localObject = null;
Class localClass2 = (Class)localMap2.get(str);
if (localClass2 != null)
{
localObject = localEntry.getValue();
if ((!localClass2.isInstance(localObject)) &amp;&amp; (!(localObject instanceof ExceptionProxy))) {
localObject = new AnnotationTypeMismatchExceptionProxy(localObject.getClass() + "[" + localObject + "]").setMember(
(Method)localAnnotationType.members().get(str));
}
}
localLinkedHashMap.put(str, localObject);
}
UnsafeAccessor.setType(this, localClass1);
UnsafeAccessor.setMemberValues(this, localLinkedHashMap);
}
view raw AnnoInvo-jdk7u151.txt hosted with ❤ by GitHub
As one can see, only the 2 last calls actually set the member attributes type and memberValues. defeaultReadObject is not used at all. Before, the type check for the annotation class is performed. If it fails, an InvalidObjectException is thrown and type and memberValues remain null.

References

  1. https://www.thezdi.com/blog/2017/12/19/apache-groovy-deserialization-a-cunning-exploit-chain-to-bypass-a-patch
  2. http://www.zerodayinitiative.com/advisories/ZDI-17-044/
  3. http://wouter.coekaerts.be/2015/annotationinvocationhandler
  4. https://github.com/pwntester/JRE8u20_RCE_Gadget/blob/master/src/main/java/ExploitGenerator.java
  5. https://gist.github.com/frohoff/24af7913611f8406eaf3
  6. https://github.com/frohoff/ysoserial/blob/master/src/main/java/ysoserial/payloads/Groovy1.java
Posted by at