Use extensions – but beware the pitfalls of optional packages in Java class loading

The Java Extension Mechanism is described in the Java Tutorial as a "standard, scalable way to make custom APIs available to all applications running on the Java platform." As described in Understanding Extension Class Loading, "the extension framework makes use of the class-loading delegation mechanism" with extension classes loaded after the bootstrap classes in rt.jar (and related JARs) but before the classes loaded from the typical classpath.

 The extension directory works a bit like the classpath in that its part of the class loading mechanism and classes available within JARs in the extension directory are made available to Java applications. There are some key, differences, however, and some of these are highlighted next.

Characteristic	Classpath	Extension Mechanism (Optional Packages)
Scope	Typically Application-Specific -classpath/-cp java.class.path Executable JAR Manifest's Class-Path Potentially All JREs on Host CLASSPATH environment variable	All JVMs Running in Specific JRE java.ext.dirs All Host's JREs Solaris: /usr/jdk/packages/lib/ext Linux: /usr/java/packages/lib/ext Windows:%SystemRoot%\Sun\Java\lib\ext
How Specified	.jar Files Explicitly specified by name (including .jar) Wildcard (*) matching all all JAR files with .jarextensions .class Files Directory containing.class files specified	All JAR files (even if extension other than .jar or no extension at all) in designated directories are loaded.
Class Loading Order	After bootstrap and extensions loading.	After bootstrap but before classpath.

One of the most significant observations worth some more emphasis is that the extension mechanism will pick up all JAR format files in the extension directory even if the file does not have a .jar extension. The implication of this is that while one can change the name of a JAR located in a classpath directory to have an extension other than .jar so that the wildcard does not pick it up, this technique will not work with the extension directory.
I'm going to use some simple examples in this post to demonstrate some of these differences. The next two code listings are for a very simple HelloWorld class and a main application class called Main that uses theHelloWorld class.

Listing 1. HelloWorld.java

public class HelloWorld  
{  
   @Override  
   public String toString()  
   {  
      return "Hello, World!";  
   }  
}

Listing 2. Main.java

import static java.lang.System.out;  
  
public class Main  
{  
   public static void main(final String[] arguments)  
   {  
      out.println(new HelloWorld());  
   }  
}  

To demonstrate a primary difference between classpath and the extension mechanism (optional packages), I will archive the compiled HelloWorld.class file into a JAR called HelloWorld.jar and place it in a different directory than the compiled Main.class file. To demonstrate the use of the traditional classpath, I place the HelloWorld.jar file in a directory calledC:\hello and will access that JAR via wildcard (*) for Main to use. This is demonstrated in the next two screen snapshots.

The two previous images demonstrate that the Java Main application can still load the HelloWorld.classfile even though I had deleted it from the current directory because the Java launcher was explicitly told (via the -classpath option) to look for it in C:\hello. Using the extensions mechanism (optional packages), it is possible to have the class loaded without it being in the same directory and without explicit classpath specification. This is shown in the next screen snapshot.

The previous screen snapshot demonstrates that the Java launcher doesn't even need the HelloWorld.class in the same directory or specified on its classpath when that class is inside a JAR that is in the extensions (optional packages) directory. This is often cited as a benefit of using the extensions mechanism because all applications using that JRE (or potentially all applications on the host) can see the same classes without need to explicitly specify them on the classpath.

With the traditional classpath approach of instructing an application to load classes from JARs, the JAR file containing the .class file needs to end with the .jar extension. The next screen snapshot demonstrates what happens when the HelloWorld.jar is renamed HelloWorld.backup in the same classpath-referenced directory.

The last image demonstrates that a NoClassDefFoundError is encountered when the JAR file in the classpath-referenced directory does not have a .jar extension. Perhaps a bit surprisingly, the extensions (optional packages) mechanism does not work the same way. Instead, all JAR files in the extensions specified directory are loaded regardless of their extension and regardless of even if they have a file extension. This is demonstrated in the next screen image.

The previous image demonstrates that renaming the JAR file that resides within the extensions directory to not have any file extension whatsoever does not prevent the classloader from loading the classes of that JAR. In other words, the classloading mechanism loads all JAR files in the specified extensions directory based on file type rather than on file name or extension. As the Optional Packages Overview summarizes, "There is nothing special about any particular JAR file itself or the classes it contains that makes it an installed optional package. It is an installed optional package by virtue of its location in jre/lib/ext."
There are some risks and downsides associated with placing too many class definitions in JARs inside the extensions directory. It can be maddening to wonder why NoSuchMethodErrors, for example, are occurring when one can see that a class specified explicitly on the classpath has the method in question. I have previously written about one of the many potential causes of NoSuchMethodError, but forgotten outdated and obsolete class definitions residing inside of JAR files in the extensions directory are another potential cause. This is demonstrated next.
The next two code listings show revised versions of Main.java and HelloWorld.java. In particular,HelloWorld has an all-new method that the new version of Main invokes. In this case, I'm going to leave the newly compiled HelloWorld.class file in the same directory when I run the Main to demonstrate that the old, busted version of HelloWorld.class in the JAR in the extensions directory takes precedence over the new hotness HelloWorld.class in the current directory.

Listing 3. Revised Hello World.java (New Method)

public class HelloWorld  
{  
   @Override  
   public String toString()  
   {  
      return "Hello, World!";  
   }  
  
   public String directedHello(final String name)  
   {  
      return "Hello, " + name;  
   }  
}  

Listing 4. Revised Main.java

import static java.lang.System.out;  
  
public class Main  
{  
   public static void main(final String[] arguments)  
   {  
      final HelloWorld helloWorld = new HelloWorld();  
      out.println(helloWorld);  
      out.println(helloWorld.directedHello("Dustin"));  
   }  
} 

The last image demonstrates that the now obsolete class definition of HelloWorld in the extensions directory takes precedence over the new class definition of HelloWorld in the same directory. Even when I specify the current directory on the classpath, the old version in the extensions directory takes precedence. This is shown in the next screen snapshot, which also shows that the JAR in the extensions directory that is "hiding" the newer JAR and its class's newer method is still not even named with a .jar extension.

The example just demonstrated is not even the most difficult situation a forgotten JAR in the specified extensions directory (or directories) can cause. In that example, at least I had a NoSuchMethodError to alert me to a problem. A potentially even more difficult situation to debug can exist when the old class definition has the same method signature but has an outdated method implementation. In such cases, there may be no error, exception, or throwable of any kind, but the application logic will simply not work correctly or as expected. The old functionality could exist in the code base for some time before it's even recognized as an issue, especially if unit tests and other testing is lacking.
Use of the extensions directory can make things easier on developers because classes in JAR files residing in the extensions directory (or directories) are available to all applications in the JRE associated with the extensions directory (or with all JREs on the host if the operating system-based host-wide extensions directory is used). However, there are definite risks associated with too liberal use of the directory. It can be easy to forget that outdated class definitions residing in JARs in that directory are preventing classloaders from loading the newer and seemingly obvious versions of the class definitions. When this happens, the very extensions (optional packages) mechanism that made developers' lives easier now make it more difficult.
Elliotte Rusty Harold provides a warning about use of the extensions (optional packages) mechanism, "While this seems convenient, it is also a long-term mistake... Sooner or later (probably sooner), you'll load the wrong version of a class from a place you aren't even thinking about and waste hours debugging." The Java Tutorial also recommends caution (I added the emphasis), "Since this mechanism extends the platform's core API, its use should be judiciously applied. Most commonly it is used for well standardized interfaces such as those defined by the Java Community Process, although it may also be appropriate for site wide interfaces."
Although the extensions (optional packages) mechanism is similar to the classpath mechanism and both are used as part of class loading, the differences between the two are important to note. In particular, it is important to remember that all JAR files (even if they don't have .jar file extensions) that reside in the directory referenced as an extension directory will be loaded. Renaming these JARs and even changing their file extension will not be sufficient to have the classloading ignore them. With classpath, on the other hand, renaming the JAR is sufficient to prevent loading when the classpath specifies individual JAR files explicitly and changing the file extension is typically sufficient to prevent loading even when the classpath uses the wildcard (*) to specify all JARs in a directory.
There are situations when the extensions (optional packages) mechanism is the appropriate choice, but these seem fairly rare. It is also important to keep the extensions (optional packages) mechanism in mind when dealing with unexplained NoSuchMethodErrors so that one can check it out to see if the offender lives in that directory or directories.

This story, "Java extension mechanism loads all JARs" was originally published by marxsoftware.blogspot.com.