Personal tools
You are here: Home Members martin Martins Weblog Reflection from Scala - Heaven and Hell
Document Actions
  • Print this page
  • Add Bookmarklet

Reflection from Scala - Heaven and Hell

Bringing some dynamic pieces to the statically typed language. Watch your steps...

For some project I need a way to compile some piece of code at runtime, load and execute it, and later forget it. The use case is somewhere in between up-front compilation and interpretation.

Compilation

Compilation is not that difficult. Put the scala-compiler.jar on the classpath, fill the Settings data structure, take the strangely named "Global" and run the compiler, which will think a little bit about your code and put a lot of nice class files in an output directory.

import java.io.File
import scala.tools.nsc._
import scala.tools.nsc.reporters._

// ...

def compile(srcDir: File, classesDir: File, sources: File*) {
val settings = new Settings()
settings.sourcepath.value = srcDir.getPath
settings.outdir.value = classesDir.getPath
object compiler extends Global(settings)
val run = new compiler.Run
run.compileFiles((sources map io.AbstractFile.getFile).toList)
if (compiler.reporter.hasErrors) {
throw new RuntimeException("Compilation failed")
}
}

For smaller scripting stuff or tighter binding between script and compiled code you could probably better use the Scala Interpreter classes (see for example this tutorial [German]).

Using the Scripts

Now for the slightly more interesting stuff: How to execute the brand-new classes compiled from the "script" sources. I use a template pattern here to do the compilation, execute some user supplied function with the newly created classes, and clean up afterwards.

Here's how this should be used.

val baz = ...
val scriptSourceDir = ...

val fooBarResult = withScripts(scriptSourceDir) { cl =>
implicit val classLoader = cl

val foo: Foo = New("MyFoo")("constructor arg1", "constructor arg2")
foo.bar(baz)

 }

That is, withScripts provides a ClassLoader that includes the compiled scripts. This is used by the function New, which I will discuss below. Pretty cool, isn't it?

And that's the implementation.

def withScripts[R](srcDir: File)(op: ClassLoader=>R): R = withScripts(srcDir, None)(op)

def withScripts[R](srcDir: File, parentClassLoader: ClassLoader)(op: ClassLoader=>R): R =
withScripts(srcDir, Some(parentClassLoader))(op)

private def withScripts[R](srcDir: File, parentClassLoader: Option[ClassLoader])(op: ClassLoader=>R): R = {
var tmpDir = File.createTempFile("ssi", ".tmp")
tmpDir.delete()
if (! tmpDir.mkdirs()) {
println("Can't mkdirs on " + tmpDir)
}
try {
// TODO: probably must pass the parentClassLoader to compile
compile(srcDir, tmpDir)
val result: R = {
val classLoader = parentClassLoader match {
case None => new URLClassLoader(Array(tmpDir.toURL))
case Some(parent) => new URLClassLoader(Array(tmpDir.toURL), parent)
}
op(classLoader)
}
result
} finally {
val notDeleted = (tmpDir.listFiles() ++ Array(tmpDir)) filter {!_.delete()}
notDeleted foreach {f => println("Warning: " + f + " could not be deleted")}
}
}

I wanted to shield the users of this mini-scripting-framework from to much contact with reflection. Let's see how it works. I proudly present the implemenation of the New function used in the example above.

Reflection

First of all, when using reflection, it is quite natural to convert from a String containing a class name to a Class. Here's an implicit for it. Easy, not really necessary, but... anyway:

  implicit def string2Class[T<:AnyRef](name: String)(implicit classLoader: ClassLoader): Class[T] = {
val clazz = Class.forName(name, true, classLoader)
clazz.asInstanceOf[Class[T]]
}

Note the implicit parameter list where you can provide the class loader either implicitly, or explicitly when have to deal with different class loaders.

That was the heaven part. Now let's go to hell :-)

To create a new object from a class, you use a constructor. These can take parameters. Life could be easy, if everything was an object. Unfortunately there are a few exceptions. 8 or 9, if I count correctly: primitives. And, to make things more complex, there is autoboxing. Finally think about the slightly ill-behaved case of a class with constructors, that behave differently when called with a primitive int and when called with the int boxed to an Integer.

OK. Ground settled. Let's go into the gory details. The New function takes the class name, the constructor parameters in special packaging (explained soon) and a class loader, again possibly implicit. It converts the class name to the class, matches the actual parameters with the declared parameter types of the constructors of the class. If there is only one matching constructor, it is invoked with the actual parameters.

  def New[T<:AnyRef](className: String)(args: WithType*)(implicit classLoader: ClassLoader): T  = {
val clazz: Class[T] = className
val argTypes = args map { _.clazz } toArray
val candidates = clazz.getConstructors filter { cons => matchingTypes(cons.getParameterTypes, argTypes)}
require(candidates.length == 1, "Argument runtime types must select exactly one constructor")
val params = args map { _.value }
candidates.first.newInstance(params: _*).asInstanceOf[T]
}

private def matchingTypes(declared: Array[Class[_]], actual: Array[Class[_]]): Boolean = {
declared.length == actual.length && (
(declared zip actual) forall {
case (declared, actual) => declared.isAssignableFrom(actual)
})
}

So what is WithType, and why is it needed? Each object has getClass, hasn't it?

Yes, but not everything is an object, as described above. Once autoboxing has objectified a primitive, the information is lost, whether it was a primitive or a primitive wrapper. Therefore the original type, for example Integer.TYPE or Integer.class has to be provided to the New function. Here's the definition of WithType:

  sealed abstract class WithType {
val clazz : Class[_]
val value : AnyRef
}

case class ValWithType(anyVal: AnyVal, clazz: Class[_]) extends WithType {
lazy val value = toAnyRef(anyVal)
}

case class RefWithType(anyRef: AnyRef, clazz: Class[_]) extends WithType {
val value = anyRef
}

But I promised that it should be easy for the user of the framework. This stuff seems overly complex! And what's that toAnyRef?

The inconvenience of constructing a WithType can be reduced with implicits (you see, I have a new hammer to put in those screws):

  implicit def refWithType[T<:AnyRef](x:T) = RefWithType(x, x.getClass)
implicit def valWithType[T<:AnyVal](x:T) = ValWithType(x, getType(x))

Now the function New can be called with primitives and with objects or a mix of boths and the compiler will add the type information. And after implementing this, I have found a use for the distinction of AnyRef and AnyVal in the Scala type hierarchy. Nice stuff. Heavenly.

And back to hell: What about toAnyRef and getType? I haven't found standard library functions for boxing an arbitrary AnyVal and AnyVal currently does not support getClass. That means, I had to implement these functions myself.

def getType(x: AnyVal): Class[_] = x match {
case _: Byte => java.lang.Byte.TYPE
case _: Short => java.lang.Short.TYPE
case _: Int => java.lang.Integer.TYPE
case _: Long => java.lang.Long.TYPE
case _: Float => java.lang.Float.TYPE
case _: Double => java.lang.Double.TYPE
case _: Char => java.lang.Character.TYPE
case _: Boolean => java.lang.Boolean.TYPE
case _: Unit => java.lang.Void.TYPE
}

 def toAnyRef(x: AnyVal): AnyRef = x match {
case x: Byte => Byte.box(x)
case x: Short => Short.box(x)
case x: Int => Int.box(x)
case x: Long => Long.box(x)
case x: Float => Float.box(x)
case x: Double => Double.box(x)
case x: Char => Char.box(x)
case x: Boolean => Boolean.box(x)
case x: Unit => ()
}

Probably every Java programmer who had to do some reflection / compiler stuff has to implement this every now and then... Fortunately the Scala syntax is concise. Are there copy and paste errors? Forgotten primitive types? I hope not.

Conclusion

The result of having a simple to use way of bringing scripts into a running JVM and using them are satisfactory for me. A lot of Scala features have proven really useful for this purpose. The standard library is either not accessible enough for beginners like me or it is a little bit lacking in some areas. But that doesn't put me off, the implemenation of this code didn't take too long.

I'd like to thank paulp and DRMacIver for support on the #scala IRC channel. They gave me some directions where to look for possible solutions.

Maybe I attach the source code later, but this won't be today.

_____
tags:
Thursday, January 29, 2009 in Programming Languages  | Permalink |  Comments (4)

Array types

Posted by Anonymous User at 2009-02-05 07:14
I don't know if it applies in this case, but when doing reflection, you usually need to handle array types differently as well (especially primitive array types such as int[]). For that:
if (clazz.isArray()) {
if (clazz.getComponentType().isPrimitive()) {
... //special handling of primitive array types based upon component type
}
}

Hope that helps.
-Chris Hansen

Array types

Posted by Martin Kneißl at 2009-02-09 18:15
Chris, thanks for your comment.
To make it short: The code works for some array cases, but not for all, mainly because of array autoboxing in Scala, which sometimes escape my tricks with implicits to capture the static type of the argument. You end up with an BoxedAnyArray and lose the information whether the element type is primitive or boxed. I have extended the New function to support the cases where the boxed array uniquely selects a constructor method, but the code is not really cleaned up enough to be shown in public :-) .
To fix these or any other case where there are ambiguous I would have to introduce a precedence on argument list types, which I might do some day. Until then client code will have to explicitly specify the constructor to use. The New function is not useful in these cases.

now in trunk

Posted by Anonymous User at 2009-06-19 17:03
Hey Martin - I just checked some similar code into trunk as scala.reflect.Invocation. It might still need some tweaking but the syntax lets us say stuff like

obj o 'method(arg1, arg2, arg3)

by virtue of the strategic use of implicits. I didn't credit you in the source since I'm not big on overdoing author lists and since this is about how I would have written it anyway, but if that troubles you at all let me know and I'll be happy to add your name. (This is "paulp" from #scala.)

Looking forward to using 2.8

Posted by Martin Kneißl at 2009-06-23 11:02
Great news, Paul. I am really looking forward to using 2.8, soon. I'm just too lazy to play around with snapshots...

Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: