Updating Functions

In computer science often updates to bindings / environments are described by updating substitution functions at specific points. This blog post shows a way to express such updates in Scala.

Yesterday dinesh asked in #scala on IRC whether Scala supported updateable functions

(11:46:16) dinesh_: well for example, if you have f(x) = x, and then apply the function update f[5 := 10]
(11:46:50) dinesh_: then f[5 := 10](x) = x, except f[5 := 10](5) = 10

In the following discussion assume f to be defined as

def f(x: Int) = x

The obvious built in way to express the function update in Scala would be to use PartialFunction.orElse. One would like to write

{ case 5 => 10 } orElse f

Unfortunately the type inferencer will not derive the function argument type in the partial function literal and PartialFunction.orElse requires its argument to be a PartialFunction, not a "normal" Function1. So you had to write instead

({ case 5 => 10 }: PartialFunction[Int,Int]) orElse ({ case x => f(x) }: PartialFunction[Int,Int])

which is not that elegant.

Implicits and the pimp-my-library pattern comes to the rescue. Define

object FunctionUpdate {
  implicit def function2RichFunction[A,B](f: Function1[A,B]) = new RichFunction(f)
  class RichFunction[A, B](f: Function1[A,B]) {
    def updated(pf: PartialFunction[A, B]): Function1[A, B] =
      pf orElse { case x => f(x) }
  }
}

And now you can write

import FunctionUpdate.function2RichFunction

def f(x: Int) = x
assert(f(1) == 1)
assert(f(5) == 5)

val g = f _ updated { case 5 => 10 }
assert(g(1) == 1)
assert(g(5) == 10)

Why Google Wave might wash away privacy

Google Wave might become another major threat to your privacy due to its design. Make an informed and conscious decision on how you want to use it.

A short disclaimer upfront: I must confess that I have very limited knowledge of the wave protocol besides that from the Wave keynote. If there are technical errors in this article please feel free to report them in comments, such that I can post an updated article in the future.

What is Google Wave?

Google Wave is a promising new Web application and protocol trying to improve online communication and collaborative activities. It allows for multiple users to cooperatively change documents called "Waves". A wave is composed of "Wavelets" that are the unit of access control.

The documents are stored online on servers. There isn't one single central server for all collaborating wave users, but each Wave user will have to log in to their wave server to send and receive updates to their waves. When users of different wave servers ride on the same Wave, updates to the Wave are distributed to any of the participating Wave servers, such that all users which are assigned to a particular Wavelet have an eventually consistent view of the document (or the Wavelets they are assigned to).

The distribution of the changes is immediate. The Google Wave keynote presentation focuses largely on the speed of update distribution (and this is very impressive).

How does Google Wave affect my privacy?

In principle, Wave has a lot in common with the way e-mail and XMPP works. Communication is not user to user but server to server. That is, the (e-mail, XMPP, Wave) server has access to the information of its users.

My major privacy concerns with Google Wave are twofold. Firstly, even though the protocol is open, how many independent Wave servers will there be? Fortunately there are already people working on easy installation of alternative Wave servers. But, given the success of GMail, I don't take it for granted that reasonably small groups will have their own Wave servers. On the contrary, I guess most Wave users will be on Google's servers.

Whenever you participate in a Wave that has at least one Google user on it, you will give the information in that Wave to Google. Is this a problem, now? I'm not sure. I still think Google is not evil. The problem lies in the future. Google will keep some of the information you give them forever, willingly or not, for example on backups.

When Google merges with some other company with less philanthropic mission, they might start to change privacy policies. When Google is hacked, the information will spread. When a Google account on your Waves is hacked (which is rather likely), your information leaks, too. When governmental institutions approach Google, they might disclose your information. When data protection laws are weakened (there is such a trend in Germany and in the USA after 9/11) Google will have to give your information away.

You don't know which of your information will mean what to criminals, the police, a future employer, your insurance company.

Of course in the above, "Google" could be any large Wave provider.

The second major concern is the quality of information given away. The small, fast incremental updates and the replay facility mean, that all your typing errors, temporary wordings, etc. are visible to any of the users on the wave and all of their servers.
If you think you shouldn't worry about that and are a software developer, think of Wave as your source code repository. I don't want to show all of my stupid mistakes to all co-developers immediately. I'd rather clean up the mess before I commit. And I don't want suggestions like "People coding this way should read the GoF book immediately" from some marketing bot. But something like that is likely to happen: Why should Google promote Wave other than to make money of it? How could they make money from Wave? By using your information. Whether it is for direct marketing or in a more subtle way, you will have to pay for using Wave in one way or the other.

The functionality that has been shown in the Wave keynote is impressive. I'd like to use that functionality, better today than tomorrow. But the privacy issues should at least be discussed openly before committing fully to that service. Users should be able to effectively control which information they give away, and their choice should be consciously. Until there is a model to control your Waves you should always assume you are on camera. For those in my age, just google yourselves to check if Google represents well what you think you have been years ago.

Further reading: "Googling Security" by Greg Conti.

Interview "Cold Calls" - "Unerwünschte Telefonwerbung"

Die Regionalnachrichten "17:30" von SAT.1 NRW haben mich heute zu unerwünschter Telefonwerbung interviewt, erfragt, was man tun kann, und ob ich glaube, dass die aktuelle Gesetzesänderung wirken wird.

Heute morgen bekam Ute einen Anruf, man wolle mich zum Thema Telefonwerbung interviewen. Ich war einigermaßen überrascht, da ich mich nicht für einen Experten auf diesem Gebiet halte. Heute mittag bekam ich dann Besuch von Carsten Isherwood von Westcom und seinem Team, die das Interview mit mir für SAT.1 NRW aufgezeichnet haben.

Vor zwei Jahren war das Aufkommen an Telefonwerbung bei uns so hoch und so unerträglich, dass ich eine Linkliste mit Anlaufpunkten veröffentlicht habe, an die sich von Werbeanrufen genervte Opfer wenden können. Das hat wohl schließlich dazu geführt, dass Westcom sich an mich gewandt hat. Ich habe die Liste nochmal bereinigt, so dass (Stand jetzt) alle Links funktionieren.

Gegen Firmen, die unwerwünschte Telefonwerbung betreiben, kann seit heute eine Geldbuße von bis zu 50.000 € pro Einzelfall verhängt werden. Ich verspreche mir von dieser Gesetzesänderung eine abschreckende Wirkung, allerdings nur dann, wenn genügend Betroffene die Möglichkeit zur Beschwerde nutzen, z. B. bei der Bundesnetzagentur. Dort gibt es ein Formular zur Meldung von unerlaubter Telefonwerbung (PDF). Je mehr Leute dieses Formular nutzen, um die richtigen Informationen von den Telemarketingfirmen zu bekommen und sich beschweren, desto eher könnte die Unsitte der Telefonwerbung eingedämmt werden.

Telefonwerbung ohne Anzeige der anrufenden Rufnummer wird separat mit einer Geldbuße von bis zu 10.000 € belegt. Hier wird die Ermittlung der Verursacher natürlich relativ schwierig werden...

Das neue Gesetz ist auf jeden Fall ein Schritt in die richtige Richtung. Allerdings steht zu befürchten, dass zu wenige Geldbußen verhängt werden können, um die Telefonwerbung unwirtschaftlich zu machen, oder die Call-Center / Werbefirmen ins Ausland verlegt werden. Ich bin gespannt.

Ebenfalls gespannt bin ich, wie der Beitrag geschnitten wird. Leider kann ich die Sendung nicht live verfolgen, sondern muss auf die Veröffentlichung im Internet warten.

Update:

Das Video ist nun online. Von den 15 Minuten aufgenommenen Materials sind leider nur ein paar Sekunden übrig geblieben.

Fast Forward to Scala

A quick introduction to Scala for Java programmers I have presented recently.

Over the last months I have been asked several times what this wonderful programming language is that I'm so excited about, and why one should care. I decided to prepare some slides and gave a talk to a mixed group, mostly developers, to show why I think Scala is relevant for Java programmers.

Although the slides need my voice and hand-weaving to get the point across I have uploaded the slides to SlideShare. Have a look and some fun!

Fast Forward To Scala

Unfortunately SlideShare happily removed the animations and destroyed part of the layout...

A Scala Wrapper for Apache Velocity

Groovy has string interpolation in two forms: GStrings and Groovy Templates. Scala has -- nothing in that area, unless you count String concatenation or printf-like format strings. But the Java world has template engines, here is how to use one with additional comfort.

When a program has to generate text where most text is fixed and only some portions have to be dynamic, using templates is a natural approach. Groovy has GStrings and Groovy Templates, the Unix shells have "here documents", other scripting languages have different approaches to string interpolation.

The Scala library is still -- how shall I put it -- lean in that area and has no special support for text generation. But of course you can leverage Java libraries for that. So let's go and show the Groovy people that Scala has no problem generating some text!

I have used Apache Velocity with some success before, that's why I choose Velocity as the template engine. But I'd hate to have to create a resource file for each template in my program, even for those five-liners. Therefore I have put together a little wrapper library that prettifies using Velocity a little bit.

Here's some usage example to explain what it lets you do:

import eu.kneissl.scala.velocity.Conversions._
assert ("Hello ${yourname}. My name is ${myname}".merge('yourname->"Guest", 
                                                        'myname->"Martin") 
  == "Hello Guest. My name is Martin"

assert ("#foreach( $i in $is )$i#end".merge('is->(1 to 3)) 
  == "123")

case class Person (name: String)
val people = List(Person("Peter"), Person("Paul"), Person("Mary"))
val template: Template = "Hello ${yourname}. My name is ${myname}"
people foreach { person => println(template.merge('yourname->person.name, 'myname->"Martin") }

The Wrapper is described in a little more detail on its project page. Have fun!

QCon London 2

More from QCon 2009: Tony Hoare on science and engineering. Architecting for scalability. Microsoft surface. Scala introduction. Reconciliation. Michael Feathers. The Obama campaign.

Tony Hoare on the spectrum between scientist and engineer.

In his opening keynote Tony Hoare talked about the spectrum between(computer) scientist and engineer. He underlined the different goals
and methods of both and how important it is that both sides communicate to benefit from each other. Interestingly the way he presented the topic was nearly identical to the workshop with Sam Aaron I attended yesterday. While Sam tried to polarize between "modernism" and "wabi-sabi" Tony Hoare did it between "modernism" (for the scientist) and "post-modernism" (for the engineer).

One of Tony's topics, that one has to have formal specification for programs to talk about correctness of programs, of course caused discussion on
how to achieve formal specifications. One of the attendees suggested to use test driven development as a way to formalize requirements and
Tony Hoare somehow agreed.

Panel discussion on architecting for scalability.

For me the nuggets in this discussion were

- When you have a scalable architecture, linear gains / penalties
don't shift the ultimate limit (but costs...)
- While language might change the way you think about problems its
implementation does typically only linearly affect performance (and
then see above)
- When doing SOA, watch out for services that provide too much data
to important consumers, as the network is a bottleneck.
- Use real data when doing performance tests because artificial test
data might be too random or clustered in ways different from real data
such that data alignment / indexes / whatever are used differently
leading to false results.

And it has been great to have Cameron Purdy in the panel who really nails some things down...

Microsoft Surface.

I had the pleasure of trying out this giant iPod touch at the Microsoft booth. It's a huge difference seeing this thing on the web
and trying it out in reality. I'm quite convinced that this is a technology that could have real value if it becomes affordable.

You can use that thing to interact with the displayed information. Not only a single person, but many can play with the virtual cards, photos, whatever is displayed.

Have you ever tried to get a design right by starting in some UML editor? That doesn't work for me well. I need some large sheets of paper to evaluate the first ideas. When working with multiple people I like to use small cards or post-its and shuffle them around on a table or whiteboard.

If we had a Microsoft Surface and a "tolerant" UML tool, we could try things without the need to manually transfer the result from / into
the computer. By "tolerant" I mean a tool that supports incremental development of the model with relaxed model checking to ease "playing"
with the model. Basically what I liked in Booch notation: cloudy, incrementally drawable notation that incrementally can be made more
precise and formal.

Scala introduction by Jonas Boner.

Jonas tried to do a Scala introduction that goes behind basic syntax and "hello world", and succeeded in doing this. For people like me
that already know that level of Scala there was not too much of new information in that talk. I had wished for more detail on how well
Scala does in real world applications and what the issues are you have to look for compared to other languages. Nevertheless I like Jonas'
style of presentation.

Changing the reconciliation process.

Brian Oliver from Oracle did not exist and even lesser talked about a better way to do reconciliation. That's a joke of course: Oracle
obviously had forced him to include a really ridiculous disclaimer at the start of his presentation.

It was a great presentation that covered the functional requirements on reconciliation, what's difficult about it, and how you could use
data grids to improve the situation.

Michael Feathers on Testing and Programming Languages.

By accident I attended the recording of this interview and I'm glad that I did. Check the video when it is online, he has a lot to
say. Also check out his blog (http://michaelfeathers.typepad.com/)

 On the IT support of the Obama campaign.

Martin Fowler and N.N. presented how they supported Barack Obama in his campaign by working on the IT services for his supporters and
volunteers. I learned a lot about the american election system which is rather different from what we have in Germany (Is it really such different?
Have to think about that...). I wonder if I like the idea how much influence it might have whether the parties hire people that are good
at creating web sites or not. But that's obviously one aspect of how politics works.

QCon London 1

Reflections on the tutorial day of the QCon conference in London.

Sitting in the lounge of QCon London looking at the Westminster Abbey, the sun is shining, with a cup of coffee, I'll take a short reflection on yesterday's tutorials.
The first tutorial was by Linda Rising on "Influence Strategies". This tutorial was great: An almost perfect mix of scientific background, practical everyday context, and not the least, a great amount of warmth and humour. The techniques taught in the tutorial are quite applicable, so watch out! I might start using them on you...
The other tutorial by Sam Aaron on "Aesthetical Programming" did not really fulfil my expectations. There were some interesting discussions on aesthetics applied to computer programs, but overall the practical aspect was not covered enough, in my opinion. The most interesting part of the discussion went on "modernism" vs "wabi-sabi" and made me think about how strictness and regularity of the language influences
the changeability of programs. I had hoped for more practical advice on how to improve readability of code. Sam has made his case for DSLs,
of course, and that has been well received. The tutorial could have been improved with more concrete listings and examples. Nevertheless,
I have a lot of interesting things to reflect on when I'll read the handouts again with some distance to the tutorial.
On the conference itself: It's very international, in the first tutorial we had people from at least twelve countries from all across the world. The rooms are convenient, but as I carry a power-sucking-Dell-laptop I had wished for power supplies. Food was fine -- that's important, as I learned from Linda's session, and the location is superb: Westminster Abbey is across the street and a lot of things to see besides computers and geeks within walking distance.
I'm looking forward to having the conference talks today.

Scala Changes the Java World

Javamagazin, the major German developer magazine for the Java platform, titles "Lift the curtain: Scala changes the Java world".

OK, that's my translation of the German cover title "Vorhang auf: Scala verändert die Java-Welt" of Javamagazin (issue 4.09). Nevertheless, one of my colleagues has told me: "Now I think you bet on the right horse. This seems not like any other niche language."

Being on the front page of the magazine might give Scala adoption by mainstream developers another stimulus.

Note that the Javamagazin site is not updated to the latest print version at the time of writing this blog entry.

Reflection from Scala - Heaven and Hell

Bringing some dynamic pieces to the statically typed language. Watch your steps...

For some project I need a way to compile some piece of code at runtime, load and execute it, and later forget it. The use case is somewhere in between up-front compilation and interpretation.

Compilation

Compilation is not that difficult. Put the scala-compiler.jar on the classpath, fill the Settings data structure, take the strangely named "Global" and run the compiler, which will think a little bit about your code and put a lot of nice class files in an output directory.

import java.io.File
import scala.tools.nsc._
import scala.tools.nsc.reporters._

// ...

def compile(srcDir: File, classesDir: File, sources: File*) {
    val settings = new Settings()
    settings.sourcepath.value = srcDir.getPath
    settings.outdir.value = classesDir.getPath
    object compiler extends Global(settings)
    val run = new compiler.Run
    run.compileFiles((sources map io.AbstractFile.getFile).toList)
    if (compiler.reporter.hasErrors) {
      throw new RuntimeException("Compilation failed")
    }
  }

For smaller scripting stuff or tighter binding between script and compiled code you could probably better use the Scala Interpreter classes (see for example this tutorial [German]).

Using the Scripts

Now for the slightly more interesting stuff: How to execute the brand-new classes compiled from the "script" sources. I use a template pattern here to do the compilation, execute some user supplied function with the newly created classes, and clean up afterwards.

Here's how this should be used.

val baz = ...
val scriptSourceDir = ...

val fooBarResult = withScripts(scriptSourceDir) { cl =>
      implicit val classLoader = cl

      val foo: Foo = New("MyFoo")("constructor arg1", "constructor arg2")
      foo.bar(baz)

 }

That is, withScripts provides a ClassLoader that includes the compiled scripts. This is used by the function New, which I will discuss below. Pretty cool, isn't it?

And that's the implementation.

def withScripts[R](srcDir: File)(op: ClassLoader=>R): R = withScripts(srcDir, None)(op)

def withScripts[R](srcDir: File, parentClassLoader: ClassLoader)(op: ClassLoader=>R): R = 
    withScripts(srcDir, Some(parentClassLoader))(op)

private def withScripts[R](srcDir: File, parentClassLoader: Option[ClassLoader])(op: ClassLoader=>R): R = {
    var tmpDir = File.createTempFile("ssi", ".tmp")
    tmpDir.delete()
    if (! tmpDir.mkdirs()) {
      println("Can't mkdirs on " + tmpDir)
    }
    try {
        // TODO: probably must pass the parentClassLoader to compile
        compile(srcDir, tmpDir)
        val result: R = {
           val classLoader = parentClassLoader match {
             case None => new URLClassLoader(Array(tmpDir.toURL))
             case Some(parent) => new URLClassLoader(Array(tmpDir.toURL), parent)
             }
           op(classLoader)
        }
        result
    } finally {
       val notDeleted = (tmpDir.listFiles() ++ Array(tmpDir)) filter {!_.delete()} 
       notDeleted foreach {f => println("Warning: " + f + " could not be deleted")}
    }
  }

I wanted to shield the users of this mini-scripting-framework from to much contact with reflection. Let's see how it works. I proudly present the implemenation of the New function used in the example above.

Reflection

First of all, when using reflection, it is quite natural to convert from a String containing a class name to a Class. Here's an implicit for it. Easy, not really necessary, but... anyway:

  implicit def string2Class[T<:AnyRef](name: String)(implicit classLoader: ClassLoader): Class[T] = {
    val clazz = Class.forName(name, true, classLoader)
    clazz.asInstanceOf[Class[T]]
    }

Note the implicit parameter list where you can provide the class loader either implicitly, or explicitly when have to deal with different class loaders.

That was the heaven part. Now let's go to hell :-)

To create a new object from a class, you use a constructor. These can take parameters. Life could be easy, if everything was an object. Unfortunately there are a few exceptions. 8 or 9, if I count correctly: primitives. And, to make things more complex, there is autoboxing. Finally think about the slightly ill-behaved case of a class with constructors, that behave differently when called with a primitive int and when called with the int boxed to an Integer.

OK. Ground settled. Let's go into the gory details. The New function takes the class name, the constructor parameters in special packaging (explained soon) and a class loader, again possibly implicit. It converts the class name to the class, matches the actual parameters with the declared parameter types of the constructors of the class. If there is only one matching constructor, it is invoked with the actual parameters.

  def New[T<:AnyRef](className: String)(args: WithType*)(implicit classLoader: ClassLoader): T  = {
    val clazz: Class[T] = className
    val argTypes = args map { _.clazz } toArray
    val candidates = clazz.getConstructors filter { cons => matchingTypes(cons.getParameterTypes, argTypes)}
    require(candidates.length == 1, "Argument runtime types must select exactly one constructor")
    val params = args map { _.value }
    candidates.first.newInstance(params: _*).asInstanceOf[T]
  }

  private def matchingTypes(declared: Array[Class[_]], actual: Array[Class[_]]): Boolean = {
    declared.length == actual.length && (
      (declared zip actual) forall {
        case (declared, actual) => declared.isAssignableFrom(actual)
      })
  }

So what is WithType, and why is it needed? Each object has getClass, hasn't it?

Yes, but not everything is an object, as described above. Once autoboxing has objectified a primitive, the information is lost, whether it was a primitive or a primitive wrapper. Therefore the original type, for example Integer.TYPE or Integer.class has to be provided to the New function. Here's the definition of WithType:

  sealed abstract class WithType {
    val clazz : Class[_]
    val value : AnyRef
  }
  
  case class ValWithType(anyVal: AnyVal, clazz: Class[_]) extends WithType {
    lazy val value = toAnyRef(anyVal)
  }
  
  case class RefWithType(anyRef: AnyRef, clazz: Class[_]) extends WithType {
    val value = anyRef
  }

But I promised that it should be easy for the user of the framework. This stuff seems overly complex! And what's that toAnyRef?

The inconvenience of constructing a WithType can be reduced with implicits (you see, I have a new hammer to put in those screws):

  implicit def refWithType[T<:AnyRef](x:T) = RefWithType(x, x.getClass)
  implicit def valWithType[T<:AnyVal](x:T) = ValWithType(x, getType(x))

Now the function New can be called with primitives and with objects or a mix of boths and the compiler will add the type information. And after implementing this, I have found a use for the distinction of AnyRef and AnyVal in the Scala type hierarchy. Nice stuff. Heavenly.

And back to hell: What about toAnyRef and getType? I haven't found standard library functions for boxing an arbitrary AnyVal and AnyVal currently does not support getClass. That means, I had to implement these functions myself.

def getType(x: AnyVal): Class[_] = x match {
    case _: Byte => java.lang.Byte.TYPE
    case _: Short => java.lang.Short.TYPE
    case _: Int => java.lang.Integer.TYPE
    case _: Long => java.lang.Long.TYPE
    case _: Float => java.lang.Float.TYPE
    case _: Double => java.lang.Double.TYPE
    case _: Char => java.lang.Character.TYPE
    case _: Boolean => java.lang.Boolean.TYPE
    case _: Unit => java.lang.Void.TYPE
  }

 def toAnyRef(x: AnyVal): AnyRef = x match {
    case x: Byte => Byte.box(x)
    case x: Short => Short.box(x)
    case x: Int => Int.box(x)
    case x: Long => Long.box(x)
    case x: Float => Float.box(x)
    case x: Double => Double.box(x)
    case x: Char => Char.box(x)
    case x: Boolean => Boolean.box(x)
    case x: Unit => ()
  }

Probably every Java programmer who had to do some reflection / compiler stuff has to implement this every now and then... Fortunately the Scala syntax is concise. Are there copy and paste errors? Forgotten primitive types? I hope not.

Conclusion

The result of having a simple to use way of bringing scripts into a running JVM and using them are satisfactory for me. A lot of Scala features have proven really useful for this purpose. The standard library is either not accessible enough for beginners like me or it is a little bit lacking in some areas. But that doesn't put me off, the implemenation of this code didn't take too long.

I'd like to thank paulp and DRMacIver for support on the #scala IRC channel. They gave me some directions where to look for possible solutions.

Maybe I attach the source code later, but this won't be today.

Family Polymorphism in Scala

How to keep types together for a set of associated types.

Today I have experimented with abstract types in Scala. Abstract types are types references that will be defined by subclasses.

The example

As an interesting exercise I found the graph example in the "Family Polymorphism" paper by Erik Ernst (2001). He defines two kinds of graphs, one ordinary and one with edges that can be disabled. For those that are too lazy to dig through the paper, one of the problems discussed is how to define a polymorphic message on edges without drilling holes into type safety and without hurting reusability by introducing massive C++ templating.

I use abstract types to define a type structure for Graphs with Nodes and Edges. This structure is then refined in two concrete sets of implementations. The interesting point is, that Scala allows me to avoid mixing up the types from the different implementations.

abstract class Graph {
    type Node <: AbstractNode
    type Edge <: AbstractEdge
    
    def mkNode() : Node
    def connect(n1: Node, n2: Node) : Edge
    
    abstract class AbstractEdge(val n1: Node, val n2: Node) 

    trait AbstractNode {
      def touches(edge: Edge): Boolean = edge.n1 == this || edge.n2 == this
    }
  }

Graph is the abstract definition of the structure. It contains abstract types for the Nodes and Edges, and Node and Edge factory methods, as well as the method touches, which will be redefined polymorphically for the "disabled edges" kind of graph below.

  class BasicGraph extends Graph {
    type Node = BasicNode
    type Edge = BasicEdge
    protected class BasicNode extends AbstractNode
    protected class BasicEdge(n1:Node, n2:Node) extends AbstractEdge(n1, n2)
    
    def mkNode() = new BasicNode
    def connect(n1: Node, n2: Node) : BasicEdge = new BasicEdge(n1, n2)
  }

Nothing special for BasicGraph: It just contains concrete classes for the Nodes and Edges and defines corresponding factory methods.

  class OnOffGraph extends Graph {
    type Node = OnOffNode
    type Edge = OnOffEdge
    protected class OnOffNode extends AbstractNode {
      override def touches(edge: Edge): Boolean = edge.enabled && super.touches(edge)
    }
    protected class OnOffEdge(n1:Node, n2:Node, var enabled: Boolean) extends AbstractEdge(n1, n2)
    def mkNode() = new OnOffNode
    def connect(n1: Node, n2: Node) : OnOffEdge = new OnOffEdge(n1, n2, true)

  }

OnOffGraph has the extension to provide for disabled edges. Note that the method OnOffNode.touches can be defined without the need to downcast the edge, because the abstract type Edge is defined to be an OnOffEdge in OnOffGraphs. The type system ensures that you cannot have a BasicEdge connected to OnOffNodes, so we are safe here!

Following some testing code:

    val g = new BasicGraph
    val n1 = g.mkNode()
    val n2 = g.mkNode()
    val e = g.connect(n1,n2)
    assert(n1 touches e)
    assert(n2 touches e)
    val g2 = new BasicGraph
    // g2.connect(n1,n2) // ERROR: can't link to node of other graph

The way BasicGraphs are defined also prevents edges across graphs (Node and Edge are "path-dependent types"). Whether this is desired depends on the context; one could define a super class and an OuterEdge parallel to BasicEdge that would allow to span different graphs.

    val og = new OnOffGraph
    val on1 = og.mkNode()
    val on2 = og.mkNode()
    val oe = og.connect(on1, on2)
    // val mixed = og.connect(n1, n2) // ERROR: og.connect not applicable to g.Node

    assert(on1 touches oe)
    assert(on2 touches oe)
    // println(on2 touches e) // ERROR: on2.touches not applicable to g.Edge
    oe.enabled = false;
    assert (! (on2 touches oe), "After disabling, edge virtually has gone")
    assert (! (on1 touches oe), "After disabling, edge virtually has gone")

As desired, edges can be disabled for OnOffGraphs and the different node types are not compatible such that attempts to disable BasicEdges or to access an "enabled" field via runtime type BasicEdge are statically prevented.

    def addSome(graph: Graph): Graph#Edge = {
      val n1, n2 = graph.mkNode()
      graph.connect(n1,n2)
    }
    
    val e2 = addSome(g)
    val oe2 = addSome(og)
    // oe2.enabled = false // type OnOffGraph not retained, graph.Edge not possible
    ()

It is still possible to write code in terms of unspecific Graphs. In the current Scala implementation it is not possible to refer to a path-dependent type starting at a method parameter. Therefore, when the addSome method is applied to an OnOffGraph, the information that the result is always an OnOffEdge, is not retained by the type system. But you can get near that by making addSome generic:

def addSome[G <: Graph](graph: G): G#Edge = {
      val n1, n2 = graph.mkNode()
      graph.connect(n1,n2)
    }
    
    val e2 = addSome(g)
    val oe2 = addSome(og)
    oe2.enabled = false // now OK.

Conclusion

The possibility to define related types in a way similar to the one above enables the programmer to retain more static type information and certainly reduces the number of required casts. This is most probably also applicable for product / customization scenarios where multiple related classes have to be extended in a compatible way.

No Closures in Java 7?

Currently it seems that Java 7 will not get closures. Oh dear.

According to Hamlet D'Arcy's blog Java 7 will not get closure support. That's bad news to me. It should have closures instead of the new style "for" loop added in Java 5. It should have closures in Java 6. Now it seems that it will not get closures in Java 7.

Closures are not that difficult to understand. At least when you compare them to anonymous inner classes in Java. Others disagree. I don't follow the reasoning of the closure opponents when they say that because there are stupid Java programmers out there you should limit the Language trying to prevent them from doing too much harm. That's just impossible. Incompetent programmers will shoot themselves in the foot in any language.

Fortunately there are other languages on the JVM that can use the real strength of Java: libraries, portability, and (to some extent) tooling.

iPhone bei simyo

Mein Lieblings-Mobilfunk-Reseller bietet jetzt das iPhone 3G an. Wenn es denn copy&paste hätte, 64GB Speicher und eine Java VM, dann könnte ich mich schwer zurückhalten. So ist es cool, schick, aber doch noch nicht vollständig.

Was durfte ich heute lesen? Simyo, mein Lieblings-Mobilfunk-Reseller, bietet über seinen Hardware-Partner das iPhone 3G an. Gar nicht mal so teuer (für ein iPhone):

Apple iPhone 3G 16GB Schwarz:     674,90 Euro.

Damit die Musiksammlung und ein paar Bilder und Videos darauf passte, es also als Ersatz für den iPod durchgehen könnte, müsste das iPhone schon 64GB Speicher haben. Und solche neumodischen Dinge wie Copy&Paste will Apple uns in einem Smart-Phone auch (noch) vorenthalten... Siehe zu diesem Thema auch Mikes Bemerkungen auf seinem Blog.

Außerdem bin ich immer noch der Meinung, das ein Telefon (insbesondere ein Smartphone) richtige Tasten haben sollte. Und eine Schnur :-) (nein, das nun wieder nicht: Hänn di koi Schnur?)

Pressemitteilung von simyo

Limited Visibility

When driving my car I do not like foggy weather with limited visibility. In programming languages it is the other way round: Too much detail visibility will hinder evolution of software systems. Unfortunately Java is rather weak in controlling visibility. Scala improves the situation a little bit.

Usually one uses the term "information hiding" to describe the technique of limiting the visibility of implementation details to users of software components / libraries. The Booch book [reference to be added] has nice pictures of a cat from the view of the user (grandma: purr, purr) and the implementor (gears and levers). Information hiding is very important for two reasons. First, it makes it easier to use functionality because the API is more clearly defined. Second, the provider of functionality can change implementation details more easily without fearing to break client code.

Unfortunately Java's visibility model is not that helpful in practice. You can limit visibility to the class itself (private), the same package (not subpackages or sibling packages), to more specific classes in the inheritance hierarchy (protected) or to make the class or its element world visible (public). Interface methods are always public.

It is not practical in larger Java programs to make only a facade of a larger software system or library visible and to define limited visibility between the parts of the software system. A typical scenario is that you have a structure of packages that use helper functions from a utility package. The utility functions are not useful in a context outside this package structure. But Java cannot prevent client software to use the helpers from the utility package. Only convention can try to do this and external tools can emit warnings.

Scala discards the "package friendly" or default visibility of Java and lets the programmer specify to which level of enclosing scope an element is visible, giving more control than possible in Java. Another interesting feature is that you can restrict access to the same object (as opposed to the same class).

package chb13.fooapp

package processing {
  class FooProcessing {
    def foo {
      chb13.fooapp.utils.FooHelper.onlyForFriends
    }
  }
}

package utils {
  private [fooapp] object FooHelper {
    def onlyForFriends {
        System.out.println("psst...")
    }
  }
}

Note the access from FooProcessing.foo to FooHelper.onlyForFriends. A method from chb13.bar.BarProcessing would not have access to FooHelper methods.

Note also that there is another package declaration syntax possible in Scala as used above. You can nest package declarations and define classes and objects in several packages in the same source file. My opinion on this feature is, that it is only useful for presentations and documentation like this. In real software systems where you need packages to group and organize functionality you should not have different packages in the same file. The situation may change when we move to repository based software development environments and give up files to organize source code, but I do not see this coming in the near future.

Less spectacular and known from many languages: You can rename members on import, resolving name conflicts without having to resort to fully qualified class names. It is also possible to place imports more flexibly than in Java. Rewriting part of the previous example, we could give our helper method a shorter name that is also more meaningful in the given context:

package processing {
  class FooProcessing {
    def foo {
      import chb13.fooapp.utils.FooHelper.{onlyForFriends => psst}
      psst
    }
  }
}

Conclusion

When first introduced to Java I always was a little bit put off by the fact that you could access almost every class from everywhere using fully qualified names, even without declaring access in a file preamble.

Scala in principle still has this possibility, but only if you do not use the access qualifiers to limit visibility. The combination of the access qualifiers and the more flexible syntax for imports allows the provider of some functionality to limit access to internals and therefore ensuring better maintainability and at the same time allows for the clients to access the published functionality using names of their choice.

By the way: I didn't forget the other two languages. XSLT is currently on hold until I receive a book on it (ordered, but not delivered, yet). Groovy will get a blog entry real soon now. And Scala deserves at least one negative ^H^H^H less positive entry in the near future, but its weaker points are harder to describe...

To Type Or Not To Type - Functions in Scala and Groovy

Does a static type system help even though the code becomes longer / more ugly?

Continuing my exploration of Groovy and Scala I have finished chapter 8 of "Programming in Scala", which is on functions and closures. Most of the function / closure stuff is not that surprising. Which means that this blog entry will be rather short.

Nested functions

Scala supports nested function declarations, that is named functions declared within a named function. These have access to the names in the enclosing function(s). Nested functions are useful, when you have functions that are only useful in a limited context to improve readability. In this case it is also nice that you do not have to pass parameters explicitly from the outer function to the inner one. Program layout then resembles the call hierarchy.

def testNested() {
    val context=5
    def foo(i: Int) = i + context
    assertEquals(8, foo(3))
  }

In this example (part of a JUnit test case) testNested sets up some execution context for the inner function and then evaluates it. No need to define "context" as a parameter to "foo".

Groovy does not support nested function declarations. I think Pascal had them and I missed them a lot when I started to program C about 1990. But if you really need them, you can emulate nested functions using closures in Groovy, see below.

Closures

Both Scala and Groovy allow for anonymous functions. In both languages these are not only allowed to refer to parameters but also to refer to variables in scope when the function is defined, thus creating closures. In Groovy you can always omit the parameter types. In Scala you often have to define the parameter types.

// Scala
val incr = (i: Int) => i + 1
assertEquals(2, incr(1))

// Groovy
def incr = { x -> x + 1 }
assertEquals(2, incr(1))

Note the syntactical similarity. Scala defines anonymous functions with the double arrow and calls them "function literals", Groovy uses curly braces with single arrow and calls them "closures". Here I find it easier to read the Groovy variant, partially because the type can be left out.

Both languages support shortcuts for anonymous function definitions.

Groovy allows to drop the parameter declaration and arrow when the function takes only a single parameter, which is then called "it" implicitly:

// Groovy
def incr = { it + 1 }
assertEquals(2, incr(1))

Scala allows to drop the parameter list and arrow when the parameters are only used once. Every parameter is then replaced by an underscore. The parameter types are then declared inline.

// Scala
val incr = (_: Int) + 1
val add = (_: Int) + (_: Int)
assertEquals(2, incr(1))
assertEquals(7, add(3, 4))

The Scala code looks a little bit ugly, or at least strange. What do you get for the additional typing? Better content assist in Eclipse for example. Better byte code, because the compiler has more knowledge on which methods are going to be called. Less stupid errors, when applying functions to unsupported actual arguments. On the other hand, duck typing also has its merits.

Fortunately the parameter type declarations can be omitted in Scala, too, provided that the compiler can infer the types automatically from the context.

// Scala
val someNumbers = List(-1, 0, 1, 2)
assertEquals(List(-1), someNumbers.filter(x => x < 0))
// or, because x is used only once in the function literal:
assertEquals(List(-1), someNumbers.filter(_ < 0))

This again looks very similar to equivalent Groovy code:

// Groovy
def someNumbers = [ -1, 0, 1, 2 ]
assertEquals([-1], someNumbers.findAll{it < 0})

One important difference: Scala is type checked. Assume you type + instead of < in the predicate closure above. Scala won't compile: You get "type mismatch, found: Int required: Boolean". In Groovy you get something else...

// Broken Groovy
def someNumbers = [ -1, 0, 1, 2 ]
assertEquals([-1, 1, 2], someNumbers.findAll{it + 0})
//                                           ~~~~~~     means it != 0 

I don't want to stress this to much, because there are enough other possibilities to make errors that pass type checking, but the combination of Groovy Booleans and missing static type system is generating really strange effects, here.

Partial Application of Functions

Both languages support partial application of functions, that is, creating a new function by binding only some of the function parameters, but not all. The result is a new function with the remaining unbound parameters. This sounds strange when you hear this concept for the first time, but has some interesting real-world applications. Missed that feature the day before in Java and had to define some anonymous inner classes with 90% syntax and 10% functionality instead.

The concept is called "currying" or, in German "schönfinkeln", referring to the inventors of the concept. Groovy can only bind the leftmost parameter of a function when currying, Scala supports binding arbitrary parameters.

// Groovy
def add = { a, b -> a + b }
def incr = add.curry(1)

"incr" is a closure with a single parameter that adds 1 (the parameter to "curry") to its parameter when applied. Possibly also interesting: I used the closure syntax to emulate a local function definition (see above) for "add".

// Scala
def add(a: Int, b: Int) = a + b
val incr = add(1, _: Int)// Scala

def sub(a: Int, b: Int) = a - b
val decr = sub(_: Int, 1)
assertEquals(2, decr(3))

The "incr" definition is essentially the same in Scala, slightly better readable because of syntax support for currying (the underscore denotes an unbound parameter, that is the parameter for the resulting function. The value 1 in bound to the first parameter of add, the second parameter stays unbound, resulting in a function with one parameter.

The "decr" definition above is not possible using currying in Groovy, because it binds the second parameter of "sub".

Tail recursion optimization

Scala supports tail recursion optimization, that is the transformation of a recursive function into iterative code, provided that the recursive call is the last statement in the recursive function. You might think of the callee borrowing the stack frame of the caller because the caller does not need it anymore since it is about to return.

In Scala the tail recursion optimization is only done when directly calling the same named function. Indirect recursion or calling throug curried functions will not be optimized, according to the book. This is more than Java does (nothing) but less than other languages. The reason seems to be a restriction of the JVM. I'll have to see which limitation this is in practice.

Conclusion

Just taking the visual appearance of the code, Groovy and Scala both do well for function literals / closures. I prefer the underscore as wildcard character to the magical "it" parameter and the half-hearted currying support of Groovy. On the other hand I somewhat dislike the syntax for supplying the type information for function literals in Scala.

I do not know the additional API calls for either language. The static type system and resulting IDE support makes it easier to find candidate API calls in Scala through code completion. I almost wrote that I assume Scala to be faster than Groovy thanks to the type system, but without any benchmark this is not a valid statement.

No, this was not so short. But fun to write.

Rational Numbers in Scala

I just finished another chapter in the Programming in Scala book: "Functional objects". C++ strikes back: The return of operator overloading...

Todays chapter of Programming in Scala introduced "functional objects", in other words: immutables. The task was to implement rational numbers in Scala. Here are my highlights.

Conciseness: Scala is not Cobol

The language is concise (as advertised). Not like Cobol (shudder) nor like Ada. Static types are only required in a few places, mainly as parameter types. Type inference works well, saves some typing, and (more important) removes clutter from the code. The declaration of method return type is optional, but improves readability. The return keyword is also optional which also improves readability for this class, which has only very short methods.

For example the multiplication method:

// part of class Rational
def multiply(that: Rational) =
  new Rational(this.numer * that.denom, this.denom * that.denom)

You can call that method either Java-like:

val result = new Rational(1,3).multiply(new Rational(3,5))

or with infix notation

val result = new Rational(1,3) multiply new Rational(3,5)

Operator overloading is also possible and useful for this example:

// part of class Rational
def *(that: Rational): Rational = this multiply that

Now multiplication can be written more naturally:

val result = new Rational(1,3) * new Rational(3,5)

Development Enviroment

I use the Eclipse plugin for Scala (version 2.7.2RC5). It is usable but rather limited in functionality compared to the Java development tools. There are lots of bugs: incremental compilation not properly synchronizing with the editors, incorrect diagnostics, and more. Syntax highlighting is OK, completion uses the static type information and is quite useful for novices like me.

Integration with Java

As expected you can call Java code without problems. After defining my Rational class I wondered how this would be usable from Java code. First I used javap to check the signatures of the class. Type definition and constructors look like Java, same for "conventional" methods as the "multiply" method above. The class implements ScalaObject, that means you have to include the scala jars on the classpath. Operator overloading results in methods with dollar signs in their names. Nevertheless they are callable from Java.

Actually it is fun to mix the different JVM languages in the same workspace. I implemented the class in Scala. I wrote the unit test in Scala but using the JUnit framework, because I do not know the testing framework of Scala. I did some additional tests in Groovy. Looking back at the C/C++ world this is really awesome: You can mix and match languages almost seamlessly. No core dumps, yet :-)

Conclusion

Todays lesson was another step up the Scala. No major obstacles encountered. Writing this blog entry took longer than implementing Rational with some JUnit tests. Of course I had the book as cheat sheet.

I can fully recommend "Programming in Scala" (at least from the first six chapters I have read). This book by Martin Odersky, Lex Spoon and Bill Venners is one of the best Programming Language introductions I have read. The examples are well chosen. There are few concepts that are used before described, and always with forward reference to the chapter where they will be introduced in depth. The book is available as e-Book (no DRM, but with watermark). The printed version can be pre-ordered and is due this month.

Checking a String for upper case characters

Trying three programming languages for the Java Platform at the same time shows interesting features of the different languages. Here solutions for checking a String for upper case characters in Java, Groovy, and Scala.

The Task

A simple task, taken from the Programming in Scala book: Check whether a String contains any upper case character.

Solutions

Java

// Java
String aString = "abCde";
boolean upperFound = false;
for (char c : aString.toCharArray()) {
    if (Character.isUpperCase(c)) {
        upperFound = true;
        break;
    }
}

Scala

// Scala
val aString = "abCde"
val upperFound = aString.exists(_.isUpperCase)

Shorter, clearer, no control structures. That's the way -- I like it.

Groovy

So what about Groovy. Should be very similar to the Scala solution, since Groovy has closures, too, shouldn't it?

Let's try the it the Scala way, nearly 1:1:

// Groovy (1)
def aString = "abCde"
def upperFound = aString.any{it.isUpperCase()}
// RUNTIME ERROR!

Why doesn't it work? Groovy treats aString as a sequence of Strings, not Characters, and String has no method "isUpperCase".

OK, make it more explicit:

// Groovy (2)
def aString = "abCde"
def upperFound = aString.any{(it as Character).isUpperCase()}
// RUNTIME ERROR!

This one is scary: Even though there is an explicit conversion to Character, the method isUpperCase on java.lang.Character is not chosen. Groovy complains, that there is a method isUpperCase(char) and a method isUpperCase(int).

Solution: explicitly use the static helper method on Character and convert the iteration value to Character.

// Groovy (3)
def aString = "abCde"
def upperFound = aString.any{Character.isUpperCase(it as Character)}

Conclusion

This is of course a kind of a micro-benchmark for the expressiveness of the languages. Runtime performance has not been evaluated, so no statement on that, here.

The Java solution is clearly the most verbose and least clear solution.

The Scala solution is extremely short and clear. It is also statically type checked, even though there is no explicit type information given. The compiler can infer the types starting from the string literal.

The Groovy solution is reasonable short and readable. But the solution was not straightforward. Why does the iteration on String yield String as element type? Why does the call to isUpperCase fail in Listing "Groovy (2)"?

A New Programming Language Every Year

After some abstinence I have started to learn programming languages again. And it's fun! At least one of the languages is even fun-ctional.

I used to learn a programming language a year, approximately, when I was a student. Programming can be a lot of fun, kind of an intellectual puzzle, and also a bit of an art, to find "aesthetically pleasing" ways of expressing information and algorithms.

Then comes the time when you have to do something to earn a living. Even if you have a job in the software industry that will not mean that you can always choose the prettiest programming language. Sometimes you'll get the chance of learning some new language during business hours when there is an urgent need in some project you are assigned to. But languages that are not mainstream are not in this category.

Well, what about learning in your leasure time? Not so easy when you have little children: They have their demands and after a day in the office and after having spent some time with the children you are often simply too tired to toy around with a programming language. That's why I haven't learned too many languages in the last years.

I am going to compensate this now by learning 2-3 languages in parallel... I can't remember when I did this the last time, but I'll give it a try (and hope that I won't confuse too many APIs...)

Here are the candidates

• XSLT (Does this even count as a "programming" language? At least it is a formal language, that'll do for me.)
• Groovy
  
• Scala

Why are these currently interesting for me?

XSLT just because there are many (too many!) XML documents out there, that have to be mangled in some way. This one is also most probably directly useful for my job. I intend to use it for tree data like I use the Unix stream editor for line oriented files. I do not assume that I become a real friend of XML, though...

Groovy because of the hype around it. I had already started to experiment with it about 2 years ago just before version 1.0 of Groovy. At that time I had the impression that the language is rather fragile, but this might have changed by now.

Last week I read some examples in Scala and I am really impressed by this language. I like the combination of

• integration in the Java Platform
• support for functional programming
• static type system
• concise and regular syntax.
  

I'll document some of my experiences in this blog, stay tuned.

I hate trackback spam!

Checking our web space I had to discover that we went low on space due to such a dull thing as trackback spam. Is there a way to get away without being spammed, or do we have to accept a certain level of spam?

Suddenly - in a phase when I did not publish anything on our pages - disk usage of the site went up to its limits. After logging in it became clear that my four blog entries got thousands of trackbacks.

If anyone needed some biology lessons or medicine or poker experience: The trackbacks could have provided useful information...

Fortunately trackbacks have to be acknowledged on this system and even more fortunately they can be disabled completely. Maybe at some time I will connect to a spam filter for comments / trackbacks, but probably this is not worth the trouble and I just keep them turned off.

Spam is really the plague of the internet. But when you really think about it, it may just be the other side of the coin of the well appreciated freedom and partial anonymity of the internet. How could you stop spam other than by introducing strict control and accountability of every step an individual makes in the internet - big brother would be watching us.

Probably we have to roll out our own web of trust (again) to get the most important information through the net without delay while maintaining free communication with people neither known nor recommended to us, yet, and live with lots of spam in exchange for the freedom of not having to sign every packet of information with a key assigned to us by Mr. Schäuble (our equivalent of a Secretary of Homeland Security).

Software Engineering Radio

Where some software engineering professionals and gurus share their knowledge and opinions.

On http://se-radio.net/ Markus Völter and his friends publish a very interesting audio podcast every 10 days. Very useful to broaden one's view on software engineering and software architecture.

Rechner mit Schokoladenkühlung

Wenn Obst am Arbeitsplatz nicht mehr ausreicht.

Alle, die mich kennen, wissen, dass ich am besten arbeite, wenn ich ausreichend mit Schokolade versorgt bin: Schokolade macht glücklich, anderes Essen nur dick.

Zur Zeit wird bei uns in der Firma ein Obstbüfett angeboten. Eine tolle Sache. Jetzt muss ich nur noch die Schokoladenkühlung für meinen Rechner beantragen, dann habe ich immer leckeres Schokoladenfondue.

Anscheinend hat aber noch niemand probiert, den Rechner mittels Schokolade zu kühlen: Google weiß nichts darüber.

Schokoladenfüllung ist aber auch nicht schlecht...