In an earlier post I asked for an implementation of the following Java interface in Scala:

public interface Iterator2 extends java.util.Iterator {}

The solution was to use a combination of self types and existential types. However, some comments made me aware of Ticket #1737 where a more general form of the problem is discussed. Given the following Java code:

public interface A<T> {
    void run(T t);
    T get();
}

public abstract class B implements A {}

How could B be implemented in Scala? My previous approach is not sufficient here since for implementing the run() method, one needs to be able to refer to the existential type in run()’s parameter list:

class C extends B {
  this: A[_] =>

  def run(t:???) {} // What should go here for t's type?
  def get = "hello world"
}

So what we need is a way to name the existential type used in the self type (that is the type parameter to A) such that we can refer to it in the parameter list of the run() method. Here is my shot:

class D {
  type Q = X forSome { type X; }
}

class C extends B {
  this: A[D#Q] =>

  def run(t:D#Q ) {}
  def get: D#Q = "hello world"
}

Unfortunately this results in a compilation error with Scala 2.7.5

error: class C needs to be abstract, since method run in trait A of type (T)Unit is not defined

and in a AssertionError for Scala 2.8.0.r18604-b20090830020201

Exception in thread "main" java.lang.AssertionError: assertion failed: michid.iterator.A[T]
at scala.Predef$.assert(Predef.scala:107)
at scala.tools.nsc.symtab.Types$TypeRef.transform(Types.scala:1417)
at scala.tools.nsc.symtab.Types$TypeRef$$anonfun$baseTypeSeq$3.apply(Types.scala:1588)
...

I created Ticket #2091 for these problems.

Yesterday I gave a presentation at Jazoon 09 about using Scala for scripting RESTful web applications with Apache Sling.

In the session I showed how to take advantage of Scala to create RESTful web applications with Apache Sling. I demonstrated how to uses its DSL capability and support for XML literals to create type safe web site templates. In contrast to conventional web site template mechanisms (e.g. JSP), this does not rely on a pre-processor but rather uses pure Scala code.

There are Session slides and support material available here. The support material contains a fully workable demo application. A Scala scripting bundle for Sling is also included.

Well, I wasn’t aware of Ticket #1737 when I was trying to find a solution to the problem from my previous post. Thanks to Jorge Ortiz for pointing this out. However, I reviewed my approach to solving this and didn’t find sever limitations. Maybe someone else does…

When I initially stumbled on this, I remembered that existential types where introduced into Scala for coping with Java’s raw types. But there is an additional twist here, we need to tell the compiler that our MyIterator implementation actually ‘is an instance of a raw type’. So combining existential types with self types led me to the following solution:

class MyIterator extends Iterator2 {
  this: java.util.Iterator[_] =>
  def hasNext = true
  def remove = throw new Error
  def next = "infinity"
}

We can now safely use instances of MyIterator.

  def test1(it: MyIterator) = {
    println(it.next)
  }

  def test2(it: java.util.Iterator[_]) = {
    println(it.next)
  }

  val it = new MyIterator
  val v: String = it.next
  println(v)

  test1(it)
  test2(it)

The approach using existential types in combination with self types makes sure that values returned from the next method always are typed correctly.

This is something I stumbled on recently when trying to implement javax.jcr.NodeIterator in Scala.

Assume you are using a library which exports an Iterator2 interface:

public interface Iterator2 extends java.util.Iterator {}

Note that Iterator is a raw type and Iterator2 does not take any type parameters. So how would you implement Iterator2 in Scala?

Here is a start:

class MyIterator extends Iterator2 {
  def hasNext = false
  def remove = throw new Error
  def next: Nothing = throw new Error
}

But if the next method should return an actual value, what would be it’s return type? It turn’s out that any other type than Nothing results in a compiler error:

error overriding method next in trait Iterator of type ()E;
method next has incompatible type ()Any

So how would you implement Iterator2?

Function memoization is an optimization technique to avoid repeated calculation of function values which have been calculated by a previous evaluation of the function. In this post I show how function memoization can be implemented in Scala. Although straight forward at a first glance, effectively memoizing recursive functions requires some second thoughts. For the sake of simplicity I only discuss functions of arity one.

Assume we have a function strSqLen which calculates the square of the length of a string.

def strSqLen(s: String) = s.length*s.length

Assume further, that for some reason evaluating above function is processor intensive. One way to speed things up is to cache result values and to look them up on subsequent invocations of strSqLen. This can be done either by the function itself or by the caller. The first approach has the drawback that each and every function has to implement caching and that clients have no control over the caching mechanism. The latter approach puts all the burden on the client programmer and produces potential boilerplate. To overcome these issues we need a way to construct a memoized function having the same type as a given function.

In Scala a function of arity one is an instance of Function1. We can thus sub-class a Function[-T, +R] to Memoize1[-T, +R] which represents the memoized function. (Note the concise syntactic sugar T => R which is synonym to the more verbose Function1[T, R].)

class Memoize1[-T, +R](f: T => R) extends (T => R) {
  import scala.collection.mutable
  private[this] val vals = mutable.Map.empty[T, R]

  def apply(x: T): R = {
    if (vals.contains(x)) {
      vals(x)
    }
    else {
      val y = f(x)
      vals + ((x, y))
      y
    }
  }
}

object Memoize1 {
  def apply[T, R](f: T => R) = new Memoize1(f)
}

When applied to an argument of type T, apply checks whether the function value is in the cache. If so, it returns that value. If not, it calls the original function f, puts the function value into the cache and returns it. Note that the member vals which contains the cached function values has object private visibility (private[this]). A less restrictive visibility would cause type checking to fail since mutable.Map[A, B] is invariant in both its type arguments while Memoize1[-T, +R] is contravariant in T and covariant in R (as is Function1[-T, +R]). However since vals is accessed from its containing instance only, it cannot cause problems with variance. We tell this to the compiler by declaring the field object private.

We can now easily create and use a memoization of strSqLen like this:

val strSqLenMemoized = Memoize1(strSqLen)
val a = strSqLenMemoized("hello Memo")
val b = strSqLen("hello Memo")
assert(a == b)

Going recursive

Memoization of recursive functions is possible but might not expose the desired effect. Consider the following recursive implementation of the factorial function.

def fac(n: BigInt): BigInt = {
  if (n == 0) 1
  else n*fac(n - 1)
}

Calculating the factorials from 200 down to 0 does not take advantage of the memoization at all.

val facMem = Memoize1(fac)

for (k <- 200 to 0 by -1) {
  println(facMem(k))
}

While facMem(200) caches the factorial of 200, it does not cache the results of its intermediate recursive invocations since the recursive calls invoke the original function instead of the memoized. When it comes to calculating facMem(199) there is no gain from memoization here since – although calculated before – facMem(199) is not cached.

To improve this we need a more flexible implementation of fac. We want its recursive calls to be available on the outside such that they are available for caching.

def facRec(n: BigInt, f: BigInt => BigInt): BigInt = {
  if (n == 0) 1
  else n*f(n - 1)
}

Here the caller needs to pass a function for calculating factorials to factRec. She is therefore free to pass a memoized function to speed up recurrent invocations. However, it seems we are back to square one: we need to pass a function for calculating factorials do facRec in order for it to calculate factorials. As it turns out this is not the case. By passing facRec to itself recursively, we can construct the desired factorial function.

var fac: BigInt => BigInt = null
fac = facRec(_, fac(_))

First we declare function fac to map from BigInt to BigInt. Then we partially apply facRec to fac which yields the desired factorial function.

We can generalize and factor out this construction process into an object Y like this:

object Y {
  def apply[T, R](f: (T, T => R) => R): (T => R) = {
    var yf: T => R = null
    yf = f(_, yf(_))
    yf
  }
}

Along the same lines we can provide functionality for creating memoized versions of recursive functions. Instead of just recursively invoking the passed function, we pass a memoized version of it.

object Memoize1 {
  // ... same as above

  def Y[T, R](f: (T, T => R) => R) = {
    var yf: T => R = null
    yf = Memoize1(f(_, yf(_)))
    yf
  }
}

Using Memoize1.Y we can calculate the factorials from 200 down to 0 while taking full advantage of memoization of all intermediate recursive invocation.

def facRec(n: BigInt, f: BigInt => BigInt): BigInt = {
  if (n == 0) 1
  else n*f(n - 1)
}

val fac = Memoize1.Y(facRec)

for (k <- 200 to 0 by -1)
  println(fac(k))

The kind of comments I keep getting on my static meta-programming with Scala blogs are often along the lines of: “The ability to encode Church Numerals in Scala still seems uselessly academic to me, but cool none-the-less”. In this blog I will show how meta-programming can be applied in practice – at least theoretically.

In my previous blogs I introduced a technique for static meta-programing with Scala. This technique uses Scala’s type system to encode values and functions on these values. The Scala compiler acts as interpreter of such functions. That is, the type checking phase of the Scala compiler actually carries out calculations like addition and multiplication.

In this post I show how to apply meta-programming for two practical problems: conditional compilation and loop unrolling. The examples make use of type level encoding for booleans and natural numbers. While I introduced an encoding for natural numbers before, I use an alternative method which is more powerful in this post. Previously it was not possible to build nested expressions having expressions as operands themselves. The new encoding supports such expressions. However, in general the new encoding depends on the -Yrecursion compiler flag which is experimental and as of now only available in the Scala trunk. The type level encoding for booleans is along the same lines as the one for natural numbers.

Conditional Compilation

Conditional compilation is useful for example for enabling or disabling debugging or logging statements. Ideally code which is excluded by a compile time condition does not have any effect on the run-time behavior of the rest of the code. That is, the rest of the code behaves exactly as if the excluded code were not there at all. Optimizing compilers generally remove code which is unreachable. This is where meta-programming fits in: type level encoded functions (meta-functions) are evaluated at run-time. The result of the evaluation is a type. Now we only need to trick the compiler into compiling a block of code or not compiling it depending on that type.

Lets first define meta-booleans and some operations on them (full code here):

object Booleans {
  trait BOOL {
    type a[t <: BOOL, f <: BOOL] <: BOOL
    type v = a[TRUE, FALSE]
  }
  final class TRUE extends BOOL {
    type a[t <: BOOL, f <: BOOL] = t
  }
  final class FALSE extends BOOL{
    type a[t <: BOOL, f <: BOOL] = f
  }
  trait IF[x <: BOOL, y <: BOOL, z <: BOOL] extends BOOL {
    type a[t <: BOOL, f <: BOOL] = x#a[y, z]#a[t, f]
  }
  trait NOT[x <: BOOL] extends BOOL {
    type a[t <: BOOL, f <: BOOL] = IF[x, FALSE, TRUE]#a[t, f]
  }
  trait AND[x <: BOOL, y <: BOOL] extends BOOL {
    type a[t <: BOOL, f <: BOOL] = IF[x, y, FALSE]#a[t, f]
  }
  trait OR[x <: BOOL, y <: BOOL] extends BOOL {
    type a[t <: BOOL, f <: BOOL] = IF[x, TRUE, y]#a[t, f]
  }

  // aliases for nicer syntax
  type ![x <: BOOL] = NOT[x]
  type ||[x <: BOOL, y <: BOOL] = OR[x, y]
  type &&[x <: BOOL, y <: BOOL] = AND[x, y]
}

The following pre-processor object contains an implicit method for converting a value of type TRUE to an Include object whose apply method executes a block of code. Similarly it contains an implicit method for converting a value of type FALSE to an Exclude object whose apply method simply does nothing. The strange line where null is being cast to B is a trick for getting a witnesses of a value of type B.

object PreProc {
  def IF[B] = null.asInstanceOf[B]

  object Include {
    def apply(block: => Unit) {
      block
    }
  }

  object Exclude {
    def apply(block: => Unit) { }
  }

  implicit def include(t: TRUE) = Include
  implicit def exclude(f: FALSE) = Exclude
}

Using these definitions is quite convenient now:

object IfDefTest {
  import PreProc._

  type LOG = TRUE
  type ERR = TRUE
  type WARN = FALSE

  def errTest() {
    IF[(LOG && ERR)#v] {
      println("err")
    }
  }

  def warnTest() {
    IF[(LOG && WARN)#v] {
      println("warn")
    }
  }

  def main(args: Array[String]) {
    errTest()
    warnTest()
  }
}

Running the above code will print err but wont print warn to the console.

Loop Unrolling

Another application for static meta-programming is loop unrolling. When the number of iterations of a loop is small and only depends on quantities known at compile time, run time performance might profit from unrolling that loop. Instead of resorting to copy paste, we can use similar techniques like above.

Again let’s first define meta-naturals and their operations (full code here):

object Naturals {
  trait NAT {
    type a[s[_ <: NAT] <: NAT, z <: NAT] <: NAT
    type v = a[SUCC, ZERO]
  }
  final class ZERO extends NAT {
    type a[s[_ <: NAT] <: NAT, z <: NAT] = z
  }
  final class SUCC[n <: NAT] extends NAT {
    type a[s[_ <: NAT] <: NAT, z <: NAT] = s[n#a[s, z]]
  }
  type _0 = ZERO
  type _1 = SUCC[_0]
  type _2 = SUCC[_1]
  type _3 = SUCC[_2]
  type _4 = SUCC[_3]
  type _5 = SUCC[_4]
  type _6 = SUCC[_5]

  trait ADD[n <: NAT, m <: NAT] extends NAT {
    type a[s[_ <: NAT] <: NAT, z <: NAT] = n#a[s, m#a[s, z]]
  }
  trait MUL[n <: NAT, m <: NAT] extends NAT {
    trait curry[n[_[_], _], s[_]] { type f[z] = n[s, z] }
    type a[s[_ <: NAT] <: NAT, z <: NAT] = n#a[curry[m#a, s]#f, z]
  }

  // aliases for nicer syntax
  type +[n <: NAT, m <: NAT] = ADD[n, m]
  type x[n <: NAT, m <: NAT] = MUL[n, m]
}

The pre-processor object defines a trait Loop having an apply method which takes a block of code as argument. Again there are two implicit conversion methods. One which converts the zero type to a Loop with an empty apply function. An another one which convert the type N + 1 to a a Loop with an apply function which executes the block once and then applies itself to the type N.

object PreProc {
  def LOOP[N] = null.asInstanceOf[N]

  trait Loop[N] {
    def apply(block: => Unit)
  }

  implicit def loop0(n: ZERO) = new Loop[ZERO] {
    def apply(block: => Unit) { }
  }

  implicit def loop[N <: NAT](n: SUCC[N])(implicit f: N => Loop[N]) = new Loop[SUCC[N]] {
    def apply(block: => Unit) {
      block
      null.asInstanceOf[N].apply(block)
    }
  }
}

Again using this is easy and convenient:

object LoopUnroll {
  import PreProc._

  def unrollTest() {
    // The following line needs the -Yrecursion 1 flag
    // LOOP[(_3 x _2)#v] {
    LOOP[_6] {
      println("hello world")
    }
  }

  def main(args: Array[String]) {
    unrollTest()
  }
}

The above code prints the string “hello word” six times to the console.

Conclusion

Scala’s type system is powerful enough for encoding commonly encountered functions. Together with Scala’s strong capability for creating internal DSLs, this results in convenient techniques for static meta-programming. Such techniques can be applied to practical problems – at least in theory. In practice Scala’s support is not (yet?) there. For one the technique presented here depends on an experimental compiler flag (-Yrecursion). Further the types required for meta-programming might causes an exponential growth in compilation time which is not desirable. And finally an analysis with c1visualizerwith showed, that the compiler seems not to remove all unnecessary calls.

In my previous post about Meta-Programming with Scala I suspected that there was no way to express partial function application in Scala’s type system. However Matt Hellige proofed me wrong in his comment.

His solution uses a trait for partially applying a function to some of its arguments. An abstract type exposed by the trait represents the resulting function which takes the remaining arguments.

object Partial {
  // Partial application of f2 to x
  trait papply[f2[_, _], x] {
    type f1[y] = f2[x, y]
  }

  // apply f to x
  type apply[f[_], x] = f[x]

  trait X
  trait Y
  trait F[A1, A2]

  // Test whether applying the partial application of
  // F to X to Y equals in the type F[X, Y]
  case class Equals[A >: B <: B, B]
  Equals[apply[papply[F, X]#f1, Y], F[X, Y]]
}

Having this solved we can define a type which encodes multiplication on the Church Numerals.

    trait curry[n[_[_], _], s[_]] {
      type f[z] = n[s, z]
    }

    // Multiplication for this encoding
    type mult[m[_[_], _], n[_[_], _], s[_], z] = m[curry[n, s]#f, z]

A full working example is available from my code page. Note, the code takes forever (i.e. some minutes) to compile. Matt also noted an issue with squares. With my version of the compiler (Ecipse plugin 2.7.2.r15874-b20080821120313) the issue does not show up however.

In this post I deviate a bit from the topic of my recent posts about Meta-Programming with Scala. I will have more to say about the latter topic in upcoming posts however.

Recently I read this rather fascinating post about a Type-safe Builder Pattern in Scala. When Heinz Kabutz mentioned the builder pattern in his latest issues of the The Java Specialists’ Newsletter I decided to try to come up with a type safe version for Java.

What I finally came up with is not strictly a builder but something I rather call an initializer. The initializer contains the initial state required by the target object. The state is accumulated within the initializer. Only when the state is complete can it be passed to the targets object’s constructor. Java’s type system prevents passing a initializer with an incomplete state to the target class’s constructor.

public class Foo {
  private final int a;
  private final int b;

  public Foo(Initializer<TRUE, TRUE> initializer) {
    this(initializer.a, initializer.b);
  }

  private Foo(int a, int b) {
    super();
    this.a = a;
    this.b = b;
  }

  public String toString() {
    return "a = " + a + ", b = " + b;
  }

  public static class Initializer<HA, HB> {
    private int a;
    private int b;

    private Initializer() {
      super();
    }

    private Initializer(int a, int b) {
      super();
      this.a = a;
      this.b = b;
    }

    public static Initializer<FALSE, FALSE>create() {
      return new Initializer<FALSE, FALSE>();
    }

    public Initializer<TRUE, HB> setA(int a) {
      this.a = a;
      return new Initializer<TRUE, HB>(a, this.b);
    }

    public Initializer<HA, TRUE> setB(int b) {
      this.b = b;
      return new Initializer<HA, TRUE>(this.a, b);
    }

    static abstract class TRUE {}
    static abstract class FALSE {}
  }

}

The basic technique is the same as for the Type-safe Builder Pattern in Scala: the phantom types TRUE and FALSE are used to keep track of the state. Only a complete state will result in a Initialiter instance which subsequently can be passed to Foo’s constructor.

Here is how this is used:

public class Main {
  public static void main(String[] args) {
    Initializer<?, ?> initializer = Initializer.create();

    // Foo.create(initializer);           // won't compile
    // Foo.create(initializer.setB(1));   // won't compile
    // Foo.create(initializer.setA(1));   // won't compile

    Foo foo = new Foo(initializer.setA(1).setB(2));
    System.out.println(foo);
  }
}

This was sitting on my desk for quite a while now while I was busy with other things. Finally I came around to write things up.

In my last post I showed how to encode the Church numerals and addition on them with Scala’s type system. I mentioned that this approach does not seem to scale. In this post I show the problems I faced while I tried to extend the approach to multiplication. This particular example shows that Scala’s type system does not seem to support partial function application which in general is crucial for defining more complex functions base on simpler ones. But before delving into Scala, let’s review the church numerals in lambda calculus.

Define a lambda term for each natural number n

where stands for the n-fold application of s. Think of s standing for successor and z for zero. Then the number n is simply the n-fold successor of zero.

Addition can now be define as

.

Here the n-fold successor of zero is used as zero element for m. Again this can be thought of as the m-fold successor of the n-fold successor of zero.

Multiplications is similar but instead of using a different value for zero, a different successor function is used:

This boils down to the m-fold application of the n-fold successor function.

As in my previous post the Church numerals are encoded in Scala’s type system as follows

type _0[s[_], z] = z
type _1[s[_], z] = s[z]
type _2[s[_], z] = s[s[z]]
type _3[s[_], z] = s[s[s[z]]]
type _4[s[_], z] = s[s[s[s[z]]]]

I did not succeed encoding multiplication however. The most straight forward encoding – which closely follows the one for addition – looks like this

type mul[m[s[_], z], n[s[_], z], s[_], z] = m[n[s[_], z], z]

However the Scala compiler complains:

  s[_$1] forSome { type _$1 } takes no type parameters,
  expected: one

Unlike addition, we need to pass a partially applied function here – namely the n-fold successor function. Since the above does not work, I’m not sure if there even is a syntax for expressing partial function application in Scala’s type system.

In another approach I tried to introducing a formal parameter for the n-fold successor function:

type mul[m[n[s[_], z], z], n[s[_], z], s[_], z] = m[n, z]

The compiler does not complain here which is encouraging. However this breaks on application

abstract class Zero
abstract class Succ[T]

type zero = mul[_0, _0, Succ, Zero]
type one = mul[_1, _3, Succ, Zero]

The first application results in a kind error

the kinds of the type arguments (Meta._0,Meta._0,Meta.Succ,Meta.Zero) do not conform to
the expected kinds of the type parameters (type m,type n,type s,type z).
Meta._0's type parameters do not match type m's expected parameters:
type s has one type parameter, but type n has two

The second application causes an assertion failure of the Scala compiler (see Ticket #742).

For a better understanding let’s analyze what mul[_1, _3, Succ, Zero] expands to:

mul[_1, _3, Succ, Zero] =
_1[_3, Zero] =
_3[Zero]

While this looks somewhat correct, it is certainly not. _3 takes two arguments but gets one. This is exactly what the Scala compiler was complaining about.

In the solution to a puzzle I posted earlier, I mentioned that the taken approach might be a first step towards meta-programming in Scala.

While the approach to determine the depth of a type plays nicely with the Church encodings of natural numbers proposed in the paper Towards Equal Rights for Higher-kinded Types, defining arbitrary arithmetic operators seems problematic. Addition does not pose a problem. But there seems to be no easy generalization to other arithmetic operators. This is despite the fact that the authors of the above paper mention that Scala’s kinds correspond to the types of the simply typed lambda calculus.

In this post I will explain how to represent the natural numbers and an addition operator using Scala’s type system. In a later post I will show why this approach does not easily scale to the other arithmetic operators like multiplication and why I think that this might not be a problem of Scala’s type system in general but rather a shortcoming of its syntax. Finally I noted that Scala 2.7.1.RC1 dropped the contractiveness requirement for implicits. This might open up another way for meta-programming. I probably write more on this in yet another post.

The following code shows a Church encoding of the natural numbers in Scala’s type system. (The full source is available from my code page).

  type _0[s[_], z] = z
  type _1[s[_], z] = s[z]
  type _2[s[_], z] = s[s[z]]
  type _3[s[_], z] = s[s[s[z]]]
	// ...

A natural numbers is encoded as a type function which takes two arguments: a successor function and a zero element. The n-th natural number is then the n-fold application of the successor function to the zero element. Such natural numbers can be instantiated by passing them an actual type for its successor function and its zero element. The depth function form my last post can then be used to evaluate such a number;

  abstract class Zero
  abstract class Succ[T]
  type two = _2[Succ, Zero]
  println(depth(nullval[two]))  // prints 2

We can now define an addition operator like this:

  type plus[m[s[_], z], n[s[_], z], s[_], z] = n[s, m[s, z]]
  type +[m[s[_], z], n[s[_], z]] = plus[m, n, Succ, Zero]

Basically plus takes two Church numerals n and m and composes them such that m becomes the zero element of n. The second line defines an infix + operator:

  println(depth(nullval[_2 + _2])) // prints 4
  println(depth(nullval[_2 + _3])) // prints 5

Let’s analyze in detail what _2 + _3 expands to in order to better understand above code:

  _2 + _3 =
	plus[_2, _3, Succ, Zero] =
	_3[Succ, _2[Succ, Zero]] =
	_3[Succ, Succ[Succ[Zero]]] =
	Succ[Succ[Succ[Succ[Succ[Zero]]]]]

So passing _2 + _3 to the depth function will indeed print 5 since this its structural depth.

From here one might want to generalize this to other arithmetic operations. I tried multiplication but failed so far. In a next post I will explain my closest shot(s) at multiplication and I will explain what I think are the difficulties with this approach.