Language Support in the language you're supporting (or eating your own dogfood)

Last post I talked about how to implement support for a new language in vNext, specifically I showed (more or less) how to implement support for F#. This time, we'll take a look at what I had to do to rewrite the language support itself into F#. But before I can do that, I have to explain a bit how vNext today handles languages through language support packages.

First lets assume that we've already build a FSharpSupport project, written in C#. This project works according to specs, implement all the necessary methods (read my previous post if you don't know which ones these are), and provide us with assemblies that we can use. Also, we have a sample project (let's just call this Sample, that's set up to use the FSharpSupport package. The Sample project is a simple vNext console app that just outputs Hello from F#, and nothing more.

When we (in the directory of Sample) run k run, this is aproximately what happens:

  1. K starts up, sets up it's environment, and hooks into .NET events such as AssemblyResolve etc.
  2. K boots up the "Application Host", it's responsible for running our application.
  3. The Application Host requests the assembly for "Sample" (Assembly.Load("Sample")).
  4. Compiling Sample requires FSharpSupport, so Assembly.Load("FSharpSupport") is called.
  5. FSharpSupport is returned (either by compiling C# sources, or discovering it's binary, doesn't matter for this explanation).
  6. FSharpSupport is used to compile Sample.
  7. Sample is returned to the "Application Host".
  8. The application host runs Sample.

This works as you'd expect, without any problems. However, if we want FSharpSupport to be written in F#, one does arise. Namely the fact that you can't really have two assemblies that have the same name, and in vNext, sources are preferred over nupkegs.

Infinite recursion in the compiler

Let's take a look at what actually happens if you try to build FSharpSupport with itself. First, we have converted FSharpSupport sources to F#, and we have a precompiled working version written in C# from earlier. Both have the same name, FSharpSupport. One is simply a project sitting as sources in a directory, the other is a nuget-package installed into the kpm central cache.

The project.json of FSharpSupport looks like this:

{
  "dependencies": {
    "Microsoft.Framework.Runtime": "1.0.0-*",
    "FSharp.Compiler.Service": "0.0.58",
    "FSharp.Core.Open.FS31": "3.1.1.5"
  },

  "code": ["provider.fs"],

  "language": {
    "name": "F#",
    "assembly": "FSharpSupport",
    "projectReferenceProviderType": "FSharpSupport.FSharpProjectReferenceProvider"
  },

  "frameworks" : {
    "net45" : { 
      "dependencies": {
      }
    }
  }
}

Now, if I were to do kpm build on this, you'd eventually get to a point where it calls Assembly.Load("FSharpSupport"). And since vNext prefer sources over binaries, that'd try to compile FSharpSupport (which is incidentally also what we are in the middle of), which in turn would lead to a new call to Assembly.Load("FSharpSupport"), which would lead to trying to compile the code and so on and so forth. Eventually, the kpm dies with an StackOverflowException.

Feeding vNext with our assembly location

One little known fact about vNext (or kre specifically) is the ability to feed it "library paths" using a --lib command line flag. kpm for instance already utilizes this, having it's sources in a lib/Microsoft.Framework.PackageManager sub-folder of the klr itself. So, if we spesify the location of our built version of FSharpSupport when we do kpm build, that version will get picked up by Assembly.Load("FSharpSupport"), because the --lib has higher precedence than source projects. Great then. We just use kpm --lib <path-to-FSharpSupport> build, right?

Unfortunately, this doesn't work. And to explain why it doesn't work, we have to look under the covers of what actually happens when you run kpm build in the command line.

Note: I'll be explaining how this works on Windows. The same should mostly apply to *nix/osx, with the exception that it runs shell-scripts of the *.sh kind, instead of *.cmd.

When you do kpm build, that get's translated into a call to the script kpm.cmd. kpm.cmd looks like this:

@Echo OFF
SETLOCAL  
SET ERRORLEVEL=

CALL "%~dp0KLR.cmd" --lib "%~dp0;%~dp0lib\Microsoft.Framework.PackageManager" "Microsoft.Framework.PackageManager" %*

exit /b %ERRORLEVEL%  
ENDLOCAL  

Now, in case you don't read batch scripts (I mean, who does these days anyways?), the gist of this is that it takes your arguments and fires

klr --lib "<klrdir>;<klrdir>\lib\Microsoft.Framework.PackageManager" "Microsoft.Framework.PackageManager" <args>  

So, if we were to simply do kpm --lib "<FSharpSupportPath>" build, that would end up with a command with two --lib args, and one of the would be too late. So what we need to do is call klr directly. What you need to do is call the following:

klr --lib "<klrdir>;<klrdir>\lib\Microsoft.Framework.PackageManager;<FSharpSupportPath>" "Microsoft.Framework.PackageManager" <args>  

Going crazy

Now that we can actually (though quite painfully) build our language support in our own language, let's simplify things. Having to write out that command line every time you want to build is annoying, so I made a crazy build-script in FAKE. You can see how it works here. Feel free to steal it.

As always, if there's any questions, please do leave a comment :).