HS2LIB TUTORIAL

AUTOMATING THE CREATION OF STATIC LIBRARIES FROM HASKELL PROGRAMS

Tamar Christina
tamar@zhox.com http://hackage.haskell.org/package/Hs2lib

September 4, 2011

PART OF THE VISUAL HASKELL 2010 TOOLCHAIN

This is a shortened version of the manual showing only the introduction and examples. Full manual to follow.

Copyright 2011 Tamar Christina http://code.zhox.com/haskell/hs2lib Licensed under the BSD License, Version 3.0 (the License); you may not use this le except in compliance with the License. You may obtain a copy of the License at http://www.opensource.org/licenses/bsd-license. php. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an as is basis, without warranties or conditions of any kind, either express or implied. See the License for the speci c language governing permissions and limitations under the License. First printing, September 2011

Contents
1 Introduction 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples 2.1 Simple Arithmetic 2.2 User data types . 2.3 Callbacks . . . . . 2.4 Sessions . . . . . 7 7

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

9 . 9 . 23 . 24 . 25

CONTENTS

Release Notes
Version 0.5.5 - (Started) implementing a better test mechanism Version 0.5.4 - fixed a major marshalling bug having to do with lists of pointer types. Version 0.5.3 - Fixed an error with parsing pragmas - Renamed pragma hs2c# to hs2cs - Fixed an alignment issue with stdcall - Fixed an issue with lists and type synonyms Version 0.5.2 - No longer frees Strings, in order to prevent heap corruptions on calls from C#. Version 0.5.1 - Library bug fix which cauzed an error in marshalling Version 0.5.0 - Added support for memory leak tracing with the --debug flag % Added support for qualified importing. It is now possible to export values and types with the same name. (unfinished) % - Added the ability to export polymorphic types. (unfinished) % - Added support for infix constructors (unfinished) - Generates free functions StablePtr Version 0.4.8 - Major restructuring of code to allow users to extend/override the default type conversions of the tool. - Support for custom type translations Version 0.4.5 - Brand new Haskell code generators - Support for lists inside an IO wrapper (e.g. IO [Int]) - Brand new implementation in NativeMapping - Avoids unsafePerformIO as much as possible Version 0.4.4 - Fixed Include paths - NO support for datastructures with multiple constructors where one constructor has the same name as the datatype (codegen naming limitation) - Fixed codegen issue with callbacks - Added better IO Error handling Version 0.4.3 (Initial Release)

hs2lib manual

NO support for NO support for NO support for Does not allow NO support for

infix constructors lists inside Applied types (e.g. Maybe [String]) lists inside tuples (e.g. (Int, [String])) for custom translation of types. quantified types. (e.g. M.Map)

Introduction

Creating a static library from a Haskell program is no easy task. It requires you to create marshalling code1 to translate all your Haskell structures. Create foreign export declarations, write some initialization code and more. Hs2Lib aims to make this much easier by providing very simple annotations that can be used to mark functions that needs to be exported. Calling C functions from Haskell is very popular and has a lot of tooling support. The tool c2hs generates the appropriate Haskell data types and bindings from a C header le. Hs2Lib however does the inverse and much more. Since this tool is part of the back end of Visual Haskell 2010, it is essential that we be able to generate C# code as well. The tool itself is platform independent, but has mostly been tested only on windows.

De ning instances of the Storable instance for your datatypes. More on this in Chapter ??

1.1

Motivation
2 3

Throughout the years it has become increasingly easier to create shared libraries from Haskell programs when using GHC2 . The task however remains quite involved. The canonical example is the Adder program3 :

module Adder where

Syntax highlighting for Haskell provided by Alessandro Vermeulens lhs2tex-hl. Check it out at http://alessandrovermeulen.me/projects/lhs2texhl/

adder ::Int Int IO Int adder x y  return px y q foreign export stdcall adder ::Int

Int IO Int

Compiling the .DLL Adder.dll requires just three short commands:

ghc -c Adder.hs ghc -c StartEnd.c4 ghc -shared -o Adder.dll Adder.o Adder_stub.o StartEnd.o

StartEnd.c contains RTS Initialization and termination code. More about this in Chapter ??

hs2lib manual

This is somewhat disingenuous, the Haskell FFI report5 speci es Char,Int,Double,Float and Bool as basic foreign types exported by the prelude. The Adder example is thus limited to using types that are prede ned to be a Foreign type. No custom marshallers are de ned 6 since we do not export any user types. The steps that in general need to be taken to compile an arbitrary Haskell le are in a nutshell:
RTS FFI Storable GHC DLL

marshalling (similar to serialization) is the process of transforming the memory representation of an object to a data format suitable for storage or transmission. It is typically used when data must be moved between di erent parts of a computer program or from one program to another.

DEF

LIB

Figure 1.1: Compilation steps to produce a static lib RTS De nes the runtime system initialization and termination function. Without this le we would have no way of starting and stopping the Haskell runtime. FFI Declare the appropriate foreign export de nitions. Also create wrappers over the to be exported functions, which then exposes them using types that are supported by the Foreign Function Interface. Storable De ne marshalling code for user data types. This is done by providing a proper Storable instance for every data type that needs to be exported. DEF (optional) By default, everything that is a function is exported by GHC in the Exports table by GHC. This includes closures etc. To hide all the noise we can de ne our own DEF le, which then states which functions are to be put in the static libs export table. LIB (optional) If we want to use MSVC++ and compile C++ code which uses the Haskell DLL, then we also need a .LIB le for the linker, which is created in this step GHC The nal step is to use a set of GHC commands to link and compile everything together into a neat little package. Doing these steps takes a long time. They are also quite repetitive and easy to automate. Which is exactly what Hs2Lib does. 7

NOTE: This tool does not use, nor keep the stubs generated from GHC. It produces its own includes.

2
2.1

Examples

This chapter shows 4 examples of small case studies to illustrate how to use Hs2lib. The rst one will be rather elaborate, showing how to call this tool from all three code generators. The rest will all be a bit more compact.

Simple Arithmetic

The rst example involves a set of simple arithmetic functions.

module Arith where -- @@ Export summerize::r Int s r Int s Int summerize x y  sum px y q -- @@ Export single::Int r Int s single x  r 1 . . x s

Figure 2.1: Simple Haskell Arithmetic The code is quite simple. summerize takes two list of integers and calculates their collective sum, whereas single takes an integer x and returns a list which contains the enumeration from 1 to x. The functions to be annotated have already been marked using the export annotation. In order to compile this example the only command needed is hs2lib Arith. PS C:\Examples> hs2lib Arith 9

10

hs2lib manual

Linking main.exe ... The directory should now contain the following les Directory: C:\Examples Mode ----a---a---a---a---a---a---a--LastWriteTime ------------5/30/2011 1:19 AM 5/30/2011 1:17 AM 5/30/2011 1:19 AM 5/30/2011 1:20 AM 5/30/2011 1:20 AM 5/30/2011 1:20 AM 5/30/2011 1:20 AM Length -----534 159 1973 6874624 711 1250 5630 Name ---Arith.hi Arith.hs Arith.o Hs2lib.dll Hs2lib.h Hs2lib_FFI.h HSHs2lib.a
1 2

The default namespace used by the tool is "Hs2lib"1 . The library produced has also already been stripped2 . By default, manual RTS initialization needs to be done. If we were to look at the exports table for Hs2lib.dll3 we would see the following output Microsoft (R) COFF/PE Dumper Version 10.00.40219.01 Copyright (C) Microsoft Corporation. All rights reserved.

This can be changed by using the -n ag

The only extra shrinkage that can be gained now is by packing the le using something like upx. The smaller size comes with some slight overhead in start-up time. UPX will not be executed by Hs2lib 3 dumpbin /EXPORTS Hs2lib.dll

Dump of file Hs2lib.dll File Type: DLL Section contains the following exports for Hs2lib.dll 00000000 4DE2D4AC 0.00 1 8 8 characteristics time date stamp Mon May 30 01:20:12 2011 version ordinal base number of functions number of names name HsEnd@0 HsStart@0 _HsEnd@0 _HsStart@0 _single@8 _summerize@16 single@8 summerize@16

ordinal hint RVA 1 2 3 4 5 6 7 8 ... 0 1 2 3 4 5 6 7 00002299 00002258 00002299 00002258 0000204C 00001F54 0000204C 00001F54

examples

11

We can see that thanks to the .DEF4 le only the functions we wanted have been exported. Along with the RTS initialization functions HsStart and HsEnd.

See Section ?? for details

C With GCC and Netbeans

The rst call well make is from C, using the gcc compiler and the Netbeans 6.8 IDE. We start o by creating an new C project as depicted in gure 2.2

Figure 2.2: New project dialog of Netbeans 6.8 In the le main.c add the following code: #include <stdio.h> #include <stdlib.h> #include "Hs2lib_FFI.h" int main(int argc, char** argv) { HsStart(); int* foo foo[0] = foo[1] = foo[2] = = (int*)malloc(sizeof(int)*3); 2; 5; 10;

printf("Sum result: %d\n", summerize(3, foo, 3, foo)); free(foo); int count; int* bar = single(8, &count);

12

hs2lib manual

printf("Single result: (%d)\n", count); int i; for(i = 0; i < count; i++) printf("\t%d\n", bar[i]); free(bar); HsEnd(); return (EXIT_SUCCESS); }

This test tests the functions summerize and single. It also tests the ability to pass lists back and forth. If we try to run this project building will fail. This is because theres still some settings we need to change. So lets open up the project settings dialogue.

Figure 2.3: Linker dialog of Netbeans 6.8

The rst thing we need to change is the linker settings. In particular the Additional Library Directories and Libraries values need to be changed to the appropriate values. 5 In addition to the Linker, the Include Directories of the C Compiler settings also have to be changed.

In this case, C:\Examples and HSHs2lib respectively.

examples

13

Figure 2.4: C Compiler Include Path dialog of Netbeans 6.8 As mentioned before, Hs2lib comes with a set a prede ned includes that support the generated code. These les are located in %AppData%\cabal\Hs2lib-version \Includes in windows6 . Next to the Hs2lib include directory, we also need to include the folder in which the generated header les were placed for the current project7 . The last thing that we need to change is the Run Directory. This sets the CWD of the executable. This needs to be set to the same folder which has our generated .DLL le, so that it is now in the path of the executable.

Note that most compilers dont expand this path. So Its recommended to expand them before hand and then add them to the Include Directories. You can do this by just opening up the folder in explorer, which will show you the full path. 7 C:\Examples in this case

Figure 2.5: Run CWD Path dialog of Netbeans 6.8 The project can now be compiled and run, if all goes well, the following output is displayed:

14

hs2lib manual

Figure 2.6: Arith results of Netbeans 6.8

Microsoft Visual C++

This section shows how to use the generated dll from a C++ compiler such as Microsofts Visual C++. While the C compiler used the generated ".a" le, the MSVC++ compiler requires a di erent interface le: A Library import le (.lib). This can be easily generated, so lets start with that. If the tool lib.exe is on your path, Hs2lib can be instructed to generate the .lib le automatically by using the lib or -M ags. PS C:\Examples> hs2lib --lib Arith Linking main.exe ... Done. There will now be a le Hs2lib.lib among the outputs. If lib.exe is not on your path then you need to manually generate the .lib le. To do this we need the .DEF le that was used by the tool. By default Hs2lib automatically deletes this le when performing its cleanup actions. To instruct it to keep this le, the ag incl-def or -d can be used. PS C:\Examples> hs2lib --incl-def Arith Linking main.exe ... Done. The next step is to open a Visual Studio command prompt and navigate to the folder containing the .DEF le. The .lib le can be easily generated then using the command lib /DEF:Hs2lib.def C:\Examples>lib /DEF:Hs2lib.def Microsoft (R) Library Manager Version 10.00.40219.01 Copyright (C) Microsoft Corporation. All rights reserved. LINK : warning LNK4068: /MACHINE not specified; defaulting to X86 Creating library Hs2lib.lib and object Hs2lib.exp With the .lib le at hand, we can now create an empty MSVC++ Project.

examples

15

Figure 2.7: New C++ Project Dialog Visual Studio 2010 We want a Win32 Console project, and also want an empty project.

Figure 2.8: Empty C++ Project Wizard Visual Studio 2010 The C++ Source le contains the following code: #include <stdio.h> #include <stdlib.h> #include "Hs2lib_FFI.h" int main() {

16

hs2lib manual

HsStart(); int* foo foo[0] = foo[1] = foo[2] = = new int[3]; 2; 5; 10;

printf("Sum result: %d\n", summerize(3, foo, 3, foo)); delete foo; int count; int* bar = single(8, &count); printf("Single result: (%d)\n", count); int i; for(i = 0; i < count; i++) printf("\t%d\n", bar[i]); free(bar); HsEnd(); return 0; } Just like before we test the exact same functions and expect the exact same results to be returned. Before we can run this however we need to change a few settings again. First of which, is the Linker dependencies.

Figure 2.9: Linked Input settings Visual Studio 2010

examples

17

We add Hs2lib.lib to the Additional Dependencies value of the Linker. This allows it to nd our exported functions. The next step is modify the Include- and Library Directories, we add the same values we added in Figure 2.4 and Figure 2.3.

Figure 2.10: VC++ Directories Visual Studio 2010

And lastly we need to modify the working directory using the same values as in Figure 2.5.

Figure 2.11: VC++ Current Working Directory Visual Studio 2010

If we were to compile and run this project, We would unfortunately receive the following error message:

18

hs2lib manual

Figure 2.12: VC++ Heap corruption error

This is because of a mismatch between memory allocators. The call to free is corrupting the heap because the free function being used has a di erent allocation style then the allocato(malloc) that reserved the memory. The di erence is in the version of msvcrt.dll being used. GHC which uses GCC to link, links against the msvcrt.dll 8 which is the standard C runtime on windows. Unfortunately, every version of Microsoft Visual C++ brings with it a new version of the runtime. The version of visual studio being used here9 uses the runtime msvcrt100.dll10 . These di erent versions of the runtime use di erent incompatible allocators. The standard solution/convention is to always allocate and deallocate memory on the callers side 11 . I prefer not to use this approach, It would require much more work to allocate memory for the Haskell data types. Instead, I provide in Hs2lib speci cally for this case a DLL12 which provides us with the same free function being used by msvcrt.dll. It is infact a simple redirect which just re-exports the free function. This Support le exports a function freeNative which should be used to free memory allocated in Haskell Land. 13 Using this alternative free function, our project nally runs correctly.

http://en.wikipedia.org/wiki/ Microsoft_Windows_library_files# Msvcrt.dll

9

Visual Studio 2010 is the 10th version of visual studio. 10 http://msdn.microsoft.com/ en-us/library/abx4dbyh(v=VS.100) .aspx 11 It is probably for this reason that Win32 APIs require you to give a preallocated structure which will be lled in by the call instead of just returning a structure that it allocated. 12 WinDllSupport.dll
13

Include the le WinDllSupport.h and link to WinDllSupport.lib. Keep in mind that de calling convention for this DLL is Cdecl.

Figure 2.13: VC++ Arith result

examples

19

Visual C#
The last part of this case study is using the C# code generator. Despise what intuition would suggest, this is actually quite straight forward. Hs2lib can generate the C# binding automatically for us. To do this, we need to pass it the c# ag.

PS C:\Examples> hs2lib --c# Arith Linking main.exe ... Done.

This generates a le Hs2lib.cs containing everything we need. We can now create the new empty C# project.

Figure 2.14: C# New project dialog

After we have our empty project, we can now add a reference to the FFI.dll managed dll. This contains a plethora of helper functions to use the DLL from C#14 . Everything both generated and in this helper library will be in the WinDll namespace. The code that has just been generated can be found in WinDll.Generated and the types in WinDll.Generated.Types.

14

Highlights include a free function, methods to read Maybe values and much more.

20

hs2lib manual

Figure 2.15: C# Add Reference Dialog

15

15

After adding the references we can add the two les we need for this project.

Remember to rst expand the path depicted by %AppData%.

Figure 2.16: C# Existing les Dialog

We need to add both the DLL and the Hs2lib.cs le that was generated. By adding the DLL we can instruct MSBuild to copy the DLL to the output folder. This enables us to not have to set the Working directory of the project. First click on Hs2lib.DLL in the solution explorer and set its build action to Copy if newer.

examples

21

Figure 2.17: C# Hs2lib.DLL property Dialog In Program.cs add the following lines using using using using System; WinDLL.Generated; WinDLL.Utils; System.Runtime.InteropServices;

namespace CsArith { class Program { static void Main(string[] args) { Hs2lib.HsStart(); unsafe { IntPtr fooPtr = Marshal.AllocHGlobal(sizeof(int) * 3); int* foo = (int*)fooPtr.ToPointer(); foo[0] = 2; foo[1] = 5; foo[2] = 10; Console.WriteLine("Sum result: {0}", Hs2lib.summerize(3, foo, 3, foo)); Marshal.FreeHGlobal(fooPtr); int count; int* bar = Hs2lib.single(8, &count); Console.WriteLine("Single result: ({0})", count);

22

hs2lib manual

int i; for (i = 0; i < count; i++) Console.WriteLine("\t{0}", bar[i]); FFI.free(bar); } Hs2lib.HsEnd(); return; } } }

Before we can compile this code we need to enable unsafe code compilation in the compiler. We can do this by going into the projects property window.

Figure 2.18: C# Project property Dialog

After this we can nally run the program, and the result should be the same as before.

examples

23

Figure 2.19: C# Result Dialog

2.2

User data types

This case studies deals with how to export and read user de ned data types using Hs2lib. As with all preceding examples, well only cover the C examples. The example used here is the evaluation of simple expressions. module Expr where data Expr

 Add Expr Expr | Sub Expr Expr | Mul Expr Expr | Div Expr Expr | Parens Expr | Var Var | Val Int | Let String Expr Expr

type Var  String type Env  rpVar,Int qs -- @@ Export = eval -- | Evaluate an expression, throwing an exception if the variable is not found. foldExpr ::Expr Int foldExpr  foldE r s where foldE ::Env Expr Int foldE env pAdd e1 e2 q  pfoldE env e1 q pfoldE env e2 q foldE env pSub e1 e2 q  pfoldE env e1 q pfoldE env e2 q foldE env pMul e1 e2 q  pfoldE env e1 q pfoldE env e2 q foldE env pDiv e1 e2 q  pfoldE env e1 q div pfoldE env e2 q foldE env pParens e q  foldE env e foldE env pVar nm q  case lookup nm env of Nothing error p"Variable " nm " could not be found."q Just val val foldE env pVal val q  val

24

hs2lib manual

foldE env pLet x v e q

 let v  foldE env v env  px,v q : env

1 1 1

in foldE env 1 e This time while compiling were going to instruct Hs2lib to rename the project. This can be done by setting the default namespace16 PS C:\Examples> hs2lib .\Expr.hs -n Eval Linking main.exe ... Done. This creates a le Eval.DLL among other things. The exports of this DLL includes a function named eval. By specifying a name after the Export statement we can on an individual basis rename functions. Ill just assume that the user is familiar with C and no explain how to call the function. By looking at the prototype of the function its clear that its pretty straight forward. // eval :: Expr -> Int extern CALLTYPE(int) eval (Expr_t* arg1); struct Expr { enum ListExpr tag; union ExprUnion* elt; } Expr_t ;

16

This can be set with -n and will instead of using the name Hs2lib for output les, use the name supplied for all les.

2.3

Callbacks

Hs2lib supports Higher-order functions by making them callbacks.

-- @ Export myadd::pInt Int q Int myadd  p $ q

Int

Figure 2.20: Example exportable Higher-order function The generated header le would contain the following de nition // myadd :: CBF1 -> Int -> Int extern CALLTYPE(int) myadd (CBF1_t arg1, int arg2); // Callback functions typedefs // type CBF1 = Int -> Int typedef __stdcall int (*CBF1_t)(int); This is again rather straight forward, the higher-ordered arguments just become function pointers. One important thing is, if you use any lists or IO action in the higher-ordered argument. (e.g. pInt r Int sq Int Int) then the argument must be in IO (e.g. pInt IO r Int sq Int IO Int).

examples

25

The reason is that creating and reading arrays to lists is an IO operation. If you omit this the tool will still compile, but be forced to use unsafePerformIO. Calling this function is very easy, the following code illustrated how. int __stdcall iadd(int i){ return i+i; } ... int ret = myAdd(iadd,8); printf("results 1: %ld\n", ret )

2.4

Sessions

One particular neat thing is that you can pin certain pieces of Haskell data in memory. This information is not accessible outside of Haskell and thus doesnt require any marshalling information to be generated. This is handy for when you need to implement session or caching in your haskell code. The following code example should explain it: type Context  StablePtr pIORef q type Env  Context r Int s -- @ Export initEnv::IO Env initEnv  newStablePtr  newIORef -- @ Export freeEnv::Env IO pq freeEnv  freeStablePtr -- @ Export add::Env Int IO pq add env val  do env_value deRefStablePtr env modifyIORef env_value ppval : q $! q -- @ Export = sumPrime sum 1 ::Env IO Int sum 1 env  do env_value deRefStablePtr env current readIORef env_value return $ sum current main::IO Int main  do env initEnv add env 1 add env 1

rs

26

hs2lib manual

add env 1 add env 1 val sum 1 env freeEnv env return val When called, initEnv will return a void pq which you cant do anything with but pass as arguments to the Haskell functions. This is used extensively in Visual Haskell itself.