It’s somewhat amazing to me that in 2010 there should still be any problems or confusion surrounding how to build Windows Installer (.vdproj) projects in MSBuild.  But, as Edna Mode says, “Yet here we are, darling.”

First problem: figuring out the syntax.  Well that was the first, last, and in-between problem, but when I typed the following into PowerShell:

devenv.exe /?

I got Visual Studio to fire up and show a message box with some command-line help.  Unfortunately, the version of Visual Studio was 2005, not 2008 as I might have expected.  Turns out my $env:path environment variable was riddled with references to *Microsoft Visual Studio 8* when really I wanted *Microsoft Visual Studio 9.0*.

OK, I can fix that temporarily by navigating to:

c:\program files\microsoft visual studio 9.0\common7\ide

And re-running the command

devenv.exe /?

Yet I still got the IDE to open up with a message box full of command-line and switch information.

Aha! I thought.  Perhaps I need to run the .com version of the program, not the .exe.  I am operating in a command-line environment, after all.

Changing to:

devenv.com /?

Showed me an inline help list which matched the IDE message box, but at least I’m operating within PowerShell 100% now.

OK, so about the format of the command line to build a VDPROJ setup project.  At first I thought that it would be something simple like:

<Exec Command=”$(DevEnv) $(ProjectFile) /Build ‘Debug’”/>

Where $(DevEnv) was the path to devenv.com and $(ProjectFile) was the path to my .vdproj file.  Well, two problems.  First was the single- vs. double-quoting issue.  The Exec task’s Command attribute takes a quoted string.  It turns out you have to use XHTML " pseudo-characters to delineate the arguments.

The second problem was more subtle.  As it turns out, you have to pass in the SOLUTION file first, then you have to tell devenv what PROJECT FILE you want built.  Very counter-intuitive.

Here’s my final target in my .msbuild file:

<Target Name="Foo">
        <PropertyGroup>
            <DevEnv>$(ProgramFiles)\Microsoft Visual Studio 9.0\Common7\IDE\devenv.com</DevEnv>
            <SolutionFile>$(MSBuildProjectDirectory)\MySolution.sln</SolutionFile>
            <ProjectFile>$(MSBuildProjectDirectory)\MySetupProject\MySetup.vdproj</ProjectFile>
            <Configuration>Debug</Configuration>
        </PropertyGroup>
        <Exec
                Command="&quot;$(DevEnv)&quot; &quot;$(SolutionFile)&quot; /Rebuild &quot;$(Configuration)&quot; /Project &quot;$(ProjectFile)&quot; /ProjectConfig &quot;$(Configuration)&quot; /Log"
                ContinueOnError="false"
                IgnoreExitCode="false"
                WorkingDirectory="$(MSBuildProjectDirectory)" />
    </Target>

I still have to do some work to do a clean prior to the rebuild, and I’m not sure why the configuration needs to be specified twice, but this has gotten me working this weekend. Hope this helps some of you out.