Extracting WiX Bundled MSI Files Without Running The Installer

This article is one in my series focusing on what I learned creating a Windows Installer using the WiX toolkit’s “Managed Bootstrapper Application” (Custom .NET UI) framework. Each post builds upon the previous posts, so I would suggest you at least skim through the earlier articles in the series.

Here’s an index into the articles in this series:

You can extract an embedded .msi from your bundle while it’s running if you’re using a custom bootstrapper application, then extract the contents of that .msi by using the WiX SDK.

Below is a walkthrough of just extracting the files and laying them down on the filesystem. It does not run any of the CustomActions (I’ll post how to do that later). I do this all through a class I call MsiFileExtractor – the full source is at the end of this posting, so I’ll walk through the methods a little at a time.

First, you need to know a couple of things (which I’ve put into the cxtr):

  • The path to the executing bundle file, in which the msi files are embedded. You can get this through the WixBundleOriginalSource engine variable
  • This list of features to be installed for each MSI. I’m assuming you don’t just want to install the defaults and that you want to select which specific features to install.

Using my MsiFileExtractor class, you’ll need to pass into the cxtr the target folder where you want the files extracted into, the list of features to be installed for each MSI, and a reference to the MBA object itself. Then RunInstall() and watch the magic happen.

Here’s a high-level overview of what happens:

Step 1: extract MSIs from bundle

The ExtractComponentInstallersFromBundle method is used to extract the .msi files from inside the bundle .exe and place them into a temporary working folder. This is actually pretty straightforward using the WiX Unbinder class. This throws the MSI files into a AttachedContainer subfolder of our temp working folder.

step 2: explode MSIs

Now, we loop through each of the MSIs you want to install and use the InstallPackage method to actually extract the files from them into your target folder.

To do this, I used the WiX InstallPackage class, which wraps our MSI package. Keep in mind, the MSI package is basically just a database with the files embedded. So we’ll interact with the database using SQL statements in order to change the behavior of the installer.

step 2a: set the install location in the MSI database

The first thing we need to do is update the install folder. In your .wix files, you should have defined a bunch of Directory targets, like this:

  <Fragment>
    <Directory Id="TARGETDIR" Name="SourceDir">
      <Directory Id="ProgramFilesFolder">
        <Directory Id="APPLICATIONFOLDER" Name="MyCompany">
          <Directory Id="ProductFolder" Name="Demo Product Installer">
            <Directory Id="Cmp1Folder" Name="Component1" />
            <Directory Id="Cmp2Folder" Name="Component2">
                <Directory Id="Cmp3Folder" Name="Component3" />
            </Directory>
          </Directory>
        </Directory>
      </Directory>
    </Directory>
  </Fragment>

Note that I have a top-level folder APPLICATIONFOLDER defined just under the ProgramFilesFolder. All of my files get installed at or below this level. So I need to change the location of this folder in the MSI database to allow me to install somewhere other than Program Files. The way I achieve this is by setting it to “.” (the current folder), which will override everything above it in the folder hierarchy as well. I do this in the UpdateMSIDirectoryTableToUseNewApplicationFolder method by changing the DefaultDir value in the Directory table.

step 2b: set the installable features in the MSI database

As I mentioned above, the Level value for each feature determines if it’s installed by default of not. Anything that’s a Level 1 will be installed. Anything higher will not. So if all you want to install is the Level 1 features, you can skip this step. Otherwise, we need to modify the level values in the MSI’s database to change it to 1 for those features we want and to something higher than 1 for the features we don’t. I do this in the UpdateMSIFeatureTableToSetInstallableFeatures method by changing the Level value in the Feature table.

step 2c: create a Session

Since we’ll basically be using MSI/WiX to “install” these files, and they require a Session to run, we need to create our own session object. This is done using the OpenPackage method — but we need to tell it to ignoreMachineState, otherwise it will look for this MSI in the system registry and get all confused. I also set the UI level to Silent to prevent the MSI’s UI from showing. Make sure to store the old UI level so you can restore it when you’re done.

You also need to set the Properties for the session. I do this in SetEngineVarsInSession by copying the properties I want from the Bootstrapper’s StringVariable. Modify this as you need.

step 3c: the real work

Now we’re ready to actually extract the files, which I do in ExtractPackageFiles. I do this by searching for all of the features in the Feature table with a Level value of 1 (see above about setting that value), then finding all of the entries in the FeatureComponents table for those features, then finding all of the entries in the File table for those components and keeping a list of everything that needs to be installed.

Then, I call InstallPackage.ExtractFiles with that list and BAM I have installed files!

step 3: cleanup

We nuke the temp working folder to keep things tidy.

And here’s the full source code:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Microsoft.Deployment.WindowsInstaller;
using Microsoft.Deployment.WindowsInstaller.Package;
using Microsoft.Tools.WindowsInstallerXml;
using Microsoft.Tools.WindowsInstallerXml.Bootstrapper;
using FileAttributes = System.IO.FileAttributes;

/// <summary>
/// Action will extract the files in the provided MSIs into the provided folder.
/// </summary>    
internal class MsiFileExtractor 
{
    private readonly string _bundlePath;
    private readonly string _targetFolder;
    private readonly Dictionary<string, List<string>> _msiAndFeaturesToInstall;
    private readonly BootstrapperApplication _bootstrapperApplication;

    /// <summary>
    /// </summary>
    /// <param name="targetFolder">File path to the parent folder where files should be installed</param>
    /// <param name="msiAndFeaturesToInstall">collection of msi names and the features in each MSI to be installed</param>
    /// <param name="bootstrapperApplication">instance of the MBA</param>
    public MsiFileExtractor(string targetFolder, Dictionary<string, List<string>> msiAndFeaturesToInstall, BootstrapperApplication bootstrapperApplication)           
    {
        _bootstrapperApplication = bootstrapperApplication; 
        _bundlePath = _bootstrapperApplication.Engine.StringVariables["WixBundleOriginalSource"]; 
        _targetFolder = targetFolder;
        _msiAndFeaturesToInstall = msiAndFeaturesToInstall;        
    }

    /// <summary>
    /// Extract the bundle's embedded MSIs, extract the files from each and run 
    /// any custom actions
    /// </summary>        
    public void RunInstall()
    {
        var workingDir = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());

        //step 1: extract MSIs from bundle
        ExtractComponentInstallersFromBundle(_bundlePath, workingDir);

        //step 2: explode MSIs
        foreach (var msiName in _msiAndFeaturesToInstall.Keys)
        {
            var featuresToInclude = _msiAndFeaturesToInstall[msiName];
            InstallPackage(workingDir, msiName, featuresToInclude);
        }

        //step 3: cleanup
        if (Directory.Exists(workingDir))
        {
            var di = new DirectoryInfo(workingDir);
            ClearReadOnly(di);

            Directory.Delete(workingDir, true);
        }

    }

    /// <summary>
    /// Recursively clear the ReadOnly attribute from all files and subfolders
    /// for the given folder.
    /// </summary>
    /// <param name="directory"></param>
    private void ClearReadOnly(DirectoryInfo directory)
    {
        directory.Attributes &= ~FileAttributes.ReadOnly;

        foreach (var file in directory.GetFiles())
        {
            file.Attributes &= ~FileAttributes.ReadOnly;
        }

        foreach (var subDir in directory.GetDirectories())
        {
            ClearReadOnly(subDir);
        }
    }

    /// <summary>
    /// For a single MSI, extract the files and run any custom actions
    /// </summary>
    private void InstallPackage(string workingDir, string msiName, List<string> featuresToInclude)
    {
        var msiPath = Path.Combine(workingDir, "AttachedContainer");
        var msiFilePath = Path.Combine(msiPath, msiName);

        Console.WriteLine("Extracting package: {0}", msiName);

        using (var msiPackage = new InstallPackage(msiFilePath, DatabaseOpenMode.Transact) { WorkingDirectory = _targetFolder })
        {
            UpdateMSIDirectoryTableToUseNewApplicationFolder(msiPackage);

            UpdateMSIFeatureTableToSetInstallableFeatures(msiPackage, featuresToInclude);

            //Set the InternalUI level on the MSI so that it doesn't show when we open the Session object
            var prevUILevel = Microsoft.Deployment.WindowsInstaller.Installer.SetInternalUI(InstallUIOptions.Silent);
            try
            {
                using (var session = Microsoft.Deployment.WindowsInstaller.Installer.OpenPackage(msiPackage, ignoreMachineState: true))
                {

                    SetEngineVarsInSession(session);
                    ExtractPackageFiles(session, msiPackage);

                }
            }
            finally
            {
                Microsoft.Deployment.WindowsInstaller.Installer.SetInternalUI(prevUILevel);
            }

            msiPackage.Close();
        }

    }

    private void ExtractPackageFiles(Session session, InstallPackage msiPackage)
    {

        //using a hashset to ensure distinct list
        var fileKeysToInstall = new HashSet<string>();
        foreach (var feature in session.Features)
        {
            var featureLevel = msiPackage.ExecuteIntegerQuery(
                string.Format("SELECT `Level` FROM `Feature` WHERE `Feature` = '{0}'",
                              feature.Name))
                .First();

            if (featureLevel != 1) continue;

            var featureComponents =
                msiPackage.ExecuteStringQuery(
                    string.Format("SELECT `Component_` FROM `FeatureComponents` WHERE `Feature_` = '{0}'",
                                  feature.Name));

            foreach (var installableComponent in session.Components
                .Where(cp => featureComponents.Contains(cp.Name)))
            {
                var componentFileKeys = msiPackage.ExecuteStringQuery(
                    string.Format("SELECT `File` FROM `File` WHERE `Component_` = '{0}'",
                        installableComponent.Name));

                foreach (var fileKey in componentFileKeys)
                {
                    fileKeysToInstall.Add(fileKey);
                }
            }
            msiPackage.ExtractFiles(fileKeysToInstall);
        }

        //temp folder is left behind, so we need to clean it up
        //see http://sourceforge.net/p/wix/bugs/2330/
        var witempPath = Path.Combine(_targetFolder, "WITEMP");
        if (Directory.Exists(witempPath))
        {
            ClearReadOnly(new DirectoryInfo(witempPath));
            Directory.Delete(witempPath, true);
        }
    }

    /// <summary>
    /// Override the APPLICATIONFOLDER in the MSI database so
    /// that files are installed to the target folder.
    /// </summary>        
    private void UpdateMSIDirectoryTableToUseNewApplicationFolder(InstallPackage msiPackage)
    {
        //Update folders
        msiPackage.WorkingDirectory = _targetFolder;

        //The "APPLICATIONFOLDER" special name is used to denote the application top-level folder.
        //In this case, the user has selected their own top-level folder, so we need to
        //replace the existing value with the "." special value, which denotes the extracted
        //folder.               
        var installFolderName = "APPLICATIONFOLDER";
        if (msiPackage.Directories.ContainsKey(installFolderName))
        {
            var record = new Record(".", installFolderName);
            msiPackage.Execute("UPDATE `Directory` SET `DefaultDir` = ? WHERE `Directory` = ?",
                               record);
        }

        msiPackage.UpdateDirectories();
        msiPackage.Commit();
    }

    private void UpdateMSIFeatureTableToSetInstallableFeatures(InstallPackage database, List<string> featuresToInstall)
    {
        bool installDefault = !featuresToInstall.Any();
        if (installDefault) return;   

        var packageFeatures = new List<string>();

        if (database.Tables.Contains("Feature"))
        {
            packageFeatures.AddRange(database.ExecuteStringQuery("SELECT `Feature` FROM `Feature`"));

            foreach (var feature in packageFeatures)
            {
                //a Level of "1" will be installed by default, anything higher than that will not,
                // so by setting all others to "100", they will not be installed/extracted
                int newInstallLevel = featuresToInstall.Contains(feature) ? 1 : 100;

                Console.WriteLine("Setting feature {0} to install level {1}", feature, newInstallLevel);

                using (var record = new Record(newInstallLevel, feature))
                {
                    database.Execute("UPDATE `Feature` SET `Level` = ? WHERE `Feature` = ?",
                                     record);
                }
            }
        }
        else
        {
            throw new Exception("Feature table not found");
        }
        database.Commit();

    }

    private void SetEngineVarsInSession(Session session)
    {
        var propertiesToCopy = new[] {"PROP1", "PROP2"};
        foreach (var property in propertiesToCopy)
        {
            var propValue = _bootstrapperApplication.Engine.StringVariables[property];
            session[property] = propValue;
        }
    }

    private void ExtractComponentInstallersFromBundle(string bundlePath, string tmpFolder)
    {
        Unbinder unbinder = null;

        try
        {
            unbinder = new Unbinder();
            unbinder.Message += MessageEventHandlerMethod;
            unbinder.Unbind(bundlePath, OutputType.Bundle, tmpFolder);
        }
        finally
        {
            if (null != unbinder)
                unbinder.DeleteTempFiles();
        }

    }

    private void MessageEventHandlerMethod(object sender, MessageEventArgs e)
    {
        Console.WriteLine("MESSAGE: ID: {0}, LEVEL: {1}, RESOURCE: {2}, LINES: {3}",
                   e.Id,
                   e.Level,
                   e.ResourceName,
                   e.SourceLineNumbers);
    }

}
Posted in WiX

How I Wired My Humidifier Into My Nest Thermostat

A little while back, I wrote about my first year using a Nest thermostat and since then several people have asked me for details on how I wired up my whole-house humidifier. I hesitated for a long time to post much detail, since it’d be really easy to completely fubar your HVAC, humidifier and Nest all in one fell swoop. But, after much mental consternation, I’ve decided to provide some detail, but with this really big disclaimer:

BIG OL’ DISCLAIMER:

I am not an electrician, an HVAC specialist, a trained Nest consultant or frankly anyone you should trust. I provide this information solely as a reference implementation and provide no warranty that it’s accurate or even safe. You should consult an electrician or specialist before doing anything you read here. Incorrect wiring can cause massive damage to your HVAC system, or even result in fire.

In determining how to wire things up, I relied on advise from the good folks at The Thermostat Forums, and you can also look to the DIY StackExchange site.

What’s a Humistat?

From my original post, you may recall:

A “Humidistat” is the thermostat equivalent for a whole-home humidifier. It generally has an external sensor that reads the current humidity levels of the outside air, as well as a built-in sensor for the inside air. When the humidity inside is too low, such as in the winter when the heater is drying out the air, it will evaporate water into the hot air leaving the furnace and into the house.

My Wiring

Again, from my original post:

Now, connecting the Nest to the humidifier was not a straight-forward process. Humidistats come in two types, which basically comes down to 1-wire or 2-wire controls. If yours only uses 1 wire, good news, you can easily and quickly hook it up to the Nest. But most utilize 2 wires, much like a light switch, where a low voltage signal travels up one wire, the humidistat acts as a switch, closing the circuit for the voltage to travel down the second wire back to the humidifier to signal it should turn on. So in order to use the Nest, which only has 1 wire to control the humidifier, you must install a relay circuit, which, if done wrong, can damage your HVAC main board, your Nest and your humidifier — so Nest asks that you have a professional do the work. However, I am comfortable doing that kind of wiring and was able to install the relay and get the Nest hooked up on my own.

So, here’s what the original wiring looked like:

You’ll notice on the HVAC control board, there are five terminals, labeled R, C, W, Y and G. My understand, which is confirmed by this eHow Guide and this Thermostat Wiring Explained article, the terminals are:

  • R: Power (usually has a red wire)
  • C: Common
  • W: Heating (usually a white wire)
  • Y: Cooling (usually a yellow wire)
  • G: Fan

I used a 6AZU2 relay, which seems to be a commonly used relay for people doing exactly this (see the comments on the Amazon page). I used crimp-on spade connectors on the end of the wires to connect them to the relay terminals.

Here is what the new wiring looks like:

So when the Nest sends the signal on the “*” wire, the relay closes the circuit between the two lines on the humidifier, engaging it.

You can see someone else describe the same scenario, with a wiring diagram, on this site

.

Everyday Usability Design: The Good and Bad – All At Once

Very often, particularly while helping someone non-technical with a technical problem, I think to myself “this would have been so much simpler/less confusing if the designer would have put just a bit more thought into the user experience.” Similarly, I occasionally come across something where I say “man, why don’t more people do this.”

Yesterday, I had each of those reactions — to the same piece of technology! So I had to share. What was this magical piece of technology that mixed both the good and bad design experiences? Why, a gas pump, of course.

Take a look at this portion of the pump user interface. On the top, you have a credit card reader, and on the bottom, a keypad.

The Good:

The credit card magnetic stripe reader. They are everywhere: gas pumps, ATMs, grocery stores — pretty much every brick-and-mortar retail outlet you can find has some form of this device. And it seems like every time I use one, I swipe my card, only to realize I had the mag stripe on the wrong side and have to flip it and swipe again.

So what’s so good about this one? Well, take a look at the little helper pictures on the device. Did you see it? You can swipe the card with the mag stripe on… wait for it… either side. There is basically no way the user can do it wrong. The manufacturer has created a pit of success for me to fall into by adding a second receiving device on the other side. I LOVE IT!

This was the first thing I noticed on the gas pump, and I rejoiced. It put me in a better mood.

The Bad:

Immediately after swiping my card, the pump asked me to enter my zip code for credit card verification, followed by the dreaded “Do you want a car wash too?” prompt. Entering the zip code for credit card verification was straight forward — I had a set of what is clearly a numbered keypad to use. So I tapped it in. No problem.

Now I get the “Do you want a car wash?” prompt. In my head, as I recall this little episode playing out, I hear this 80s era computer voice asking “Shall we play a game?”

Ok, I can do this — “just press ‘No'” I tell myself. So I reach down to the keypad and… um.. where’s the ‘No’ key?

Now, it’s hard to tell from the picture, but the 0-9 keys, plus the yellow “Clear” and green “Enter” are all raised, like an early 90s telephone. So this is a very familiar interface to me. The yellow and green colored buttons stand out to me — clearly the designer gave them color to draw my eye to them. Then there are these stickers on the left and right side, which are also of various colors.

Here’s how it played out in my head:

Ok, so, now, how to say “no”? Let’s see, what are my options? There’s no clear “No” key, but there is a “No” sticker next to the “Clear” key and they are both yellow — so maybe that’s it. But wait, there’s a red “Cancel” label next to the green “Enter” key, and red usually means No — but green means confirm and the key itself is green. Better go with the yellow “Clear” key.

I hit the key — the machine beeps. The screen refreshed: “Do you want a car wash?”. Ok, maybe it didn’t register… so I pushed it again. And back comes the car wash prompt. (or, in my head “How about Global Thermonuclear War?”….) Clearly, this is not the right button.

To make a short story less long, here’s where I landed: Turns out, those “stickers” on the side aren’t just labels — they’re actual buttons. I probably should have taken the picture from a slight angle to make it more clear how much the buttons stick out — but to me, it was not at all obvious that these were interactive components, especially since they are juxtaposed immediately with what are clearly interactive components. So I eventually hit the yellow “No” sticker and went on my way.

There is so much going on with this little 3″ x 3″ square of user interface.

  • keys that are buttons
  • stickers that are buttons
  • stickers in yellow, red, green and white (what do those colors mean???)
  • keys in black, yellow and green

As a user, I don’t know what this all means, which means I’m likely to get it wrong, at least once. Had the buttons all been the same physical design, I would have immediately known which button to push.

It makes me wonder if the two parts of this user interface were designed by the same people.