GaragePi: Experimenting with languages

In my previous posts, (Garage Pi – My Raspberry Pi Playground, GaragePi v2: Temperature and Light, and GaragePi v3: Data Persistence and Visualization and Other Small Improvements) I described the Raspberry Pi-based system I had setup in my garage to detect when the garage door was left open or the light was left on and turn on an LED inside the house. It also reads the current temperature in the garage. This was all originally written in Python (with a node.js-based website added in the third post), but I wanted to play around with other languages on the Pi, so I followed it up with a port to JavaScript and .Net (using Mono).

I’m not going to go into too much detail in this post, because (as I hope to write up a post here shortly), I’ve since abandoned the RaspPi platform for this project and converted it over to an Arduino-based solution.

Some general notes:

In both cases, I tried to keep the layout of the code similar to the Python layout to provide better side-by-side comparisons of the codebase. In some cases, this may result in code that doesn’t match the languages’ general styling.

The hardware didn’t change for these — just the software. I also didn’t change the website or MongoDB, which let me switch back and forth between languages and compare results using the same db and website. (I added a “source language” field to the db records to help call out any oddness.)

All of this was done about a year ago (Feb 2015) — I’m just lazy and never got around to writing it up.

.NET/Mono Implementation

The .NET implementation, using Mono, was extremely straightforward. Now, in all fairness, this is the language I spend all day writing at my day job, so it’s clearly the one I’m most familiar with, but this is running on Linux and interacting with the hardware — this is supposed to be hard!

I used the “raspberry-sharp-io” library to interact with the GPIO pins. I had to add RaspPi 2 pin support and wrote a driver for the 1-wire temp sensor, both of which have since been merged to the main repository (open source pull requests for the win!). I also used a NuGet package for the MongoDB client.

If you’re thinking about going this route, this may not be a good guide. Since the time that I wrote this code, Microsoft has shifted .NET in major ways, including (pre-release) support for the next version of the framework to run natively on Linux. Additionally, Windows10 IoT edition will run natively on the RaspPi, so you could do this without Linux. At the time I went through this excersize, there wasn’t good GPIO/1-wire support in the Win10 IoT releases — that’s likely changed.

I initial expected the performance of running in .NET/Mono to be bad. While I’ve lost the raw numbers I collected, the performance generally was very good, both for CPU and memory utilization

You can find my code on GaragePi GitHub repo in the “dotNet” folder: https://github.com/johnmwright/GaragePi

JavaScript Implementation

Wow — this did not go well. Since several of my sensors rely on precise timing (the Ultrasonic Range Finder in particular), it did not fit well with the JavaScript callback model. I ended up using the r-pi-usonic npm module which is a Node module written in C so that it can better handle the timing. Note: the r-pi-usonic project is no longer maintained..

I also used the rpi-gpio npm module for GPIO pin interactions. For some reason, I had to pull it from a personal clone of the GitHub repo instead of from the npm website, though I don’t recall why. I think the version on npm was very stale at the time.

Ultimately, this solution never fully worked for me. It would periodically just die and I never got all the kinks worked it. It also took me significantly more time (like 2 orders of magnitude or more) to implement than Python and .NET combined!

I also investigated projects like Cylon.js, but ended up going deep into a rabbit hole only to hit a major roadblock. Again, this was a year ago, which in JS terms is lifetimes, so your millage may vary.

A big thanks to Clark Sell, Brett Slaski and Brandon Martinez, who worked through some of the uglier JavaScript bits with me. These guys are all on the That Conference Staff and we have a #microcontrollers Slack channel! You should consider going to That Conference is you’re reading my post and live anywhere near the US Midwest!

You can find my code on GaragePi GitHub repo in the “JavaScript” folder: https://github.com/johnmwright/GaragePi

GaragePi v3: Data Persistence and Visualization and Other Small Improvements

In my previous posts, Garage Pi – My Raspberry Pi Playground and GaragePi v2: Temperature and Light, I described the Raspberry Pi-based system I had setup in my garage to detect when the garage door was left open or the light was left on and turn on an LED inside the house. It also reads the current temperature in the garage.

Since then, I’ve expanded the system by:

  • Persisting the data into a MondgoDB database
  • Adding a separate NodeJS-based website to view the current readings and a graph of recent readings
  • Improved the logic for my SonicController

If you haven’t already, I suggest at least skimming through my previous posts.

Storing to a MongoDB database

I wanted to store the readings in a database so that I can review historical data. And, since one of the drivers of this project was for me to “play” with and learn technologies I haven’t used before, I wanted to utilize a Document Database. I also knew that I wanted to port my project to other languages (C# and NodeJS), so wanted a DB that was easily supported by each of those.

Installing the Software

I decided to use MongoDB and installed the version from Raspian (ie: I did apt-get install mongodb). There are several other resources online on how to get things installed and running, so I won’t detail them here. I just used the defaults for everything.

Note: I’m running the ‘jessie’ version of Raspian, which is basically the release candidate for the next version after ‘wheezy’. Unless you’ve specifically upgraded to ‘jessie’, you’re probably running ‘wheezy’ and may run into some issues with outdated packages.

I also installed pymongo, the mongodb client libraries for Python.

Updating My Code

Persisting the data was actually pretty straightforward.

First, I added my import statement for the client libs:

     from pymongo import MongoClient

Then, in my main loop method, I create an instance of the db client using the db name garagePi_database:

  dbclient = MongoClient()
  db = dbclient.garagePi_database

Note that if this db doesn’t already exist, it will be created as soon as you write a record to it.

Next, I create a record object, which is basically a key-value store. I made the choice not to store the raw readings and use the interpreted values instead. (ie: Instead of storing the distance measurement to the garage door, I’m storing whether the door is open or not). This is because I expect these values to vary when I start using other languages and I didn’t want to have to deal with that. I’m also storing the language used to store the data.

record = { "doorOpen" : garageDoorIsOpen,
           "lightOn"  : lightIsOn,
           "temp_F"   : temp_f,
           "timestamp": timestamp,
           "sourceLanguage": "python"
}

And finally, I save the record and print it’s generated Id:

 readingId = db.readings.insert(record)
 print("    readings posted to db with id {}".format(readingId))

Storing Less Data

Now, that’s all great, but I quickly realized this was storing a LOT of records (1 every 5 second, since that’s my polling interval), and I didn’t want that.

Instead, I really just want to know:

  • when the door is open (or recently closed)
  • when the light is on (or recently turned off)
  • what the temperature is over time

For the first two it’s pretty simple logic: if the value is true for garageDoorIsOpen or lightIsOn, or if that value has changed since the last reading, then save the record. This means I also need to keep a copy of the previous reading to determine if the value has changed.

The “temperature over time” requirement is a little more tricky. Ultimately, I decided I really only wanted readings every 15 minutes for the temperature. So I added some logic to determine if the current time’s minute value is 0, 15, 30, or 45 and save the record — but only once per minute. This only kicks-in if the door is closed and the light is off.

Also, since I’m actually writing this post after I completed the port to other languages, I’m storing the timestamp in UTC to avoid date parsing and formatting issues when writing and consuming the timestamps in multiple languages.

So, the culmination of all this logic looks like this:

from datetime import datetime, timedelta
//....

lastRecord = None

try:
  ledRun.turnOn()

  dbclient = MongoClient()
  db = dbclient.garagePi_database

  while True:
    timestamp = datetime.utcnow()
    print("Beginning Sensor Checks {}".format( timestamp))

    //....


    record = { "doorOpen" : garageDoorIsOpen,
               "lightOn"  : lightIsOn,
               "temp_F"   : temp_f,
               "timestamp": timestamp,
               "sourceLanguage": "python"
    }

    shouldSaveRecord = False
    if lastRecord is None:
      print("       + lastRecord is None")
      shouldSaveRecord = True
    else:
      if garageDoorIsOpen or garageDoorIsOpen != lastRecord["doorOpen"]:
        print("       + garageDoorIsOpen differs from lastRecord {}".format(lastRecord["doorOpen"]))
        shouldSaveRecord = True
      if lightIsOn or lightIsOn != lastRecord["lightOn"]:
        print("       + lightIsOn differs from lastRecord {}".format(lastRecord["lightOn"]))
        shouldSaveRecord = True

      alreadyRecordedForThisMinute = timestamp.minute == lastRecord["timestamp"].minute
      if not alreadyRecordedForThisMinute and (timestamp.minute == 0 or timestamp.minute == 15 or timestamp.minute == 30 or timestamp.minute == 45):
        print("       + recording due to 15 minute period")
        shouldSaveRecord = True


    if shouldSaveRecord:
      readingId = db.readings.insert(record)
      print("    readings posted to db with id {}".format(readingId))

    lastRecord = record
    time.sleep(SAMPLE_SPEED)

Adding a Website

You’ll see that I’ve added a new “Website” folder to my GaragePi GitHub repository. I’m not going to dig too deep into how I set this up, as there are so many options out there you can choose from and a lot of in-depth walk-throughs.

I’m using a node.js based site with express as the web framework and jade as a template engine.

Since this is node.js, I have a package.json file that defines all the library dependancies, so you can do npm install to download them.

Additionally, I’ve defined a start script, so you can do npm start to get the website up and running. There’s also a config.json that defines the db name/hostname/port and http port to use.

Note: You may get a warning about mongodb not being able to load js-bson. That’s ok and can be ignored.

{ [Error: Cannot find module '../build/Release/bson'] code: 'MODULE_NOT_FOUND' }
js-bson: Failed to load c++ bson extension, using pure JS version

There are a couple of custom bits that I want to describe here, though.

Getting access to the db client

Since the website will be reading the records from the MongoDB instance, it’ll need access to the db client. Instead of having each route fetch it’s own client, I’ve modified the app.js file to create a db client object and attach it to every request:

//DB config
var DbServer = mongodb.Server;
var Db = mongodb.Db;
var dbServer = new DbServer(dbconfig.host, dbconfig.port);
var db = new Db(dbconfig.dbName, dbServer, { safe: true });


// Website config
var app = express();


// Make our db accessible to our router
app.use(function (req, res, next) {
    req.db = db;  // <-- attach the db client to the request object
    next();
});

Fetching the Data

Since I’m using Express and Jade, I need to define the route (webpage path) and fetch the db records there. Then, I’ll define the view (layout) later.

My routes are in a routes folder, and I used the index.js file that comes with the out-of-the-box install and define the default \ route. Inside there, I define a function to fetch the db records for the last x days:

var recordsSince = function(db, dataSinceDate, callback) {

    db.open(function(err) {
        if (err) throw err;

        db.collection("readings", function(err, collection) {
            if (err) throw err;

            collection
                .find({ "timestamp": { "$gt": dataSinceDate } })
                .sort({ "timestamp": -1 })
                .toArray(function(err, results) {
                    if (err) throw err;                        
                    db.close();
                    callback(results);
                });
        });
    });
};

Then in the parent function, I call that fetch method to get the last couple of days worth of data, and call-out the most recent one so I can display it separately:

var numDaysToShow = 2;
var currentDate = new Date();
var dataSinceDate = new Date(currentDate.setDate(currentDate.getDate() - numDaysToShow));

recordsSince(req.db, dataSinceDate, function(records) {

    var tempReadings = records;
    var mostRecent = records[0];

    res.render('index', {
        reading: mostRecent,
        readings: tempReadings,
        showRecords: false
    });
});

Note: I have a showRecords variable here to toggle logic in my view to show a table of the raw values. This is useful while troubleshooting my graphing logic.

The res.render('index'...) call tells Jade to render the index view and bind the reading, readings and showRecords variables and make then available in the view.

That’s it for the route.

Displaying the Data

Now to display that data. Again, using Jade, I’ve defined some views in the views folder. I have an error.jade for showing error status (5xx and 4xx HTTP status). There’s a layout.jade that has the shared layout for all of the pages, including the page title, script and css includes (I’m including Bootstrap here) and defines a content block for the child pages to populate.

Then, in my index.jade, I define the content for my page using Jade’s markup language. Mine is broken into three logical sections: Current Status, Data Graph and Optionally Display the Datatable

Lesson Learned: Jade allows you to use tabs or spaces to start your lines, but not both. The editor I was using defaulted to using tabs, so I frequently got runtime errors saying something like “You used both spaces and tabs”. I ended up changing the editor’s settings to use four spaces when I hit tab instead of a tab character.

Current Status

The Current Status section displays the most recent record’s data via the bound reading value we passed in from the route above. So anywhere you see reading.XXX, that’s going to display the XXX property of the bound record.

Also note that you apply CSS styles to an HTML tag using .style notation, so span.glyphicon.glyphicon-download is rendered as <span class="glyphicon glyphicon-download">.

You’ll also see conditional logic using case when default logic blocks.

Also note that if you are only displaying the bound field, you can follow the HTML element with = and the bound field, like this dd= reading.timestamp. But if you’re adding any other stuff, like additional text, you have to use #{} to reference your bound item, like this: dd #{reading.temp_F} °F

Here’s the Current Status section:

  h1 Current Status
  dl.dl-horizontal
    dt Garage Door:
    case reading.doorOpen
      when false
        dd
          span.glyphicon.glyphicon-download
          span &nbsp;Closed
      default
        dd
          span.glyphicon.glyphicon-upload
          span Open
    dt Light:
    case reading.lightOn
      when true
        dd On
      default
        dd Off
    dt Temp:
    dd #{reading.temp_F} °F
    dt Last Record:
    dd= reading.timestamp

Which looks like this:

Data Graph

Then, in the second section, I use the D3 Javascript library to graph the data in the db records.

This consists of two parts:
The Jade markup, which pulls in an external javascript file graph.js and the d3 library, creates a <graph /> element, then executes the renderChart() function from the graph.js file, pointing it at the <graph /> element:

  script(src='http://d3js.org/d3.v3.min.js')
  script(src='/javascripts/graph.js' type='text/javascript')

  graph/
  script.
    renderChart('graph', !{JSON.stringify(readings)})

Now, the real work happens in the graph.js file (which is in the public\javascripts folder of the site). Here, I graph three data points: temperature (as a line graph), whether the garage door was open or not (as a red dot) and if light was one or not (as a yellow dot). Like this:

I’m going to try not to paste the entire file here, but this method took me a while to get right, so I’m going to describe it in some detail for my own reference and for anyone who wants to try to do the same on their own.

The contents of the file can be found in my GitHub repo and I’ll just describe pieces here.

Step 1: Set the dimensions of the canvas / graph
In this chunk, I define the screen size of the canvas onto which the graph is drawn.

Step 2: Define function to parse the date / time using ISO format
Here, I create a method to parse the date values in the database for use by the D3 library to plot the data. Note that the data is stored in UTC in the database in ISODate format, so this parse logic will convert it into a javascript date in the local timezone

// Define function to parse the date / time using ISO format
var parseDate = d3.time.format.utc("%Y-%m-%dT%H:%M:%S.%LZ").parse;

Step 3: Set the display ranges
Here, I define the displayable ranges for the x axis (time) and y (for temp) and yDoorAndLight (for door and light values, which use a different scale) axis. Note that I set a floor of 0 for the yDoorAndLight displayable values, but later I convert the false values to -1 for these data points. This will plot them below the visible graph, thus effectively not plotting the false values.

Step 4: Define the axes
Here, I define the actual displays for the x and y axis (what the tick values are, if the axis line is on the left/right or top/bottom, etc). Note that I call .scale(x) and .scale(y) to scale the axis based on the display ranges defined above.

Note that this doesn’t draw the axes — it just defines them.

Step 5: Adds the svg canvas
Now we start to actually draw stuff on the screen. First, we have to create an svg canvas to draw the other items onto. Note the d3.select(tagName) call (where tagName is the tag passed into the function call from my Jade template). This is where we define where on the page this graph will go (it’ll anchor to that provided tagName), how big it is (based on the dimensions we defined in Step 1), etc.

Step 6: change the shape of the data to work with the graphs
Here, we walk through all of the data points and modify the value to match how we want them plotted. Specifically, I call the parseDate function we defined in Step 2 to convert the dates into localtime. Then, for the doorOpen and lightOn properties, I convert the true/false values into -1 for false and either 1 or .95 for true values. This will stack them toward the top of the graph canvas, but not on top of each other. In the next step we’ll set the domain range for the values to be from 0 to 1, so they’ll be graphed with 1 as the very top and 0 as the very bottom.

// change the shape of the data to work with the graphs
data.forEach(function(d) {
    d.timestamp = parseDate(d.timestamp);
    var doorPlotPoint = -1; //outside the domain, so not shown
    if (d.doorOpen) {
        doorPlotPoint = .95;
    }
    d.doorOpen = doorPlotPoint;

    var lightPlotPoint = -1;//outside the domain, so not shown
    if (d.lightOn) {
        lightPlotPoint = 1;
    }
    d.lightOn = lightPlotPoint;


});

Step 7: Scale the range of the data
Here, I define the domain of the data for my three axis. This is used by d3 to scale and plot the values. Values that fall outside the domain are not shown. For the temperatures, I set a +/- 10 degree margin above and below the data points to make it look a little nicer and to keep the temperature datapoints from overlapping with the door/light data points.

// Scale the range of the data
var maxTemp = d3.max(data, function(d) { return d.temp_F; });
var minTemp = d3.min(data, function(d) { return d.temp_F; });
x.domain(d3.extent(data, function(d) { return d.timestamp; }));
y.domain([minTemp - 10, maxTemp + 10]);
yDoorAndLight.domain([0, 1]);

Step 8: Add the temp scatterplot
Here, I plot dots for each of the temperature readings. This helps define where there is data and where there is just a line (we’ll draw that in a minute). This is useful when my collection script dies and I have a big gap in my data.

Notice my call to .data(data). This is how I handoff the data records for it to use. Then I call .attr("cx", function (d) { return x(d.timestamp); }) to tell it what to plot for the x axis and .attr("cy", function (d) { return y(d.temp_F); }) to plot the y axis.

Step 9 & 10: Add the door scatterplot & light scatterplot
Similar to the temp scatterplot, I’m plotting the door open/closed dots. The main difference to the temp scatterplot is that I provide yDoorAndLight for the y axis values. I also use differ fill colors for each (red and yellow).

Step 11: Add the temperature valueline path.

Here, I define the valueline function for plotting the temperature values on the graph. I use .interpolate("linear") to connect the dots with a direct line. I then provide functions to the .x() and .y() calls to return the datapoint properties to use from the data records:

// Define the temperature line
var valueline = d3.svg.line()
    .interpolate("linear")
    .x(function(d) { return x(d.timestamp); })
    .y(function(d) { return y(d.temp_F); });

Then I use that function to plot the temperature data as a line:

// Add the temperature valueline path.
svg.append("path")
    .attr("d", valueline(data))
    .attr("stroke", "blue")
    .attr("stroke-width", 2)
    .attr("fill", "none");

Step 12 & 13: Add the X Axis & Y Axis
Finally, we draw the actual x and y axis lines. Note that on the y axis, I use .attr("transform", "rotate(-90)") to turn is sideways and add .text("Temperature (°F)"); to provide the axis label.

Tip: In order for the ° character to display correctly, I had to modify my layout.jade file to set the charset to utf-8, like this:

meta(http-equiv='content-type', content='application/xhtml+xml; charset=utf-8')

Optionally Display the Datatable

The final section in the index.jade file will optionally show a datatable with the raw records. Note that it starts with if showRecords, and that in the route I provide a true/false value for showRecords. That will toggle whether this table is shown or not:

  if showRecords
    hr

    div
      table.table.table-striped
        thead
          tr
            th Id
            th Lang
            th LightOn
            th DoorOpen
            th Temp
            th Timestamp
        tbody
          each record, i in readings
            tr
              td #{record._id}
              td #{record.sourceLanguage}
              td #{record.lightOn}
              td #{record.doorOpen}
              td #{record.temp_F}
              td #{record.timestamp}

Improving the SonicController Logic

A made a few tweeks to the logic in my SonicController class since my last post. Specifically, I was getting some weird readings occasionally and needed to smooth those out, and every so often, the call to get a distance reading would never return. So, instead of taking a single distance reading, I’m now taking multiple readings, dropping the high and low reading and averaging out the remaining before returning that back to the caller. I’ve also added a timeout to keep it from never returning.

You can see the commit with these changes here

And here’s my new class:

import RPi.GPIO as GPIO
import time

class SonicController:

  SPEED_OF_SOUND = 34000 #cm/s

  def __init__(self, triggerPin, echoPin):
    self.triggerPin = triggerPin
    self.echoPin = echoPin

    print("Initializing Ultrasonic Range Finder")

    GPIO.setup(self.triggerPin, GPIO.OUT, pull_up_down = GPIO.PUD_DOWN)
    GPIO.setup(self.echoPin, GPIO.IN, pull_up_down = GPIO.PUD_DOWN)

    GPIO.output(self.triggerPin, False)
    print("Waiting For Sensor To Settle")
    time.sleep(2)

  def _readDistanceOnce(self):

    print("    Distance Measurement In Progress")
    READING_TIMEOUT = 2 #sec
    maxTime = time.time() + READING_TIMEOUT

    GPIO.output(self.triggerPin, True)
    time.sleep(0.00001)
    GPIO.output(self.triggerPin, False)


    pulse_start = time.time()
    while GPIO.input(self.echoPin)==0 and pulse_start < maxTime:
      pulse_start = time.time()

    pulse_end = time.time()
    while GPIO.input(self.echoPin)==1 and pulse_end < maxTime:
      pulse_end = time.time()

    if pulse_end > maxTime:
      print("  PULSE READ TIMED OUT")

    pulse_duration = pulse_end - pulse_start
    roundtrip_duration = pulse_duration * self.SPEED_OF_SOUND
    one_way_distance = roundtrip_duration/2
    print("    Distance: {0:0.2f} cm".format(one_way_distance))
    return one_way_distance

  def readDistance(self):

    #
    # Take multiple readings in order to counter the affects of
    # bad data due to non-realtime OS.  Take a bunch of readings,
    # throw out the min and max, then average the rest.
    # 
    numReadingsToTake = 8
    print("    Taking {} Distance Measurements".format(numReadingsToTake))
    measurements = []
    for x in range(0, numReadingsToTake):
      thisReading = self._readDistanceOnce()
      measurements.append(thisReading)

    maxReading = max(measurements)
    minReading = min(measurements)
    measurements.remove(maxReading)
    measurements.remove(minReading)

    average = sum(measurements)/len(measurements)

    print("    Average Distance: {0:0.2f} cm".format(average))
    return average

  def teardown(self):
    print("Tearing down Ultrasonic Range Finder")
    GPIO.output(self.triggerPin, False)

Allowing The User To Select The Install Folder

This article is one in my series focusing on what I learned creating a Windows Installer using the WiX toolkit’s “Managed Bootstrapper Application” (Custom .NET UI) framework. Each post builds upon the previous posts, so I would suggest you at least skim through the earlier articles in the series.

Here’s an index into the articles in this series:

During an installation, it’s often expected that the end user can change the installation folder from the default value. Obviously, you’ll need some UI controls to allow the user to select the destination folder they want to use, but what do you do with it once you have it?

Changing the install folder location for your installation has two main tasks:

  • Overriding the folder path at install
  • Determining the override used at modify/uninstall/upgrade time

I’ll walk through both in this article.

Overriding the Install Folder Path

Let’s look at how the default install folder is set. In your individual MSI WiX projects, you’ll have a Directory fragment defined that will look something like this:

  <Fragment>
    <Directory Id="TARGETDIR" Name="SourceDir">
      <Directory Id="ProgramFilesFolder">
        <Directory Id="MYINSTALLFOLDER" Name="Wrightfully Blog Examples">

          <!-- ... subfolders -->   
          <Directory Id="dirLicenses" Name="Licenses" />
          <Directory Id="dirdoc" Name="doc" />
        </Directory>
      </Directory>
    </Directory>
  </Fragment>

Notice that some of the Ids are in ALLCAPS. This means they are exposed as variables that can be set by the WiX Engine’s Engine.StringVariables array. The override will replace that node in the structure and all child folders will branch off the new path. Overrides will not affect any Directory nodes defined above or beside it in the xml structure — only it and it’s children.

So, if I want to override the install folder defined in my Directory fragment, I’d do something like this:

var pathOverride = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),
                           "My Alternate Location");

Engine.StringVariables["MYINSTALLFOLDER"] = pathOverride;

Then, when the actual MSIs install, they will substitute the value for MYINSTALLFOLDER to use my new value at install time. But first, you have to tell WiX to pass the MYINSTALLFOLDER along to the MSI, which you do in the <MsiPackage> declaration of your <Bundle>, like this:

  <MsiPackage Compressed="yes" SourceFile="$(var.MyMis.TargetPath)">
      <MsiProperty Name="MYINSTALLFOLDER" Value="[MYINSTALLFOLDER]"/>
  </MsiPackage>

Determining Override Used in Later Runs

Unfortunately, if you override the folder at install time, that’s not persisted by the installation system, so you don’t know what the value was set to during later runs — such as when the user clicks Modify/Uninstall in Add/Remove Programs, or when the user goes to upgrade with a newer version of the installer.

So, in order to correctly re-override the value during future runs, we need to store it somewhere and then go grab that value when we run. And we do this with a registry key. You could do this directly in the Bootstrapper Application (managed code), but I prefer to use the WiX functionality to manage the registry key and ensure it’s removed when the software is uninstalled, etc. So in an MSI Feature that will always be installed with my software, I put this in my product definition:

<Feature Id="feat_Documentation" Title="Documentation" Level="1" Display="hidden">
    <ComponentGroupRef Id="compgroup_Docs" />

    <Component Directory='MYINSTALLFOLDER'>
        <RegistryValue Root='HKLM' Key='SOFTWARE\Wrightfully\BlogInstaller'
                   Name='MyInstallFolder' Value='[MYINSTALLFOLDER]'
                   Type='string'></RegistryValue>
    </Component>
</Feature>

This will create the registry key and populate it with the value of MYINSTALLFOLDER. Then, in my bootstrapper app, I can read the values using this code:

var existingInstallLocation =
    Registry.GetValue(@"HKEY_LOCAL_MACHINE\SOFTWARE\Wrightfully\BlogInstaller",
                    "MyInstallFolder", "").ToString();
if (!string.IsNullOrEmpty(existingInstallLocation))
{
     Engine.StringVariables["MYINSTALLFOLDER"] = existingInstallLocation;
}

If the registry value exists (is not an empty string), then I use that value for [MYINSTALLFOLDER]. Otherwise, I use the default value.

TIP: Since the default value for MYINSTALLFOLDER isn’t known by the Bootstrapper App (it’s defined in each MSI’s Directory fragment), I just always set it to a default value at startup of my bootstrapper (or to the override found in the registry) — which means the value in the Directory fragments is never actually used.

ANOTHER TIP: Make sure you do some general validation on the user-entered values to make sure it’s a valid path. Otherwise, the MSI installs will blow up, and you definitely don’t want to deal with that.