At Ngenic we strongly believe in the quality of our products, we put our heart and soul into their perfection! This is especially true for Ngenic Tune; we love it, it's our baby. Lucky for us, there's a ton of ad hoc evidence suggesting that Tune performs its function exceedingly well: positive reviews on social media, comments from satisfied customers, successful pilot projects, personal experience, and so on.
When you love something as much as we love Tune, it's easy to convince yourself that it works. But love has a tendency to cloud your judgment. Therefore we decided it would be interesting to try and see if our belief in Tune would hold up in the harsh light of scientific scrutiny. Can we devise an empirical test and prove our hypothesis rigorously enough for publication in a peer-reviewed scientific journal? It turns out the answer is yes!
To that end, we contacted a leading expert in the field of indoor comfort: Professor David P. Wyon of the International Center for Indoor Environment and Energy, Department of Civil Engineering, Technical University of Denmark (DTU). We asked Dr. Wyon if there was a way we could empirically test the performance of Tune, from a comfort perspective. Would it be possible to put a number on how well Tune does its job?
Thermal comfort is defined as that condition of mind which expresses satisfaction with the thermal environment. Wherever you happen to be reading this, pause for a moment and reflect on your thermal state. Are you a little too warm, a little too cool? Many academic and professional studies have demonstrated the benefits of a constant, comfortable indoor temperature. Unfortunately, it is an all too common occurrence that the heat pump of a home is incorrectly calibrated or incapable of fine-tuning, resulting in a situation where the occupants experience varying degrees of discomfort due to an unstable indoor temperature.
At Ngenic we claim to improve this situation by controlling the heat pump with our proprietary control algorithm "Tune", providing increased thermal comfort during a greater part of the day by eliminating uneven temperatures.
A Field Intervention Study
The advice we received from Dr. Wyon was to conduct an experiment, specifically, a single-blind field intervention study with cross-over design. To accomplish this we recruited a pool of test subjects and pseudo-randomly divided them into two groups. We then put them on an intervention schedule: alternating between the conditions "Tune on" and "Tune off" at approximately week-long intervals, one group in each condition in any given week, so that the results obtained in each condition would be equally affected by any external factors we could not control, such as the weather. In the condition with Tune disabled, the heat pump was allowed to operate on its default setting.
Here is where Dr. Wyon got really clever. The smart-phone application which accompanies an Ngenic Tune system allows users to adjust their preferred indoor temperature, we'll call this the target temperature. Even with the Tune algorithm disabled, users were able to continue to adjust their target temperature even though it had no effect on the heat pump operation. That sounds devious, but remember that when Tune was off, customers were simply reverted to normal heat pump operation, and since they were not aware of the intervention, their expectation bias could not affect the result.
The assumption was made that users would adjust their target temperature up or down if they became sufficiently uncomfortable. The number of times an adjustment was made was recorded for each household as they experienced each intervention. At the conclusion of the study, these numbers were added together for each household, separately for each "Tune on" period and "Tune off" period. By comparing these two values, a statistical test was made as to whether or not the adjustment frequency differed during periods when Tune was off. If significantly fewer adjustments were made with Tune on, then we can confidently say that, yes, Tune works!
How it all Turned Out
In order to evaluate the result of our experiment and its ramifications for our hypothesis, we need to calculate a test statistic and estimate its p-value. As per Dr. Wyon's recommendation, we used something called the Wilcoxon Signed-Rank test statistic, which may sound like a rather dry entity, but at Ngenic we love this kind of thing.
This statistic requires us to calculate the within-household differences in target temperature adjustment, these can be seen as a histogram in Figure 4. The within-household difference is a simple calculation that is best explained using an example. Say a household made one adjustment to their target temperature when Tune was off and again one adjustment when Tune was on, then that household counts as a "0" in the histogram. If a household made three adjustments with Tune off and two adjustments with Tune on, then that household counts as a "-1" in the histogram.
Now, if the hypothesis is true that Ngenic customers are equally likely to adjust their target temperature regardless of whether or not Tune is on or off, this should result in a histogram with a symmetric distribution around zero. As Figure 5 shows, the distribution skews to the negative, indicating the data support rejecting this hypothesis. That means it seems like Tune is doing well in this test, but in order to know for sure, we had to calculate a p-value.
After performing some mathematical gymnastics, the p-value came out as 0.0348, which is a very good thing. Let me just say, you haven't lived until you've conducted an empirical study and calculated a p-value less than 0.05. The data science team here at Ngenic had to take a Swedish Fika to calm our nerves! Simply put, the p-value quantifies the probability that the outcome of our experiment could have happened by chance, and 3.5% is quite low in this context.
With a low p-value in hand, we can confidently make the following statement: at a statistically significant level (p-value < 0.05), the study supports the claim that Ngenic Tune has a positive effect on thermal comfort, as improving heat pump operation by operating Ngenic Tune resulted in fewer target temperature adjustments when compared to default heat pump operation. As evidenced by the predominantly negative values shown in Figure 5, users made significantly fewer preferred temperature adjustments when Tune was engaged.
In short: an empirical study has shown that Tune works!
A Couple of Discussion Points
Obviously, there are many reasons why a person might adjust their target temperature that may not be related to thermal comfort. Cleverly, the within-subject design of the field experiment eliminates the influence of any such factors on the data. The technique of alternating conditions on a weekly schedule also ensures that any often-repeated behavior sums to zero before the statistical test is performed.
The underlying dynamic in this experiment is that Ngenic improves thermal comfort while reducing energy consumption by smoothing-out unnecessary temperature variation through optimal heat pump control. Can the temperature data collected during the interventions tell us something about this dynamic?
To investigate this question, we calculated another statistical test; this time, an F-test. Figures 1 and 2 show the indoor temperature profiles of a typical test subject, with 1 showing Tune on and 2 showing Tune off. Once again, it seems like the temperatures are more even with Tune on, but once again, we need that p-value to be sure. This time, the p-value crushed any doubt, clocking in at less than 0.001 for the F-statistic. With that in mind, we can say at a statistically significant level (p-value < 0.001), it can be confidently stated that the Tune control algorithm reduces indoor temperature variation in the majority of households.
Having conceived and carried out an empirical test, as per the design of Dr. Wyon, and having analyzed and reported the results, we wrote up a white paper and a journal article. The latter was submitted as a report of a new methodology for field intervention studies to the premier journal in the field: Indoor Air. Indoor Air is a peer-reviewed journal, which means the article was thoroughly examined and audited in detail by experts in the field anonymous to the authors. They found the experiment, analysis, and conclusions to be sound, and on 2018-07-06 the article was accepted for publication in the September issue: Volume 28, Issue 5.
The formally proven hypothesis can be stated as follows: Covert operation of Tune resulted in a statistically significant decrease in the number of preferred temperature adjustments and a statistically significant decrease in the variance of the indoor temperature. Taken together, this indicates that Tune is indeed able to reduce thermal discomfort as well as save energy!
Wyon, D.P. and Ridenour, J.E. "A covert field‐intervention experiment to determine how heating controls that conserve energy affect thermal comfort." Indoor Air. 2018;28:763–767. https://doi.org/10.1111/ina.12488.