Blog Archives

How to parse Nest info (and graph it using MRTG)

How to parse Nest info (and graph it using MRTG)

This is the continuation of my previous post where I talked about setting up my Nest Thermostat.

Data Flow

Since Nest does not offer an open API, we have to cheat a little (ok, a LOT!). By using Scott Baker’s Python code, you can query the Nest web service to get your current status on the Nest (using your nest login, password and serial number).

Data is sent from the local Nest to the web. Python is used to grab the data from the web and put into my own MRTG compatible format. From the Nest data, I am graphing Temperature and Humidity, Heating/Cooling State and the Battery Levels for each of 2 thermostats.

This is all done using a perl program that creates an array from the Python program output, parses the array line by line, then creates a file that MRTG can read to do the actual graphing.

Perl Code

This perl script runs every 5 minutes on my Raspberry Pi to create the files needed for MRTG.

#!/usr/bin/perl -w
use strict;

###
### Perl Nest (using Python to scrape the web) to mrtg log file script
### Written by Kyle Platts
### Nest Gen 1 Thermostat

my $nest1Humidity;  # Nest Humidity
my $nest1Temp;      # Nest Temperature
my $nest1Voltage;   # Nest Battery Voltage
my $nest1HeatState; # Nest call for heat
my $nest1AcState;   # Nest call for A/C
my $nest2Humidity;  # Nest Humidity
my $nest2Temp;      # Nest Temperature
my $nest2Voltage;   # Nest Battery Voltage
my $nest2HeatState; # Nest call for heat
my $uptime = `uptime`;
chomp $uptime;
my @uptimeSplit = split (/ /, $uptime);
chop $uptimeSplit[5];
$uptime = ($uptimeSplit[3] . " " . $uptimeSplit[4] . " " . $uptimeSplit[5]);
## Nest 1 = Upstairs
my @nest1 = `/usr/local/bin/nest.py --user user\@you.com --password password --serial 123456789 show`;
## Nest 2 = Downstairs
my @nest2 = `/usr/local/bin/nest.py --user user\@you.com --password password --serial 123456780 show`;

foreach my $line (@nest1) {
 chomp $line;
 if ($line =~ /current_humidity/) { # line contains the humidity, split it out
 my @nest1HumSplit = split (/\:/, $line);
 $nest1Humidity = ($nest1HumSplit[1] * 100);
 }
 if ($line =~ /current_temperature/) { # line contains the temperature in C
 my @nest1TempSplit = split (/\:/, $line);
 $nest1Temp = (((($nest1TempSplit[1] * 9) / 5) + 32) * 100);
 }
 if ($line =~ /battery_level/) { # line contains the Battery Voltage
 my @nest1VoltageSplit = split (/\:/, $line);
 $nest1Voltage = ($nest1VoltageSplit[1] * 1000);
 }
 if ($line =~ /hvac_heater_state/) { # line contains the Heat State
 my @nest1HeatSplit = split (/\:/, $line);
 $nest1HeatState = $nest1HeatSplit[1];
 if ($nest1HeatState =~ /True/) {
 $nest1HeatState = 1;
 }
 else {
 $nest1HeatState = 0;
 }
 }
 if ($line =~ /hvac_ac_state/) { # line contains the A/C State
 my @nest1AcSplit = split (/\:/, $line);
 $nest1AcState = $nest1AcSplit[1];
 if ($nest1AcState =~ /True/) {
 $nest1AcState = 1;
 }
 else {
 $nest1AcState = 0;
 }
 }
}
foreach my $line (@nest2) {
 chomp $line;
 if ($line =~ /current_humidity/) { # line contains the humidity, split it out
 my @nest2HumSplit = split (/\:/, $line);
 $nest2Humidity = ($nest2HumSplit[1] * 100);
 }
 if ($line =~ /current_temperature/) { # line contains the temperature in C
 my @nest2TempSplit = split (/\:/, $line);
 $nest2Temp = (((($nest2TempSplit[1] * 9) / 5) + 32) * 100);
 }
 if ($line =~ /battery_level/) { # line contains the Battery Voltage
 my @nest2VoltageSplit = split (/\:/, $line);
 $nest2Voltage = ($nest2VoltageSplit[1] * 1000);
 }
 if ($line =~ /hvac_heater_state/) { # line contains the Heat State
 my @nest2HeatSplit = split (/\:/, $line);
 $nest2HeatState = $nest2HeatSplit[1];
 if ($nest2HeatState =~ /True/) {
 $nest2HeatState = 1;
 }
 else {
 $nest2HeatState = 0;
 }
 }
}

open (STATE, '>', '/home/pi/scripts/nest.log') || die $!;
 if ($nest1AcState or $nest1HeatState == 1) {
 print (STATE "1\n");
 }
 else {
 print (STATE "0\n");
 }
 if ($nest2HeatState == 1) {
 print (STATE "1\n");
 }
 else {
 print (STATE "0\n");
 }
 print (STATE "$uptime\n");
 print (STATE "Nest States\n");

open (VOLTAGE, '>', '/home/pi/scripts/nest_bat.log') || die $!;
 print (VOLTAGE "$nest1Voltage\n");
 print (VOLTAGE "$nest2Voltage\n");
 print (VOLTAGE "$uptime\n");
 print (VOLTAGE "Nest Battery Voltage\n");

open (NEST1, '>', '/home/pi/scripts/nest_1.log') || die $!;
 print (NEST1 "$nest1Temp\n");
 print (NEST1 "$nest1Humidity\n");
 print (NEST1 "$uptime\n");
 print (NEST1 "Nest 1 Temp and Humidity\n");

open (NEST2, '>', '/home/pi/scripts/nest_2.log') || die $!;
 print (NEST2 "$nest2Temp\n");
 print (NEST2 "$nest2Humidity\n");
 print (NEST2 "$uptime\n");
 print (NEST2 "Nest 2 Temp and Humidity\n");

Heating and Cooling states are converted to either a 1 or 0 for graphing. Temperature is in degrees C, but being an American, I have no frame of reference to what C feels like, so I convert it to Fahrenheit. I also make sure all of the values are mulitplied by 100 so I can graph with 2 decimal point accuracy.

MRTG Config File

######################################################################
# System: Nest Learning Thermostats
# Description: Nest Thermostats
# Contact: Kyle Platts
######################################################################

### Nest Status >> Descr: 'NestUpstairsTemp/Humidity'

Options[nest_1]: gauge,growright,nolegend,noinfo,absolute
LegendI[nest_1]: Temperature
LegendO[nest_1]: Humidity
ShortLegend[nest_1]: F
YTicsFactor[nest_1]: 0.01
Factor[nest_1]: 0.01
YLegend[nest_1]: Temp/Humidity
Target[nest_1]: `cat /home/pi/scripts/nest_1.log`
MaxBytes[nest_1]: 10000
YTics[nest_1]: 5
Title[nest_1]: Nest Upstairs Readings
PageTop[nest_1]: <h1>Nest Upstairs Readings</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostat</td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Upstairs Temperature & Humidity</td>
 </tr>
 </table>
 </div>

### Nest Status >> Descr: 'NestUpstairsTemp/Humidity'

Options[nest_2]: gauge,growright,nolegend,noinfo,absolute
LegendI[nest_2]: Temperature
LegendO[nest_2]: Humidity
ShortLegend[nest_2]: F
YTicsFactor[nest_2]: 0.01
Factor[nest_2]: 0.01
YLegend[nest_2]: Temp/Humidity
Target[nest_2]: `cat /home/pi/scripts/nest_2.log`
MaxBytes[nest_2]: 10000
YTics[nest_2]: 5
Title[nest_2]: Nest Downstairs Readings
PageTop[nest_2]: <h1>Nest Downstairs Readings</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostat</td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Downstairs Temperature & Humidity</td>
 </tr>
 </table>
 </div>

### Nest Status >> Descr: 'Nest Battery Voltages'

Options[nest_bat]: gauge,growright,nolegend,noinfo,absolute
LegendI[nest_bat]: Upstairs
LegendO[nest_bat]: Downstairs
ShortLegend[nest_bat]: V
YTicsFactor[nest_bat]: 0.001
Factor[nest_bat]: 0.001
YLegend[nest_bat]: Volts
MaxBytes[nest_bat]: 4000
Target[nest_bat]: `cat /home/pi/scripts/nest_bat.log`
YTics[nest_bat]: 5
Title[nest_bat]: Nest Battery Levels
PageTop[nest_bat]: <h1>Nest Battery Levels</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostat</td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Nest Battery Levels</td>
 </tr>
 </table>
 </div>

### Nest Status >> Descr: 'Nest On/Off State'

Options[nest]: gauge,growright,nopercent,noinfo,absolute
LegendI[nest]: Upstairs
LegendO[nest]: Downstairs
ShortLegend[nest]: &nbsp;
YLegend[nest]: HVAC On/Off
Target[nest]: `cat /home/pi/scripts/nest.log`
MaxBytes[nest]: 1
YTics[nest]: 2
Title[nest]: Nest HVAC Status
PageTop[nest]: <h1>Nest HVAC Status</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostats </td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Nest HVAC Status</td>
 </tr>
 </table>
 </div>

In the MRTG config file, I use Factor and Y-ticks Factor to gain 2 decimal point granularity in my graphs. Since these are static values read each time, there is no need do math and figure out the “rate” of a digital output of the thermostat. You will see the use of gauge, nopercent and absolute on the states.
I do maintain percentages on humidity though, since that is what is calculated by the thermostat.

MRTG Log Files

These are the files that actually contain the data parsed from the web and and are read by MRTG for graphing.  Each file needs 4 lines. The first line is Value 1 (In) for MRTG. Line 2 is Value 2 (Out). Line 3 contains the Uptime if you choose to display it. Line 4 contains the graph title.

Nest Temperature and Humidity File

7466
3700
24 days, 3:54
Nest 1 Temp and Humidity

This indicates that right now, my upstairs Nest is 74.66 degrees F with 37% humidity.

Graphs

How to Email MRTG Graphs using Python SMTP Library & Gmail

How to Email MRTG Graphs using Python SMTP Library and Gmail

Send mail with PythonOne of the easiest ways I have found to create automated email messages is to use the Python SMTP Library. The code is pretty straightforward and allows you to send attachments. Using a gmail account, allows email to work whether you have a local SMTP server installed and running or not (more portable).

On my last project, I am using MRTG to graph some traffic OID’s, but the server does not have Apache installed, and even if it did, it’s firewalled with no access to Port 80. To compound matters, I do not have control over the server, as it belongs to the customer.

How to get those pretty graphs out to the rest of the world? 

I investigated several ways of removing the data from the server, TFTP, FTP, SCP, etc. the goal was to email the graphs to the interested parties, so why not just email directly from the server? The files are already there, and MRTG creates the HTML for me on the fly.

Python SMTP Library

Luckily, we do have Python installed on this server that is being used for some other tasks, so I decided to use the SMTP Library. I’ve used this library in past projects to do email attachments, so I was familiar with how to use it. What I had not done, was change the message body to HTML.

I found an example on StackOverflow and used it as the basis to start coding. If you just import the MRTG generated HTML file, the email will come through with attachments, but it won’t display them properly in the email client. What I wanted, was an exact duplicate of what would be seen on a webpage.

green

Recycling Code (can we call this green code?)

I went back to another project to look at how the images were attached.

# Send an HTML email with an embedded image and a plain text message for
# email clients that don't want to display the HTML.

import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText
from email.MIMEImage import MIMEImage

# Base path for image files
basepath = '/home/gifs'

# Define these once; use them twice!
strFrom = 'user@gmail.com'
strTo = ['user1@yourcompany.com', 'user2@yourcompany.com']
strSubject = 'Your Important Message'

# Create the root message and fill in the from, to, and subject headers
msgRoot = MIMEMultipart('related')
msgRoot['Subject'] = strSubject
msgRoot['From'] = strFrom
msgRoot['To'] = ''.join(strTo)
msgRoot.preamble = 'This is a multi-part message in MIME format.'

# Encapsulate the plain and HTML versions of the message body in an
# 'alternative' part, so message agents can decide which they want to display.
msgAlternative = MIMEMultipart('alternative')
msgRoot.attach(msgAlternative)

msgText = MIMEText('This is the alternative plain text message.')
msgAlternative.attach(msgText)

# We reference the image in the IMG SRC attribute by the ID we give it below
msgText = MIMEText('Important graphic here!<br><img src="cid:image1"><br>Another important graphic here!<br><img src="cid:image2"><br>', 'html')
msgAlternative.attach(msgText)

# Use the basepath directory to find the gif files
fp = open('%s/image1.gif' % basepath, 'rb')
msgImage1 = MIMEImage(fp.read())
fp.close()
fp = open('%s/image2.gif' % basepath, 'rb')
msgImage2 = MIMEImage(fp.read())
fp.close()
# Define the image's ID as referenced above
msgImage1.add_header('Content-ID', '<image1>',)
msgRoot.attach(msgImage1)
msgImage2.add_header('Content-ID', '<image2>',)
msgRoot.attach(msgImage2)

# Send the email (this example assumes SMTP authentication is required)
smtp = smtplib.SMTP('smtp.gmail.com', 587)
smtp.ehlo()
smtp.starttls()
smtp.ehlo()
smtp.login('user@gmail.com', 'password')
smtp.sendmail(strFrom, strTo, msgRoot.as_string())
smtp.quit()

In the above code, the body is already in HTML, so reading in the MRTG generated HTML file is pretty straight forward. However, images are referenced using a Content ID. The original MRTG HTML file does not have this present.

<!-- Begin `Daily' Graph (5 Minute -->
 <div class="graph">
 <h2>`Daily' Graph (5 Minute Average)</h2>
 <img src="cnntcobd01-day.png" title="day" alt="day" />
 <table>
 <tr>
 <th></th>
 <th scope="col">Max</th>
 <th scope="col">Average</th>
 <th scope="col">Current</th>
 </tr>
 <tr class="in">
 <th scope="row">UP</th>
 <td>1959.4 kb/s (30.3%)</td>
 <td>408.7 kb/s (6.3%) </td>
 <td>278.4 kb/s (4.3%) </td>
 </tr>
 <tr class="out">
 <th scope="row">DN</th>
 <td>26.4 Mb/s (18.3%) </td>
 <td>5079.3 kb/s (3.5%) </td>
 <td>6192.8 kb/s (4.3%) </td>
 </tr>
 </table>
 </div>
<!-- End `Daily' Graph (5 Minute -->

Python can fix this!
Since we have to read the MRTG HTML from a file anyways, why not add the cid: to the image source in the HTML?

# We reference the image in the IMG SRC attribute by the ID we give it below
# Open up the existing HTML file and add the CID before sending the email
with open('%s/target.html' % basepath, 'r') as f:
 newlines = []
 for line in f.readlines():
 newlines.append(line.replace('<img src="', '<img src="cid:'))
f.close()
with open('%s/target.html' % basepath, 'w') as f:
 for line in newlines:
 f.write(line)
f.close()
f = open('%s/target.html' % basepath, 'r')
body = ''.join(f)
f.close()

I’ve replaced <img src=” with <img src=”cid:

This now creates the appropriate Content ID within the HTML message so all the images are displayed inline.

Working Code

# Send an HTML email with an embedded image and a plain text message for
# email clients that don't want to display the HTML.

from datetime import date, timedelta
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText
from email.MIMEImage import MIMEImage

# Base path for image files
basepath = '/opt/mrtg'

# Define these once; use them twice!
strFrom = 'user@gmail.com'
strTo = ['user1@youcompany.com', 'user2@yourcompany.com']
strSubject = 'Your Important Message'

# Create the root message and fill in the from, to, and subject headers
msgRoot = MIMEMultipart('related')
msgRoot['Subject'] = strSubject
msgRoot['From'] = strFrom
msgRoot['To'] = &quot;, &quot;.join(strTo)
msgRoot.preamble = 'This is a multi-part message in MIME format.'

# Encapsulate the plain and HTML versions of the message body in an
# 'alternative' part, so message agents can decide which they want to display.
msgAlternative = MIMEMultipart('alternative')
msgRoot.attach(msgAlternative)

msgText = MIMEText('This is the alternative plain text message.')
msgAlternative.attach(msgText)

# We reference the image in the IMG SRC attribute by the ID we give it below
# Open up the existing HTML file and add the CID before sending the email
with open('%s/target.html' % basepath, 'r') as f:
 newlines = []
 for line in f.readlines():
 newlines.append(line.replace('<img src="', '<img src="cid:'))
f.close()
with open('%s/target.html' % basepath, 'w') as f:
 for line in newlines:
 f.write(line)
f.close()
f = open('%s/target.html' % basepath, 'r')
body = ''.join(f)
f.close()
# print body
msgText = MIMEText(body, 'html')
msgAlternative.attach(msgText)
# Use the basepath directory to find the gif files
fp = open('%s/target-day.png' % basepath, 'rb')
msgImage1 = MIMEImage(fp.read())
fp.close()
fp = open('%s/target-week.png' % basepath, 'rb')
msgImage2 = MIMEImage(fp.read())
fp.close()
fp = open('%s/target-month.png' % basepath, 'rb')
msgImage3 = MIMEImage(fp.read())
fp.close()
fp = open('%s/target-year.png' % basepath, 'rb')
msgImage4 = MIMEImage(fp.read())
fp.close()
fp = open('%s/mrtg-l.png' % basepath, 'rb')
msgImage5 = MIMEImage(fp.read())
fp.close()
fp = open('%s/mrtg-m.png' % basepath, 'rb')
msgImage6 = MIMEImage(fp.read())
fp.close()
fp = open('%s/mrtg-r.png' % basepath, 'rb')
msgImage7 = MIMEImage(fp.read())
fp.close()
# Define the image's ID as referenced above
msgImage1.add_header('Content-ID', '<target-day.png>',)
msgRoot.attach(msgImage1)
msgImage2.add_header('Content-ID', '<target-week.png>',)
msgRoot.attach(msgImage2)
msgImage3.add_header('Content-ID', '<target-month.png>',)
msgRoot.attach(msgImage3)
msgImage4.add_header('Content-ID', '<target-year.png>',)
msgRoot.attach(msgImage4)
msgImage5.add_header('Content-ID', '<mrtg-l.png>',)
msgRoot.attach(msgImage5)
msgImage6.add_header('Content-ID', '<mrtg-m.png>',)
msgRoot.attach(msgImage6)
msgImage7.add_header('Content-ID', '<mrtg-r.png>',)
msgRoot.attach(msgImage7)

# Send the email (this example assumes SMTP authentication is required)
smtp = smtplib.SMTP('smtp.gmail.com', 587)
smtp.ehlo()
smtp.starttls()
smtp.ehlo()
smtp.login('user@gmail.com', 'password')
smtp.sendmail(strFrom, strTo, msgRoot.as_string())
smtp.quit()

End Result

This is an 8 pair ADSL2+ EFM bonded group running at 144 Megbits/s downstream and 6.7 Megabits/s upstream.

MRTG GRAPH