Category Archives: Raspberry Pi

MRTG Tips & Tricks

MRTG

MRTG Tips and Tricks

I spent some time recently creating some new graphs for a special project. It’s an 8 pair ADSL2+ EFM boded group. There were several items that I had to use that are not what you would call “standard” MRTG options. The following are some of the things I had to do to make everything work and display correctly.

64 Bit Counters using SNMP v2c

The particular OID’s I am using are 64 bit counters. By default, MRTG uses SNMP v1, which cannot read 64 bit counters. By specifying 5 colons and a 2 on your target, you are telling MRTG to use SNMP v2c.


Target[target]: 1.3.6.1.4.1.664.5.70.7.1.1.1.9.440029&1.3.6.1.4.1.664.5.70.7.1.1.1.10.440029:public@192.168.1.10:::::2

Add OID’s Together

These particular Enterprise OID’s are split out into Unicast, Broadcast and Multicast bytes. In order to get the total traffic, all the OID’s must be added together. No problem! MRTG can do that. Simply specify your first 2 OID’s, then use the + sign to add them together. In this case, I am adding 3 sets of OID’s together. Transmit and Receive counters for Unicast, Broadcast and Multicast.


Target[target]: 1.3.6.1.4.1.664.5.70.7.1.1.1.9.440029&1.3.6.1.4.1.664.5.70.7.1.1.1.10.440029:public@192.168.1.10:::::2 + 1.3.6.1.4.1.664.5.70.7.1.1.1.11.440029&1.3.6.1.4.1.664.5.70.7.1.1.1.12.440029:public@192.168.1.10:::::2 + 1.3.6.1.4.1.664.5.70.7.1.1.1.13.440029&1.3.6.1.4.1.664.5.70.7.1.1.1.14.440029:public@192.168.1.10:::::2

Router Uptime

Once you do any arithmetic on the OID’s, MRTG no longer grabs the router uptime. You must add router uptime to the configuration file to get the router uptime to display properly again in the graphs.

RouterUptime[target]: <community string>@<ip address>

Timezone

The timezone of the unit under monitor and the server are different. In order for the times on the graph to be accurate, you must override the server locale with that of the unit under test. MRTG uses the Solaris 10 locale’s to determine timezones. Here, I am using Mountain time.

Timezone[target]: US/Mountain

Asymmetric Speeds
Since these are ADSL2+ physical links, the train rates for transmit and receive are different. MRTG can handle this too! You must specify what the transmit and receive link speeds are, so MRTG can calculate the percentages correctly. This is done by using the MaxBytes1 and MaxBytes2 parameters in the config file.

Take the EFM group bitrate (144,752,000 bps) divided by 8 to get 18094000 BYTES/s and 6,464,000 bps divided by 8 to get 808000 BYTES/s.

MRTG will also show a line on your graph to indicate the uppermost bound of the lower speed link. This makes it easier to read.

MaxBytes1[target]: 808000
MaxBytes2[target]: 18094000

Final MRTG Config File


## ADSL EFM Interface 1/4/29
Target[target]: 1.3.6.1.4.1.664.5.70.7.1.1.1.9.440029&1.3.6.1.4.1.664.5.70.7.1.1.1.10.440029:public@192.168.1.10:::::2 + 1.3.6.1.4.1.664.5.70.7.1.1.1.11.440029&1.3.6.1.4.1.664.5.70.7.1.1.1.12.440029:public@192.168.1.10:::::2 + 1.3.6.1.4.1.664.5.70.7.1.1.1.13.440029&1.3.6.1.4.1.664.5.70.7.1.1.1.14.440029:public@192.168.1.10:::::2
RouterUptime[target]: public@192.168.1.10
Timezone[target]: US/Mountain
WithPeak[target]: mw
Options[target]: growright, bits
LegendI[target]: UP
LegendO[target]: DN
MaxBytes1[target]: 808000
MaxBytes2[target]: 18094000
Title[target]: Traffic Analysis EFM Connection
PageTop[target]: <h1>Traffic Analysis EFM Connection</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Target</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>EFM Group</td>
 </tr>
 <tr>
 <td>Max Speed:</td>
 <td>144 MBits/s Downstream 6.7 Mbits/s Upstream</td>
 </table>
 </div>

Advertisements

How to parse Nest info (and graph it using MRTG)

How to parse Nest info (and graph it using MRTG)

This is the continuation of my previous post where I talked about setting up my Nest Thermostat.

Data Flow

Since Nest does not offer an open API, we have to cheat a little (ok, a LOT!). By using Scott Baker’s Python code, you can query the Nest web service to get your current status on the Nest (using your nest login, password and serial number).

Data is sent from the local Nest to the web. Python is used to grab the data from the web and put into my own MRTG compatible format. From the Nest data, I am graphing Temperature and Humidity, Heating/Cooling State and the Battery Levels for each of 2 thermostats.

This is all done using a perl program that creates an array from the Python program output, parses the array line by line, then creates a file that MRTG can read to do the actual graphing.

Perl Code

This perl script runs every 5 minutes on my Raspberry Pi to create the files needed for MRTG.

#!/usr/bin/perl -w
use strict;

###
### Perl Nest (using Python to scrape the web) to mrtg log file script
### Written by Kyle Platts
### Nest Gen 1 Thermostat

my $nest1Humidity;  # Nest Humidity
my $nest1Temp;      # Nest Temperature
my $nest1Voltage;   # Nest Battery Voltage
my $nest1HeatState; # Nest call for heat
my $nest1AcState;   # Nest call for A/C
my $nest2Humidity;  # Nest Humidity
my $nest2Temp;      # Nest Temperature
my $nest2Voltage;   # Nest Battery Voltage
my $nest2HeatState; # Nest call for heat
my $uptime = `uptime`;
chomp $uptime;
my @uptimeSplit = split (/ /, $uptime);
chop $uptimeSplit[5];
$uptime = ($uptimeSplit[3] . " " . $uptimeSplit[4] . " " . $uptimeSplit[5]);
## Nest 1 = Upstairs
my @nest1 = `/usr/local/bin/nest.py --user user\@you.com --password password --serial 123456789 show`;
## Nest 2 = Downstairs
my @nest2 = `/usr/local/bin/nest.py --user user\@you.com --password password --serial 123456780 show`;

foreach my $line (@nest1) {
 chomp $line;
 if ($line =~ /current_humidity/) { # line contains the humidity, split it out
 my @nest1HumSplit = split (/\:/, $line);
 $nest1Humidity = ($nest1HumSplit[1] * 100);
 }
 if ($line =~ /current_temperature/) { # line contains the temperature in C
 my @nest1TempSplit = split (/\:/, $line);
 $nest1Temp = (((($nest1TempSplit[1] * 9) / 5) + 32) * 100);
 }
 if ($line =~ /battery_level/) { # line contains the Battery Voltage
 my @nest1VoltageSplit = split (/\:/, $line);
 $nest1Voltage = ($nest1VoltageSplit[1] * 1000);
 }
 if ($line =~ /hvac_heater_state/) { # line contains the Heat State
 my @nest1HeatSplit = split (/\:/, $line);
 $nest1HeatState = $nest1HeatSplit[1];
 if ($nest1HeatState =~ /True/) {
 $nest1HeatState = 1;
 }
 else {
 $nest1HeatState = 0;
 }
 }
 if ($line =~ /hvac_ac_state/) { # line contains the A/C State
 my @nest1AcSplit = split (/\:/, $line);
 $nest1AcState = $nest1AcSplit[1];
 if ($nest1AcState =~ /True/) {
 $nest1AcState = 1;
 }
 else {
 $nest1AcState = 0;
 }
 }
}
foreach my $line (@nest2) {
 chomp $line;
 if ($line =~ /current_humidity/) { # line contains the humidity, split it out
 my @nest2HumSplit = split (/\:/, $line);
 $nest2Humidity = ($nest2HumSplit[1] * 100);
 }
 if ($line =~ /current_temperature/) { # line contains the temperature in C
 my @nest2TempSplit = split (/\:/, $line);
 $nest2Temp = (((($nest2TempSplit[1] * 9) / 5) + 32) * 100);
 }
 if ($line =~ /battery_level/) { # line contains the Battery Voltage
 my @nest2VoltageSplit = split (/\:/, $line);
 $nest2Voltage = ($nest2VoltageSplit[1] * 1000);
 }
 if ($line =~ /hvac_heater_state/) { # line contains the Heat State
 my @nest2HeatSplit = split (/\:/, $line);
 $nest2HeatState = $nest2HeatSplit[1];
 if ($nest2HeatState =~ /True/) {
 $nest2HeatState = 1;
 }
 else {
 $nest2HeatState = 0;
 }
 }
}

open (STATE, '>', '/home/pi/scripts/nest.log') || die $!;
 if ($nest1AcState or $nest1HeatState == 1) {
 print (STATE "1\n");
 }
 else {
 print (STATE "0\n");
 }
 if ($nest2HeatState == 1) {
 print (STATE "1\n");
 }
 else {
 print (STATE "0\n");
 }
 print (STATE "$uptime\n");
 print (STATE "Nest States\n");

open (VOLTAGE, '>', '/home/pi/scripts/nest_bat.log') || die $!;
 print (VOLTAGE "$nest1Voltage\n");
 print (VOLTAGE "$nest2Voltage\n");
 print (VOLTAGE "$uptime\n");
 print (VOLTAGE "Nest Battery Voltage\n");

open (NEST1, '>', '/home/pi/scripts/nest_1.log') || die $!;
 print (NEST1 "$nest1Temp\n");
 print (NEST1 "$nest1Humidity\n");
 print (NEST1 "$uptime\n");
 print (NEST1 "Nest 1 Temp and Humidity\n");

open (NEST2, '>', '/home/pi/scripts/nest_2.log') || die $!;
 print (NEST2 "$nest2Temp\n");
 print (NEST2 "$nest2Humidity\n");
 print (NEST2 "$uptime\n");
 print (NEST2 "Nest 2 Temp and Humidity\n");

Heating and Cooling states are converted to either a 1 or 0 for graphing. Temperature is in degrees C, but being an American, I have no frame of reference to what C feels like, so I convert it to Fahrenheit. I also make sure all of the values are mulitplied by 100 so I can graph with 2 decimal point accuracy.

MRTG Config File

######################################################################
# System: Nest Learning Thermostats
# Description: Nest Thermostats
# Contact: Kyle Platts
######################################################################

### Nest Status >> Descr: 'NestUpstairsTemp/Humidity'

Options[nest_1]: gauge,growright,nolegend,noinfo,absolute
LegendI[nest_1]: Temperature
LegendO[nest_1]: Humidity
ShortLegend[nest_1]: F
YTicsFactor[nest_1]: 0.01
Factor[nest_1]: 0.01
YLegend[nest_1]: Temp/Humidity
Target[nest_1]: `cat /home/pi/scripts/nest_1.log`
MaxBytes[nest_1]: 10000
YTics[nest_1]: 5
Title[nest_1]: Nest Upstairs Readings
PageTop[nest_1]: <h1>Nest Upstairs Readings</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostat</td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Upstairs Temperature & Humidity</td>
 </tr>
 </table>
 </div>

### Nest Status >> Descr: 'NestUpstairsTemp/Humidity'

Options[nest_2]: gauge,growright,nolegend,noinfo,absolute
LegendI[nest_2]: Temperature
LegendO[nest_2]: Humidity
ShortLegend[nest_2]: F
YTicsFactor[nest_2]: 0.01
Factor[nest_2]: 0.01
YLegend[nest_2]: Temp/Humidity
Target[nest_2]: `cat /home/pi/scripts/nest_2.log`
MaxBytes[nest_2]: 10000
YTics[nest_2]: 5
Title[nest_2]: Nest Downstairs Readings
PageTop[nest_2]: <h1>Nest Downstairs Readings</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostat</td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Downstairs Temperature & Humidity</td>
 </tr>
 </table>
 </div>

### Nest Status >> Descr: 'Nest Battery Voltages'

Options[nest_bat]: gauge,growright,nolegend,noinfo,absolute
LegendI[nest_bat]: Upstairs
LegendO[nest_bat]: Downstairs
ShortLegend[nest_bat]: V
YTicsFactor[nest_bat]: 0.001
Factor[nest_bat]: 0.001
YLegend[nest_bat]: Volts
MaxBytes[nest_bat]: 4000
Target[nest_bat]: `cat /home/pi/scripts/nest_bat.log`
YTics[nest_bat]: 5
Title[nest_bat]: Nest Battery Levels
PageTop[nest_bat]: <h1>Nest Battery Levels</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostat</td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Nest Battery Levels</td>
 </tr>
 </table>
 </div>

### Nest Status >> Descr: 'Nest On/Off State'

Options[nest]: gauge,growright,nopercent,noinfo,absolute
LegendI[nest]: Upstairs
LegendO[nest]: Downstairs
ShortLegend[nest]: &nbsp;
YLegend[nest]: HVAC On/Off
Target[nest]: `cat /home/pi/scripts/nest.log`
MaxBytes[nest]: 1
YTics[nest]: 2
Title[nest]: Nest HVAC Status
PageTop[nest]: <h1>Nest HVAC Status</h1>
 <div id="sysdetails">
 <table>
 <tr>
 <td>System:</td>
 <td>Nest Thermostats </td>
 </tr>
 <tr>
 <td>Maintainer:</td>
 <td>Kyle Platts</td>
 </tr>
 <tr>
 <td>Description:</td>
 <td>Nest HVAC Status</td>
 </tr>
 </table>
 </div>

In the MRTG config file, I use Factor and Y-ticks Factor to gain 2 decimal point granularity in my graphs. Since these are static values read each time, there is no need do math and figure out the “rate” of a digital output of the thermostat. You will see the use of gauge, nopercent and absolute on the states.
I do maintain percentages on humidity though, since that is what is calculated by the thermostat.

MRTG Log Files

These are the files that actually contain the data parsed from the web and and are read by MRTG for graphing.  Each file needs 4 lines. The first line is Value 1 (In) for MRTG. Line 2 is Value 2 (Out). Line 3 contains the Uptime if you choose to display it. Line 4 contains the graph title.

Nest Temperature and Humidity File

7466
3700
24 days, 3:54
Nest 1 Temp and Humidity

This indicates that right now, my upstairs Nest is 74.66 degrees F with 37% humidity.

Graphs

Nest Thermostat

Nest Thermostat

My wife got me a Nest thermostat (www.nest.com) for Christmas. I had it installed later that day.

For those of you who don’t know, the Nest is a learning thermostat with wireless capabilities. It has the capability to learn a schedule for you, lower the temp when you are away, and allow access remotely.

First Impression

It is slick! Setup was a breeze. Once on the wireless network, I created my Nest account and I was good to go. I loaded up the client on my iPad and I can now control my Nest from anywhere in the world.

Reports

The Nest has a daily energy use report so you can see what is happening in your house. It keeps track of the local weather too, so it knows if it was a warm or cold day.

20130128-154621.jpg

At a Glance

This is my favorite screen. It shows you all the data in one screen: inside temperature, outdoor temperature and humidity.

20130128-154832.jpg

Nest Installed

Here are a pair of pictures of the Nest installed. You can see my Dallas 1-Wire sensor hanging out underneath it.

20130128-155344.jpg

20130128-155353.jpg

Second Impression

I liked the Nest so much, I had to buy a second one for downstairs to run the hydronic floor. With my Arduino monitor, I could see that the difference in air temp would cause the floor to turn on. With 2 Nest’s, I am able to run the same schedule on both, so when the air temp drops overnight, the floor doesn’t run unnecessarily.

Monitoring the Nest

With the use of Scott Baker’s Nest.py program (Python API for Nest) and a little bit of Perl code, I can monitor the Nest’s in MRTG. The Nest does not have a published API, and it does not repsond to SNMP queries, so you have to query the web to get data, rather than query the device sitting on your local network. I hope Nest will open this up to developers at some point.

Here is the data flow.

Nest Thermostat — internet — Nest Web Portal
Raspberry Pi MRTG Server — python code — perl parse code — Nest Web Portal

Arduino Temperature Monitoring Update

Update

Having lived with my monitoring system for a little over a month, I can say that I found this project to be very fun. I now can see when and how long each of the hydronic floors is active and the temperatures around the different parts of the system.

From the data I have collected, I was able to experiment with the pump speeds. Each pump has three speeds. Monitoring the outlet and inlet temperatures showed that using the lowest speed left the most heat in floors. Previously, from my seat of the pants experience, it seemed that the middle setting was better, but the data shows that it is low and slow that works better.

Arduino Temperature Monitor

Radiant Floor Monitoring

My house has in-floor radiant heat installed in both the basement and garage floors. I’ve always wondered how often and when the circulator pumps run. I investigated getting some hour meters to see how much time accumulated for each pump, but most hour meters are run off of 12 Volts, and the pumps are 120 volt AC connected to 24 volt AC thermostats.

Doing some more investigation, I found an Arduino microcontroller could be used to monitor the pump status quite easily.

Arduino

Arduino is an 8 bit microcontroller powered by a 16 Mhz ATMEL ARM processor. Each Arduino contains several General Purpose Input Output pins that you can use to connect to various devices and sensors.

The more I looked into the Arduino, the more capabilities I found it had. You can quite literally do anything with an Arduino!

To the Lab!

My original idea was to use an Arduino Uno to monitor the 24 volt AC that controls the relays to the pumps. I had to get an optocoupler to connect 24 volt AC to an Arduino friendly 5 volt DC signal. I purchased a dual optocoupler and started to breadboard a circuit. Once I was satisfied with the circuit, I moved it to a prototyping shield and soldered it all up.

Temperature Sensors

The initial build was to include 2 temperature sensors, one colocated with each hydronic thermostat using the extra wires to the thermostats. I originally looked at using analog temp sensors, but I was concerned that the long cable runs would skew the readings. I then decided on the Dallas 1-Wire digital temp sensors. Using a single pair of wires, it is possible to power and read the sensors on a bus. You simply attach each additional sensor to the bus.

I found another blog post that showed temperature sensors epoxied to the water pipes in a solar heating set up. This gave me the idea to expand the system to monitor not only the ambient temperature, but also the inlet and outlet temperature of each zone and the boiler.

Rather than using epoxy, I had some thermal compound left over from a PC project, so I used it to thermally bond the temp sensor to the water pipe. I used zip ties to mechanically mount the sensors to the pipes. This seems to work quite well, as the temp sensors are consistent with the mechanical temperature gauges installed in the pipes.

How to monitor it all

The next step was to determine how I wanted to monitor all of the inputs. From my research, it looked like the Arduino is usually hooked up to a host PC or an ethernet sheild is used. Being the network professional that I am, the ethernet shield was my first choice.

Having decided upon ethernet as the transport protocol, I found an ethernet shield with Power over Ethernet built in! This allowed me to run just a single Cat 5 cable to the Arduino, and I didn’t have to get AC to the wall.

The most popular method of accessing data on the Arduino is to create a web server. While looking into the web server, I found an SNMP package. Once I saw I could do SNMP to the Arduino, I decided to go ahead and use MRTG to monitor the inputs. 4 birds killed with a single stone!

Arduino PoE

20130128-145437.jpg

Temperature Sensors

20130128-145545.jpg

The Whole Setup

20130128-150350.jpg