GroveStreams Another Data Service

In my continuing search for sites to put my house data on I encountered Grovestreams.com on the Arduino forum.  Well, since I have a cool setup to try these things out I decided to give it a try.  At first I got totally lost in the jargon inherent to cloud services.  These things talk about streams of data, component architecture, registering a service; you know, the kind of language that simply puts people that make things to sleep.  I persevered.

Once I started to understand what they meant by messing up example after example, I was ready to try some code.  I grabbed their python example and updated it to save one of my devices.  Since I'm a power usage freak, I chose my real time measurement of power usage.  It worked.  That was actually really cool, so I added outside temperature.  It worked too.  Then I went a little nuts and added the rest of my house data to it.

Then I was on a roll.  I modified the code again to use the techniques I used on the other cloud services.  I put it under the APSscheduler to do one minute updates, added it to the init table so it would restart if it had trouble and let it go.  It's running right now keeping track of the data I collect around the house for me.  So, what about a chart to look at?  Well, they have a dashboard like some of the other services and I created one to hold some charts; these charts are embeddable:

Pretty slick.  We've seen this before, but it's nice to see a new service like this implement it first instead of years later. They also have other 'widgets', gauges and things. What impressed me about this site is the versatility.  They have capabilities that I won't have time to try out. They can derive data from other data and present it.  It should be possible to build a chart based on my power usage and the billing charges multiplied together and actually show graphically how much money it's costing me to keep cool in the summer.  That's actually a little depressing when you think about it.

They even have alerts that can send SMS and email to you.  I created an alert to tell me when my house power usage goes over 10kWh.  It was annoying when it came in the first time and even more annoying when it came in the second time.  I actually have real time notification that I'm wasting money that comes in on my phone.  I could easily expand this to alert me when the garage door is open after sundown.  That would help a lot keeping the pack rats from stealing my tools and burying them in a mound outside.

Here's the code I'm actually running to update my house data on their site:

The python Sketch Script

#!/usr/bin/python
import time
from time import sleep
from datetime import datetime
import sys
from apscheduler.scheduler import Scheduler
import logging
from simplejson import encoder as jsonEncoder
import httplib
import StringIO
import gzip
import sqlite3
import pprint

# I keep all current values in a data base
DATABASE='/home/pi/database/desert-home'

# These are the secret numbers I use to get into
# the grovestreams service
org = "putyourownsecretthinghere";
api_key = "samewithyoursecretapikey";

#If you want lots of messages and debug information
# set this to true
DEBUG = False

# the site accepts compression, might as well use it
def compressBuf(buf):
zbuf = StringIO.StringIO()
zfile = gzip.GzipFile(mode = 'wb', fileobj = zbuf, compresslevel = 9)
zfile.write(buf)
zfile.close()
return zbuf.getvalue()

def updateGrovestreams():
# This is VERY different from their examples.  I named
# my streams with something I could understand and read
# Probably not the best way to do it, but it works really
# well for me.
component_id = "desert-home-id"
rpowerStream_id = "power_usage"
otempStream_id = "outside_temp"
apowerStream_id = "apparent_power"
voltageStream_id = "voltage"
currentStream_id = "current"
pfactorStream_id = "power_factor"
itempStream_id = "inside_temp"
ptempStream_id = "pool_temp"
pmotorStream_id = "pool_motor"

# This object really helps when displaying
# the rather complex data below
pp = pprint.PrettyPrinter(depth=10)

#get the millis since epoch
# in unix the epoch began back in 1970, look it up
now = datetime.now()
nowEpoch = int(time.mktime(now.timetuple())) * 1000

#assemble feed and convert it to a JSON string
feed = {};
feed['feed'] = {}
feed['feed']['component'] = []
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()

comp = {}
comp['stream'] = []
comp['componentId'] = component_id
feed['feed']['component'].append(comp)
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()

# Now I'm going to fill in the stream values, open database
# I took a brute force approach to building the dictionary that
# is converted into JSON.  I could have been much more elegant
# in building it, but the folks just starting out would have
# had a tough time understanding it
dbconn = sqlite3.connect(DATABASE)
c = dbconn.cursor()
# So, you make a stream to stuff things into.  It's actually
# a python dictionary that we'll pass to a JSON encoder a ways
# down into the code.  I'll be adding entries to this as I pull
# items out of the database
stream1 = {}  
stream1['streamId'] = rpowerStream_id
stream1['time'] = [] 
stream1['data'] = []
comp['stream'].append(stream1)

current_value = c.execute("select rpower from power").fetchone()[0]  
stream1['time'].append(nowEpoch)
stream1['data'].append(float(current_value))
# this is a cool way to debug this kind of thing.
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()
# notice how I get and item out of the database
# and add it to the dictionary.  I'll do this
# several times
stream2 = {}
stream2['streamId'] = otempStream_id
stream2['time'] = []
stream2['data'] = []
comp['stream'].append(stream2)
current_value = c.execute(
"select currenttemp from xbeetemp").fetchone()[0]
stream2['time'].append(nowEpoch)
stream2['data'].append(float(current_value))

stream3 = {}
stream3['streamId'] = apowerStream_id
stream3['time'] = []
stream3['data'] = []
comp['stream'].append(stream3)
current_value = c.execute(
"select apower from power").fetchone()[0]
stream3['time'].append(nowEpoch)
stream3['data'].append(float(current_value))

stream4 = {}
stream4['streamId'] = voltageStream_id
stream4['time'] = []
stream4['data'] = []
comp['stream'].append(stream4)
current_value = c.execute(
"select voltage from power").fetchone()[0]
stream4['time'].append(nowEpoch)
stream4['data'].append(float(current_value))

stream5 = {}
stream5['streamId'] = currentStream_id
stream5['time'] = []
stream5['data'] = []
comp['stream'].append(stream5)
current_value = c.execute(
"select current from power").fetchone()[0]
stream5['time'].append(nowEpoch)
stream5['data'].append(float(current_value))

stream6 = {}
stream6['streamId'] = pfactorStream_id
stream6['time'] = []
stream6['data'] = []
comp['stream'].append(stream6)
current_value = c.execute(
"select pfactor from power").fetchone()[0]
stream6['time'].append(nowEpoch)
stream6['data'].append(float(current_value))

stream7 = {}
stream7['streamId'] = itempStream_id
stream7['time'] = []
stream7['data'] = []
comp['stream'].append(stream7)
current_value = c.execute(
"select avg(\"temp-reading\") from thermostats").fetchone()[0]
stream7['time'].append(nowEpoch)
stream7['data'].append(float(current_value))

stream8 = {}
stream8['streamId'] = ptempStream_id
stream8['time'] = []
stream8['data'] = []
comp['stream'].append(stream8)
current_value = c.execute(
"select ptemp from pool").fetchone()[0]
stream8['time'].append(nowEpoch)
stream8['data'].append(float(current_value))

stream9 = {}
stream9['streamId'] = pmotorStream_id
stream9['time'] = []
stream9['data'] = []
comp['stream'].append(stream9)
tmp = c.execute("select motor from pool").fetchone()[0];
if (tmp == 'High'): # a little special handling for the pool motor
motor = 2
elif (tmp == 'Low'):
motor = 1
else:
motor = 0
stream9['time'].append(nowEpoch)
stream9['data'].append(int(motor))

# all the values are filled in, close the database
dbconn.close() # close the data base
# This will print the entire database I just constructed
# so you can see what is going on
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()
# exit() # I put this in for debugging.  It exits before
# the JSON string is constructed and sent off to grovestreams
# Of course you want to keep it commented until needed
#
# And this is where the JSON string is built
encoder = jsonEncoder.JSONEncoder()
json = encoder.encode(feed);
# and this will print it so you can see what is happening
if DEBUG:
print json # for debugging
print
sys.stdout.flush()

#Upload the feed
try:
print "Updating GroveStream ", time.strftime("%A, %B %d at %H:%M:%S")
sys.stdout.flush()
conn = httplib.HTTPConnection('www.grovestreams.com')

url = '/api/feed?&org=%s&api_key=%s' % (org, api_key)

compress = True
if compress:
body = compressBuf(json)
headers = {"Content-type": "application/json", "Content-Encoding" : "gzip"}
else:
body = json
headers = {"Content-type": "application/json", "charset":"UTF-8"}

conn.request("PUT", url, body, headers)

response = conn.getresponse()
status = response.status

if status != 200 and status != 201:
try:
if (response.reason != None):
print('reason: ' + response.reason + ' body: ' + response.read())
sys.stdout.flush()
else:
print('body: ' + response.read())
sys.stdout.flush()
except Exception:
print('HTTP Fail Status: %d' % (status) )
sys.stdout.flush()

except Exception as e:
print('HTTP Send Error: ' + str(e))
sys.stdout.flush()
   
finally:
if conn != None:
conn.close()

# I just discovered the statement below.
# someday I'll have go figure out what it really does.
if __name__ == '__main__':
print "started at ", time.strftime("%A, %B, %d at %H:%M:%S")
sys.stdout.flush()
logging.basicConfig()

#------------------Stuff I schedule to happen -----
scheditem = Scheduler()
scheditem.start()
# every minute update the data store on Xively
scheditem.add_interval_job(updateGrovestreams, seconds=60)
#
# A couple of people asked me why I put this statement in
# since I have it scheduled to happen every 60 seconds already
# Well, when you're debugging something it sucks to have to
# wait 60 seconds to see if you fixed it, so I do it 
# first, then let the scheduler take care of the rest.
#
updateGrovestreams()
while True:
time.sleep(20) #This doesn't matter much since it is schedule driven
 

Yes, in my usual form, it's got way too many comments.  It's also very inelegant; I'm not a big fan of strange statements that run really well but take an hour to understand so I just brute forced the URL and body creation of the request sent.  Notice that there isn't a library with obscure calls and methods and stuff.  This is all python code that I simply stole from them and added my particular items to.  Makes it nice that I don't have to learn how to use some special library.  I also put in a variable to control debugging so you can see what is happening.  It gets a little complex at a couple of points.

I didn't stop there.  I took their arduino example and modified it to update some items as well.  It worked first try.  The example had a little trouble since Grovestreams used Strings as a way of making the code more easily read and it started losing memory.  The poor little arduino has so little memory that I knew it would die over time, so I sent them a note describing the problem.  Guess what?  They responded!

Yes, in this 21st century world of minimal text messages, they actually answered me and came up with a new example that works better.  Unfortunately, I didn't save that one as an illustration for you, you'll just have to go to their site and get it yourself.  However, I did save the first version I tried:

The Arduino Sketch

/*

 Arduino GroveStreams Stream Feed via Ethernet

 The GroveStreams client sketch is designed for the Arduino and Ethernet.
 A full "how to" guide for this sketh can be found at https://www.grovestreams.com/developers/getting_started_arduino_temp.html
 This sketch updates several stream feeds with an analog input reading,
 from a temperature probe, via the GroveStreams API: https://www.grovestreams.com/developers/apibatchfeed.html
 The Arduino uses DHCP and DNS for a simpler network setup.
 The sketch also includes a Watchdog / Reset function to make sure the
 Arduino stays connected and/or regains connectivity after a network outage.
 Use the Serial Monitor on the Arduino IDE to see verbose network feedback
 and GroveStreams connectivity status.

 GroveStreams Setup:

 * Sign Up for Free User Account - https://www.grovestreams.com
 * Create a GroveStreams organization and select the Arduino blueprint
 * Enter a unique MAC Address for this network in this sketch under "Local Network Settings" 
 *    (Newer shields have the mac address on a sticker on the shield. Use that.)
 *    (A MAC address can also be generated within a GroveStreams organization: tools - Generate MAC Address)
 * Enter the GroveStreams org uid under "GroveStreams Settings" in this sketch 
 *    (Can be retrieved from a GroveStreams organization: tools - View Organization UID)
 * Enter the GroveStreams api key under "GroveStreams Settings" in this sketch  
 *    (Can be retrieved from a GroveStreams organization: click the Api Keys toolbar button, 
 *     select your Api Key, and click View Secret Key)

 Arduino Requirements:

 * Arduino with Ethernet Shield or Arduino Ethernet
 * Arduino 1.0 IDE

 Network Requirements:

 * Ethernet port on Router    
 * DHCP enabled on Router
 * Unique MAC Address for Arduino

 Additional Credits:
 Example sketches from Arduino team, Ethernet by David A. Mellis

 */

#include <SPI.h>
#include <Ethernet.h>
#include <MemoryFree.h>


// Local Network Settings
byte mac[] = {0x90, 0xA2, 0xDA, 0x00, 0x33, 0x33};        // Change this!!! Must be unique on local network. 
                                               // Look for a sticker on the back of your Ethernet shield.

// GroveStreams Settings
String gsApiKey = "You\'re going to need your own number here";   //Change This!!!
String gsOrg = "another number you need to put in";      //Change This!!!
String gsComponentName = "Temperature";                     //Optionally change. Set this to give your component a name when it initially registers.

char gsDomain[] = "grovestreams.com";   //Don't change. The Grove Streams domain. 
String gsComponentTemplateId = "temp";  //Don't change. Tells GS what template to use when the feed initially arrives and a new component needs to be created.
                                        // The blueprint is expecting "temp".

//GroveStreams Stream IDs. Stream IDs tell GroveStreams which component streams the values will be assigned to.
//Don't change these unless you edit your GroveStreams component definition and change the stream ID to match this.
String gsStreamId1 = "s1";   //Temp C - Random Stream. 
String gsStreamId2 = "s2";   //Temp F - Random Stream. Don't change.
String gsStreamId3 = "s3";   //Temp C - Interval Stream (20 second intervals). Don't change.
String gsStreamId4 = "s4";   //Temp F - Interval Stream (20 second Intervals). Don't change.

const int updateFrequency = 20 * 1000; // GroveStreams update frequency in milliseconds (the GS blueprint is expecting 20s)
const int temperaturePin = 0;          // Then Temperature pin number.    

// Variable Setup
String myIPAddress;  //Set below from DHCP. Needed by GroveStreams to verify that a device is not uploading more than once every 10s.
String myMac;        //Set below from the above mac variable. The readable Mac is used by GS to determine which component the feeds are uploading into.

long lastConnectionTime = 0;    //Don't change. Used to determine if the Ethernet needs to be reconnected. 
boolean lastConnected = false;  //Don't change. 
int failedCounter = 0;          //Don't change. 

// Initialize Arduino Ethernet Client
EthernetClient client;


void setup()
{
  // Start Serial for debugging on the Serial Monitor
  Serial.begin(9600);

  // Start Ethernet on Arduino
  startEthernet();
  randomSeed(analogRead(1));
}

void loop()
{

  // Print Update Response to Serial Monitor
  if (client.available())
  {
    char c = client.read();
    Serial.print(c);
  }

  // Disconnect from GroveStreams
  if (!client.connected() && lastConnected)
  {
    Serial.println("...disconnected");
    Serial.println();
    showMem();

    client.stop();
  }

  // Update sensor data to GroveStreams
  if(!client.connected() && (millis() - lastConnectionTime > updateFrequency))
  {
    String tempC = getTemperatureC();
    Serial.print (tempC);
    String tempF = getTemperatureF();
    Serial.print (tempF);
    updateGroveStreams(tempC, tempF);
  }

  // Check if Arduino Ethernet needs to be restarted
  if (failedCounter > 3 ) {
    
    //Too many failures. Restart Ethernet.
    startEthernet();
  }

  lastConnected = client.connected();
}

void updateGroveStreams(String tempC, String tempF)
{
  Serial.println("\nin update routine");
  if (client.connect(gsDomain, 80))
  {         

    //Assemble the url used to pass the temperature readings to GroveStreams
    // The Arduino String class contains many memory bugs and char arrays should be used instead, but
    // to make this example simple to understand we have chosen to use the String class. 
    // No none memory issues have been seen with this example to date.
    //We are passing temperature readings into two types of GroveStreams streams, Random and Interval streams.
    String url = "PUT /api/feed?&compTmplId=" + gsComponentTemplateId + "&compId=" + myMac + "&compName=" + gsComponentName;
    url += "&org=" + gsOrg + "&api_key=" + gsApiKey;

    url += "&" + gsStreamId1 + "=" + tempC;  //Temp C - Random Stream
    url += "&" + gsStreamId2 + "=" + tempF;  //Temp F - Random Stream
    url += "&" + gsStreamId3 + "=" + tempC;  //Temp C - Interval Stream (20 second intervals)
    url += "&" + gsStreamId4 + "=" + tempF;  //Temp F - Interval Stream (20 second intervals)

    url += " HTTP/1.1";
    Serial.print(url);
    client.println(url);  //Send the url with temp readings in one println(..) to decrease the chance of dropped packets
    client.println("Host: " + String(gsDomain));
    client.println("Connection: close");
    client.println("X-Forwarded-For: "+ myIPAddress); //Include this line if you have more than one device uploading behind 
                                                      // your outward facing router (avoids the GS 10 second upload rule)
    client.println("Content-Type: application/json");
    client.println();

    if (client.available())
    {
      //Read the response and display in the the console
      char c = client.read();
      Serial.print(c);
    }

    lastConnectionTime = millis();

    if (client.connected())
    {     
      failedCounter = 0;
    }
    else
    {
      //Connection failed. Increase failed counter
      failedCounter++;

      Serial.println("Connection to GroveStreams failed ("+String(failedCounter, DEC)+")");   
      Serial.println();
    }

  }
  else
  {
     //Connection failed. Increase failed counter
    failedCounter++;

    Serial.println("Connection to GroveStreams Failed ("+String(failedCounter, DEC)+")");   
    Serial.println();

    lastConnectionTime = millis(); 
  }
}

void startEthernet()
{
  //Start or restart the Ethernet connection.
  client.stop();

  Serial.println("Connecting Arduino to network...");
  Serial.println();  

  //Wait for the connection to finish stopping
  delay(1000);

  //Connect to the network and obtain an IP address using DHCP
  if (Ethernet.begin(mac) == 0)
  {
    Serial.println("DHCP Failed, reset Arduino to try again");
    Serial.println();
  }
  else
  {

    Serial.println("Arduino connected to network using DHCP");
    Serial.println();

    //Wait to ensure the connection finished
    delay(1000);

    //Set the mac and ip variables so that they can be used during sensor uploads later
    myMac =getMacReadable();
    Serial.println("MAC: " + myMac);

    myIPAddress = getIpReadable(Ethernet.localIP());
    Serial.println("IP address: " + myIPAddress);
  }

}

String getMacReadable() 
{
  //Convert the mac address to a readable string
  char macstr[20];
  snprintf(macstr, 100, "%02x:%02x:%02x:%02x:%02x:%02x", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
  return String(macstr);
}

String getIpReadable(IPAddress p)
{
  //Convert the ip address to a readable string
  String ip;
  for (int i =0; i < 3; i++)
  {
    ip += String(p[i], DEC);
    ip += ".";
  }
  ip +=String(p[3], DEC);
  return ip;
}

String getTemperatureF() 
{
  //Get the temperature analog reading and convert it to a string
  float voltage, degreesC, degreesF; 

  //voltage = (analogRead(temperaturePin) * 0.004882814);
  //degreesF = degreesC * (9.0/5.0) + 32.0;
  degreesF = float(random(70, 120));

  char temp[20] = {0}; //Initialize buffer to nulls
  dtostrf(degreesF, 12, 3, temp); //Convert float to string

  String stemp = temp;
  stemp.trim();  //Trim off head and tail spaces
  return stemp; 
}

String getTemperatureC() 
{
  //Get the temperature analog reading and convert it to a string
  float voltage, degreesC, degreesF; 

  //voltage = (analogRead(temperaturePin) * 0.004882814);
  //degreesC = (voltage - 0.5) * 100.0;
  degreesC = float(random(20, 40));

  char temp[20] = {0}; //Initialize buffer to nulls
  dtostrf(degreesC, 12, 3, temp); //Convert float to string

  String stemp = temp;
  stemp.trim();  //Trim off head and tail spaces
  return stemp; 
}

void showMem(){
  char Dbuf [100];
  
  strcpy_P(Dbuf,PSTR("Mem = "));
  Serial.print(Dbuf);
  Serial.println(freeMemory());
}

This worked really well, but like I said above, it will die over time because the Streams library will run the arduino out of memory.  I left the code in it that I used to check memory usage and also the random number generation I used to test it.

Notice that the call to send the data is different?  Of course you did.  The call is part of their batch update API that is much simpler to use than the JSON call I used in the python example.  This makes getting the data up there much easier for the arduino folks.  They're even working on improvements to make the API more easily understood.  The python folk seem to like infinitely complex data structures, so I'm leaving that example as it is.

They even have a full blown tutorial on hooking the arduino up to their service.  Unlike a lot of the tutorials out there, this one can actually be read.  You don't even have to click on page after page like the darn instructables we've all learned to love.

I really like this site.  Like I said, I haven't gotten past the very surface of its capabilities, and probably never will, but it was nice to see a site where the folks running it actually care if people can use it.