Ethernet, Running Out of Connections

When I added thingspeak <link>as a repository for my data online, I started to notice that my House Controller <link> was freezing up and rebooting pretty often.  This isn't as bad a problem as one would think since the arduino only takes a few seconds to reboot and will continue right where it left off with only a minute or so worth of data lost.  However, it drove me nuts after a day or so of watching it randomly reboot for no apparent reason.

I moved the controller over to my breakfast table and plugged it into a laptop to monitor what was happening.  When the problem happened, it seemed to be at 10 minute intervals, but sometimes it would be at a minute interval.  What the heck?  I finally noticed that it was happening when the controller was doing a lot of internet interaction.  This made sense because I had recently added updates to thingspeak, so maybe I had screwed that up somehow.

Well, I finally had a brainstorm and realized that I was running out of connections on the ethernet card.  I use a 5100 based ethernet card for the arduino.  I use one of the older arduino ethernet cards, and I have heavily modified it so I can tell if the ethernet actually connects OK, and these devices only support four simultaneous connections..  See my page on the Thermostat <link> for details on the modifications I did. I connect to these devices using ethernet:  North home thermostat, South home thermostat, Dyndns to publish IP address, Dyndns to get my current IP address, Cosm, emoncms, thingspeak, and a connection for the web server that runs on the arduino itself. That's a possibility of eight connections, and somehow I must be using them all up.

I was.  I use timers for connecting to devices, and over time, I had too many of them trying to connect, or already connected at the same time.  The web server running on the little machine is always connected; the thermostats are connected every thirty seconds as clients; Cosm, emoncms, and thingspeak, connect every minute; Dyndns is connected every ten minutes and again a few seconds later if my external IP address changes due to my ISP.  Obviously, these were happening at the same time occasionally (often) and the board was running out of connections and pausing which caused the watchdog timer to fire.

I can't emphasize enough how useful timers are.  These little machines don't have an operating system that can handle scheduling tasks or switching between actions that some projects require.  You have to handle all this stuff yourself.  If you want something to happen at noon, you can either wait until your clock hits noon, or set a timer to check it for you.  If you are waiting, you can't do anything else.  Using timers to schedule things allows the little devices to behave like a machine many times its capabilities.  If you're into automating things, you should examine timers extensively.

What should I do?  There are lots of solutions, things like a table of connections that I can check to see if one is available, add a call to the ethernet library to tell me if I have a connection left to use, try and connect then check the return value for the right error, etc.  I decide not to do any of those.  Instead, I staggered the timers such that they split the minute up into intervals where each device could have a clear shot at getting a connection.  This should be fine since the various interactions only take a couple of seconds max and the odds of inserting a new bug using any of the other techniques was extremely high.  My original code to set up the various server interactions was:

Original Timer Setup

void initTimers(){
  
  Alarm.timerRepeat(30, updateThermostats); 
  Alarm.timerRepeat(60, updatePachube);     
  Alarm.timerRepeat(60, updateEmoncms);
  Alarm.timerRepeat(60, updateThingspeak);
  Alarm.timerRepeat(10*60,checkPublicIp);   
  
 /* other timers to control devices follow this */

So, you can see that every minute, I try to update at least 4 devices and every ten minutes, I add a couple more.  The odds of running out of connections (maximum 4) are high; it's a wonder the board stayed up for more than a couple of minutes.

There isn't any good solution to doing this differently using the same technique.  If I stagger the times, it will just be rebooting less often because they will eventually fall in the same couple of seconds; I know, I tried that first.  So, I redid the way the timers are set up:

Modified Timer Setup

void initTimers(){
  // I want the various timers to work on the minute boundary
  while(second() != 0){
    delay(500);
    doggieTickle(); // so the watchdog doesn't time out on me
  }
  // I have to stagger the timers for the ethernet
  Alarm.timerOnce(5, updateThermostats);                           
  Alarm.timerOnce(10, updatePachube);
  Alarm.timerOnce(15, updateEmoncms);
  Alarm.timerOnce(20, updateThingspeak);
  Alarm.timerRepeat(10*60,checkPublicIp);

First, I wait until the time hits a minute boundary by waiting until seconds == zero.  I have to keep the watchdog timer reset during this period to make sure I don't reboot the board.  Then I set timers for the offset from the minute boundary that I want for the various devices.  When these timers expire, I set timers for the next time the device is supposed to run.  For example, the thermostats are handled like this:

The Arduino Sketch

void updateThermostats(){  
// this will interogate the thermostats to make sure data is current

  Alarm.timerOnce(30, updateThermostats);  // check again in 30 seconds.

  getThermostat(0, "status", true);
  getThermostat(1, "status", true);
}

And, then when it cycles around again, a new timer will be set to do the same thing.  Basically, when I initialize the timers, I'm just kick starting an action that carries itself on after that.  This way the devices fire off at certain times within a minute so they don't conflict with each other as much, which should leave me with a free connection or two at all times.

Most people that do this kind of thing use a home computer to collect and forward data; that isn't a solution for me.  I don't want to tie up a large machine doing this kind of service since they can be useful for other things (surfing and games).  That's why I built the house controller in the first place.  Also, laptops and desktops take a long time to reboot and that doesn't go well with experimental software.  Mainly though, what the heck kind of pleasure is there in using a great big ol' machine to do something like this?

For my next project, I'm going to move this device up to the latest version of the arduino IDE.  I've been using IDE 21 for a long time now and the latest is 1.0.3 or something.  They changed a lot of stuff and redid most of the libraries, and I have changed some of the libraries to support my work, so this may be a challenge.