Trackbacks disabled

Minor note here that I turned off trackbacks because some enterprising idiot has found a way to use it to send spam. I have very little patience for advertising and even less patience with spam. Besides, after having seen thousands of viagra and male enhancement emails in my daily job, it surprises me that there are still people out there who think I’d be gullible enough to respond.

I’d like to believe the content here, what little there is, is useful to someone and it would be nice if trackbacks reflected the fact that others value what’s posted here. But, when that feature gets abused in this manner, I’m not letting it make it through to the readers. If I don’t want to see it, I’m sure you don’t want to either.

We now return to our regularly scheduled programming…

Complete code for sun_switch project

Some time ago, I started documenting my project to automatically switch on/off some external lighting on the house. You can read the full rationale behind the project in this post. I’ve been delayed in getting this posted because I’ve had other personal projects that have been a higher priority lately.

Today, I’m posting the (nearly) final code for the entire project. All you need to play with this is a WWVB receiver, an Arduino, the TimeLord library, and an optional LED. The LED would be attached to the ssrPin output (with a current limiting resistor) to give an indication of whether the lights would be on or off. Initially, the output defaults to having the lights on. This is so that a power outage in the middle of the night will default to the on state until the system receives a valid time from the WWVB receiver. The LED that’s built into the Arduino is not used for this because I used it in another part of the code to indicate the stability of the received signal. That LED should turn on/off at a rate of once per second if the received signal is clean and free of errors.

There are probably a few bugs in the code at this point, but my testing has shown it to be fairly reliable – enough so that I’m going to be working on a project enclosure as well as some modifications to the electrical in the garage to put this project to good use. I think this code demonstrates a wide range of the capabilities of the arduino. There is some use of the default libraries as well as some custom programming that utilizes the atmega168 hardware directly.

I don’t think I’ll draw the ire of the Arduino haters on this one, but you never know. Personally, I think the flexibility of the platform is something that most of ‘haters’ don’t understand. You don’t have to use any of the libraries if you don’t want to and you can pretty much just skip forward to writing straight C for the mega168 if that’s what you want to do. Personally, I wouldn’t pass up an opportunity to use a development platform with those capabilities; especially when you can avoid spending extra time and money on creating custom boards.

If you want more details and explanations of the code, please read my prior posts.

Source: sun_switch

Translating WWVB time to local time

We’re almost at a point where I can share the final code. One of the minor hurdles I had to tackle was converting UTC time with a day of year value into a more standard Gregorian local time. It turns out that the TimeLord library doesn’t have the facilities for doing this conversion on it’s own – probably because it was designed to work with a real time clock module instead of a WWVB receiver with my clock code.

Below is the code I developed for doing the conversion. This function is much longer than I would have liked, and I try to keep things small so it’s easier to debug. We start off with a definitition of what time zone we’re in. I’m in GMT-5, so the definition goes as follows:

#define timezone -5

Also, we’ll need our definition of ‘struct time t’ which carries the UTC version of our current time:

struct TIME {
  uint8_t seconds;
  uint8_t minutes;
  uint8_t hours; 
  uint16_t doy; // day of year
  uint16_t year;
  uint8_t leapYear;
};

volatile struct TIME t; // our only global variable

If I recall correctly, those are the only definitions that are missing from the following code block. This code tries to generically calculate local time without the added issue of having to jump forward or back due to DST and it should work for all timezones. Note also that a good chunk of the code is trying to compensate for the local day/month/year changing due to crossing midnight on a particular day. I probably didn’t need to worry about that since the actual sunrise/sunset values will only change by about a minute from one day to the other. I guess I was being a little OCD when I wrote this.

// compute gregorian date from time struct t
void getGregorianDate(byte cdate[])
{
  uint8_t months[] = {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
  uint16_t daysLeft = t.doy;
  uint8_t curMonth = 1;
  int8_t curHour, curDay, 
         curYear = t.year - 2000;
  
  if (t.leapYear == 0)
    months[1] = 28;
    
  while (daysLeft > months[curMonth-1])
  {
    daysLeft -= months[curMonth-1];
    curMonth++; 
  }
  
  // convert our stored UTC time to local time
  if (((int8_t) t.hours + timeZone) < 0)
  {
    // adjust for negative timeZone constant with rollovers 
    curHour = t.hours + 24 + timeZone;
    curDay = daysLeft - 1;
    
    if (curDay < 1) // roll back the month
    {
      // we're actually on the last day of the previous month
      curMonth--;
      curDay = months[curMonth - 1];
    }
    
    if (curMonth < 1) // roll back to prior year
    {
      curDay = months[11];
      curMonth = 12;
      curYear--;
    }
  }
  else // positive calculation with possible rollovers
  {    // note that even if Timezone is a negative constant, this code still works
    curHour = t.hours + timeZone;
    curDay = daysLeft;
    if (curHour > 23) // day
    {
      curHour -= 24;
      curDay++;
    }
    if (curDay > months[curMonth - 1]) // month
    {
      curDay = 1;
      curMonth++;
    }
    if (curMonth > 12) // year
    {
      curMonth = 1;
      curYear++;
    }
  }
  cdate[tl_second] = t.seconds;
  cdate[tl_minute] = t.minutes;
  cdate[tl_hour] = curHour;
  cdate[tl_day] = curDay;
  cdate[tl_month] = curMonth;
  cdate[tl_year] = curYear;
}

Could this be better? Absolutely. I just haven’t taken the time to go through and simplify it yet. There are too many variables being used, for one. I could eliminate all of the curDay, curMonth, and curYear references and use the cdate[] array in their place. I’m also not sure we really have to worry so much about whether the end result of adding the timezone results in a positive or negative value. There’s something there that makes me think I could possibly cut the code for calculating the day, month, and year in half, but the technique for doing so isn’t clear to me just yet. The one good thing I can say about it is that it works great in it’s current form, so I’m leaving it well enough alone for the moment.

Calculating Sunrise/Sunset

So, now that we have most of the components of this project posted (including this post), I can finally give you the details of what we’re trying to accomplish. My house is situated in a smallish town without a lot of street lighting and the nearest light is too far away to effectively illuminate the house. Even worse than that, I have a driveway that slopes back towards the house creating a small, dark area just in front of the garage door. The problem with this arrangement is that it’s very easy for someone to spend a lot of time trying to steal gas from the one car I have that doesn’t fit in the garage. (funny tip: If you’re trying to steal gas, don’t go after a 4 cylinder, 35 MPG car like the guy in my neighborhood did. There is, on average, less than 4 gallons of fuel in the tank!) To help with this problem, I do have a pair of recessed lights on the house over the garage, but I don’t have a way to effectively turn them on and off at the right time of day. Currently, I have to leave the lights on all the time due to an unpredictable personal schedule of work, family, and friends which isn’t very efficient.

What about a light sensor? I did think of that solution, but that has its own set of problems. Being in the northeast, we get a fair bit of snow in the winter time. That snow can reflect a lot of light and cause the sensor to turn the lights off at the wrong time. Taking it a step further, if someone’s willing to resort to minor criminal activity, nothing would stop them from figuring out some way to fool the sensor since it would have to be mounted on the outside of the house where it would be accessible and visible.

My solution is to use a combination of a RTC, the WWVB time signal, and the TimeLord library to accurately compute sunrise and sunset. The resulting circuit would automatically turn the lights on at sunset and turn them off at sunrise. Not only would this be an energy savings over the course of each day, but with advancements in LED technology, I could reduce my energy use to just pennies per month and get the benefit of having a properly lighted driveway. Furthermore, an addition of some wireless communications could allow me to plot the energy usage over the course of the year. That last feature isn’t planned for this version but it wouldn’t be difficult to do.

So, how do you compute sunrise and sunset? Not being afraid of a little math, I started looking online at the equations necessary to accomplish this. I found this page that had all of the calculations and put together some proof of concept code to see how accurate it would be on the arduino. Being spoiled from working on PC’s I was a little surprised to find that the answers didn’t make much sense. In particular, the trig functions in the avr-libc package tended to give me the most inaccurate values. Furthermore, the calculation of the number of days since the last epoch was giving me a number that was too large to be represented on the microcontroller.

My next thought was to add a floating point processor to the circuit, but the problem with that is 1) added chips take up space, making the project bulky …and… 2) added chips drive up the overall cost of the project. Sometimes the best path forward is the simplest, so I did a search online to see if anyone else had already solved the problem. As it turns out, the TimeLord library does exactly what I needed it to do and it’s very easy to use. My only suggestion for the authors of this library would be to improve the documentation. Sometimes it’s hard to tell if values should be passed as local time or UTC. Beyond that, I would also add functions so that the user can pass either local or UTC without having to do the conversion. However, it did solve my problem without needing any extra circuitry.

I will be posting the full code to this project at a later date. For the time being, though, I’m posting the pieces in the same way I developed the final code which is to say that I put together several small and easy to debug code modules and then later integrated all of them into one large project. The benefit of doing it this way is that it’s much easier and faster to debug small modules than it is to try and put the whole project together in one attempt. Here is the proof of concept code for this week:

#include <TimeLord.h>

// current date - BTW - don't do this if you can avoid it.  Globals are evil.
int curYear = 2010;
int curMonth = 11;
int curDay = 12;

// west longitude and north lattitude - approximate - 
//      I'm not giving you my address in an online code example anyway... :-)
double Lw = -78;
double Ln = 43;

void setup() 
{
  // put your setup code here, to run once:
  Serial.begin(9600);
  
  TimeLord tardis;
  tardis.TimeZone(-5 * 60);
  byte day[] = { 0, 0, 12, curDay, curMonth, curYear }; // noon
  tardis.Position(Ln, Lw);
  if (tardis.SunRise(day))
  {
    Serial.print("Sunrise: ");
    Serial.print((int) day[tl_hour]);
    Serial.print(":");
    Serial.println((int) day[tl_minute]);
  }
  
  if (tardis.SunSet(day))
  {
    Serial.print("Sunset: ");
    Serial.print((int) day[tl_hour]);
    Serial.print(":");
    Serial.println((int) day[tl_minute]);
  }
}

void loop() {}

The next post in this series will have the fully integrated and tested code. I still have some modifications to make to the electrical in my garage to enable this to work, so it may be a while before I can post pictures of the final project. My goal is to have this done and installed sometime before the end of April.

Arduino as a RTC

So, I’ve been putting together this project where I’m receiving accurate time via a WWVB receiver (see previous post). To really make use of this, I need some way to keep track of the time during those times of the day when the WWVB signal isn’t available.

Initially, I looked at the chronodot from Adafruit industries, but one of the things I didn’t like about it was the fact that the battery mounts to the back side of the breakout board. I know that this was done to reduce the cost of the breakout board, but it makes the end result kinda bulky. Besides that, changing out that battery could be tricky depending on how you mount it in your project enclosure. Also, the fact that it’s round doesn’t help when have to mount it into my project case, either.

In addition to considering the chronodot, I also have been waiting on a similar product from Sparkfun.com which is based on the ds3234 real time clock module. Their version, at the time I started development, was simply just the bare timekeeping chip and the breakout for it was unavailable. Fast forward a few months and now the breakout is available, but yet again, the battery is mounted to the back of the board :-(

Not being too sure of the best way to proceed, it hit me one day that I might be able to program the arduino to perform the real time clock functions. Yeah, the crystal that’s on the board may drift with temperature by a few parts per million, but if I’m receiving a valid WWVB signal at some point of the day, I might not need extreme accuracy at all. In fact, after considering the requirements of my project, it wouldn’t really matter if the time drifted by up to 5 minutes over the course of the day.

Below is the proof of concept code I wrote:

// What do you do when an accurate, temperature compensated real time 
//    clock chip breakout is unavailable from your favorite vendors?  
//    Well, you first consider the less accurate battery backed 
//    alternatives until you later realize that there's not really a 
//    big difference between using the non-temp-compensated rtc and 
//    just programming the arduino to do it directly. 
//
//   So - that's what this is.  A real time clock implemented based on 
//    16bit Timer1 and the cpu's clock frequency.  Since both the 
//    dedicated rtc and the this rtc are both dependent on temperature 
//    variations and since both are using crystals for their time source, 
//    the amount of error should be acceptible.  Also, if you integrate 
//    wwvb receiver code with this code, your accuracy should be more 
//    than close enough...

#include <avr/io.h>
#include <avr/interrupt.h>

#define ISR_TIMER1_COUNT 15625 // 16.0MHz clock / (prescaler = 1024)

struct TIME {
  int seconds;
  int minutes;
  int hours; 
};

volatile TIME t;  // global time struct

void setupTimer1(void)
{
  // Setup Clear Timer on Compare Match mode.  We should interrupt each 
  //    time TCNT1 is equal to the value in OC1A register.  TCNT1 will 
  //    automaticall be reset to 0 each time a compare match happens.  
  //    Because this is done in hardware, the interrupt frequency should 
  //    be very stable (no interrupt service routine overhead to deal 
  //    with).
  
  // Please refer to the atmega168 data sheet for an explanation of the 
  //    registers and the values chosen here.
  
  // WGM13:0 = 4  for CTC on OCF1A value.  Prescaler set to divide by 1024.
  TCCR1A &= ~(1<<COM1A1) &   // Clearing bits
            ~(1<<COM1A0) &
            ~(1<<COM1B1) &
            ~(1<<COM1B0) &
            ~(1<<WGM11) &
            ~(1<<WGM10);
            
  TCCR1B &= ~(1<<ICNC1) &    // Clearing bits
            ~(1<<ICES1) &
            ~(1<<WGM13) &
            ~(1<<CS11); 
            
  TCCR1B |= (1<<WGM12) |    // Setting bits
            (1<<CS12) |
            (1<<CS10);
  
  OCR1A = ISR_TIMER1_COUNT;
  
  
  // OCIE1A interrupt flag set
  TIMSK1 |= (1<<OCIE1A);
  
  // Start counter at 0, not that it would matter much in this case...
  TCNT1 = 0;
}

// This interrupt service routine gets called once per second
ISR(TIMER1_COMPA_vect)
{
  t.seconds++;
  if (t.seconds > 59)
  {
    t.minutes++;
    t.seconds = 0;
  }
  if (t.minutes > 59)
  {
    t.hours++;
    t.minutes = 0;
  }
}

void setup()
{
  Serial.begin(9600);
  
  t.seconds = 0;  // initialize our time struct
  t.minutes = 0;
  t.hours = 0;
  
  setupTimer1();
  sei();    // allow interrupts globally
}

void loop()
{
  int sec = t.seconds;
  while (sec == t.seconds) {delay(100);} // wait for t.seconds to increment
  
  Serial.print("Time: ");
  Serial.print(t.hours);
  Serial.print(":");
  Serial.print(t.minutes);
  Serial.print(":");
  Serial.println(t.seconds);
}

You’ll note here that very little of the actual arduino library is used at all. I decided to use the 16bit Timer1 on the microcontroller directly with an interrupt to keep track of the time. When I started monitoring the program, I wrote down the current time on the computer. After 24 hours, I came back and again compared the time on the computer with the time that was coming out of my program and discovered that the time drift was only 3 seconds over the course of a full day! If it performs that well, I guess I don’t need a separate clock chip.

In the next post, I’ll be writing about the TimeLord library.

WWVB – Accurate time reception via a radio receiver

I’ve been working on a project that integrates a few ideas into one really cool project. The details of the project itself will become apparent in a few more posts as I get closer to completing the project, but I don’t want to put it all out there at once. It would make this post overly long, so please be patient as the details are revealed.

Part of this project depends on using a WWVB receiver to receive an accurate time/date signal from the broadcast station in Colorado, USA. The NIST time signal is a national radio signal that propagates along the ground throughout much of the United States. This signal is just a simple carrier wave that’s modulated to provide a digital stream of time data to anyone with a receiver. Once the signal is fully received and decoded, you have a time reference that’s just as accurate as your GPS. The nice thing about using WWVB is that the signal easily propagates through walls and you don’t need a clear view of the sky.

For this project, I’m using a WWVB receiver module I purchased at Sparkfun.com. This module only requires three connections: power, ground, and signal out. No complicated interfaces here, that’s for sure.  You may note in the comments on the Sparkfun site that if you live on the east coast, it’s recommended to get the longer loopstick antenna from Digikey.  I should note here that I live in western NY and I’m having no problems receiving the signal between the hours of 10pm (22:00 hours) and 12:30pm UTC.  My receiver isn’t even near a window.  The signal, in my case, has to penetrate through two walls of my house to be received and that’s not even considering the fact that there are a lot of other houses in my neighborhood that the signal has to get through.

So, now that we have our radio receiver, we need a microcontroller and some code to be able to receive and decode the signal.  I chose an arduino pro mini for the microcontroller platform.  It’s simple to program, tiny, and more than capable of doing what I need for this project.  Of course, I also had to add a regulated 5V supply and a few other parts to make the project complete.  More on that later.

As for code, there are many examples online to look at.  I searched and found several examples, but I didn’t like the way they were written.  Very few, if any, showed good programming practices such as code reuse and small, tightly maintained functions that were clearly commented and documented.  Being frustrated with this situation, I wrote my own code from scratch.  It was easier than I expected:

note: wordpress does not make posting C code easy. If you notice any clipped or improperly formatted sections in the code below, let me know so I can fix it.:

#include 

#define wwvb 10  // input pin

#define pulseMarker 0 // all because processing doesn't understand 'enum'...
#define pulseOne 1
#define pulseZero 2
#define pulseInvalid -1

int8_t wwvbLastState;
int8_t ledPin = 13;
int8_t ledState = 0;

struct WWVBFRAME
{
  int8_t minutes;
  int8_t hours;
  int16_t doy;
  int16_t year;
  int8_t leapYear;
};

void setup()
{
  Serial.begin(9600);
  pinMode(wwvb, INPUT);
  pinMode(ledPin, OUTPUT);

  // At the start of our program, we have no idea what state our input is in
  //  or even where we are in the sequence.  Wait for two consecutive
  //  markers...
  wwvbLastState = digitalRead(wwvb);
  while (digitalRead(wwvb) == wwvbLastState) {} // wait for a transition to
                                                //  happen

}

/* note:  0.8S is a marker.  There are two to three consecutive markers at
              the start of the minute.
          0.2S is a zero
          0.5S is a one

          The signal transitions low for the start of each pulse.

          In this program, pulse types are represented as an int value
          according to the following:  0 = marker; 1 = logical one;
          2 = logical zero.  An enum would have been a more elegant solution,
          but processing doesn't have enums as far as I can tell (gives a
          compile error).
*/

void loop()
{

  // Rant:
  // Somethings *really* not right when you can declare this variable *two*
  //  different ways *and* one of those ways is kinda broken.  For instance,
  //  saying "WWVBFRAME wwvbFrame" works if you never have to pass it to a
  //  function.  To get passing to a function to work, you have to change the
  //  declaration to "struct WWVBFRAME wwvbFrame".  This is broken, imho.
  //  Either complain ALL THE TIME about the incorrect declaration or ACCEPT
  //  BOTH.  The last thing I want to do is to waste time debugging code that
  //  fails mysteriously without any explanation!
  struct WWVBFRAME wwvbFrame;
  int8_t dataValid = 0;

  resetWWVBFRAME(&wwvbFrame);

  while(1)
  {
    if (dataValid > 0)
    {
      Serial.print("valid: ");
      showTime(&wwvbFrame);
    }
    resetWWVBFRAME(&wwvbFrame);
    dataValid = 0;

    waitMinuteMarker(); // wait for next frame start...

    wwvbFrame.minutes = wwvbGetMinutes();
    if (wwvbFrame.minutes < 0) continue; // error - go back to the beginning
                                         //  of the while loop  

    wwvbFrame.hours = wwvbGetHours();
    if (wwvbFrame.hours < 0) continue;

    wwvbFrame.doy = wwvbGetDayOfYear();
    if (wwvbFrame.doy < 0)  continue;

    if (wwvbSkipDUT1() < 0) continue;

    wwvbFrame.year = wwvbGetYear();
    if (wwvbFrame.year < 0) continue;

    wwvbFrame.leapYear = wwvbGetLeapYear();
    if (wwvbFrame.leapYear < 0) continue;     // If we made it here, we have
      dataValid = 1;                                       // valid data
  }
} 

void resetWWVBFRAME(struct WWVBFRAME *w)
 {   
  w->minutes = -1;
  w->hours = -1;
  w->doy = -1;
  w->year = -1;
  w->leapYear = -1;
}

void showTime(struct WWVBFRAME *w)
{
  Serial.print("DOY: ");
  Serial.print((int) w->doy);
  Serial.print(" Year: ");
  Serial.print((int) w->year);
  Serial.print(" LeapYear: ");
  Serial.print((int) w->leapYear);
  Serial.print(" Time(UTC): ");
  Serial.print((int) w->hours);
  Serial.print(":");
  Serial.println((int) w->minutes + 1);
}

int8_t wwvbGetMinutes(void)
{
  int8_t minutes = 0; int8_t temp = 0;
  minutes = wwvbGetNumber(3, 10);
  if (minutes < 0) return minutes;

  temp = getPulseByType(); // discard unused pulse

  temp = wwvbGetNumber(4, 1);
  if (temp < 0) return temp;  

  minutes += temp;
  return minutes;
}

int8_t wwvbGetHours(void)
{
  int8_t hours = 0; int8_t temp = 0;

  temp = getPulseByType();
  if (temp != pulseMarker) return -1;

  temp = getPulseByType();  // discard unused pulses
  temp = getPulseByType();

  hours = wwvbGetNumber(2, 10);
  if (hours < 0) return -1;

  temp = getPulseByType(); // discard unused pulse

  temp = wwvbGetNumber(4, 1);
  if (temp < 0) return -1;  

  hours += temp;
  return hours;
}

int16_t wwvbGetDayOfYear(void)
{
  int16_t doy = 0; int8_t temp = 0;

  temp = getPulseByType();
  if (temp != pulseMarker) return -1;

  temp = getPulseByType();  // discard unused pulses
  temp = getPulseByType();

  doy = wwvbGetNumber(2, 100);
  if (doy < 0) return -1;

  temp = getPulseByType(); // discard unused pulse

  temp = wwvbGetNumber(4, 10);
  if (temp < 0) return -1;
  doy += temp;

  temp = getPulseByType();
  if (temp != pulseMarker) return -1;

  temp = wwvbGetNumber(4, 1);
  if (temp < 0) return -1;

  return doy + temp;
}

// DUT1 is not used in this project.
int8_t wwvbSkipDUT1(void)
{
  // note: it may be a good idea to beef up the error checking here
  int8_t i;
  for (i = 0; i < 5; i++) // two unused pulses plus the DUT1 sign bits
    getPulseByType();     // skip over them...

  int8_t temp = getPulseByType(); // verify marker
  if (temp != pulseMarker) return -1;

  for (i = 0; i < 4; i++) // skip the DUT1 value
    getPulseByType();

  return 0;
}

int16_t wwvbGetYear(void)
{
  int16_t year; int8_t temp;
  temp = getPulseByType(); // discard unused bit

  year = 2000 + wwvbGetNumber(4, 10);
  if (year < 2000) return -1;

  temp = getPulseByType();
  if (temp != pulseMarker) return -1;

  temp = wwvbGetNumber(4, 1);
  if (temp < 0) return -1;
    return year + temp;
} 

int8_t wwvbGetLeapYear(void)
{
  uint8_t leapYear= getPulseByType(); // unused pulse

  leapYear = getPulseByType();
  if ((leapYear == pulseInvalid) ||
      (leapYear == pulseMarker))
    return -1;
  else if (leapYear == pulseOne)
    return 1;

  return 0;
}

/* wwvbGetNumber() - this is one of the core functions that makes the
    whole thing work.  This function gets called to retrieve a sequence
    of pulses from the wwvb stream and convert those pulses into an
    integer.  Typically, the numbers are encoded as binary bits and their
    position in the stream determines if they are to be multiplied by 1,
    10, or 100.  In addition, there is some error checking going on to make
    sure we are receiving a valid signal from teh receiver.  The count
    variable is expected to start at 1 instead of 0 and indicates the number
    of pulses to be evaluated.  As is common in the rest of the code,
    we return -1 if we detect an error.
*/
int16_t wwvbGetNumber(int8_t count, int multiplier)
{
  int8_t i; int8_t result = 0; count--;
  for (i = count; i >= 0; i--)
  {
    int8_t pulse = getPulseByType();
    if ((pulse == pulseInvalid) ||
        (pulse == pulseMarker))
      return -1;

  // Note: Lesson learned: never send a floating point function to do an
  //    integer operation.  In converting the numbers, I originally used
  //    a conversion trick I had learned from programming on pc's where
  //    you set up a loop counting down to zero and use the counter variable
  //    with the pow() function from the math library to determine the value.
  //    In our case, a one in the data stream indicates a value of 2^i.
  //    However, in 'arduino-land', this does *not* work. Somehow when you
  //    try to compute anything equal to or above 2^2, a rounding error
  //    occurs and internally you get back something like 3.9 which, when
  //    cast back to an int, gives you 3 instead of 4!  Bit shifting is fool
  //    proof.

 if (pulse == pulseOne)
      result = result + (multiplier * (1 << i)); 
    // Old Method: result =  result + multiplier * pow(2, i);  
  }
  return result;
}

// Wait for the double marker signal that indicates the start of the 
//  broadcast frame
void waitMinuteMarker()
{
  while (1)
  {
    waitMarker();
    if (getPulseByType() == pulseMarker)
      return;
  }
}

// Wait indefinitely for a marker to be received.  Used when we know the 
//    frame is garbage due to noise.
void waitMarker()
{
  while(1) // simple function to wait for a marker - this also takes care of  
  {        //  potential errors in reception at the same time.
    int8_t pulse = getPulseByType();
    if (pulse == pulseMarker) // marker
      return;
  }
}

/* readPulse() - this function waits for a transition on the input pin 
    from the wwvb receiver and then computes the duration that the input 
    pin was held low.  Later, this value is used to determine what kind 
    of pulse was received.
*/
long readPulse()
{
  long pw = 0;
  
  // read the value of digital pin 10. Wait until the pin transitions low.  
  if (digitalRead(wwvb) == 1)
    while (digitalRead(wwvb) != 0) {}
  
  if (ledState == 0)  // blink integrated LED to show activity
  {
    ledState = 1;
    digitalWrite(ledPin, HIGH);
  }
  else
  {
    ledState = 0;
    digitalWrite(ledPin, LOW);
  }
  // Store the time value  
  long m = millis();
    
  // when it changes from low to high, store the current time value
  while (digitalRead(wwvb) != 1) {}
  pw = millis() - m;
    
  return pw;
}

/* getPulseByType() - Critical function that calls readPulse() to read 
      one pulse from the wwvb stream.  Thresholds are then applied to 
      the received pulse width to determine what kind of pulse was 
      received.  The tolerance of the thresholds determine how much 
      error there can be in the signal before we decide that there is 
      too much noise in the signal to continue processing the current 
      frame.
*/
int8_t getPulseByType()  // use thresholds to interpret pulse width
{
   long  pw = readPulse();
   if (pw > 900)          // error conditions. Pulse must be between 150 
     return pulseInvalid; //   and 900mS long. 
   else if (pw < 150)
     return pulseInvalid;
   else if (pw > 650)
     return pulseMarker;  
   else if (pw > 350)
     return pulseOne;  
   else 
     return pulseZero;  
}

The above code has been fully debugged and works great.  Sure, it could use some improvements like better error checking in a few places, but it handily gets the job done.  I’ve left in some comments that I used to help me remember how to decode the signal that I could have removed, but then thought it would probably make the program more readable if I left them in.

In our next post, we’ll be integrating this with a real time clock module so that we can have an updated and accurate source of time 24 hours a day.  Currently, atmospheric and man-made noise make it impossible to depend on a solid signal throughout the day.  However, I can say that I do get at least one valid WWVB frame per day, so that coupled with a RTC module should give me time keeping abilities that are more than close enough for my project.

Dropped or Missing Keystrokes in IE 7 or 8

TIP: If you’ve recently upgraded to IE7 or IE8 and found that your keyboard is suddenly dropping characters when you compose a new email in gmail, you may be experiencing a problem due to the Adobe Shockwave Flash plugin.  I recently got a new laptop with Windows 7 installed and was getting very frustrated with this problem.  Unfortunately, online posts about the issue didn’t address the problem directly.  In most cases, users were instructed to disable all plugins and see if that fixed the problem.

In my particular case, I simply disabled the Adobe Shockwave Flash plugin and the problem went away.  I would note, though, that I only experienced the issue in gmail when I was composing a new message or a response to an existing message.  Otherwise, IE worked normally.

Forged Emails…

Note: this is a re-publishing of a post I had on my blog back in 2007.  Lately I’ve been getting quite a few questions regarding this topic, so I decided it would be good to bring this material back to the forefront.

Every once in a while, I get asked the following question: “I just got an email that says I tried to send an email message to someone I’ve never heard of and it bounced back. Why?” Or it goes something like this: “I got a message that says I tried to send a virus to someone and the email was rejected. How is that possible?” Well, the answer is a little technical, but I’m going to show you how this happened.

The short answer is that someone created a fake email that shows you as the sender. The reasons for doing this are usually related to adware, spyware, or viruses. Basically, they’re trying to install a piece of software onto as many systems as possible without the users knowing about it. The end result can range from something that’s just plain annoying to stealing the contents of your bank account and you can bet someone is getting paid to do it. By naming another valid email account as the sender, they raise the potential to get their virus/adware/spyware installed onto at least one computer. If the recieving email account exists and there isn’t any antivirus software scanning the email, it might get installed on that user’s computer if the reciever opens the message along with any attachments that are inside it. If the message bounces back to the email account named as the sender, the person named as the sender would naturally be confused by a message he or she didn’t send. Consequently, they investigate the content of the original message and wind up unintentionally installing the nastyware on their computer instead. It’s a win-win situation for the person who’s trying to infect as many machines as possible even though it leads to confusion for the recipients.

That leads us to the question of how it’s possible to create a message and name a false To: and From: address. To start, the malicious user connects to a SMTP server somewhere on the internet and issues the necessary commands to send a message with your email address in the From: field. This is possible because most SMTP servers do not require the user to authenticate themselves before issuing the necessary commands to send an email. Therefore, any anonymous user on the internet can send an email that pretends to come from anyone else in the world! This problem was introduced with RFC821 in the early days of email. If you go ahead and read that document, it will tell you how to manually send an email, but it’s rather long and complicated. Here’s an example of how to open a connection to a mail server, create a fake message, and get the mail server to deliver it: (Note that the ‘>’ characters have been added here to mark where I manually entered commands to the SMTP server. In reality, those characters don’t actually appear on the screen. Names of servers and email addresses have been changed to protect private information and I emphasize that this is only an example – not a real message I may have tried to send):

C:\> telnet smtp.somedomain.com 25

220 smtp.somedomain.com ESMTP Server (Microsoft Exchange Internet Mail S
ervice 5.5.2657.72) ready
> HELO nastyware.com
250 OK
> MAIL FROM: John.Smith@redmail.com
250 OK - mail from <John.Smith@redmail.com>

> RCPT TO: Joe.Johnson@bluemail.com
250 OK - Recipient <Joe.Johnson@bluemail.com>
> DATA
354 Send data.  End with CRLF.CRLF
Subject: Your password has been updated.
Date: September 10, 2005
From: support@somebank.com
To: Joe.Johnson@bluemail.com

Your password was recently updated in our system.  If you did not change your 
password, please use the link below to notify us that someone has illegally 
accessed your banking information.  Thank you.

http://www.S0mebank.com/support/unauthorized_access.asp

.
250 OK
> QUIT
221 closing connection

Connection to host lost.

C:\>

As you can see from the above example, I lied about who I am and why the message was being sent. Furthermore, I named Jonh.Smith@redmail.com as the sender, but the actual body of the email indicates that email was sent by support@somebank.com. This is so the person who eventually recieves this email will think their bank is trying to alert them to some kind of unauthorized access. If this message bounces because the recipient’s email isn’t real or because their spam filter rejects it, John.Smith@redmail.com will recieve the message instead. It won’t bounce back to the support@somebank.com address because the addresses in the body of the message aren’t used for delivering the message. Those addresses are only used by mail client software such as Outlook, Netscape, or Thunderbird for replying to the messages after they are delivered. Digging a little deeper, you’ll also notice that the url at the bottom of the message body has a zero where there should be an ‘o’ so that when the user clicks on the link, they will be directed to a website that looks like the website of their bank. The reality is that the website is most likely run by the person who forged the email and is designed to lure the recipient into revealing their account information such as an account number and password.

Unfortunately, RFC821 allowed this sort of abuse and had no protections in place to prevent it. There were no provisions in the specification that would require end-users to provide any proof of their identity before an email message would be accepted for delivery. Furthermore, most implementations of the protocol didn’t check to see that the addresses given in the MAIL FROM: and RCPT TO: portions of the protocol matched those addresses given later on in the body of the message – the From:, To:, and Reply-To: lines. Consequently, it was very easy to create a message like the one above and nearly every mail server on the internet would happily accept the message for delivery.

Fortunately, many email providers today are starting to require that users authenticate to the server before the server will accept an email message for delivery. In addition to that, some mail servers have restrictions in place to help verify the email addresses specified on the MAIL FROM: line match those used in the body of the message. Another methon being adopted is to have the mail server verify that the MAIL FROM: address belongs to a valid account.

However, even with those restrictions in place, you will still get the occasional fake message because most mail servers don’t have a reliable way of verifying the authenticity of a message when it gets passed from one mail server to the next. The typical example is where Joe sends an email to John. The message goes to bluemail.com’s email server for routing and delivery. That server then passes the message to redmail.com’s mail server for delivery to the end user. Redmail.com’s mail server doesn’t have any way to verify that Joe.Johnson@bluemail.com is a real mailbox. It can’t because only bluemail.com’s servers know what addresses are registered in the system. As a result, redmail.com’s servers accept the message for delivery.

Thankfully, this situation is slowly changing and the holes in the system are being plugged. Unfortunately, until all of the insecurities in the system are eradicated, you’ll still get fake emails and there isn’t much that can be done about it in the short term. In closing, if you avoid opening attachments and refuse to just click on the links you recieve in your email, you will be much better protected from these kinds of problems.

Disclaimer: The above is an example of a malicious message and is not intended to represent an actual email that may have been sent or recieved. The information given is a result of a real email message, but the content has been changed significantly to illustrate the themes mentioned in this posting.

Update monitoring under Debian Linux

I’ve recently been migrating some of my servers at work from SuSE Linux over to Debian based distributions.  We can’t do this in every instance because enterprise vendor support for Debian doesn’t exist for every product we use.  We like migrating over to Debian for the following reasons:

  • Security patches are freely available and frequent.  One complaint I’ve had with Novell is the fact that they charge a yearly fee for product updates.  Many of those updates are the same updates that are free to everyone not using Novell’s products.
  • Debian, for a community based distribution, has a very slow turn over rate.  As a sysadmin, we already spend enough of our time trying to keep up with just the product and security patches that are necessary for the day to day usage of the systems, so we don’t want to invest additional time in upgrading base operating systems if we can avoid it.

In addition to the above reasons, I’ve also discovered that it’s pretty easy to get Debian to automatically inform me when I need to download/install updates.  For those of you who are running X windows/KDE/Gnome, you already know that there are desktop utilities that will do the same thing.  However, in a server environment,  you don’t necessarily have the luxury of running a full GUI desktop, so command line solutions are the only tools available.

This brings me to the point of my post.  I wanted to have a cron job give me a report of what updates and security patches were available to be installed on my system.  I further wanted the server to give me this information in an email message so that I don’t have to login to the server on a regular basis just to find out if updates are available (this is what I like to refer to as babysitting the server and I just don’t have time nor patience for it).  As it turns out, this is fairly easy to do with a bit of python code, but the details aren’t widely known.

In Ubuntu based distributions when you open an ssh connection to the server, it will happily display the following:

15 packages can be updated.
18 updates are security updates.

What’s curious here is that the system is obviously running some program in the background to update /etc/motd with the information I want to have in my report, but further searches online failed to identify what mechanism is responsible for the update.  After digging around on the web for an hour trying to figure this out, I eventually downloaded the source code for the update-notifier package which is the code base for the graphical desktop widget that prompts the user to install updates.

The source code for that package allowed me to eventually figure out that /usr/lib/update-notifier/apt-check was being run in the background.  This little python script spits out a very simple pair of numbers.  The first number is the total number of packages that can be upgraded and the second number is the number of security patches available.  Further reading of the script showed that it took another command  line option which would cause it to list each of the packages that needed to be upgraded.

Hmm.  I’ve never really written python before, but how hard could it be?  So after some fiddling with the script, I ended up with the following modified version of apt-check:

#!/usr/bin/python

#nice apt-get -s -o Debug::NoLocking=true upgrade | grep ^Inst

import apt_pkg
import os
import sys
from optparse import OptionParser
from gettext import gettext as _
import gettext
SYNAPTIC_PINFILE = "/var/lib/synaptic/preferences"

def clean(cache,depcache):
    # mvo: looping is too inefficient with the new auto-mark code
    #for pkg in cache.Packages:
    #    depcache.MarkKeep(pkg)
    depcache.Init()

def saveDistUpgrade(cache,depcache):
    """ this functions mimics a upgrade but will never remove anything """
    depcache.Upgrade(True)
    if depcache.DelCount > 0:
        clean(cache,depcache)
    depcache.Upgrade()

def _handleException(type, value, tb):
    sys.stderr.write("E: "+ "Unkown Error: '%s' (%s)" % (type,value))
    sys.exit(-1)

# -------------------- main ---------------------

# be nice
os.nice(19)

# setup a exception handler to make sure that uncaught stuff goes
# to the notifier
sys.excepthook = _handleException

# gettext
APP="update-notifier"
DIR="/usr/share/locale"
gettext.bindtextdomain(APP, DIR)
gettext.textdomain(APP)

# check arguments
parser = OptionParser()
parser.add_option("-p",
                  "--package-names",
                  action="store_true",
                  dest="show_package_names",
                  help="show the packages that are going to be installed/upgraded")
parser.add_option("-r",
                  "--report",
                  action="store_true",
                  dest="create_report",
                  help="Report on packages that need upgrading and summarize results")
(options, args) = parser.parse_args()
#print options.security_only

# init
apt_pkg.init()

# get caches
try:
    cache = apt_pkg.GetCache()
except SystemError, e:
    sys.stderr.write("E: "+ _("Error: Opening the cache (%s)") % e)
    sys.exit(-1)
depcache = apt_pkg.GetDepCache(cache)

# read the pin files
depcache.ReadPinFile()
# read the synaptic pins too
if os.path.exists(SYNAPTIC_PINFILE):
    depcache.ReadPinFile(SYNAPTIC_PINFILE)

# init the depcache
depcache.Init()

if depcache.BrokenCount > 0:
    sys.stderr.write("E: "+ _("Error: BrokenCount > 0"))
    sys.exit(-1)

# do the upgrade (not dist-upgrade!)
try:
    saveDistUpgrade(cache,depcache)
except SystemError, e:
    sys.stderr.write("E: "+ _("Error: Marking the upgrade (%s)") % e)
    sys.exit(-1)

# check for upgrade packages, we need to do it this way
# because of ubuntu #7907
upgrades = 0
security_updates = 0
for pkg in cache.Packages:
    if depcache.MarkedInstall(pkg) or depcache.MarkedUpgrade(pkg):
        # check if this is really a upgrade or a false positive
        # (workaround for ubuntu #7907)
        if depcache.GetCandidateVer(pkg) != pkg.CurrentVer:
                upgrades = upgrades + 1
                ver = depcache.GetCandidateVer(pkg)
                for (file, index) in ver.FileList:
                    if (file.Archive.endswith("-security") and
                        file.Origin == "Ubuntu"):
                        security_updates += 1

# print the number of upgrades
if options.show_package_names:
    pkgs = filter(lambda pkg: depcache.MarkedInstall(pkg) or depcache.MarkedUpgrade(pkg), cache.Packages)
    sys.stderr.write("\n".join(map(lambda p: p.Name, pkgs)))
elif options.create_report:
    sys.stderr.write("\n===========================================\n")
    sys.stderr.write("Packages to be upgraded:\n    ")
    pkgs = filter(lambda pkg: depcache.MarkedInstall(pkg) or depcache.MarkedUpgrade(pkg), cache.Packages)
    sys.stderr.write("\n    ".join(map(lambda p: p.Name, pkgs)))
    sys.stderr.write("\n===========================================\n")
    sys.stderr.write("packages needing upgrades: %s\n" % (upgrades))
    sys.stderr.write("security updates: %s\n" % (security_updates))
else:
    # print the number of regular upgrades and the number of
    # security upgrades
    sys.stderr.write("%s;%s" % (upgrades,security_updates))

sys.exit(0)

The above script isn’t quite finished yet.  The report it gives me has some visual artifiacts that could be eliminated from the resulting email.  Here’s a sample report:

Reading package lists... 0%

Reading package lists... 100%

Reading package lists... Done

Building dependency tree... 0%

Building dependency tree... 0%

Building dependency tree... 1%

Building dependency tree... 50%

Building dependency tree... 50%

Building dependency tree

Reading state information... 0%

Reading state information... 0%

Reading state information... Done

===========================================
Packages to be upgraded:
 libssl0.9.8
 openssl
===========================================
packages needing upgrades: 2
security updates: 0

For the moment, I’ll leave the finishing touches as an exercise to the reader.  I’ll make the changes when I have more free time in my schedule, but for the time does the bare minimum that I needed.

Another update…

You may have noticed that I finally got around to porting the site over to WordPress.  It took as long as it did because life has been rather busy for me and spending time on this isn’t really a necessity.

I’m hoping the comment system is a bit more robust than it was with BlogEngine.Net.  Feel free to post comments – I will do my best to keep up with what everyone is saying, however, I’m still sticking with no-links in the comments for the time being.  I would much rather write interesting posts here rather than act as a gatekeeper of the comments, so lets hope it works better this time around.  I basically gave up on the old blog because it was time consuming to keep up with the rather ineffective comment system.

For those of you who may think I’ve sold out and went with the masses on WordPress, I can honestly say that it would be very difficult for me to write an original blog system that has the flexibility and features of WordPress.  Ultimately, there’s a reason it’s so popular and I really can’t complain at all.   Customizing the site is very easy and it just works.

Here’s to hoping I can stop writing meta-posts about blog software and get to something that’s a lot more interesting.