thu, 02-feb-2012, 16:56

In my last post I discussed the January 2012 cold snap, which was the fifth coldest January on record in Fairbanks. I wondered how much of this was due to the arbitrary choice of “month” as the aggregation level for the analysis. In other words, maybe January 2012 was really remarkable only because the cold happened to fall within the margins of that particular month?

So I ran a query against the GHCN daily database for the Fairbanks Airport station. This data set won’t allow me to answer the exact question I’m interested in because the data for Fairbanks only goes back to 1948 and because I don’t have the data for the last day in January 2012. But I think the analysis is still valid, even if it’s not perfect.

The following query calculates the average temperature in Fairbanks for every 31-day period possible, and ranks it according to a descending sort of the average temperature for those periods. The results show the start date, end date, the year and month that the majority of data appears in (more on that later), and the average temperature over the period.

One other note: the temperatures in this post are in degrees Celsius, which is why they’re different than those in my previous post (in °F).

SELECT date(dte - interval '15 days') AS start,
       date(dte + interval '15 days') AS end,
       to_char(dte, 'YYYY-MM') AS yyyymm,
       round(avg_31_days::numeric, 1),
       rank() OVER (ORDER BY avg_31_days) AS rank
FROM (
    SELECT dte,
           avg(tavg_c) OVER (
              ORDER BY dte
              ROWS BETWEEN 15 PRECEDING AND 15 FOLLOWING
           ) AS avg_31_days
    FROM (
        SELECT dte,
               (sum(CASE WHEN variable = 'TMIN'
                    THEN raw_value * 0.1
                    ELSE NULL END)
              + sum(CASE WHEN variable = 'TMAX'
                    THEN raw_value * 0.1
                    ELSE NULL END)) / 2.0 AS tavg_c
        FROM ghcnd_obs
        WHERE station_id = 'USW00026411'
        GROUP BY dte
    ) AS foo
) AS bar
ORDER BY rank;

The funky CASE WHEN ... END stuff in the innermost query is because of the structure of the data in the GHCN database. The exciting part is the window function in the first subquery that calculates the average temperature for a 31-day window surrounding every date in the database.

Some of the results:

start end yyyymm avg (°C) rank
1964-12-10 1965-01-09 1964-12 -36.9 1
1964-12-09 1965-01-08 1964-12 -36.9 2
1964-12-11 1965-01-10 1964-12 -36.9 2
1971-01-06 1971-02-05 1971-01 -36.8 4
1964-12-08 1965-01-07 1964-12 -36.7 5
1971-01-07 1971-02-06 1971-01 -36.6 6
1971-01-05 1971-02-04 1971-01 -36.5 7
1964-12-12 1965-01-11 1964-12 -36.4 8
1968-12-21 1969-01-20 1969-01 -36.3 9
1968-12-22 1969-01-21 1969-01 -36.3 10
...        
2012-01-14 2012-02-13 2012-01 -33.9 42
2012-01-13 2012-02-12 2012-01 -33.9 43

There are some interesting things here. First, the January we just went through doesn’t show up until the 42nd coldest. You can also see that there was a very cold period from mid-December 1964 through mid-January 1964. This “even” appears five times in the first ten coldest periods. The January 1971 event occurs three times in the top ten.

To me, this means a couple things. First, we’re diluting the rankings because the same cold weather event shows up multiple times. If we use the yyyymm column to combine these, we can get a better sense of where January 2012 fits. Also, if an event shows up on here a bunch of times, that probably means that the event was longer than the 31-day window we’ve set at the outset. If you look at the minimum and maximum dates for the 1964 event, the real even lasted from December 8, 1964 through January 11, 1964 (35 days).

If we use this query as a subquery of one that groups on yyyymm, we’ll get a ranking of the overall events:

SELECT yyyymm, min_avg,
       rank() OVER (ORDER BY min_rank)
FROM (
    SELECT yyyymm,
           min(avg_31_days) AS min_avg,
           min(rank) AS min_rank
    FROM (ABOVE QUERY) AS foobie
    GROUP BY yyyymm
) AS barfoo
ORDER BY min_rank;

Here’s the results of the new query:

yyyymm min_avg (°C) rank
1964-12 -36.9 1
1971-01 -36.8 2
1969-01 -36.3 3
1951-01 -34.2 4
1996-11 -34.0 5
2012-01 -33.9 6
1953-01 -33.7 7
1956-12 -33.5 8
1966-01 -33.3 9
1965-01 -33.2 10

Our cold snap winds up in sixth place. I’m sure that if I had all the data the National Weather Service used in their analysis, last month would drop even lower in an analysis like this one.

My conclusion: last month was a really cold month. But the exceptional nature of it was at least partly due to the coincidence of it happening within the confines of an arbitrary 31-day period called “January.”

tags: SQL  weather 
wed, 01-feb-2012, 18:41

January 2012 was a historically cold month in Fairbanks, the fifth-coldest in more than 100 years of records. According to the National Weather Service office in Fairbanks:

January 2012 was the coldest month in more than 40 years in Fairbanks. Not since January 1971 has the Fairbanks area endured a month as cold as this.

The average high temperature for January was 18.2 below and the average low was 35 below. The monthly average temperature of 26.9 below was 19 degrees below normal and made this the fifth coldest January of record. The coldest January of record was 1971, when the average temperature was 31.7 below. The highest temperature at the airport was 21 degrees on the 10th, one of only three days when the temperature rose above zero. This ties with 1966 as the most days in January with highs of zero or lower. There were 16 days with a low temperature of 40 below or lower. Only four months in Fairbanks in more than a century of weather records have had more 40 below days. The lowest temperature at the airport was 51 below on the 29th.

Here’s a figure showing some of the relevant information:

The vertical bars show how much colder (or warmer for the red bars) the average daily temperature at the airport was compared with the 30-year average. You can see from these bars that we had only four days where the temperature was slightly above normal. The blue horizontal line shows the average anomaly for the period, and the orange (Fairbanks airport) and dark cyan (Goldstream Creek) horizontal lines show the actual average temperatures over the period. The average temperature at our house was -27.7°F for the month of January.

Finally, the circles and + symbols represent the minimum daily temperatures recorded at the airport (orange) and our house (dark cyan). You can see the two days late in the month where we got down to -54 and -55°F; cold enough that the propane in our tank remained a liquid and we couldn’t use our stove without heating up the tank.

No matter how you slice it, it was a very cold month.

Here’s some of the R code used to make the plot:

library(lubridate)
library(ggplot2)
library(RPostgreSQL)
# (Read in dw1454 data here)
dw_1454$date <- force_tz(
    ymd(as.character(dw_1454$date)),
    tzone = "America/Anchorage")
dw_1454$label <- 'dw1454 average'
# (Read FAI data here)
plot_data$line_color <- as.factor(
    as.numeric(plot_data$avg_temp_anomaly > 0))
plot_data$anomaly <- as.factor(
    ifelse(plot_data$line_color == 0,
        "degrees colder",
        "degrees warmer"))
plot_data$daily <- 'FAI average'

q <- ggplot(data = plot_data,
    aes(x = date + hours(9))) # TZ?
q + geom_hline(y = avg_mean_anomaly,
        colour = "blue", size = 0.25) +
    geom_hline(y = avg_mean_pafg,
        colour = "orange", size = 0.25) +
    geom_hline(y = avg_mean_dw1454,
        colour = "darkcyan", size = 0.25) +
    geom_linerange(aes(ymin = avg_temp_anomaly,
        ymax = 0, colour = anomaly)) +
    theme_bw() +
    scale_y_continuous(name = "Temperature (degrees F)") +
    scale_color_manual(name = "Daily temperature",
        c("degrees colder" = "blue",
          "degrees warmer" = "red",
          "FAI average" = "orange",
          "dw1454 average" = "darkcyan")) +
    scale_x_datetime(name = "Date") +
    geom_point(aes(y = min_temp,
        colour = daily), shape = 1, size = 1) +
    geom_point(data = dw_1454,
        aes(x = date, y = dw1454_min,
            colour = label), shape = 3, size = 1) +
    opts(title = "Average Daily Temperature Anomaly") +
    geom_text(aes(x = ymd('2012-01-31'),
        y = avg_mean_dw1454 - 1.5),
        label = round(avg_mean_dw1454, 1),
        colour = "darkcyan", size = 4)
tags: R  temperature  weather 
sun, 22-may-2011, 08:06
Arduino temperature display

Arduino temperature display

I’ve had an Arduino-based weather station since June 2009, but one problem with it has been that there hasn’t been any easy way to display the data in real time without going to the database (or the raw import files) to see what the latest observations were. I wrote a quick web page for this (home display), and this is convenient if I want to see what the temperatures are around the house when I’m not at home. But sitting in the living room, it’d be nice to know what the temperatures are just by looking.

The choices for displays aren’t all that great. Arduino can drive the classic seven segment LED displays (like old calculators, for those who remember what a calculator is), which are big and bright, but it requires a lot of pins, and even more power to light enough of them to display the amount of data I need (at least four temperatures, and possibly other sensor values like pressure or light). There are graphical LCD panels, which you control pixel by pixel but are expensive and aren’t all that large. And then there are text LCD displays that use the Hitachi HD44780 controller (or one based on it). These are more like old terminal screens where you can write to each character position on the screen. Common sizes are 16 characters across x 2 rows and 20 x 4 displays.

I chose a white on black, 16 x 2 display (SparkFun LCD-00709). It requires 5 volts to power the display and the LED backlight, and uses six of the digital pins on the Arduino board. The Arduino Uno (and earlier Duemilanove) have fourteen digital pins (two of which are the serial TX/RX pins), so one could hook up two of these displays to a single Arduino. The hookup for a single display, and the code I’m using follows. The Arduino Cookbook was invaluable for getting the everything working.

LCD / Arduino circuit diagram

The Arduino code reads four comma-delimited numbers from the serial port. They’re formatted as integer values representing the temperature in Fahrenheit multiplied by ten (50.2°F = 502). If the value is negative, the minus sign appears at the end of the number (-40.0°F = 400-). Here’s the Arduino code that displays the incoming data to the LCD:

#include <LiquidCrystal.h>

const int numRows = 2;
const int numCols = 16;
const int NUMBER_OF_FIELDS = 4; // t_west, t_back, t_up, t_down
int fieldIndex = 0;
int values[NUMBER_OF_FIELDS]; // temp * 10

LiquidCrystal lcd(12, 11, 5, 4, 3, 2);

void setup() {
  lcd.begin(numCols, numRows);
  lcd.print("SWINGLEYDEV.COM");
  Serial.begin(9600);
}

void displayVal(int val) {
  int ival;
  int dval;
  if (val >= 0) {
    ival = floor(val / 10.0);
    dval = val - (ival * 10);
  } else {
    ival = ceil(val / 10.0);
    dval = -1 * (val - (ival * 10));
  }
  if (ival >= 0) {
    lcd.print(" ");
  }
  lcd.print(ival);
  lcd.print(".");
  lcd.print(dval);
}

void loop() {
  if (Serial.available()) { // Read message
    delay(100);
    while (Serial.available()) {
      char ch = Serial.read();
      if (ch >= '0' && ch <= '9') {
        values[fieldIndex] = (values[fieldIndex] * 10) + (ch - '0');
      } else if (ch == ',') {
        if (fieldIndex < NUMBER_OF_FIELDS - 1) {
          fieldIndex++;
        }
      } else if (ch == '-') {
        values[fieldIndex] = values[fieldIndex] * -1;
      } else {
        lcd.clear();
        lcd.setCursor(0, 0);
        lcd.print("W "); // West outdoor sensor
        displayVal(values[0]);
        values[0] = 0;
        lcd.setCursor(9, 0);
        lcd.print("B "); // Back outdoor
        displayVal(values[1]);
        values[1] = 0;
        lcd.setCursor(0, 1);
        lcd.print("U "); // Upstairs
        displayVal(values[2]);
        values[2] = 0;
        lcd.setCursor(9, 1);
        lcd.print("D "); // Downstairs
        displayVal(values[3]);
        values[3] = 0;
        fieldIndex = 0;
      }
    }
  }
}

To make this work, I need a program on the computer the Arduino is plugged into that will read the weather data from the database and dump it to the serial port that the display Arduino is plugged into. Here’s that code:

#! /usr/bin/env python

import serial, time, shutil, os
import psycopg2

connection = psycopg2.connect(host='HOSTNAME', database='DB')
cursor = connection.cursor()
ser = serial.Serial('/dev/ttyACM0', timeout=1)

def make_arduino_str(value):
    integer = int(round(value * 10))
    if value < 0:
        return "{0}-".format(integer * -1)
    else:
        return "{0}".format(integer)

while 1:
    query = "SELECT west, back, down, up FROM arduino_view;"
    params = ()
    cursor.execute(query, params)
    if cursor.rowcount:
        rows = cursor.fetchall()
        for row in rows:
            (west, back, down, up) = row
        (west, back, down, up) = map(make_arduino_str,
                (west, back, down, up))
        message = "{0},{1},{2},{3}\n".format(west, back, up, down)
        ser.write(message)
    time.sleep(60)
ser.close()
cursor.close()
connection.close()

Each minute, it reads the temperatures from a database view that consolidates the latest readings from the weather sensors (arduino_view), formats them in the manner the Arduino display code expects, and sends them to the serial port for display.

It works really well, and looks good. I wish the display itself were larger, however. I’ll probably get a second LCD panel so I can display more data, but if someone would make a 20 x 4 display that was twice as large as the one I’ve got now, I’d buy it. The 16 x 2 is easy to read from the couch (five feet away), but isn’t readable from the kitchen.

tags: Arduino  weather  lcd 
sat, 21-may-2011, 07:29
Back cabin, high water

Back cabin, high water

I started measuring the depth of Goldstream Creek a little over a year ago this week. Each morning I measure down from a particular spot on the bridge over the Creek to the top of the water or ice and report this (plus daily high and low temperature, precipitation, snowfall and snow depth) to the Weather Service. The following plot (if nothing shows up, click the following links to view a PNG or PDF version) shows the depth of the Creek at our swimming hole on the top, and the daily high and low temperatures on the bottom. The dark cyan line on the top plot is the height of the bridge (about six feet below the entrance to our house), and the dark cyan line on the lower plot is the freezing point.

At it’s highest, the Creek was just over two feet from the bottom of the bridge, the slough flooded into the dog yard about two thirds of the way across the lowest point, and the Creek seemed dangerously close to topping the banks. The photo at the top shows the back cabin during the high water event.

This year’s breakup was similar to last year: ground and meltwater from the surrounding area started flooding on top of the ice and over the course of a couple weeks, it eroded the ice below until the water level rapidly dropped to more normal summertime depths. One interesting note is that we seem to get a large pulse of water (the rise starting around March 11th) before the snow has started melting, which would seem to indicate that the first pulse is coming from groundwater sources. We don’t start getting afternoon temperatures above freezing until the beginning of April, and this is when snowmelt starts bringing the level up even higher.

When the level begins to drop, it’s pretty dramatic as we go from bankfull conditions to almost nothing in a week. This year we’ve still got wide shelves of ice hanging on the banks six feet above the level of the water.

The plot also shows the brief ice storm in late November where we got a couple inches of rain that froze on the roads and brought the Creek up slightly.

In the past, we’ve had more dramatic breakups where the initial springtime pulse of ground and meltwater breaks up all the ice in a couple days and sends it past our house crashing and grinding, but even the more gradual melting pattern of the last two years is impressive in how quickly the Creek rises and falls.

sun, 15-may-2011, 11:25
Battery, Arduino, XBee

Battery, Arduino, XBee

Several years ago when I started messing around with Arduino and building my own weather station, I bought a few XBee radio chips with the idea of setting up some more remote sites without having to run wires out to the sensor stations. I spend several frustrating hours trying to get the radios to talk to each other, but couldn’t get them to work (I think this was because they were in API mode instead of AT mode). Then I got Building Wireless Sensor Networks and Arduino Cookbook and finally got everything working.

At the same time, the batteries in our UPS at work needed to be replaced, and rather than immediately recycling them, I took a couple home to see if they still had enough juice in them to work in a remote sensing capacity. They’re 12 volt lead-acid batteries (which may make them unsuitable for winter) that were rated for 5 Amp-hours when they were new. That should be plenty of power to drive an Arduino, XBee and a couple sensors. The XBee chips have some data pins on them, so I may be able to eliminate the Arduino from each sensor station, depending on the type of sensor I use.

The current setup, shown in the photo at the top, is designed to see how long a remote station can run on one of the batteries without any sensors and at springtime temperatures (typically between 20°F and 60°F at our house). The Arduino is reading the input voltage and sending it wirelessly to an XBee coordinator plugged into a SparkFun explorer board and connected to my small eeeBox computer.

Here’s the circuit diagram:

Circuit diagram

The analog input pins on the Arduino board are expecting voltages between 0 and 5 volts, so I’m using three resistors as a voltage divider to reduce the input voltage (nominally 12 volts, but potentially as high as 15V) to a range safe for the board. The sketch looks like this (almost identical to the sketch in the Arduino Cookbook except that the formula for resistorFactor is incorrect in the book):

const float referenceVolts = 5;

const float R1 = 2000; // A0 to V+
const float R2 = 1000; // A0 to GND

const float resistorFactor = 1023.0 * (R2/(R1 + R2));

const int batteryPin = 0; // A0

void setup() {
   Serial.begin(9600); // to XBee
}

void loop() {
   int val = analogRead(batteryPin);
   float volts =
     (val / resistorFactor) * referenceVolts * 10;
   Serial.println(volts);
   delay(1000);
}

Because the Arduino transmit pin (D1) is connected to the XBee receive pin (DIN) and the Arduino receive (D0) is connected to the XBee send (DOUT), anything the Arduino sends to the serial port is redirected to the XBee, which immediately transmits it wirelessly to the XBee receiver chip in the house.

At the moment, I’m only able to have the sender and receiver about 50 feet from each other before losing the signal, but I think that a pair of boards with whip antennas will work better than the chip antennas I’m currently using. I also set up my network such that the XBee Pro board (which has a supposed range of a mile) is the coordinator receiving messages, but I think it will work better as the sender. When the battery experiment is over, I’ll reverse the position of the XBee boards and see if I get better performance. I’d like to have a station out at the red cabin and potentially somewhere on the hillside, and for that to work the radios will need to be able to communicate over several hundred feet.

The circuit diagram was generated using circuit macros, which take circuit diagrams written in the m4 language and turn them into PIC files which are converted into LaTeX code using dpic. That sounds complicated (and it sort of is), but I much prefer describing the circuits in a text file than to trying to draw them using XCircuits or some other graphical tool. If you’re curious, you can download the code and a Makefile for generating the images.

Here’s what the data looks like so far:

Voltage over time

So far all I can see is a diurnal pattern associated with temperature: it was about 56°F at 6:30 PM last night when the data started, dropped to a minimum of 27°F at around 5:30 AM, and it’s been rising since, hitting 50°F at the end of the period shown on the plot.


<< 0 1 2 3 4 5 6 7 8 9 10 11 12 >>
Meta Photolog Archives