Ben’s Updates – Optimizations &c.

Hi, everyone. It’s been awhile since I’ve posted and quite a bit has happened, so let’s get started.

Refactoring

When I last posted, the codebase… worked, but it was a mess; everything was crammed into setup() and loop(), making changing one sensor’s functionality a bit of a mess. Sure, architecture can slow your code down, but just a little organization can’t hurt, right?

I figured that I might as well leverage C++ and make each sensor a class implementing an abstract base class Sensor. I didn’t do much more than that; even the ABC is just a way of enforcing consistent method names. I don’t know a lot of C++, but I don’t really need vtables, range for-loops, and the like for this program anyway.

Doing this made things a bit more manageable; I still want to break up the update() loop so that I can easily change when or if data is written. This is because, aside from raw data reads and serialization, my main source of optimization after this will be to skip some SD writes.

Optimization

Skipping SD writes is pretty simple; I keep a running tab of the variance and mean. If, accounting for gravity, it doesn’t seem to have been moving at any point in the data we just collected, we throw out the data and keep track of how many times we did that. When the shark does start moving, we just write “skipped X writes” to the SD card and continue as usual.

Raw data reads and serialization looks a bit more painful. There’s plenty of overhead and cruft in the libraries that we’re using – for example, reading from the accelerometers reads from the registers, packs them into a short int, stores them, and then converts the ints into floats based on the scale. In reality, we don’t need anything but those 6 bytes of raw accelerometer data, and while that example is relatively minor, the benefits stack over time.

Still, doing that requires a lot of investment; low-level code to read the sensors, a new file format to store the raw data, a Python script to convert the data back into a CSV. All that is for benefits that, while significant, are pretty late-game in the grand scheme of things. We’ve got other things to worry about… namely, storing and serving said data for analysis.

The Server

Due to the amount of data we’re collecting and the rate at which we’re collecting it, I expect us to be collecting (at the most) about 3 gigabytes a month. That’s not huge at first, but we want to re-use these tags and host them on a server so that no one has to constantly keep a local copy on their hard drives.

One solution we have is to adapt tagbase to our needs, replacing the Microsoft access interface with SQL; we actually know the people who put it together and can get some help from them. There are people who have worked with servers before; I haven’t, and I don’t think anyone has really used SQL yet. This will be a fun experience, indeed…

Anyways…

I’ve got plenty of stuff to do here, both now and over the summer. I got approved for a grant, so I’ve got summer housing both at my college and the Chesapeake Bay… we’ll see how this goes. See you guys in a month or so.

Ben’s Update – Introductions

William’s already introduced me, so I’ll be brief. I’ve joined the project to work on the more design-oriented part of the software end. William has done a great work with the hardware; for the most part, I can view the chips as black boxes and get to work on the program itself. This means optimizing the code to run as efficiently as possible – a feat which, given that I’ve worked mainly in languages like Ruby and Python, is definitely new ground for me!

My work at this point has come down to re-working William’s code to cache data to the SD card. As William has explained it to me, we want to write to the SD card in large bursts instead of bit by bit; we also don’t need to write things like temperature or time data to the SD card very frequently. The solution is pretty simple: hold the accelerometer data in an array and then write it once the array is  full.

Unfortunately, our Arduino has a ridiculously low amount of memory (about 2kb of SRAM), so at this highly unoptimized point we can only hold about 512 bytes of data, and as a result, we’re still writing to the SD card every 3 seconds or so. We can reduce this by removing debugging code through conditional compilation and moving some of our strings into flash memory with PROGMEM.

William also found a library called Narcoleptic that should put the board to sleep while it waits to get data, but I haven’t looked into it yet. I’ll see if I can get anything from that next week.

–Ben