Arrrrggghh! SAS is Evil!

Just another day (or two) of torturing data. Like I mentioned a couple of days ago, a week back I decided to update a data set to include the last year or so of data (the data sources I use were recently updated). Like most "simple" jobs, it's turned out to be much more of a hairball than I expected. Although the program I used was fairly simple to rewrite, I realized that I had to update not one, not tWo, but THREE datasets in order to bring everything up to the present.

Caution: SAS Geekspeak ahead

One of the data sets is pretty large (it was about 70 gigabytes, but with the updates and indexing I've done, it's almost 100 gig). So, adding the new data and checking it took quite a while (no matter how efficiently you code things, SAS simply takes a long time to read a 70 gigabyte file). I thought I had everything done except for the final step. Unfortunately, the program kept crashing due to "insufficient resources."

For the unitiated, when manipulating data (sorting, intermediate steps on SQL select statements, etc...) SAS sets up temporary ("scratch") files. They're supposed to be released when SAS terminates, but unfortunately, my system wasn't doing that. So, I had over 180 gigabytes of temporary files clogging up my hard drive. This means that there wasn't enough disk space on my 250 gigabyte drive for SAS to manipulate the large files I'm using.

Of course, I only realized this when my program crashed AFTER EIGHT HOURS OF RUNNING! TWICE!

I've now manually deleted all the temporary files, and I'm running the program overnight to see if this fixes the problem.

Ah well - if it was easy, anyone could do it.

update (next morning): Phew! It ran - it seems the unreleased temporary files were the issue. On to the next problem.

Research and Computers

My research productivity was way down since moving to Unknown University. Getting set up at a new school always takes more time than I expect, so the fall wasn't that productive. And the first part of the Spring semester was tied up with prepping for the CFA classes I've been teaching, which took far longer than I'd planned on.

But now that that's over, I'm back to research. So far this semester, I've submitted two papers to journals and gotten papers accepted at three conferences (granted, they're smaller regional conferences rather than the national ones, but it's a start). And this week, I should finish up a third (smaller) paper that's been in progress for FAR too long.

Now that I've cleared the deck of my older papers, I get to work on some newer ideas (caution: nerd alert ahead).

Without revealing too much information (after all, I am blogging pseudonymously), I've been working on a project lately that requires me to make pairwise comparisons between each and every security analyst who makes forecasts in a given two-year period. This results in over 25 million pairwise comparisons (and that's just for a two-year period). If you think this is a huge amount of data, you're right - it is.

But the truly amazing thing is how easily my computer can handle this. My setup is pretty basic - I'm running SAS on a 6 month old desktop machine with 2 gigs of ram. The program runs through one iteration of the analysis in less than two minutes. In the grad school days of yore, it would require a day or two on the mainframe, and I'd have to break the job into multiple parts to avoid crashing the system.

Life in the time since the meteor hit and the dinosaurs died is good.