Alias often...

It's often very frustrating to retype shell commands over and over.  One thing that I sometimes forget to do is to create aliases for common commands.  I just added an alias for something I do quite often - search the shell history for some command I typed in either the previous or current session.

Edit your config file of choice (I updated .profile) and add the following:

alias hgrep='history | grep'

Now I can just type hgrep along with the search phrase of choice.  ie,

somemachine:someuser:/opt/somedir/home/username: hgrep 'start'
  235  ../bin/startup.sh
  250  ../bin/startup.sh
  256  ../bin/startup.sh
 1023  hgrep 'startserv'
 1024  hgrep 'start'

Toyota Tacoma compass display fix...

The compass and temperature overhead display in my 2007 Toyota Tacoma had stopped working a couple of years ago.  I called the dealership and found out that they don't fix that issue - they just replace the unit. A new compass unit, and the replacement time, costs about $300 (at least it did a couple years ago). I didn't want to  pay that much, so I just left it alone.  

Recently I found a YouTube video that shows someone repairing the unit.  The issue that the repair video was addressing is that some of the resistors have weak solder connections, and re-soldering the resistors with poor contact will fix the issue. It turned out to be that easy for me, and I was happy to have a video as a guide.  The person who made the video warned against putting too much stress on some of the plastic tabs, and that warning kept me from breaking tabs.

I removed the overhead display, removed  the display from the housing, and inspected the resistors. There were a number of surface mount resistors, but only two resistors (the ones marked 510) looked like they might not be making good contact.  I used a multimeter to check all of the resistors on the board to make sure they were okay.  Perhaps that was overkill - I wouldn't expect that the resistors would be bad - but it gave me an excuse to use the multimeter and that was good enough for me. Next, I heated up the existing solder with my soldering iron. I figured that would  restore good contact between the board and the resistors.  Then I added a tiny amount of solder to ensure that the connection was good.  I rechecked the resistors with the multimeter, and then reinstalled the overhead display.  I started the truck, and nothing displayed. I was pretty disappointed for a second, but I pushed the button on the display and there it was - temperature and compass direction!  After two years or so of not having a functioning display I had forgotten that the button cycles between Celsius, Fahrenheit, and turning the display off.

Everything worked great, and I saved $300!

AWS S3 Sample Using TransferUtility.DownloadDirectory...

I need to write a utility for a project that I'm working on, and the utility will need to download all of the files in an S3 directory. Luckily the AWSSDK provides an easy way to do this with the TransferUtility.DownloadDirectory method.

The following is a simple example usage of the DownloadDirectory method.  

public class S3Downloader
{

   public void DownloadS3Directory(string bucketName, string s3Directory, 
                                   string localDirectory)
   {
      var s3Config = new AmazonS3Config
      {
         ServiceURL = "s3-us-west-2.amazonaws.com",
         CommunicationProtocol = Protocol.HTTP
      };

      using (var s3Client = new AmazonS3Client(
                                   new EnvironmentAWSCredentials(), 
                                   s3Config))
      {
         using (var transferUtility = new TransferUtility(s3Client))
         {
            var ddr = new TransferUtilityDownloadDirectoryRequest
            {
               BucketName = bucketName,
               LocalDirectory = localDirectory,
               S3Directory = s3Directory
            };

            ddr.DownloadedDirectoryProgressEvent += DisplayProgress;
            transferUtility.DownloadDirectory(ddr);                
         }
      }
   }

   private void DisplayProgress(object sender, 
                                DownloadDirectoryProgressArgs args)
   {
      Console.WriteLine(args);
   }
}

public class Program
{
   public static void Main(string[] args)
   {
      string bucketName = "mybucket";
      string s3Directory = "/archived/files/2013-07";
      string localDirectory = @"C:\Temp\s3test";

      var s3Downloader = new S3Downloader();
      s3Downloader.DownloadS3Directory(bucketName, 
                                       s3Directory, 
                                       localDirectory);
   }
}

Here is an example of what is written to the console:

Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 8192
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 16384
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 24576
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 32768
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 40960
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 49152

Line endings and views, although unrelated, can both cause confusion...

Line Endings

Boy how I wish there weren't a difference between Unix and Windows line endings. I recently needed to search through a bunch of map reduce output (tons of reduce files in text format), so I decided to bulk insert them into a SQL database table.

I wrote a bulk insert statement for each of the reduce files (well, some SQL code wrote the bulk insert statements based on a list of file names), and I quickly encountered errors when attempting to execute the statements. The bulk insert statements looked something like this:

BULK INSERT ReduceData_View  FROM 'path\part-r-00000' 
WITH (FIRSTROW = 1,FIELDTERMINATOR = '\t', ROWTERMINATOR = '\n');

The reduce files are rows of data that are tab separated. The frustrating part is that I had failed to remember that the map reduce job was done on a Linux machine, and therefore each file contained Unix style line endings.  I had been viewing the files in Notepad++ (which will display different file formats without complaint), so it was confusing to me when I would see errors. I changed the ROWTERMINATOR value to '\r\n', and everything worked fine.  

Lesson - pay attention to the source system type when processing data files.  I didn't spend a long time on this, but it was still frustrating.

SQL Views

I had created a view to insert into, because I had an identity column on my target table.  It seemed like an easy way to avoid the problem of bulk insert trying to insert into the identity column.  The problem occurred when I ran the SQL to do the inserts, and some of the columns from the source files had values that were too wide for the table columns.  I changed the target table columns to use varchar(max), since some of the source data had values that were larger than 4000 chars.  I ran the bulk insert again and I still had errors due to the data being too large for one of the target columns.  What the heck?  What I hadn't realized is that the view I created had information for the columns in the target table from when the target table was first created.  The table had the column width changed to varchar(max), but the view thought the table column width was varchar(4000).  I ran the SQL to recreate the view and then the bulk insert worked just fine.