All Things IT Blog » .NET http://www.enusbaum.com/blog My little nerded out corner of the Internets! Thu, 02 Sep 2010 01:43:18 +0000 en hourly 1 http://wordpress.org/?v=3.0.1 .NET StringBuilder — Fast, but not as fast as you think! http://www.enusbaum.com/blog/2009/05/28/net-stringbuilder-fast-but-not-as-fast-as-you-think/ http://www.enusbaum.com/blog/2009/05/28/net-stringbuilder-fast-but-not-as-fast-as-you-think/#comments Thu, 28 May 2009 20:18:47 +0000 eric http://www.enusbaum.com/blog/?p=294 I recently ran into a situation where I was tasked to profile some .NET code and do some optimizations anywhere hot spots popped up. I was amazed to find out that one of the BIGGEST offenders in our code block was a simple call to StringBuilder.Append(char). I had to take a step back and scratch my head and wonder if my profiler was confused.

I re-ran some tests using the StopWatch class to hard code some metrics into the application and they also confirmed the findings. What’s up? How could a class that everyone says you can use to your hearts content when it came to string concatenation was failing me?

Turns out, it was a mix of misuse and a common misconception about the StringBuilder Class.

One of the first things you learn while picking up .NET is that the StringBuilder Class is your friend when it comes to concatenating large strings in memory. It beats the pants off of String.Concat and String.Format, while also being a mutable object in the Framework utilizing an in-memory buffer.

I used JetBrains dotTrace to help profile the application and it was very evident from the get-go that StringBuilder was causing the whole process to slow down.

The nature of my application was basically reading in a text buffer 1 character as a time, and using the StringBuilder as an output buffer. So for a 1k file, The method Append(char) would be called 1024 times. A 600k file would call Append(char) 614,400 times.

So why was I getting burned in execution time? The issue turned out to be two fold.

First, there’s overhead cost to the call. I don’t care how lightweight your method is, if you’re calling it SIX HUNDRED THOUSAND TIMES, it’s going to take a bit. Let alone a method who handles a string buffer in memory and string manipulation. So basically, no matter how fast StringBuilder actually is, it’s not a free call and you should consider the fact that the call still has overhead when architecting your solution.

Architecture brings me to my second point. While writing each character individually made sense initally, it seems that it was just lazy :P The optimized route would have been calling Append with a SUBSTRING of the input buffer, this way we avoid the overhead of multiple calls by writing all the neccisary data in one big blob.

So 600,000 calls to StringBuilder.Append(char) becomes only a few hundred calls to StringBuilder.Append(string.Substring(start, count)). Sure, the Substring Virtual Method itself has overhead, but it’s still less than the thousands of calls to Append(char) that we’re saving ourself :)

Conclusion?

StringBuilder is fast, but it’s not free. Take this into consideration when utilizing it while appending large data sets in small chunks. :)

Cheers!

]]> http://www.enusbaum.com/blog/2009/05/28/net-stringbuilder-fast-but-not-as-fast-as-you-think/feed/ 1 Reading ID3 Tags using C# http://www.enusbaum.com/blog/2007/02/21/reading-id3-tags-using-c/ http://www.enusbaum.com/blog/2007/02/21/reading-id3-tags-using-c/#comments Wed, 21 Feb 2007 21:55:09 +0000 eric http://www.enusbaum.com/blog/2007/02/21/reading-id3-tags-using-c/ I'm currently working on an application which will read through a directory full of MP3's downloaded from USENET and then sort them into sub-folders titled after the "Artists - Album" stored within the ID3 tag.

I currently only have code that can read ID3v1 tags but am working on updated code which will allow for ID3v2+ tags to be read as well.

I use a STRUCT to hold the ID3 data and then use a quick little routine to parse the raw byte[] array containing the ID3 data read from the file into my STRUCT.

The STRUCT:

PLAIN TEXT
C#:
  1. private struct ID3Tag
  2. {
  3. [MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
  4. public byte[] bTagHeader; //0-2
  5. [MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
  6. public byte[] bTrackName; //3-32
  7. [MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
  8. public byte[] bArtistsName; //33-62
  9. [MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
  10. public byte[] bAlbumName; //63-92
  11. [MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
  12. public byte[] bYear; //93-96
  13. [MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
  14. public byte[] bComment; //97-126
  15. [MarshalAs(UnmanagedType.ByValArray, SizeConst = 1)]
  16. public byte[] bGenres; //127
  17. }

Reading the last 128 bytes of the MP3 file into a byte[] array:

PLAIN TEXT
C#:
  1. private byte[] ReadID3FromFileToByte(string sLocalFile)
  2. {
  3. using (FileStream oFS = new FileStream(sLocalFile, FileMode.Open, FileAccess.Read))
  4. {
  5. using (BinaryReader oBR = new BinaryReader(oFS))
  6. {
  7. oBR.BaseStream.Position = (oFS.Length - 128);
  8. return oBR.ReadBytes((int)oFS.Length);
  9. }
  10. }
  11. }

The routine used to convert by the byte[] array into the STRUCT:

PLAIN TEXT
C#:
  1. private ID3Tag ID3BytesToID3Struct(byte[] bRawID3)
  2. {
  3. GCHandle hRawID3 = GCHandle.Alloc(bRawID3, GCHandleType.Pinned);
  4. ID3Tag id3NewTag = (ID3Tag)Marshal.PtrToStructure((hRawID3.AddrOfPinnedObject()), typeof(ID3Tag));
  5. hRawID3.Free();
  6. return id3NewTag;
  7. }

And the code which brings it all together:

PLAIN TEXT
C#:
  1. byte[] bFileData = ReadID3FromFileToByte(sMyInputFile);
  2. ID3Tag myID3 = ID3BytesToID3Struct(bFileData);

I hope this is able to help someone else out in their efforts to read an ID3 tag using C#.

Cheers!

]]> http://www.enusbaum.com/blog/2007/02/21/reading-id3-tags-using-c/feed/ 6 Why does the Microsoft .NET implementation of GZip compression suck? http://www.enusbaum.com/blog/2007/01/22/why-does-the-microsoft-net-implementation-of-gzip-compression-suck/ http://www.enusbaum.com/blog/2007/01/22/why-does-the-microsoft-net-implementation-of-gzip-compression-suck/#comments Mon, 22 Jan 2007 23:11:29 +0000 eric http://www.enusbaum.com/blog/2007/01/22/why-does-the-microsoft-net-implementation-of-gzip-compression-suck/ I've been working on a self contained patch generator using C#. For the patch data payload, I wanted to use GZip (or deflate) to compress the payload, this way if you generated a patch for a 1MB file, the patch file wouldn't be +1MB (5 bytes per change).

I did some quick tests to compare compression ratios on a simulated data payload:

Original Uncompressed Payload: 288k
GZipped using System.IO.Compression: 171k
Zipped using WinZip with "Super Fast" Compression: 105k
7-Zipped using 7-Zip with "Maximum" Compression: 61k

I had to do some research and dig into the MSDN libraries before the answer was finally revealed from the horses mouth:

"The compression functionality in DeflateStream and GZipStream is exposed as a stream. Data is read in on a byte-by-byte basis, so it is not possible to perform multiple passes to determine the best method for compressing entire files or large blocks of data. The DeflateStream and GZipStream classes are best used on uncompressed sources of data. If the source data is already compressed, using these classes may actually increase the size of the stream."

It's a bit disappointing that Microsoft itself would not recommend it's own built in compression routine, and even go so far to say that it may "actually increase the size of the stream". I hope in the upcoming version of .NET 3.0, they address some shortcomings of this area of the framework. Perhaps an exposed method where you can pass a byte[] which can be compressed using multiple passes and Adaptive Huffman Encoding.

I know there are several alternatives out there, such as assemblies which support the ZIP compression format, but wasn't the goal of .NET to move away from DLL hell? ;-)

In the meantime, I'll probably be sticking with the Microsoft implementation of GZip until I take some time to create my own Huffman encoding routine. I think a good place to start would be this book titled Data Compression: The Complete Reference. I remember thumbing through it once at a Borders book store and thinking it was a really great reference. At the time though, I had no use for such a resource.

Quick note to my readers: I understand that there are plenty of online resources for Adaptive Huffman Encoding and other compression methods using C# that have already been created. Personally, I learn by doing. So when I do my own research and create something from the ground up it really helps me understand how it all works.

]]> http://www.enusbaum.com/blog/2007/01/22/why-does-the-microsoft-net-implementation-of-gzip-compression-suck/feed/ -43