I re-ran some tests using the StopWatch class to hard code some metrics into the application and they also confirmed the findings. What’s up? How could a class that everyone says you can use to your hearts content when it came to string concatenation was failing me?

Turns out, it was a mix of misuse and a common misconception about the StringBuilder Class.

One of the first things you learn while picking up .NET is that the StringBuilder Class is your friend when it comes to concatenating large strings in memory. It beats the pants off of String.Concat and String.Format, while also being a mutable object in the Framework utilizing an in-memory buffer.

I used JetBrains dotTrace to help profile the application and it was very evident from the get-go that StringBuilder was causing the whole process to slow down.

The nature of my application was basically reading in a text buffer 1 character as a time, and using the StringBuilder as an output buffer. So for a 1k file, The method Append(char) would be called 1024 times. A 600k file would call Append(char) 614,400 times.

So why was I getting burned in execution time? The issue turned out to be two fold.

First, there’s overhead cost to the call. I don’t care how lightweight your method is, if you’re calling it SIX HUNDRED THOUSAND TIMES, it’s going to take a bit. Let alone a method who handles a string buffer in memory and string manipulation. So basically, no matter how fast StringBuilder actually is, it’s not a free call and you should consider the fact that the call still has overhead when architecting your solution.

Architecture brings me to my second point. While writing each character individually made sense initally, it seems that it was just lazy The optimized route would have been calling Append with a SUBSTRING of the input buffer, this way we avoid the overhead of multiple calls by writing all the neccisary data in one big blob.

So 600,000 calls to StringBuilder.Append(char) becomes only a few hundred calls to StringBuilder.Append(string.Substring(start, count)). Sure, the Substring Virtual Method itself has overhead, but it’s still less than the thousands of calls to Append(char) that we’re saving ourself

Conclusion?

StringBuilder is fast, but it’s not free. Take this into consideration when utilizing it while appending large data sets in small chunks.

Cheers!

I had a couple goals in mind while creating my routine:

1. KEEP IT SIMPLE — A lot of routines out there WORK, but their code is too overly complicated for their own good. This over complication leads to slowness which brings me to my next goal It should be a simple class that accepts input data, with simple public accessors that are easy to understand even for the novice developer (sorry folks, no asynchronous delegates).

2. MAKE IT FAST — When dealing with large amounts of data in C#, especially when running it through an algorithm, it’s all too easy to use all the handy built in virtual methods or using other build in tools which make coding easier with speed being the sacrifice. Die hard C++ developers will point to these routines as C#’s downfall as a legitimate language when it comes to data intensive tasks.

The class I came up with is pretty simple. I use a Generic List to store a collection of “Leaf” objects, which have several basic attributes that help not only identify its value but also its place in the tree. Using this method, I was able to utilize the built in methods of the List object (I know, for shame…. but it’s easier in this instance) by firing off anonymous delegates for searches and comparators. I’m not terribly worried about using these virtual methods here only because the creation and encoding of the tree is usually the smallest task in the process

The encoding and decoding is where I decided to focus on optimizations since this is where the BULK of the work is done. The .NET Framework has several methods that make working with binary data easy. You can use the Convert.ToString() method which allows you to pass in a BASE option, thus allowing you to convert any character to it’s binary representation. My original implementation used that method and the end result as embarrassingly slow

I went back to the drawing board and thought to myself, “If I had to re-write this in C++, how would I handle the encoding?” Duh, I’d be using bitwise operators up the wazoo!

After some recoding and pulling my hair out for a couple of hours, I was able to re-write the routine using bit operations and it works! On top of all that, it’s fast as all get out! My current benchmarks had it encoding a 1MB data set in under 1 second with ~50% compression. Not too shabby! Of course, compression ratios will vary depending on how normalized the input data set is.

Now I’m sure there’s some question you have on your mind and I’ll try to address them now:

Q: Does it use a lot of memory?

A: You bet your sweet ass it does! Seriously though, it’s only the method in which I setup the class that requires the memory. I establish an input buffer within the class that you can write the ‘raw’ data to, which is then read from during encoding. In addition, during encoding I create an output buffer in memory where the ‘encoded’ data is written. So it stands to reason that if you’re encoding 100MB of data, this routine can easily gobble up 200MB of RAM or more. The rule of thumb I found was File Size * 4 would be the memory requirement. There’s optimizations you can make that would lower the memory footprint (like, build the frequency table without buffering then read the input 1 byte at a time, say from a file), but I felt that would over complicate the solution and make it too focused for one specific instance. The current implementation is kept general for a reason

Q: Could this be done faster in C++?

A: Yes, probably…. but not much faster. Although the code is written in C#, at run time the IL is compiled to x86. The bit operations we’re using would compile the exact same as a C++ routine (XOR is XOR, I don’t care what language you’re using). In addition, the encoding of the data itself is only using primitive native types which limits any cross language differences. In fact, you can paste the encode and decode routines into C++ and they work! Your only speed improvement might be in the generation of the tree itself… but even then, that’s super small overhead when compared with the amount of data you’re probably compressing.

Of course, all that applied to the Encoding (Compression) side of the house, the decompression routine is pretty slow (about 3 seconds per 1MB of decompressed data) and could probably use some more optimizations.

Q: Is this any better than using the built in GZip or Deflate classes available in System.IO.Compression?

A: It’s not even close to being in competition with those routines This is really just a proof of concept for BASIC Huffman Coding, which doesn’t take into account advanced features of modern compression routines such as pattern or content mapping. This routine is slower (especially in decompression), so I wouldn’t go making anything like this your #1 choice for a compression routine if others are available So use this for educational purposes only.

Q: What version of the .NET Framework will this work with?

A: The code here was written in Visual Studio 2008 targeting .NET 3.5. I use some Framework 3.5 specific things (such as object initializers), but nothing that would make conversion difficult. I avoided LINQ only because I’m not entirely sold on the idea and I still like using anonymous delegates. This code could be converted to Framework 2.0 with minor changes and possibly Framework 1.1, but that might require a little more effort.

Q: What is the format used to store the compressed data?

A: I encode the decompression information within the final output stream. The output format is like this:

Bytes 0 – 8: Final Output Size (Not used, but there as a checksum if needed in the future)

Byte 9: Number of Bytes in the Decode Dictionary

Bytes 10 – n: Decode Dictionary

Between the Decode Dictionary and the actual data I add the characters “BCD” (which stands for Binary Coded Data). This lets me know where the dictionary ends and the actual coded data begins. It helped during debugging and I figure it’d help anyone else out there as well while working with this routine

Q: What’s with the essay? Just give me the code!

A: Fine! Seriously though, the only reason I’m doing such a long write-up on the code is to help people who are perhaps beginning to look into this sort of code for the first time and might have questions on why I did things a certain way. Understanding WHY the code was written helps understand how it operates.

So that’s the high and low of it!

Hope this helps someone out and if you have any questions, please feel free to leave a comment!

Cheers!

Huffman.zip – Download (3k)

After fumbling around with that for an hour or so an only succeeding in creating a small square on the screen, I headed over to Microsoft.com to see if any MSDN articles might exist to help me along in my ‘ground up’ learning of XNA. I was pleasantly surprised when I found a great article titled “Your First Game: Microsoft XNA Game Studio in 2D“. This was EXACTLY what I was looking for as it starts from the ground up, assuming the reader has never done game programming before, let alone 3D game programming.

The example provided my Microsoft in this article is a simple 2D Texture of a cat that bounces around the window. I was so pleased with the ease of coding this, I thought to myself, “Heck, how hard could it be to recreate Pong?” So I set upon my task.

Several hours and many Coca-Cola bottles later I had not only my first XNA game running in Windows, but after purchasing the XNA Creators Club annual subscription from the XBox Live! marketplace for $99, I had it running on my XBox 360 as well I decided to take a little extra time and add a debug information screen as well as a small welcome screen

Controls (PC):

Up/Down for Player 1 Paddle (Left): Q/A

Up/Down for Player 2 Paddle (Right): Up Arrow/Down Arrow

Debug Information: F1

Controls (XBox 360):

Up/Down for Player 1 Paddle (Left): Left Thumbstick on Player 1 Remote

Up/Down for Player 2 Paddle (Right): Left Thumbstick on Player 2 Remote

I did run into a couple of ‘gotchas!’ while working with Game Studio. The major one I had trouble with was when you’re developing for the XBox 360 you have to account for overscan on the Television and have your game render within the ‘Safe Area‘. I noticed that when I was playing my Pong! game on my LCD TV at 720p, the edges of the game were cut off and it looked like it was stretched past the borders of my TV. After asking the fine folks in #XNA on IRC about this issue, they were able to help me out. Now I have my Pong! game account for this by setting up an XBox 360 macro which pads the edges of the play area by 50 pixels.

I’m including the Source Code for my version of Pong! which I’ve titled, “2D WndrPong!”. You can work in XNA Game Studio for free using Visual C# 2005 Express Edition along with XNA Game Studio 2.0 To deploy your games to an XBox 360, you must pair your XBox with your PC (using the “XNA Game Studio Device Center” tool) then you must purchase an XNA Creators Club Membership from the XBox 360 Marketplace. I believe the prices are $49.99 for 3 Months, $99.99 for a year. I opted for the entire year since I know it’s going to take me some time to learn

Baby Steps

2D WndrPong! Source Code – Download (36k)

I’ve spent the last week or so playing around with the .NET Framework v3.5 and I really like what I see. Microsoft has also put a lot of work into Team Foundation Server 2008 as well. I’ve setup a Virtual Machine using Virtual PC here running Team Foundation Server 2008 on Windows 2003 Server R2 and was able to get WWWinamp imported + converted without issue!

[zoomer]66|450|0|WWWinamp in Team Foundation Server 2008|1|0[/zoomer]

One thing I have noticed is that TFS does take up quite a bit of resources. I had to kick my VM up to 2GB of RAM in order for it to handle and build WWWinamp without hitting the swap file. Something to keep in mind moving forward I suppose. I’d imagine a large installation of TFS would require something along the lines of a Quad Core CPU with at least 4GB RAM.

There are several features in .NET 3.5 that I plan on implementing in WWWinamp right off the bat:

• Object Initializers – This will really help clean up the already massive code base. It’ll also speed things up a bit
• Embedded Manifest – A feature I’m surprised they didn’t implement with the release of .NET 3.0. Now the .manifest file will be embedded into WWWinamp, so you won’t have the extra file handing around if you run Windows Vista
• Extension Methods – Again, this will help clean up the code base and make things faster. I imagine it’ll take up a little more memory, but these days RAM is abundant

Now, I see LinQ and I’m trying to figure out it’s role in everything. I think it’ll make DB communication easier but I’m not a fan of the anonymous data types and implicit local variables. I think it could lead to some sloppy coding methodology if not kept in check.

I think it’s handy that you can now run a SQL like statement on a generic, but I’m wondering what sets this apart from the FindAll method that already exists in System.Collection.Generics. It does make it easier though to search through a Generic Collection for a developer who doesn’t have a solid grasp on delegates.

So, all that said, things are moving along. I’ll keep you all updated on the progress and please, send in those feature requests!

So basically, I’m asking help for what to put in the next version. If you could leave a comment with any of the following, that’d REALLY help out!

• What feature would you like to see added to WWWinamp?
• Is there any nagging bug or weird behavior you would like fixed?

I’m currently fiddling around with Visual Studio 2008 right now and am looking into the benefits C# 3.5 offers over 2.0, which the current release of WWWinamp is based upon.

Thanks for your feedback and support everyone! I look forward to your responses!

I’ve been working on the Discogs.com API Assembly for .NET Applications now for a couple days and have been able to make some progress. It’s now a bit more stable as well a a tiny bit easier to use.

I took some time and added a DEBUG class. This class allows you to find out what’s happening within the Discogs.com Assembly if you start to have issues! This new class has two properties:

• Verbose (bool) – If set to TRUE, Verbose logging will be enabled allowing you to get more precise detail on what is going on within the Assembly. Otherwise, only exceptions will be logged.
• Log (string) – This is a string containing the current debug log.

Also, it has one Method:

• LogEvent (string sEvent) – Logs the value passed in to the debug log. This way you can use the same debug log from your own applications Should help make things a little easier.

New in this version as well is better error handling in the event of 404′s or an Artist/Release isn’t found. Before if you requested something that didn’t exist, the Assembly kinda crapped out while trying to deserialize the (non-existent) XML This has all been fixed.

I’m also including a small example program (with source code) on how to use the Discogs.com API Assembly. I’ve coded the example in C#, so sorry to all those VB.NET developers out there! If one of you guys would like to translate it to VB.NET, I’d be more than happy to post it here as well.

If you do not already have Microsoft Visual Studio installed, no worries! Microsoft provides a free version for C# development called Visual C# Express and you can get it here over at Microsoft.com.

You will need to update the Reference to the Discogs.com API Assembly. It currently points to where I had it setup on my local machine.
Any and all feedback is appreciated!

Cheers!

Discogs.com API Assembly for .NET Applications v1.0 Build 2871 – Download (9k)

Discogs.com API Assembly Example Application (with Source) – Download (10k)

The issue that I ran into was I wanted to log these messages not to the file system, but to the database. Microsoft provides for this in allowing people to create custom Trace Listeners. After some heavy Googling I was able to find this article on MSDN which describes a method in which you would be able to create a custom Trace Listener. Using the code from this article, I was able to boil it down to a simple code shell which anyone could take and use within their WCF application.

This is very helpful for people looking to capture and log the incoming and outgoing SOAP messages to their WCF application without having to create a custom dispatcher. On top of that, creating a Listener provides a drag-and-drop assembly you can use on any future project you might create.

The code for this project can be found here (ZIP, 5.4k). The solution was created using Visual Studio 2005.

Cheers!

I currently only have code that can read ID3v1 tags but am working on updated code which will allow for ID3v2+ tags to be read as well.

I use a STRUCT to hold the ID3 data and then use a quick little routine to parse the raw byte[] array containing the ID3 data read from the file into my STRUCT.

The STRUCT:

[csharp]

private struct ID3Tag
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
public byte[] bTagHeader; //0-2
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
public byte[] bTrackName; //3-32
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
public byte[] bArtistsName; //33-62
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
public byte[] bAlbumName; //63-92
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
public byte[] bYear; //93-96
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 30)]
public byte[] bComment; //97-126
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 1)]
public byte[] bGenres; //127
}

[/csharp]

Reading the last 128 bytes of the MP3 file into a byte[] array:

[csharp]

private byte[] ReadID3FromFileToByte(string sLocalFile)
{
using (FileStream oFS = new FileStream(sLocalFile, FileMode.Open, FileAccess.Read))
{
using (BinaryReader oBR = new BinaryReader(oFS))
{
oBR.BaseStream.Position = (oFS.Length – 128);
return oBR.ReadBytes((int)oFS.Length);
}
}
}

[/csharp]

The routine used to convert by the byte[] array into the STRUCT:

[csharp]

private ID3Tag ID3BytesToID3Struct(byte[] bRawID3)
{
GCHandle hRawID3 = GCHandle.Alloc(bRawID3, GCHandleType.Pinned);
ID3Tag id3NewTag = (ID3Tag)Marshal.PtrToStructure((hRawID3.AddrOfPinnedObject()), typeof(ID3Tag));
hRawID3.Free();
return id3NewTag;
}

[/csharp]

And the code which brings it all together:

[csharp]

byte[] bFileData = ReadID3FromFileToByte(sMyInputFile);
ID3Tag myID3 = ID3BytesToID3Struct(bFileData);

[/csharp]

I hope this is able to help someone else out in their efforts to read an ID3 tag using C#.

Cheers!

I did some quick tests to compare compression ratios on a simulated data payload:

Original Uncompressed Payload: 288k
GZipped using System.IO.Compression: 171k
Zipped using WinZip with “Super Fast” Compression: 105k
7-Zipped using 7-Zip with “Maximum” Compression: 61k

I had to do some research and dig into the MSDN libraries before the answer was finally revealed from the horses mouth:

“The compression functionality in DeflateStream and GZipStream is exposed as a stream. Data is read in on a byte-by-byte basis, so it is not possible to perform multiple passes to determine the best method for compressing entire files or large blocks of data. The DeflateStream and GZipStream classes are best used on uncompressed sources of data. If the source data is already compressed, using these classes may actually increase the size of the stream.”

It’s a bit disappointing that Microsoft itself would not recommend it’s own built in compression routine, and even go so far to say that it may “actually increase the size of the stream”. I hope in the upcoming version of .NET 3.0, they address some shortcomings of this area of the framework. Perhaps an exposed method where you can pass a byte[] which can be compressed using multiple passes and Adaptive Huffman Encoding.

I know there are several alternatives out there, such as assemblies which support the ZIP compression format, but wasn’t the goal of .NET to move away from DLL hell?

In the meantime, I’ll probably be sticking with the Microsoft implementation of GZip until I take some time to create my own Huffman encoding routine. I think a good place to start would be this book titled Data Compression: The Complete Reference. I remember thumbing through it once at a Borders book store and thinking it was a really great reference. At the time though, I had no use for such a resource.

Quick note to my readers: I understand that there are plenty of online resources for Adaptive Huffman Encoding and other compression methods using C# that have already been created. Personally, I learn by doing. So when I do my own research and create something from the ground up it really helps me understand how it all works.

One way I had considered was converting the images into a byte[] and then just putting this into the source code. The images I was working with were only about 200 bytes long, so this could be do-able.

Upon further investigation I came across this tutorial over at DevHood.com. It seemed a better idea to just roll with Embedded Resources in the EXE, since it would give me a few more options in the long run.

Below is the code I used to retrieve an embedded resource. You pass in the file name of the resource you want, and it’ll return a byte[] with the content:

[csharp]

public byte[] Server_ReadEmbeddedResource(string sFileName)
{
Assembly asResources = Assembly.GetExecutingAssembly(); //Referrence to the current assembly
string[] sResNames = asResources.GetManifestResourceNames(); //Store list of resources for the current assembly in an array

foreach (string sResourceName in sResNames)
{
if(sResourceName.EndsWith(sFileName))
{
BinaryReader oBR = new BinaryReader(asResources.GetManifestResourceStream(sResourceName));
return oBR.ReadBytes((int)asResources.GetManifestResourceStream(sResourceName).Length);
}
}
return new byte[] { 0 };
}

[/csharp]