.NET Remoting + F5 Load Balancer == *headdesk*

Some of the topics I post on here are usually only of my interest, but once in a while I’ll give back to the community on issues that had me ripping my hair out.

Recently I was working on an application that utilized .NET Remoting to access the middle tier of the application through an F5 Load Balancer. For whatever reason, the .NET Remoting application kept dropping the connection with errors such as:

“An existing connection was forcibly closed by the remote host”

Through trouble shooting we were able to determine that it was definitely and issue with the F5 Load Balancer handling the connections being established by the .NET Remoting host.

We were able to resolve the issue by disabling the OneConnect profile on the F5 Load Balancer for the .NET Remoting VIP. Once the OneConnect profile was disable, the .NET Remoting application was able to hit the Remoting Service behind the VIP without issue.

Hope this helps anyone out there having the same issue!

Cheers!

ApocBot v0.0

ApocBot v0.0 — The First (not completely working) Open Source MTGO Trade Bot written in C#/.NET!

Now that the fanfare is over, let’s get down to the nitty-gritty.

What was with the delay?

The truth of the matter is that life just has a way of happening. My desire to play Magic The Gathering: Online waxes and wanes through the years and I started this project shortly after moving across the country. So you can say it was the product of my being bored in the evenings Over the course of a few months my interest in this project started to fade as I settled into my new job and had to travel a bit.

I had debated in my mind on how to handle the bot. I know over time it would be a maintenance nightmare as people would require support and updates, which is honestly why I decided to go Open Source with it. For some reason people feel that paying for software implies some form of support! In the end, I just stopped working on it because (a) MTGO is just a hobby, so I wasn’t looking to make money off the bot and (b) I didn’t want to have to constantly maintain and update the code to work with new versions/UI changes as they come out. That’s just the lazy bastard I am

That being said, I still feel this is an important project because it gives the community at large the ability to finally be more open about automation on MTGO (and possibly other games).

So what shape is the code currently in?

Out of the box, the code isn’t going to work. I’ll go into the details below on why/what needs to be updated to get it in running shape.

Other than updating some of the code to work with any UI/screen changes, it should still work pretty well! I know it worked the last time I tried to use it (six months ago?).. so I’m sure with some TLC, someone can get it up and running again.

And it’s free?!

Yep! Anyone is free to download and use the code for ApocBot in their own Magic Online Trading Bots as long as they adhere to the BSD License terms. If anyone is feeling giving, links back to my blog and some Magic Online gratitude would be appreciated! (My MTGO user name is “WndrBr3d“)

Download:

ApocBot and the source code are released under the BSD License. Please review the included “license.txt” file for terms and conditions. For more information on the BSD Licenses, click here.

ApocBot v0.0 (Source)

Requirements: Visual Studio 2010 and .NET 4.0

And now.. the very rough technical details on how it works…

How does it even work?!

I followed the same basic idea as a lot of the AutoIT bots that are out there in the sense that I’m snapping screen shots and determining what’s on the screen by creating checksums of specified areas. I figured, “Hey, if they can make it work in AutoIT, I’m sure I can make it work in C# even better!”

So using the idea of area checksums, ApocBot can identify what is currently being displayed on the screen and act appropriately.

The basic configuration for ApocBot to run was WindowsXP (Classic Theme) running at 1024×768 (32-bit color depth).

The Classes/Controllers

MTGOController.cs

The MTGOController class holds the primary logic for ApocBot including the main program loop and all MTGO Object definitions.

On start up, all the MTGO Objects are defined. Currently these are all hard coded but I had planned to eventually move them all to an XML file and just have them deserialized on start up (any takers on this? ).. but for now, they’re all hard-coded.

After everything is setup, the main program loop will start. The basic function is this:

1. Grab a screen shot
2. Determine the state of the bot (switch statement)
3. Based upon the state, check the screen for any known events/items to act upon
4. Sleep 250ms (to avoid CPU spike caused by constant screen polling)

ScreenController.cs

The ScreenController class holds all the logic required to grab the screen shots and calculate the area checksums.

MouseKeyboardController.cs

The MouseKeyboardController class holds all the logic required to make mouse movements and send keys to the program. Hooks into the Windows API are used to simulate the mouse events to the OS.

The Objects

There are really three main objects used by ApocBot to know what’s going on:

1. MTGOScreen - Represents a screen in MTGO, such as the “Home” or “Trade” screens (Multiple Areas, Multiple Points)
2. MTGOArea - Represents an area on a screen, such as a pop-up dialogue, card graphic or window (Multiple Points)
3. MTGOPoint - Represents a point on the screen, such as an “OK” or “Cancel” button

Objects are referenced in the system to main properties: Name and Checksum.

The checksums that are currently hard-coded might be out of date. You can generate new checksums using the AreaChecksum() method in the ScreenController.

Almost forgot! It’s named “ApocBot” because Apocalypse was my favorite Magic: The Gathering expansion!

Cheers!

• Basic Trade functionality is completed
  • Trade x for y (i.e. 4 Uncommons for 1 Rare)
  • Buy/Sell x for x (i.e. 6 Rares for 1 Ticket or 1 Ticket for 6 Rares)
• Smart Bot functionality is 70% Completed

Currently ApocBot is able to correctly identify cards that are selected during a trade and quote the stored price for the cards.  So far, so good!

Want some more info?

Okay, so here’s the big announce. I’m going to open source the Basic Trade functionality. I think it’s something that’s really fundamental and should have been included in the MTGO client to begin with. To that, I think it’s stilly to charge for such a basic function after writing it.

So now to answer some obvious questions:

Q: ZOMG! You’re going to open source it?!

A: Yes. ApocBot will be the first free, open source MTGO trade bot

Q: What license will it be released under?

A: I’m going to release it under GPL v3

Q: When?!

A: It’ll be available on this site by January 1st, 2010 I’ll need some beta testers to use it before I release the source, just to make sure I’m not going to cause more problems than good. I’ll send out a call for that within the coming weeks.

Q: What about SmartBot functionality?

A: I’m still working on SmartBot and to be honest, it has been more trouble than it’s worth. I’ll make a decision on that as soon as I think it’s at a functional point. Because I’m currently working full time, I’m only able to dedicate a couple hours a week to ApocBot so this might take a bit. Part of me open sorucing ApocBot in the first place is so the community can begin development of such a plugin or enhancement.

Q: What will I need to develop ApocBot?

A: ApocBot is written in Microsoft C# and you will need Microsoft Visual Studio 2008 (with .NET 3.5) to develop. And Magic: The Gathering Online.. of course

So that’s that! What’s funny is I was mulling this post around in my head and how to handle a MTGO bot that I knew (a) wouldn’t make a ton of money and (b) should really be free anyways.

Please feel free to e-Mail me or leave any questions you might have here on this post and I’ll be more vigilant in getting back to everyone.

Cheers!

I’ve been working on my first retail product and I’m really excited about it!

After trying to secure a 3rd party trading bot for Magic The Gathering Online (MTGO) v3 from Wizards of the Coast (WotC), I decided it was time to strike out on my own and write one. A lot of people had done so already using automation software (like AutoIT), but I went the path of a more professional software development environment and wrote it in .NET using C# and WCF.

WCF you ask? Yep! I don’t want to dive into the details too much at the moment but it will revolve around a central data hub providing real time updates to users of the application (vs. having to download and deploy a new version every time an update is published from WotC). There will be a bunch of good features as well!

I wanted to poll people on pricing though. I’ve seen prices for ‘Simple’ Magic Online trading bots for anywhere from $99 to $399! And to that, I haven’t even been able to find the price of a ‘Smart Bot’ anywhere, as it seems the people who build them keep the software to themselves.

What I’m thinking about doing is setting up a monthly pricing model. Here’s the matrix:

Trade Bot: Your basic Magic Online trading bot. “Buying/Selling x Cards for x Tickets” or “Trading x Uncommons for x Rares”

Smart Bot: Each card in your library can be assigned a specific price. You’re running a Virtual Store on Magic Online.

Some features I’m working on for the Smart Bot will be:

• User Credit System (and ability to create Credit Network spanning multiple bots)
• Interactive Website Showing Sales History / Income Information

I’m offering a time based licensing because I think it’s the most fair pricing structure for this type of software.

So I’m curious to hear back from anyone who stumbles across this blog post! I want to hear features and requests you would have for such a piece of software! I know Magic Trading bots are a shadey business at best, but I wanted to try to make a legitimate alternative to having to pay a lot of money up front for a piece of software you might not like.

I’ll try to post some screen shots of the bot in action while I’m testing it. I’m really excited about this project!

Cheers!

I re-ran some tests using the StopWatch class to hard code some metrics into the application and they also confirmed the findings. What’s up? How could a class that everyone says you can use to your hearts content when it came to string concatenation was failing me?

Turns out, it was a mix of misuse and a common misconception about the StringBuilder Class.

One of the first things you learn while picking up .NET is that the StringBuilder Class is your friend when it comes to concatenating large strings in memory. It beats the pants off of String.Concat and String.Format, while also being a mutable object in the Framework utilizing an in-memory buffer.

I used JetBrains dotTrace to help profile the application and it was very evident from the get-go that StringBuilder was causing the whole process to slow down.

The nature of my application was basically reading in a text buffer 1 character as a time, and using the StringBuilder as an output buffer. So for a 1k file, The method Append(char) would be called 1024 times. A 600k file would call Append(char) 614,400 times.

So why was I getting burned in execution time? The issue turned out to be two fold.

First, there’s overhead cost to the call. I don’t care how lightweight your method is, if you’re calling it SIX HUNDRED THOUSAND TIMES, it’s going to take a bit. Let alone a method who handles a string buffer in memory and string manipulation. So basically, no matter how fast StringBuilder actually is, it’s not a free call and you should consider the fact that the call still has overhead when architecting your solution.

Architecture brings me to my second point. While writing each character individually made sense initally, it seems that it was just lazy The optimized route would have been calling Append with a SUBSTRING of the input buffer, this way we avoid the overhead of multiple calls by writing all the neccisary data in one big blob.

So 600,000 calls to StringBuilder.Append(char) becomes only a few hundred calls to StringBuilder.Append(string.Substring(start, count)). Sure, the Substring Virtual Method itself has overhead, but it’s still less than the thousands of calls to Append(char) that we’re saving ourself

Conclusion?

StringBuilder is fast, but it’s not free. Take this into consideration when utilizing it while appending large data sets in small chunks.

Cheers!

I had a couple goals in mind while creating my routine:

1. KEEP IT SIMPLE — A lot of routines out there WORK, but their code is too overly complicated for their own good. This over complication leads to slowness which brings me to my next goal It should be a simple class that accepts input data, with simple public accessors that are easy to understand even for the novice developer (sorry folks, no asynchronous delegates).

2. MAKE IT FAST — When dealing with large amounts of data in C#, especially when running it through an algorithm, it’s all too easy to use all the handy built in virtual methods or using other build in tools which make coding easier with speed being the sacrifice. Die hard C++ developers will point to these routines as C#’s downfall as a legitimate language when it comes to data intensive tasks.

The class I came up with is pretty simple. I use a Generic List to store a collection of “Leaf” objects, which have several basic attributes that help not only identify its value but also its place in the tree. Using this method, I was able to utilize the built in methods of the List object (I know, for shame…. but it’s easier in this instance) by firing off anonymous delegates for searches and comparators. I’m not terribly worried about using these virtual methods here only because the creation and encoding of the tree is usually the smallest task in the process

The encoding and decoding is where I decided to focus on optimizations since this is where the BULK of the work is done. The .NET Framework has several methods that make working with binary data easy. You can use the Convert.ToString() method which allows you to pass in a BASE option, thus allowing you to convert any character to it’s binary representation. My original implementation used that method and the end result as embarrassingly slow

I went back to the drawing board and thought to myself, “If I had to re-write this in C++, how would I handle the encoding?” Duh, I’d be using bitwise operators up the wazoo!

After some recoding and pulling my hair out for a couple of hours, I was able to re-write the routine using bit operations and it works! On top of all that, it’s fast as all get out! My current benchmarks had it encoding a 1MB data set in under 1 second with ~50% compression. Not too shabby! Of course, compression ratios will vary depending on how normalized the input data set is.

Now I’m sure there’s some question you have on your mind and I’ll try to address them now:

Q: Does it use a lot of memory?

A: You bet your sweet ass it does! Seriously though, it’s only the method in which I setup the class that requires the memory. I establish an input buffer within the class that you can write the ‘raw’ data to, which is then read from during encoding. In addition, during encoding I create an output buffer in memory where the ‘encoded’ data is written. So it stands to reason that if you’re encoding 100MB of data, this routine can easily gobble up 200MB of RAM or more. The rule of thumb I found was File Size * 4 would be the memory requirement. There’s optimizations you can make that would lower the memory footprint (like, build the frequency table without buffering then read the input 1 byte at a time, say from a file), but I felt that would over complicate the solution and make it too focused for one specific instance. The current implementation is kept general for a reason

Q: Could this be done faster in C++?

A: Yes, probably…. but not much faster. Although the code is written in C#, at run time the IL is compiled to x86. The bit operations we’re using would compile the exact same as a C++ routine (XOR is XOR, I don’t care what language you’re using). In addition, the encoding of the data itself is only using primitive native types which limits any cross language differences. In fact, you can paste the encode and decode routines into C++ and they work! Your only speed improvement might be in the generation of the tree itself… but even then, that’s super small overhead when compared with the amount of data you’re probably compressing.

Of course, all that applied to the Encoding (Compression) side of the house, the decompression routine is pretty slow (about 3 seconds per 1MB of decompressed data) and could probably use some more optimizations.

Q: Is this any better than using the built in GZip or Deflate classes available in System.IO.Compression?

A: It’s not even close to being in competition with those routines This is really just a proof of concept for BASIC Huffman Coding, which doesn’t take into account advanced features of modern compression routines such as pattern or content mapping. This routine is slower (especially in decompression), so I wouldn’t go making anything like this your #1 choice for a compression routine if others are available So use this for educational purposes only.

Q: What version of the .NET Framework will this work with?

A: The code here was written in Visual Studio 2008 targeting .NET 3.5. I use some Framework 3.5 specific things (such as object initializers), but nothing that would make conversion difficult. I avoided LINQ only because I’m not entirely sold on the idea and I still like using anonymous delegates. This code could be converted to Framework 2.0 with minor changes and possibly Framework 1.1, but that might require a little more effort.

Q: What is the format used to store the compressed data?

A: I encode the decompression information within the final output stream. The output format is like this:

Bytes 0 – 8: Final Output Size (Not used, but there as a checksum if needed in the future)

Byte 9: Number of Bytes in the Decode Dictionary

Bytes 10 – n: Decode Dictionary

Between the Decode Dictionary and the actual data I add the characters “BCD” (which stands for Binary Coded Data). This lets me know where the dictionary ends and the actual coded data begins. It helped during debugging and I figure it’d help anyone else out there as well while working with this routine

Q: What’s with the essay? Just give me the code!

A: Fine! Seriously though, the only reason I’m doing such a long write-up on the code is to help people who are perhaps beginning to look into this sort of code for the first time and might have questions on why I did things a certain way. Understanding WHY the code was written helps understand how it operates.

So that’s the high and low of it!

Hope this helps someone out and if you have any questions, please feel free to leave a comment!

Cheers!

Huffman.zip – Download (3k)

I’m currently working on the latest version of WWWinamp after taking another hiatus from the project in order to clear my head and work on some other fun things such as the previously mentioned XNA projects

The next version of WWWinamp will be targeted for the .NET 3.5 Framework and you will need to have it installed on your machine in order to run WWWinamp. Most of the changes in this version are behind the scenes and will probably be transparent to you, the user. BUT, some visible changes in the upcoming version of WWWinamp 4.2 will be:

• Automatic Version Checking for new releases of WWWinamp
• Ability to enable or disable Automatic Version Checking
• Ability to enable or disable Windows Vista warning message
  
• Improved performance under load

I know, nothing earth shattering on the horizon. “Why is this?” you might as. Well, the honest reason is that I haven’t received any bug reports with issues that weren’t related to a misconfiguration or user error. This is a good thing! It means that WWWinamp is stable and there aren’t any horrible bugs that continue to nag users. The downside to this is that because WWWinamp is pretty feature rich as is, I’m running low on ideas that could be implemented.

An idea I’ve tossed about and am thinking about trying out is creating a WWWinamp .NET assembly for user within your own Windows or ASP.NET applications developed using the .NET framework. The assembly would handle the media library and searching of the library so you wouldn’t have to code your own database and file searching routines, as well as handling the unmanaged code required to interface with Winamp (if it’s running locally on the WWWinamp machine) or communicating via WCF to a remote instance of WWWinamp running the WCF daemon.

I think that between the WWWinamp Server and my proposed WWWinamp Assembly, that’d give both advanced and novice users full control over their setup.

As I mentioned in a previous post, I’ve moved WWWinamp into Visual Studio 2008 with Team Foundation Server. Within the latest version of Visual Studio 2008 there is a set of code analysis tools which calculate such items as Code Maintainability and Cyclomatic Complexity. I ran it on WWWinamp and it scored the following:

Maintainability Index: 83 (out of 100)

Cyclomatic Complexity: 636

Depth of Inheritance: 7

Class Coupling: 151

Lines of Code: 1,829 (!!)

After over a year, I think WWWinamp has gotten better and continues to improve with every version!

Big thanks to everyone for their support and keep your eyes peeled here for the latest version of WWWinamp!

After fumbling around with that for an hour or so an only succeeding in creating a small square on the screen, I headed over to Microsoft.com to see if any MSDN articles might exist to help me along in my ‘ground up’ learning of XNA. I was pleasantly surprised when I found a great article titled “Your First Game: Microsoft XNA Game Studio in 2D“. This was EXACTLY what I was looking for as it starts from the ground up, assuming the reader has never done game programming before, let alone 3D game programming.

The example provided my Microsoft in this article is a simple 2D Texture of a cat that bounces around the window. I was so pleased with the ease of coding this, I thought to myself, “Heck, how hard could it be to recreate Pong?” So I set upon my task.

Several hours and many Coca-Cola bottles later I had not only my first XNA game running in Windows, but after purchasing the XNA Creators Club annual subscription from the XBox Live! marketplace for $99, I had it running on my XBox 360 as well I decided to take a little extra time and add a debug information screen as well as a small welcome screen

Controls (PC):

Up/Down for Player 1 Paddle (Left): Q/A

Up/Down for Player 2 Paddle (Right): Up Arrow/Down Arrow

Debug Information: F1

Controls (XBox 360):

Up/Down for Player 1 Paddle (Left): Left Thumbstick on Player 1 Remote

Up/Down for Player 2 Paddle (Right): Left Thumbstick on Player 2 Remote

I did run into a couple of ‘gotchas!’ while working with Game Studio. The major one I had trouble with was when you’re developing for the XBox 360 you have to account for overscan on the Television and have your game render within the ‘Safe Area‘. I noticed that when I was playing my Pong! game on my LCD TV at 720p, the edges of the game were cut off and it looked like it was stretched past the borders of my TV. After asking the fine folks in #XNA on IRC about this issue, they were able to help me out. Now I have my Pong! game account for this by setting up an XBox 360 macro which pads the edges of the play area by 50 pixels.

I’m including the Source Code for my version of Pong! which I’ve titled, “2D WndrPong!”. You can work in XNA Game Studio for free using Visual C# 2005 Express Edition along with XNA Game Studio 2.0 To deploy your games to an XBox 360, you must pair your XBox with your PC (using the “XNA Game Studio Device Center” tool) then you must purchase an XNA Creators Club Membership from the XBox 360 Marketplace. I believe the prices are $49.99 for 3 Months, $99.99 for a year. I opted for the entire year since I know it’s going to take me some time to learn

Baby Steps

2D WndrPong! Source Code – Download (36k)

I’ve spent the last week or so playing around with the .NET Framework v3.5 and I really like what I see. Microsoft has also put a lot of work into Team Foundation Server 2008 as well. I’ve setup a Virtual Machine using Virtual PC here running Team Foundation Server 2008 on Windows 2003 Server R2 and was able to get WWWinamp imported + converted without issue!

[zoomer]66|450|0|WWWinamp in Team Foundation Server 2008|1|0[/zoomer]

One thing I have noticed is that TFS does take up quite a bit of resources. I had to kick my VM up to 2GB of RAM in order for it to handle and build WWWinamp without hitting the swap file. Something to keep in mind moving forward I suppose. I’d imagine a large installation of TFS would require something along the lines of a Quad Core CPU with at least 4GB RAM.

There are several features in .NET 3.5 that I plan on implementing in WWWinamp right off the bat:

• Object Initializers – This will really help clean up the already massive code base. It’ll also speed things up a bit
• Embedded Manifest – A feature I’m surprised they didn’t implement with the release of .NET 3.0. Now the .manifest file will be embedded into WWWinamp, so you won’t have the extra file handing around if you run Windows Vista
• Extension Methods – Again, this will help clean up the code base and make things faster. I imagine it’ll take up a little more memory, but these days RAM is abundant

Now, I see LinQ and I’m trying to figure out it’s role in everything. I think it’ll make DB communication easier but I’m not a fan of the anonymous data types and implicit local variables. I think it could lead to some sloppy coding methodology if not kept in check.

I think it’s handy that you can now run a SQL like statement on a generic, but I’m wondering what sets this apart from the FindAll method that already exists in System.Collection.Generics. It does make it easier though to search through a Generic Collection for a developer who doesn’t have a solid grasp on delegates.

So, all that said, things are moving along. I’ll keep you all updated on the progress and please, send in those feature requests!

So basically, I’m asking help for what to put in the next version. If you could leave a comment with any of the following, that’d REALLY help out!

• What feature would you like to see added to WWWinamp?
• Is there any nagging bug or weird behavior you would like fixed?

I’m currently fiddling around with Visual Studio 2008 right now and am looking into the benefits C# 3.5 offers over 2.0, which the current release of WWWinamp is based upon.

Thanks for your feedback and support everyone! I look forward to your responses!