C# Image Analysis (“auto-gaming”) – with source

By | 2010.08.16

For the past couple of weeks I have been playing Age of Conan with Sara.

I am Weathor, Aquilonian Priest of Mitra. MWAHAHA. :)

As always I’m looking for something to tinker with. Just playing the game is not that fun. I’ve been searching for addons for AoC, but I can’t find anything useful. Basically they don’t support any mentionable scripting on the client or server. So I thought it would be fun to do an app that integrates this. Totally useless, but fun. (Yeah, I wanted to play with image analysis…)

I figured the app needed to do three main things:

  • Make screenshot of a specific application that is running, for example Age of Conan.
    It is important that it takes the app and not the whole screen as we need to know where things are on the screen.
  • Analysis of the screenshots being taken.
    I’ve added a module interface so its easy to make new modules. You get a lot of functionality for free.
  • Send keystrokes/commands to the game.
    I’m using PostMessage Win32 API, but it seems to require the application to be active (front), maybe a protection built into the game?

After a few hours of hacking I have a working application. I’m using the new .Net 4.0 Parallel.For/ForEach feature to split the processing into mutliple threads. It actually works quite fast too, even with high resolution. Up to 10fps on 1680×1050 – the main time hog is taking the screenshot. The analysis and copying is very fast. See screenshot bellow. Note that the speed drops a bit when I set it to stretch (shrink) image to fit on the second screen. There is a lot of room for optimizing though…

The application is basically a framework for capturing screen, analyzing it and reacting to the result. The analysis module architecture makes it easy to get started for people who want to extend it. Each module lets the framework what area of the screen it is interested in and then this area is served to it.

The analysis modules are executed in parallel and/or in sequence depending on what the modules want. It also has a priority queue so it is possible to pre-process the image in one module and then do analysis in another.

As you can see from the screenshot I have 3 working modules: Health, Stamina and Mana. The modules each look at a specific line on the screen where they count the number of correctly colored pixels. Just a straight forward way of reading a percentage indicator.

image 

A simple image analysis module looks something like this:

public int Health = 0;

public override unsafe void NewImage(int imageHeight, int imageWidth)

{

    _bounds = new Rectangle(10, imageHeight - 60, 115, 1);

    Program.SharedStorage.Set("Health", Health.ToString());

    Health = 0;

}

public override unsafe AnalyzerPixelReturnData ProcessPixel(int row, int col, PixelData pixel, byte* srcByteArray, byte* dstByteArray, int imageHeight, int imageWidth)

{

    if (pixel.Red > 170

        && pixel.Green < 20

        && pixel.Blue < 20)

    {

        Health++;

    }

    return new AnalyzerPixelReturnData() { IsModified = true, NewPixelData = red };

}

Image analysis modules can choose between doing their own iteration or having it done for them. In this sample it is being done for the module. ProcessPixel() is called with row number, column number and pixel color. Only rows/cols within _bounds object are served to it.

Applications supports that the target window both moves and is resized, but if it is resized then the analysis modules probably needs to recalculate where they are looking for what.

Source code:

FacebookLinkedInTwitterOrkutDiggShare/Bookmark

5 thoughts on “C# Image Analysis (“auto-gaming”) – with source

  1. Djenad Razic

    Hi Ted, I have played with your “Thingie”. And I need to thank you for great start in that field. I am added to mentioned thing Emgu (also I was used advantage of GPU processing) and some mouse and keyboard lib which was resulting fully capable bot for my example EVE Online :) . Reprogrammed modules to use some generics and stored most of the settings to XML (also includes presets for various ships).
    If you are interested we could finish it and earn few bucks selling it online?

    Reply
    1. tedd Post author

      How fun that the code is useful for someone. I hope I have time to update the speed of screenshots soon, from what I can remember I found a way to make it neglible.

      When it comes to the code itself, please consider it BSD licensed. In short this means that you are free to use the code, including making money from it, and you are not required to release the source code you build around it – I would however encourage everyone to release some code now and then.

      Right now I have way too much to do, so no time for fun adventures in gaming or bot-gaming. :)

      Reply
  2. Toni

    Hey I was hopping you could help me out a bit.. I’m messing with your source for an assistant program. Your source, as it stands, is way beyond me so I’m happy I found something I can use almost out of the box for what I need but I can’t figure out how you’ve made the modules work. If its not too much trouble maybe you can point me in the right direction/show me how to do this?

    I’m sure this is VERY simple to do with your code but I cant figure it out. I am working with pentominoes puzzles. There are between 1 and 4 on screen in quadrants. The edges are rough and the background is textured. I figured the best way to work with them would be set a rectangle for the area in which each puzzle can be displayed and scan top to bottom, left to right looking for consecutive strings of black pixels to find the left bound, etc for upper lower and right. Then I just need to find the shape in 19×19 blocks and submit it to a string in sharedstorage so puzzle1 in sharedstorage may be
    581110110001101111011111001111010001111111 where 1 = open, 2 = blocked, and 58 = row/column count
    From there I can integrate it with the rest of my code and it is done.

    Thanks for any and all help,
    Toni

    Reply
    1. tedd Post author

      Hi

      Probably way too late reply from my side. Email notifications easily drown in the storm of incoming crap I get every day (even after spam filtering).

      Your idea of creating quadrants that you scan is sound. If you need to read numbers then I would scan all quadrants for certain key-points that only exists for certain numbers. For example 5 and 7 both have a pixel in upper left, but only 5 has a pixel bellow that. Figuring out these key points isn’t too hard and would make it easy to determine what numbers are where. Part from that letter recognition from image (OCR) is not too difficult, you can find many good libraries that do that already. Then you could just scan the whole board and read the response as text.

      Part from that I would recommend IRC or similar to get you started on understanding modules. For example #C# on Undernet (or Freenode or EFNet) have people that can answer more specific questions if you pastebin them the code you are having problems with.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *