Async/await HTTP server in C#

Download: Tedd.Demo.HttpServer.zip (slightly outdated compared to code below)

Background

Ok, so there are already alternatives for web servers in C#. Some of them are very feature rich.

But I wanted to play around with TPL and async/await a bit so I put together a quick hack of a minimal HTTP-server.

The main focus here has been on the use of async await keyword in .Net 4.5 that allows for thread conservation while keeping the code clean. .Net will automatically manage thread pool. In theory this solution could hold 10 000 connections and still use only 10 threads.

Traditional a server like this would be implemented in one of two ways:

  • Synchronous: one dedicated thread per connected client.
  • Asynchronous: Required calls to Begin* and End* to form a sort of loop. Methods are re-executed for every receive or send operation and you have to split your code into several methods.

Task Parallel Library + await

With the introduction TPL the traditional thread model is encapsulated in a managed Task with a wide variety of functionality. This in itself is handy, works great with anonymous methods and is handled by .Net managed thread pool. In .Net 4.5 a new keyword “await” was introduced which magically allows us to pause methods while waiting for a synchronous operation. During this pause the thread returns to the thread pool to do other work, and upon resume a thread is assigned the task of continuing execution of the method. The magic is of course just an illusion and behind the scenes the compiler adds some code that resembles the yield implementation. It converts the method to a state machine allowing a thread to enter and exit a method at certain positions using goto-statements and labels.

An example of how await allows us to release the thread while waiting for write operation to complete, then flush the stream. This would traditionally either block a thread (synchronous) or require you to split code into two methods (asynchronous).

private async void SendText(string text)
{
    var data = Encoding.UTF8.GetBytes(text);
    await _networkStream.WriteAsync(data, 0, data.Length);
    await _networkStream.FlushAsync();
}

The code

As always you should consider security if you use this code. I have implemented a regex-check of the file, which is the only data we pass to the OS in any way. New version don’t contain any regex, its a gaping security hole! But a regex-check like that could probably easily be vulnerable to a prematurely null-terminated string for instance. (Don’t trust code.)

Listener.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading.Tasks;

namespace Tedd.Demo.HttpServer.Server
{
    class Listener
    {
        public readonly int Port;
        private readonly TcpListener _tcpListener;
        private readonly Task _listenTask;

        public Listener(int port)
        {
            Port = port;

            // Start listening
            _tcpListener = new TcpListener(IPAddress.Any, Port);
            _tcpListener.Start();

            // Start a background thread to listen for incoming
            _listenTask = Task.Factory.StartNew(() => ListenLoop());

        }

        private async void ListenLoop()
        {
            for (; ; )
            {
                // Wait for connection
                var socket = await _tcpListener.AcceptSocketAsync();
                if (socket == null)
                    break;

                // Got new connection, create a client handler for it
                var client = new Client(socket);
                // Create a task to handle new connection
                Task.Factory.StartNew(client.Do);
            }
        }
    }
}

Client.cs

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Sockets;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using Microsoft.Win32;

namespace Tedd.Demo.HttpServer.Server
{
    class Client
    {
        private readonly Socket _socket;
        private readonly NetworkStream _networkStream;
        private readonly MemoryStream _memoryStream = new MemoryStream();
        private readonly StreamReader _streamReader;
        private readonly string _serverName = "Tedd.Demo.HttpServer";

        public Client(Socket socket)
        {
            _socket = socket;
            _networkStream = new NetworkStream(socket, true);
            _streamReader = new StreamReader(_memoryStream);
        }

        public async void Do()
        {
            // We are executed on a separate thread from listener, but will release this back to the threadpool as often as we can.
            byte[] buffer = new byte[4096];
            for (; ; )
            {
                // Read a chunk of data
                int bytesRead = await _networkStream.ReadAsync(buffer, 0, buffer.Length);

                // If Read returns with no data then the connection is closed.
                if (bytesRead == 0)
                    return;

                // Write to buffer and process
                _memoryStream.Seek(0, SeekOrigin.End);
                _memoryStream.Write(buffer, 0, bytesRead);
                bool done = ProcessHeader();
                if (done)
                    break;
            }

        }

        private bool ProcessHeader()
        {
            // Our task is to find when full HTTP header has been received, then send reply.
            for (; ; )
            {
                _memoryStream.Seek(0, SeekOrigin.Begin);
                var line = _streamReader.ReadLine();
                if (line == null)
                    break;

                if (line.ToUpperInvariant().StartsWith("GET "))
                {
                    // We got a request: GET /file HTTP/1.1
                    var file = line.Split(' ')[1].TrimStart('/');
                    // Default document is index.html
                    if (string.IsNullOrWhiteSpace(file))
                        file = "index.html";
                    // Send header+file
                    SendFile(file);
                    // Close connection (we don't support keep-alive)
                    _socket.Close();
                    return true;
                }

            }
            return false;
        }

        private void SendFile(string file)
        {
            // Get info and assemble header
            byte[] data;
            string responseCode = "";
            string contentType = "";
            try
            {
                if (File.Exists(file))
                {
                    // Read file
                    data = File.ReadAllBytes(file);
                    contentType = GetContentType(Path.GetExtension(file).TrimStart(".".ToCharArray()));
                    responseCode = "200 OK";
                }
                else
                {
                    data = System.Text.Encoding.ASCII.GetBytes("<html><body><h1>404 File Not Found</h1></body></html>");
                    contentType = GetContentType("html");
                    responseCode = "404 Not found";
                }
            }
            catch (Exception exception)
            {
                // In case of error dump exception to client.
                data = System.Text.Encoding.ASCII.GetBytes("<html><body><h1>500 Internal server error</h1><pre>" + exception.ToString() + "</pre></body></html>");
                responseCode = "500 Internal server error";
            }

            string header = string.Format("HTTP/1.1 {0}\r\n"
                                          + "Server: {1}\r\n"
                                          + "Content-Length: {2}\r\n"
                                          + "Content-Type: {3}\r\n"
                                          + "Keep-Alive: Close\r\n"
                                          + "\r\n",
                                          responseCode, _serverName, data.Length, contentType);
            // Send header & data
            await _socket.SendAsync(System.Text.Encoding.ASCII.GetBytes(header));
            await _socket.SendAsync(data);
            await _socket.FlushAsync();

        }

        /// <summary>
        /// Get mime type from a file extension
        /// </summary>
        /// <param name="extension">File extension without starting dot (html, not .html)</param>
        /// <returns>Mime type or default mime type "application/octet-stream" if not found.</returns>
        private string GetContentType(string extension)
        {
            // We are accessing the registry with data received from third party, so we need to have a strict security test. We only allow letters and numbers.
            if (Regex.IsMatch(extension, "^[a-z0-9]+$", RegexOptions.IgnoreCase | RegexOptions.Compiled))
                return (Registry.GetValue(@"HKEY_CLASSES_ROOT\." + extension, "Content Type", null) as string) ?? "application/octet-stream";
            return "application/octet-stream";

        }
    }
}

Internal time perception (aging)

It is a well known fact that as you grow older time seems to pass by more quickly. You may have memories from when you were younger and days seemed very long and holidays seemed to last forever. As you grow older time fly by and you barely have time to get home from work and make food before bedtime and back to work.

Lesser known is it that this phenomenon has been studied and that this is not just something you imagine. It is a fact for all people. And the rate at witch your internal sense of time is changing is following a logarithmic scale.

So I plotted the numbers in an Excel pivot table to get a graph everyone can read. Looking at it we see some interesting facts.

Time perception

Some fun facts

  • A 10 year old kid perceives a day as twice as long as a 40 year old.
  • The rate at witch you “loose time” decreases as you get older. This means that time when you are young is much more valuable.
    That might be something to think of when you plan to “relax later in life”.
  • By percentage relative to your internal clock, this is the distribution of your life if we assume you will live until you are 80 years old:
    • Age 1: 10% of perceived life done.
    • Age 4: 20% of perceived life done.
    • Age 8: 30% of perceived life done.
    • Age 13: 40% of perceived life done.
    • Age 20: 50% of perceived life done.
    • Age 28: 60% of perceived life done.
    • Age 40: 70% of perceived life done.
    • Age 52: 80% of perceived life done.
    • Age 64: 90% of perceived life done.
    • Age 80: Death.

I thought knowing these numbers could put it into perspective and possibly make better use of the time we have. Any break from standard procedure like for instance a holiday would change the rate of perception, so going to do something you have never done and exposing yourself for a lot of new impressions will actually make the holiday seem longer. (For you, not your children if they are young enough.)

.Net Dictionary<> speed and memory

I love writing and optimizing stuff that puts strain on memory and CPU. Lately I’ve been wondering how well the .Net Dictionary object performs, and after looking at the source code for it, Googling and SO’ing I thought it would be good to complete the project with some real-world data that could be shared.

So I made a small test app to measure the speed and memory consumption of a System.Collections.Generic.Dictionary<Int32, Int32> in .Net.

I wanted to test with random data, so a set of random data (Int32) is generated up front. I also wanted to test lookup time with random access to simulate real-world usage where L2 cache/read-ahead might not be applicable.

Test machine specs

As of mid 2011 this computer is new and can be considered a fast desktop computer. I designed it myself for high performance computations, I bought “the best” within reasonable price parameters.

  • AMD Phenom II X6 1100T (Socket-AM3, Six Core, 3,3GHZ/3,7GHz Turbo, 9MB) set at conservative overclocking settings (bad cooling at the moment): Asus AISuite reports 3,410 GHz
  • ASUS Crosshair IV Extreme
  • 16 GB of Corsair Vengeance DDR3 1600MHz 16GB CL9
  • 2xSATA disks@550MB/s.
  • Windows 7 Ultimate 64-bit
  • Almost clean install, just Visual Studio and required dev tools.

The test

The test runs on a single thread, x64 Console Application, .Net 4.0 Client Profile. My interest is purely in the Dictionary index size and speed.

The receipt

  • Create a list of 100.000.000 random unique Int32 numbers.
  • Warm up by running once with 10k samples.
  • For every size of 10, 100, 1.000, 10.000, 100.000, 1.000.000, 10.000.000
    • Add this amount of random numbers to the dictionary. The random numbers were generated during startup so we are slowed by any overhead in random routine. It is a simple int copy operation from an int[] array.
    • If the dictionary is big enough, search it for 100.000 random keys. The random keys to search for are also generated on startup, and they may or may not exist in the dictionary.

The results

The short story

  • Dictionaries in .Net are very fast in almost any case and perform extremely well with large volumes.
    100 000 random lookups in a table consisting of 10 000 000 random entries takes 0,02 seconds, the table uses 200MB of RAM.
    (Note on that: Many of these lookups would not yield matches.)
  • Memory used by a dictionary seems to be around 20-40 times for Int32 index. The ratio goes down as the dictionary gets larger.
  • Pre-allocating memory seems to almost cut insert time in half.

Hits

These tests have a lot of misses in the lookups (random range is between Int32.MinValue and Int32.MaxValue). If I recall correctly then misses are a worst-case scenario for hashtables. I re-did the tests with hits only and got about the same results, up to 30% faster in some cases. Though I have not tested this in-depth.

Details

image

The source code

static void Main(string[] args)
{
    DictionaryTest dictionaryTest;
    dictionaryTest = new DictionaryTest(100000000);
    
    // Warmup
    dictionaryTest.RunTest(new int[] { 100000 }, false);

    // Run tests
    dictionaryTest.RunTest(new int[] { 10, 100, 1000, 10000, 100000, 1000000, 10000000 }, false);
    dictionaryTest.RunTest(new int[] { 10, 100, 1000, 10000, 100000, 1000000, 10000000 }, true);
    Console.ReadKey();
}
public class DictionaryTest
{
    private int[] TestData;
    private int[] RandomPos;
    public int DummyToPreventOptimization = 0;

    public DictionaryTest(int maxSamples)
    {
        FillRandom(maxSamples);
    }

    private void FillRandom(int size)
    {
        Console.WriteLine("Generating " + size + " random samples...");
        Random rnd = new Random();
        TestData = new int[size];
        RandomPos = new int[10000];

        // We need to add only unique data
        Dictionary<int, bool> used = new Dictionary<int, bool>(size);

        for (int i = 0; i < size; i++)
        {
            int n;
            for (; ; )
            {
                n = rnd.Next(int.MinValue, int.MaxValue);
                if (!used.ContainsKey(n))
                    break;
            }
            used.Add(n, true);
            TestData[i] = n;
        }

        Console.WriteLine("Generating " + RandomPos.Length + " random lookup points...");
        for (int i = 0; i < RandomPos.Length; i++)
        {
            RandomPos[i] = rnd.Next(0, RandomPos.Length);
        }
    }

    public void RunTest(int[] sizes, bool setDictionarySize)
    {
        // Force GC to finish up.
        Stopwatch stopwatch = new Stopwatch();

        foreach (int size in sizes)
        {

            Console.WriteLine("*** Testing " + size + " samples:");
            // Fill dictionary with random data
            long memBefore = GC.GetTotalMemory(true);
            Dictionary<int, int> dictionary;
            if (setDictionarySize)
                dictionary = new Dictionary<int, int>(size);
            else
                dictionary = new Dictionary<int, int>();
            stopwatch.Reset();
            stopwatch.Start();
            for (int i = 0; i < size; i++)
            {
                dictionary.Add(TestData[i], i);
            }
            stopwatch.Stop();
            GC.Collect();
            long memAfter = GC.GetTotalMemory(true);
            long mem = memAfter - memBefore;

            Console.WriteLine(string.Format("Preset: {0}: {2} seconds", setDictionarySize, size, Math.Round((double)stopwatch.ElapsedMilliseconds / 1000d, 5)));
            Console.WriteLine(string.Format("{0} bytes memory, Ratio: {1}", mem, (double)mem / (double)size));


            if (size >= RandomPos.Length)
            {
                // Ensure GC won't trouble us
                GC.Collect();
                int maxLen = Math.Min(RandomPos.Length, size);
                stopwatch.Reset();
                stopwatch.Start();


                for (int i = 0; i < maxLen; i++)
                {
                    DummyToPreventOptimization = dictionary[TestData[RandomPos[i]]];
                }
                stopwatch.Stop();

                double avg = Math.Round(((double)stopwatch.ElapsedMilliseconds / (double)maxLen) / 1000f, 10);
                Console.WriteLine(string.Format("{0} random access: {1} seconds (avg of {2} seconds per lookup)", maxLen, Math.Round((double)stopwatch.ElapsedMilliseconds / 1000d, 5), avg));
            }
        }
    }

}

Nordic Domain Hosting AB is pure fraud (svindel)

A few months ago I got a phone call from a Swedish company claiming to be a domain registry named “Nordisk domenhandtering” (as it turned out later “Nordic Domain Hosting AB”). They said someone was trying to register my (Norwegian) company name as .eu and .se domains, and offered me to “protect” my name by buying these domains first.

When I asked follow-up questions on the girl was unable to answer properly. When I asked what company she called from she got really unsure and didn’t repeat the same name, but a similar name to Nordic Domain Hosting AB. When I asked why someone was trying to register my name she said “often other companies try to build upon existing brands.” … Well that’s funny given that my company was a small 2-man business running for a few months with a very small group of customers.

I told her I was on the way into a meeting and that she had to send me information on email. She said she could not do that, she needed a decision quickly. So I asked her to at least send some info about her company, but she couldn’t do that either.

But she had to confirm some information about my company. “Is your company name X?”. Yes. “Is your company address X?” Yes. The questions go on until finally “Do you agree to the terms given?”… I of course didn’t say yes. Instead I said something like “Look, I REALLY have to go into this meeting now. Call me back in two hours.”

A week later I get an invoice. Registration of two domain names for around $1000 kr. I checked and the domain names were still free, so I could still register them myself for 10% of that price.

I simply ignore it. So a few weeks later I get an invoice reminder. So I send them an email pointing out that I didn’t order anything, the obviousness of their fraud and all the laws they had broken. And of course saying that invoice won’t be paid.

And guess what… Today I received a letter from European Economy Control AB stating that they will take it to court if I don’t pay.

So then there is nothing else for me to do than to spend some of my valuable time ensuring that as many people as possible know about this fraud. And to see if I can have European Economy Control AB’s bank licenses revoked and accounts closed for participating in this fraud.

If they would just had stopped when I pointed out that I knew this was a scam then I wouldn’t have bothered spending time on this.

So how can you see if my side is correct? Google it: http://www.google.no/#q=Nordisk+Domain+Hosting+svindel (English speaking may need translator)

I’m just amazed that they can keep this going for so long without getting caught by the Swedish police.

 

* Post written in English to make it easier for people in all Nordic countries to read.

Resolving application path in .dll Web, Console, WinForms and WPF applications

When sharing .dll’s between desktop applications and web applications finding the current application path can be troublesome.

For desktop applications you can usually just use the path of a loaded assembly (.exe or .dll) in the application directory. For web applications that won’t work since the .dll’s are copied to a temporary directory when the web application starts. This is caused by the ShadowCopy function of AppDomains and it enables us to overwrite web application binary files while server is running.

Here are some static methods to automatically resolve desktop apps to the .exe-directory and web apps to the web folder (you know, where the .aspx-files are). They work by replacing “~” with root directory. Method used for resolving varies by type, so a check to see if we are running as web application is used.

Usage

string appPath = ResolvePath("~");

Or use directly

string image = ResolvePath("~/Images/image1.jpg");

The code

// References required:
// System.Configuration
// System.Web

/// <summary>
/// Resolves path with ~ (root of current app) to application path.
/// </summary>
/// <param name="path">Path</param>
/// <returns>Physical path</returns>
public static string ResolvePath(string path)
{
    if (System.Web.HttpContext.Current == null)
        return ResolveAppPath(path);
    return ResolveWebPath(path);
}

/// <summary>
/// Resolves path with ~ (root of current app) to application path.
/// </summary>
/// <param name="path">Path</param>
/// <returns>Physical path</returns>
private static string ResolveAppPath(string path)
{
    return path.Replace("~", System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location) + "/").Replace("/", "\\").Replace("\\\\", "\\");
}
/// <summary>
/// Resolves path with ~ (root of current web app) to physical application path.
/// </summary>
/// <param name="path">Path</param>
/// <returns>Physical path</returns>
private static string ResolveWebPath(string path)
{
    return path.Replace("~", System.Web.HttpContext.Current.Request.PhysicalApplicationPath + "/").Replace("/", "\\").Replace("\\\\", "\\");
}

Locating config file from .dll in both Desktop and Web applications

I’m using Nini for most my config needs. Nini is a .Net library for reading/writing/merging all kinds of config files. It provides a good API with type conversion, default values, etc.

To be able to access both app.config and web.config using Nini in .dll files included in test projects, desktop applications, services, web applications, webservices, etc I’ve put together a small method that automatically resolves current application configuration file by checking if it is running under web or not and resolving the correct directory+file. It also caches the response to save a few CPU cycles.

Example use with Nini

string configFile = ConfigFile;
if (!File.Exists(configFile))
    throw new Exception(string.Format("Config file \"{0}\" does not exist. Unable to continue.", configFile));
IConfigSource source = new DotNetConfigSource(configFile);

Source code for resolving config file

// References required:
// System.Configuration
// System.Web
private static string configFile = null;
/// <summary>
/// Returns config file for application
/// </summary>
public static string ConfigFile
{
    get
    {
        // Note: Untested!
        if (configFile == null)
        {

            if (System.Web.HttpContext.Current == null)
                configFile = System.Configuration.ConfigurationManager.OpenExeConfiguration(System.Configuration.ConfigurationUserLevel.None).FilePath;
            else
                configFile = "~/Web.config".Replace("~", System.Web.HttpContext.Current.Request.PhysicalApplicationPath + "/").Replace("/", "\\").Replace("\\\\", "\\");
        }
        return configFile;
    }
}

Unsafe kernel32-mapped memory Bitmap WPF

Okay, its vacation time so I thought I’d take time from vacation (and from reading news about the terror attacks in Oslo that happened a few miles from here) to do a little post.

This time is an unsafe kernel32 memory mapped bitmap image I’m constantly improving upon. I do a lot of DSP-related stuff and the best way to debug it is to make pretty pictures. Therefore much of the time I work with the FFT data in memory. To achieve speed I allocate memory using kernel23, and I use unsafe pointers to work on it. An added bonus is that it uses InteropBitmap so the picture can be directly used in WPF, great for MVVM.

Here is the current source – under constant development, but may be useful for some.

Notice ImageUIntPtr, ImageIntPtr and ImageBytePtr. In some cases I also use Double or Float pointers, but then its not useful for displaying the picture – only useful as fast memory.

The this-property shows an example of how to access the data.

using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;
using System.Runtime.InteropServices;
using System.Windows.Interop;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using PixelFormat = System.Windows.Media.PixelFormat;

namespace Tedd.AudioFingerprint.Engine
{
    /// <summary>
    /// This object holds a byte array of the picture as well as a BitmapSource for WPF objects to bind to. Simply call .Invalidate() to update GUI.
    /// </summary>
    public unsafe class Bitmap : IDisposable
    {
        // some ideas/code borowed from CL NUI sample CLNUIImage.cs
        [DllImport("kernel32.dll", SetLastError = true)]
        private static extern IntPtr CreateFileMapping(IntPtr hFile, IntPtr lpFileMappingAttributes, uint flProtect, uint dwMaximumSizeHigh, uint dwMaximumSizeLow, string lpName);
        [DllImport("kernel32.dll", SetLastError = true)]
        private static extern IntPtr MapViewOfFile(IntPtr hFileMappingObject, uint dwDesiredAccess, uint dwFileOffsetHigh, uint dwFileOffsetLow, uint dwNumberOfBytesToMap);
        [DllImport("kernel32.dll", SetLastError = true)]
        private static extern bool UnmapViewOfFile(IntPtr hMap);
        [DllImport("kernel32.dll", SetLastError = true)]
        private static extern bool CloseHandle(IntPtr hHandle);

        //[DllImport("kernel32.dll", EntryPoint = "RtlMoveMemory")]
        //private static extern void CopyMemory(IntPtr Destination, IntPtr Source, uint Length);

        private IntPtr _section = IntPtr.Zero;
        public IntPtr ImageData { get; private set; }
        public InteropBitmap BitmapSource { get; private set; }
        public int BytesPerPixel { get; private set; }
        public uint ByteLength { get; private set; }
        public int Height { get; private set; }
        public int Width { get; private set; }
        public PixelFormat PixelFormat { get; private set; }
        public readonly object LockObject = new object();

        public unsafe uint* ImageUIntPtr { get; private set; }
        public unsafe int* ImageIntPtr { get; private set; }
        public unsafe byte* ImageBytePtr { get; private set; }

        /// <summary>
        /// Initializes an empty Bitmap
        /// </summary>
        /// <param name="width">Image width</param>
        /// <param name="height">Image height</param>
        /// <param name="pixelFormat">Image format</param>
        public Bitmap(int width, int height, PixelFormat pixelFormat)
        {
            PixelFormat = pixelFormat;
            BytesPerPixel = pixelFormat.BitsPerPixel / 8;
            uint imageSize = (uint)width * (uint)height * (uint)BytesPerPixel;
            ByteLength = imageSize;
            // create memory section and map
            _section = CreateFileMapping(new IntPtr(-1), IntPtr.Zero, 0x04, 0, imageSize, null);
            ImageData = MapViewOfFile(_section, 0xF001F, 0, 0, imageSize);
            BitmapSource = Imaging.CreateBitmapSourceFromMemorySection(_section, width, height, pixelFormat, width * BytesPerPixel, 0) as InteropBitmap;
            ImageIntPtr = (int*)ImageData;
            ImageUIntPtr = (uint*)ImageData;
            ImageBytePtr = (byte*)ImageData;

            Height = height;
            Width = width;
        }

        /// <summary>
        /// Initializes an Bitmap from a System.Drawing.Bitmap
        /// </summary>
        /// <param name="width">Image width</param>
        /// <param name="height">Image height</param>
        /// <param name="pixelFormat">Image format</param>
        public Bitmap(int width, int height, System.Drawing.Bitmap bitmap, PixelFormat pixelFormat)
        {

            PixelFormat = pixelFormat;
            BytesPerPixel = pixelFormat.BitsPerPixel / 8;
            uint imageSize = (uint)width * (uint)height * (uint)BytesPerPixel;

            ByteLength = imageSize;
            // create memory section and map
            _section = CreateFileMapping(new IntPtr(-1), IntPtr.Zero, 0x04, 0, imageSize, null);
            ImageData = MapViewOfFile(_section, 0xF001F, 0, 0, imageSize);
            BitmapSource = Imaging.CreateBitmapSourceFromMemorySection(_section, width, height, pixelFormat, width * BytesPerPixel, 0) as InteropBitmap;
            ImageIntPtr = (int*)ImageData;
            ImageBytePtr = (byte*)ImageData;
            Height = height;
            Width = width;

            CloneFromBitmap(bitmap);
        }
        /// <summary>
        /// Initializes an Bitmap from a System.Drawing.Bitmap
        /// </summary>
        /// <param name="width">Image width</param>
        /// <param name="height">Image height</param>
        /// <param name="pixelFormat">Image format</param>
        public Bitmap(System.Drawing.Bitmap bitmap, PixelFormat pixelFormat)
        {

            int width = bitmap.Width;
            int height = bitmap.Height;
            PixelFormat = pixelFormat;
            BytesPerPixel = pixelFormat.BitsPerPixel / 8;
            uint imageSize = (uint)width * (uint)height * (uint)BytesPerPixel;

            ByteLength = imageSize;
            // create memory section and map
            _section = CreateFileMapping(new IntPtr(-1), IntPtr.Zero, 0x04, 0, imageSize, null);
            ImageData = MapViewOfFile(_section, 0xF001F, 0, 0, imageSize);
            BitmapSource = Imaging.CreateBitmapSourceFromMemorySection(_section, width, height, pixelFormat, width * BytesPerPixel, 0) as InteropBitmap;
            ImageIntPtr = (int*)ImageData;
            ImageBytePtr = (byte*)ImageData;
            Height = height;
            Width = width;

            CloneFromBitmap(bitmap);
        }
        // Discarded - not fast enough so should not be used.
        //[System.Runtime.CompilerServices.IndexerName("TheItem")]
        //public int this[int index]   // Indexer declaration
        //{
        //}
        //[System.Runtime.CompilerServices.IndexerName("TheItem")]
        //public int this[int x, int y]   // Indexer declaration
        //{
        //    get
        //    {
        //        int p = (int) ((x*this.BitmapSource.Width*this.BytesPerPixel) + (y*this.BytesPerPixel));
        //        return ImageData[p];
        //    }
        //}


        /// <summary>
        /// Invalidates the bitmap causing a redraw
        /// </summary>
        public void Invalidate()
        {
            //lock (LockObject)
            //{
            BitmapSource.Invalidate();
            //}
        }
        ~Bitmap()
        {
            Dispose();
        }
        private bool _disposed;

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }



        protected void Dispose(bool disposing)
        {
            lock (LockObject)
            {
                if (disposing)
                {
                    // free managed resources
                }
                // free native resources if there are any.
                if (ImageData != IntPtr.Zero)
                {
                    UnmapViewOfFile(ImageData);
                    ImageData = IntPtr.Zero;
                }
                if (_section != IntPtr.Zero)
                {
                    CloseHandle(_section);
                    _section = IntPtr.Zero;
                }
            }
        }



        public unsafe void CopyTo(Bitmap destinationBitmap, int top, int left, int height, int width)
        {
            lock (LockObject)
            {
                // Find smallest array
                //int size = (int)Math.Min(ByteLength, destinationBitmap.ByteLength) / destinationBitmap.BytesPerPixel;
                // Copy memory
                //CopyMemory(ImageData, destinationBitmap.ImageData, size);
                //Marshal.Copy(src, destinationBitmap.ImageData, 0, size);

                // First copy int64 as far as we can (faster than copying single bytes)
                Int64* src64 = (Int64*)ImageData;
                Int64* dst64 = (Int64*)destinationBitmap.ImageData;
                byte* src8 = (byte*)ImageData;
                byte* dst8 = (byte*)destinationBitmap.ImageData;

                int maxWidth = Math.Min(Math.Min(Width - left, destinationBitmap.Width - left), width + top);
                int maxHeight = Math.Min(Math.Min(Height - top, destinationBitmap.Height - top), height + top);

                for (int x=top; x<maxHeight;x++)
                {
                    for (int y = left; y < maxWidth; y++)
                    {
                        int srcp = (x*Width) + y;
                        int dstp = (x * destinationBitmap.Width) + y;
                        dst8[dstp] = src8[srcp];
                    }
                }

            }
            
        }
        public unsafe void CopyTo(Bitmap destinationBitmap)
        {
            lock (LockObject)
            {
                // Find smallest array
                //int size = (int)Math.Min(ByteLength, destinationBitmap.ByteLength) / destinationBitmap.BytesPerPixel;
                // Copy memory
                //CopyMemory(ImageData, destinationBitmap.ImageData, size);
                //Marshal.Copy(src, destinationBitmap.ImageData, 0, size);

                // First copy int64 as far as we can (faster than copying single bytes)
                int copied = 0;
                Int64* src64 = (Int64*)ImageData;
                Int64* dst64 = (Int64*)destinationBitmap.ImageData;

                int srcHeight = Height * Width;
                int dstHeight = destinationBitmap.Height * destinationBitmap.Width;
                int maxLen = Math.Min(srcHeight, dstHeight);

                int copyLength = maxLen - 8;
                int i = 0;
                while (copied < copyLength)
                {
                    dst64[i] = src64[i];
                    i++;
                    copied += 8;
                }

                // Then copy single bytes until end of data
                byte* src8 = (byte*)ImageData;
                byte* dst8 = (byte*)destinationBitmap.ImageData;

                i *= 8;
                while (copied < ByteLength)
                {
                    dst8[i] = src8[i];
                    i++;
                    copied++;
                }

            }
        }

        public unsafe void CloneFromBitmap(System.Drawing.Bitmap sourceBitmap)
        {
            throw new NotImplementedException("Untested!");
            lock (LockObject)
            {

                // First copy int64 as far as we can (faster than copying single bytes)
                int copied = 0;
                BitmapData bData = sourceBitmap.LockBits(new Rectangle(new Point(), sourceBitmap.Size), ImageLockMode.ReadOnly, sourceBitmap.PixelFormat);
                Int64* src64 = (Int64*)bData.Scan0.ToPointer();
                Int64* dst64 = (Int64*)ImageData;

                int srcHeight = sourceBitmap.Height * sourceBitmap.Width;
                int dstHeight = Height * Width;
                int maxLen = Math.Min(srcHeight, dstHeight);

                int copyLength = maxLen - 8;
                int i = 0;
                while (copied < copyLength)
                {
                    dst64[i] = src64[i];
                    i++;
                    copied += 8;
                }

                // Then copy single bytes until end of data
                byte* src8 = (byte*)bData.Scan0.ToPointer();
                byte* dst8 = (byte*)ImageData;

                i *= 8;
                while (copied < ByteLength)
                {
                    dst8[i] = src8[i];
                    i++;
                    copied++;
                }

            }
        }



        public Bitmap Clone()
        {
            lock (LockObject)
            {
                // Create new bitmap
                Bitmap bitmap = new Bitmap(Width, Height, PixelFormat);
                // Copy data into new bitmap
                CopyTo(bitmap);

                return bitmap;
            }
        }

        public static Bitmap CreateFromFile(string file)
        {
            using (Image image = Image.FromFile(file))
            {
                using (System.Drawing.Bitmap bmp = new System.Drawing.Bitmap(image))
                {

                    System.Drawing.Imaging.PixelFormat format = bmp.PixelFormat;
                    Bitmap ret = new Bitmap(image.Width, image.Height, PixelFormats.Bgr32);

                    BitmapData data = bmp.LockBits(new Rectangle(0, 0, image.Width, image.Height),
                                                   ImageLockMode.ReadOnly,
                                                   format);
                    int stride = data.Stride;
                    int offset = stride - image.Width * ret.BytesPerPixel;
                    unsafe
                    {
                        byte* src = (byte*)data.Scan0.ToPointer();
                        byte* dst = (byte*)ret.ImageData;

                        int mp = image.Height * image.Width * ret.BytesPerPixel;
                        for (int p = 0; p < mp; p++)
                        {
                            dst[p] = src[p];
                        }
                    }
                    return ret;

                }
            }
        }

        public static Bitmap CreateFromStream(Stream file)
        {
            using (Image image = Image.FromStream(file))
            {
                using (System.Drawing.Bitmap bmp = new System.Drawing.Bitmap(image))
                {

                    System.Drawing.Imaging.PixelFormat format = bmp.PixelFormat;
                    Bitmap ret = new Bitmap(image.Width, image.Height, PixelFormats.Bgr32);
                    BitmapData data = bmp.LockBits(new Rectangle(0, 0, image.Width, image.Height),
                                                   ImageLockMode.ReadOnly,
                                                   format);
                    int stride = data.Stride;
                    int offset = stride - image.Width * ret.BytesPerPixel;
                    unsafe
                    {
                        byte* src = (byte*)data.Scan0.ToPointer();
                        byte* dst = (byte*)ret.ImageData;

                        int mp = image.Height * image.Width * ret.BytesPerPixel;
                        for (int p = 0; p < mp; p++)
                        {
                            dst[p] = src[p];
                        }
                    }
                    return ret;
                }
            }
        }

        private static byte[] imageToByteArray(System.Drawing.Image imageIn)
        {
            using (MemoryStream ms = new MemoryStream())
            {
                imageIn.Save(ms, System.Drawing.Imaging.ImageFormat.Gif);
                return ms.ToArray();
            }
        }

        public void SavePNG(string filename)
        {
            lock (LockObject)
            {
                using (var fileStream = new FileStream(filename, FileMode.Create))
                {
                    SavePNG(fileStream);
                }
            }
        }

        public void SavePNG(Stream stream)
        {
            lock (LockObject)
            {
                BitmapEncoder encoder = new PngBitmapEncoder();
                encoder.Frames.Add(BitmapFrame.Create(BitmapSource));
                encoder.Save(stream);
            }
        }

        public System.Drawing.Bitmap CloneToBitmap()
        {
            lock (LockObject)
            {
                return System.Drawing.Image.FromHbitmap(ImageData);
            }
        }

        public void Blur(int xBlur, int yBlur)
        {
            Bitmap srcImage = Clone();
            int* srcIntPtr = srcImage.ImageIntPtr;
            int* dstIntPtr = ImageIntPtr;

            int xfrom = (xBlur / 2) * -1;
            int xto = xfrom + xBlur;
            int yfrom = (yBlur / 2) * -1;
            int yto = yfrom + yBlur;

            for (int x = 0; x < Height; x++)
            {
                for (int y = 0; y < Width; y++)
                {
                    // Look around
                    long avg = 0;
                    int count = 0;
                    for (int xa = xfrom; xa < xto; xa++)
                    {
                        for (int ya = yfrom; ya < yto; ya++)
                        {
                            if (x + xa >= 0 && x + xa < Height
                                && y + ya >= 0 && y + ya < Width)
                            {
                                int p = ((x + xa) * Width) + (y + ya);
                                avg += srcIntPtr[p];
                                count++;
                            }
                        }
                    }
                    int pd = (x  * Width) + y ;
                    dstIntPtr[pd] = (int)(avg / count);
                }
            }

            srcImage.Dispose();
        }

        public void SwapColor32(int oldColor, int newColor)
        {
            
            for (int x = 0; x < Height; x++)
            {
                for (int y = 0; y < Width; y++)
                {
                    int p = (x * Width) + y;
                    if (ImageIntPtr[p] == oldColor)
                        ImageIntPtr[p] = newColor;
                }
            }
        }
    }
}

Hann/Hanning & Hamming window functions in C#

Not so much output from me lately. The reason is that I’m occupied with an exciting and extremely complex hobby project. My geeksuperpower (or one of them) is that I’m able to keep extremely focused for long periods of time (months). Even when its boring, tough and seems impossible to solve the task I work my self through it. The prerequisite being that its complex and interesting enough. A quality many coders possess, for good and for bad.

Anyway, I’ve been thinking I should share some of my findings. So here goes.

Hann (Hanning) and Hamming window functions which you apply to the wave stream frame before passing it to the FFT. I looked everywhere and couldn’t find it for C#, but with some help of Wikipedia and the good people at StackOverflow I managed to put it together.

// http://en.wikipedia.org/wiki/Window_function#Window_examples public Complex[] Hamming(Complex[] iwv)
{
    int N = iwv.Length;

    // iwv[i].Re = real number (raw wave data) // iwv[i].Im = imaginary number (0 since it hasn't gone through FFT yet) for (int n = 1; n < N; n++)
        iwv[n].Re *= 0.54f - 0.46f * (float)Math.Cos((2 * Math.PI * n) / (N - 1));

    return iwv;
}

public Complex[] Hann(Complex[] iwv)
{
    int N = iwv.Length;

EDIT: See blog comments below on this method.
    for (int n = 1; n < N; n++)
        iwv[n].Re = 0.5f * (float)Math.Cos((2 * Math.PI * n) / (N - 1));

    return iwv;
}

Note: Parts of Hamming code borrowed from this file which is part of a C# project that can do autotuning.

ArrayBuilderQueue

… is that the right name?

It basically works like a combination of the StringBuilder and Queue objects. You enqueue arrays of T, and you dequeue custom sized chunks of the array. What makes it nice is that I’ve optimized memory usage so it never creates any temp arrays internally except the one returned to you when you eat (of course).

Added some comments in case you need to change anything.

/// <summary>
    /// Works like a Queue object except it enqueues arrays and lets you dequeue custom sized chunks of the arrays enqueued
    /// </summary>
    /// <typeparam name="T">Array type</typeparam>
    public class ArrayBuilderQueue<T>
    {
        private List<Array> buffer = new List<Array>();
        private int currentArrayPos = 0;
        public int Length { get; private set; }

        /// <summary>
        /// Adds an array to the queue
        /// </summary>
        /// <param name="items">Array of items to add</param>
        public void Enqueue(T[] items)
        {
            lock (buffer)
            {
                if (items == null)
                    return;
                buffer.Add(items);
                Length += items.Length;
            }
        }

        public void Clear()
        {
            lock (buffer)
            {
                buffer.Clear();
                Length = 0;
                currentArrayPos = 0;
            }
        }

        /// <summary>
        /// Dequeues items from buffer.
        /// </summary>
        /// <param name="count">Number of items</param>
        /// <returns>Array of items</returns>
        public T[] Dequeue(int count)
        {
            lock (buffer)
            {
                count = Math.Min(count, Length);
                T[] ret = new T[count];
                int eaten = 0;
                int retPos = 0;
                while (eaten < count)
                {
                    // Check how much remains of first array
                    int bl = 0;
                    if (buffer.Count > 0)
                        bl = buffer[0].Length;
                    int remaining = bl - currentArrayPos;
                    // Nothing?
                    while (remaining == 0 && buffer.Count > 0)
                    {
                        // Remove it and recalculate
                        buffer.RemoveAt(0);
                        currentArrayPos = 0;
                        if (buffer.Count > 0)
                        {
                            // Still got some, try next
                            if (buffer[0] != null)
                                remaining = buffer[0].Length - currentArrayPos;
                        }
                    }
                    // Copy data to return var
                    int eatSize = Math.Min(remaining, count - eaten);
                    if (eatSize > 0)
                        Array.Copy(buffer[0], currentArrayPos, ret, retPos, eatSize);
                    // Update internal tracking
                    retPos += eatSize; // Where we are in buffer we are copying to
                    currentArrayPos += eatSize; // Where we are in current array after eating
                    eaten += eatSize; // How much we have eaten (main loop)
                    Length -= eatSize; // Length was just decreased
                }

                return ret;
            }
        }
    }

And the unit test if you want (doesn’t cover all scenarios):

[TestMethod]
public void BufferBuilder()
{
    Common.Utils.ArrayBuilderQueue<byte> bufferBuilder = new ArrayBuilderQueue<byte>();
    bufferBuilder.Enqueue(System.Text.Encoding.ASCII.GetBytes("This is a test\r\n"));
    bufferBuilder.Enqueue(System.Text.Encoding.ASCII.GetBytes("Of the "));
    bufferBuilder.Enqueue(System.Text.Encoding.ASCII.GetBytes("BufferBuilder class.\r\n"));
    bufferBuilder.Enqueue(System.Text.Encoding.ASCII.GetBytes("Have fun!\r\n"));
    string str = null;
    str = System.Text.Encoding.ASCII.GetString(bufferBuilder.Dequeue(8)); Assert.IsTrue(str == "This is ");
    str = System.Text.Encoding.ASCII.GetString(bufferBuilder.Dequeue(8)); Assert.IsTrue(str == "a test\r\n");
    str = System.Text.Encoding.ASCII.GetString(bufferBuilder.Dequeue(7)); Assert.IsTrue(str == "Of the ");
    str = System.Text.Encoding.ASCII.GetString(bufferBuilder.Dequeue(20)); Assert.IsTrue(str == "BufferBuilder class.");
    str = System.Text.Encoding.ASCII.GetString(bufferBuilder.Dequeue(13)); Assert.IsTrue(str == "\r\nHave fun!\r\n");
    bufferBuilder.Clear();
    bufferBuilder.Enqueue(System.Text.Encoding.ASCII.GetBytes("1234567890"));
    str = System.Text.Encoding.ASCII.GetString(bufferBuilder.Dequeue(10)); Assert.IsTrue(str == "1234567890");
}

IE favorites, NTFS symbolic links and Dropbox

NTFS symlinks

Well-known to *nix-users, unknown to most Windows-users.

A file or folder can exist in more than one place, meaning the file is stored once to disk, but can have two different names on different locations. If you change it one place the change happens to the other place. I won’t bother going into details, but in short NTFS supports a few different ways to do this.

New in Windows Vista and Windows 7 is “NTFS Symlinks” which is a powerful way of linking files and directories between filesystems and over network in addition to the usual duality.

Dropbox

Dropbox is a great tool for synchronizing small amounts of files between computers (<2GB free).

Symbolic Link Creator

This application gives you a simple to use GUI for creating symlinks.
(Quick hint: Run as Administrator)

image

Synergy effect

Dropbox does a great job in synchronizing the dropbox folder, but it requires you to actively put files there. This is fine for all the stuff you can choose, but there are a few scenarious where you can’t do that easily.

In the picture above I have created a “_MachineSync” folder that contains a Favorites folder. By deleting my Favorites-folder in my user profile and linking in the Dropbox Favorite-folder all my IE favorites is synchronized to Dropbox. I’ve done the same on both my work PC and home PC, so now favorite items are instantly synced between home and work PC (as well as being accessible via web).

The same could be done with my documents, Desktop or anything else I’d like to keep in Dropbox. Just note that some folders require you to log on as a different user to be able to delete them before symlinking them in from Dropbox.