CE's Blog

Custom Memory Destination Manager for libjpeg

Christian Etter — Sat, 06 Apr 2013 23:55:54 +0000

Similar to creating an in memory source manager for libjpeg decompression, we can also create a destination manager which will store the compressed data in memory.
FOr this purpose, the destination manager contains a byte vector:

typedef struct _jpeg_destination_mem_mgr
{
    jpeg_destination_mgr mgr;
    std::vector<unsigned char> data;
} jpeg_destination_mem_mgr;

Initialization takes place in the callback mem_init_destination():

static void mem_init_destination( j_compress_ptr cinfo )
{
    jpeg_destination_mem_mgr* dst = (jpeg_destination_mem_mgr*)cinfo->dest;
    dst->data.resize( JPEG_MEM_DST_MGR_BUFFER_SIZE );
    cinfo->dest->next_output_byte = dst->data.data();
    cinfo->dest->free_in_buffer = dst->data.size();
}

When the compression has finished, we need to resize the buffer to the actual size:

static void mem_term_destination( j_compress_ptr cinfo )
{
    jpeg_destination_mem_mgr* dst = (jpeg_destination_mem_mgr*)cinfo->dest;
    dst->data.resize( dst->data.size() - cinfo->dest->free_in_buffer );
}

When the buffer size is not large enough, the library requests more data in the following callback:

static boolean mem_empty_output_buffer( j_compress_ptr cinfo )
{
    jpeg_destination_mem_mgr* dst = (jpeg_destination_mem_mgr*)cinfo->dest;
    size_t oldsize = dst->data.size();
    dst->data.resize( oldsize + JPEG_MEM_DST_MGR_BUFFER_SIZE );
    cinfo->dest->next_output_byte = dst->data.data() + oldsize;
    cinfo->dest->free_in_buffer = JPEG_MEM_DST_MGR_BUFFER_SIZE;
    return true;
}

All callbacks are configured in jpeg_mem_dest():

static void jpeg_mem_dest( j_compress_ptr cinfo, jpeg_destination_mem_mgr * dst )
{
    cinfo->dest = (jpeg_destination_mgr*)dst;
    cinfo->dest->init_destination = mem_init_destination;
    cinfo->dest->term_destination = mem_term_destination;
    cinfo->dest->empty_output_buffer = mem_empty_output_buffer;
}

Usage:

jpeg_destination_mem_mgr dst_mem;
jpeg_compress_struct_wrapper cinfo;
j_compress_ptr pcinfo = cinfo;
jpeg_mem_dest( cinfo, &dst_mem);

After encoding has finished, data can be retrieved from the data member of the destination manager.

unsigned char * pjpeg_data = dst_mem.data.data();
size_t data_size = dst_mem.data.size();

Here is a link to the entire code: jpeg_mem_dst.h

Custom Memory Source Manager for libjpeg

Christian Etter — Sat, 06 Apr 2013 16:35:49 +0000

libjpeg-turbo is an optimized plug-in replacement for libjpeg, which delivers 2-4 times the performance of the original library. However currently the current libjpeg-turbo release version have a memory source manager included when running in libjpeg v6b compatibility mode.

The following implementation fills the gap:

The callback mem_fill_input_buffer() is invoked when the library expects more data than supplied in the buffer. Since we already hold all data in the memory buffer, this can only be due to the image being prematurely terminated. There are two different ways of handling such a circumstance, either allow the library to finish processing and display the partial data, or abort processing by raising an error. The first option is activated by defining PROCESS_TRUNCATED_IMAGES.

static boolean mem_fill_input_buffer( j_decompress_ptr cinfo )
{
#ifdef PROCESS_TRUNCATED_IMAGES
    jpeg_source_mgr* src = cinfo->src;
 
    static const JOCTET EOI_BUFFER[ 2 ] = { (JOCTET)0xFF, (JOCTET)JPEG_EOI };
    src->next_input_byte = EOI_BUFFER;
    src->bytes_in_buffer = sizeof( EOI_BUFFER );
#else
    ERREXIT( cinfo, JERR_INPUT_EOF );
#endif
    return TRUE;
}

mem_skip_input_data() is called when the library wants to skip over a certain part of the data. If the data to be skipped is less than the remaining bytes in the buffer, we simple anjust the buffer pointer. If there is not enough input data, we either raise an exception or set the input data length to zero, which will result in a call to mem_fill_input_buffer(), where an EOI marker is returned.

static void mem_skip_input_data( j_decompress_ptr cinfo, long num_bytes )
{
    jpeg_source_mgr* src = (jpeg_source_mgr*)cinfo->src;
 
    if ( 1 > num_bytes )
        return;
 
    if ( num_bytes < src->bytes_in_buffer )
    {
        src->next_input_byte += (size_t)num_bytes;
        src->bytes_in_buffer -= (size_t)num_bytes;
    }
    else
    {
#ifdef PROCESS_TRUNCATED_IMAGES
        src->bytes_in_buffer = 0;
#else
        ERREXIT( cinfo, JERR_INPUT_EOF );
#endif
    }
}

In our case, there is no need to perform initialization or termination, since the buffer which is used is managed by the caller.

static void mem_init_source( j_decompress_ptr cinfo ) 
{
}
 
static void mem_term_source( j_decompress_ptr cinfo ) 
{
}

The following function performs initialization of the memory buffer. It is important to note that the bytes supplied are not being copied, and therefore the buffer must not be freed before the image processing has finished.

static void jpeg_mem_src( j_decompress_ptr cinfo, jpeg_source_mgr* src, void* buffer, long nbytes )
{
    src->init_source = mem_init_source;
    src->fill_input_buffer = mem_fill_input_buffer;
    src->skip_input_data = mem_skip_input_data;
    src->resync_to_restart = jpeg_resync_to_restart;
    src->term_source = mem_term_source;
    src->bytes_in_buffer = nbytes;
    src->next_input_byte = (JOCTET*)buffer;
    cinfo->src = src;
}

With the above manager definition, we can easily initialize and work with the memory source manager.

jpeg_decompress_struct cinfo;
jpeg_source_mgr src_mem;
jpeg_create_decompress( &cinfo );
jpeg_mem_src( &cinfo, &src_mem, (PVOID)content, (long)dwSize );
jpeg_read_header( &cinfo, TRUE );

Here is the complete header file: jpeg_mem_src.h.

RAII Wrapper for libjpeg Compression/Decompression

Christian Etter — Sun, 06 Jan 2013 22:39:14 +0000

To facilitate resource cleanup within libjpeg encoding/decoding functions, the following classes implement RAII wrappers for the jpeg_decompress_struct and jpeg_compress_struct structures.

RAII is a C++ concept which helps to ensure that all resources are correctly deinitialized/released even in case of error/exceptions.

Usage example:

jpeg_decompress_struct_wrapper cinfo;
jpeg_decompress_struct* pcinfo = cinfo;
jpeg_win32_src_mgr src_data;
 
jpeg_win32_src( cinfo, &src_data, szFilename );
jpeg_read_header( cinfo, TRUE ); 
pcinfo->dct_method = JDCT_IFAST;
/* ... */

No calls to jpeg_destroy_decompress() are necessary, stack unwinding will ensure that the jpeg_decompress_struct will be automatically destroyed, even if the encoding/decoding routine has several exit/return paths.
Since the wrapper class implements an operator for jpeg_decompress_struct*, the instance can be passed as a parameter instead of “&cinfo”.

This is the wrapper for jpeg_decompress_struct:

class jpeg_decompress_struct_wrapper
{
public:
    jpeg_decompress_struct_wrapper()
    {
        jpeg_create_decompress( &this->cinfo );
    }
 
    ~jpeg_decompress_struct_wrapper()
    {
        jpeg_destroy_decompress( &this->cinfo );
    }
 
    operator jpeg_decompress_struct*()
    {
        return &this->cinfo;
    }
private:
    jpeg_decompress_struct cinfo;
};

Wrapper for jpeg_compress_struct:

class jpeg_compress_struct_wrapper
{
public:
    jpeg_compress_struct_wrapper()
    {
        jpeg_create_compress( &this->cinfo );
    }
 
    ~jpeg_compress_struct_wrapper()
    {
        jpeg_destroy_compress( &this->cinfo );
    }
 
    operator jpeg_compress_struct*()
    {
        return &this->cinfo;
    }
private:
    jpeg_compress_struct cinfo;
};

Win32 libjpeg Source Manager

Christian Etter — Thu, 06 Dec 2012 14:37:34 +0000

I was recently running into problems using libjpeg-turbo together with the stdio FILE* API. So instead of using _wfopen_s, _tfopen_s etc. I decided to write my own source manager which does not rely on stdlib functionality and uses the Win32 CreateFile API instead.

First, we need to extend a jpeg_src_mgr structure to hold additional data.

#define JPEG_WIN32_SRC_MGR_BUFFER_SIZE ( 1 << 10 )
 
typedef struct _jpeg_win32_src_mgr
{
    jpeg_source_mgr mgr;
    unsigned int bytes_read;
    HANDLE file;
    unsigned char buffer[ JPEG_WIN32_SRC_MGR_BUFFER_SIZE ];
} jpeg_win32_src_mgr;

Then we define several callback functions which are used by the libjpeg API to retrieve data from our data source.

The win32_init_source() callback is a no-op since we perform initialization before handing over the source manager to the library.

static void win32_init_source( j_decompress_ptr cinfo ) 
{
}

Function win32_fill_input_buffer() is called when the library has consumed the entire data in the buffer. The return value should always be TRUE if no I/O suspension is needed. There is one peculiarity when handling truncated image files. To prevent the library to request further data on EOF, the recommendation is to insert an EOI marker to signal end of file.

static boolean win32_fill_input_buffer( j_decompress_ptr cinfo )
{
    jpeg_win32_src_mgr* src = (jpeg_win32_src_mgr*)cinfo->src;
    DWORD bytes_read = 0;
    if ( TRUE != ReadFile( src->file, src->buffer, JPEG_WIN32_SRC_MGR_BUFFER_SIZE, &bytes_read, NULL ) )
        ERREXIT( cinfo, JERR_FILE_READ );
    src->mgr.next_input_byte = src->buffer;
    src->mgr.bytes_in_buffer = (size_t)bytes_read;
    if ( 0 == src->mgr.bytes_in_buffer )
    {
        /* The image file is truncated. We insert EOI marker to tell the library to stop processing. */
        src->buffer[ 0 ] = (JOCTET)0xFF;
        src->buffer[ 1 ] = (JOCTET)JPEG_EOI;
        src->mgr.bytes_in_buffer = 2;
    }
    return TRUE;
}

The win32_skip_input_data() callback is called when the library wants to skip over some unnecessary data such as APP markers.
It is valid for the function to leave 0 bytes in the input buffer, in which case win32_fill_input_buffer() will be called.

static void win32_skip_input_data( j_decompress_ptr cinfo, long num_bytes )
{
    jpeg_win32_src_mgr* src = (jpeg_win32_src_mgr*)cinfo->src;
 
    if ( 1 > num_bytes )
        return;
    if ( num_bytes <= src->mgr.bytes_in_buffer )
    {
        src->mgr.next_input_byte += (size_t)num_bytes;
        src->mgr.bytes_in_buffer -= (size_t)num_bytes;
    }
    else
    {
        LARGE_INTEGER offset;
        offset.QuadPart = num_bytes - (long)src->mgr.bytes_in_buffer;
        src->mgr.bytes_in_buffer = 0;
        DWORD new_pos = SetFilePointer( src->file, offset.LowPart, &offset.HighPart, FILE_CURRENT );
        if ( INVALID_SET_FILE_POINTER == new_pos )
            ERREXIT( cinfo, JERR_FILE_READ );
    }
}

The win32_term_source() callback ensures there are no open handles left when the library is done processing.

static void win32_term_source( j_decompress_ptr cinfo ) 
{
    jpeg_win32_src_mgr* src = (jpeg_win32_src_mgr*)cinfo->src;
    if ( INVALID_HANDLE_VALUE != src->file  && NULL != src->file )
    {
        CloseHandle( src->file );
        src->file = NULL;
    }
}

Our new source manager is initialized via a call to jpeg_win32_src(), which will open the file and provide references to the callbacks defined above.

static void jpeg_win32_src( j_decompress_ptr cinfo, jpeg_win32_src_mgr * src, PCTSTR filename )
{
    src->file = CreateFile( filename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL );
    if ( INVALID_HANDLE_VALUE == src->file )
        ERREXIT( cinfo, JERR_FILE_READ );
    src->bytes_read = 0;
    src->mgr.init_source = win32_init_source;
    src->mgr.fill_input_buffer = win32_fill_input_buffer;
    src->mgr.skip_input_data = win32_skip_input_data;
    src->mgr.resync_to_restart = jpeg_resync_to_restart;
    src->mgr.term_source = win32_term_source;
    src->mgr.bytes_in_buffer = 0;
    src->mgr.next_input_byte = (JOCTET*)src->buffer;
    cinfo->src = (jpeg_source_mgr*)src;
}

Usage of the new Win32 source manager is simple:

jpeg_decompress_struct cinfo;
jpeg_win32_src_mgr src_data;
 
jpeg_create_decompress( &cinfo );
jpeg_win32_src( &cinfo, &src_data, szFilename );
jpeg_read_header( &cinfo, TRUE ); 
/* ... */

It would be quite easy to modify the source manager code listed above to support reading from other Win32 sources such as named pipes, console buffer or I/O devices.

The entire file can be downloaded here: jpeg_win32_src.h

Visual Studio 2012 Entity Data Model Wizard Annoyance

Christian Etter — Tue, 13 Nov 2012 19:02:14 +0000

Just stumbled upon an annoyance in VS 2012 Entity data model wizard. I had implemented a class and then decided to add an Entity Model to the project. Since the model was supposed to implement storage functionality for the previously written class, it was named the same as the .cs file of that class (except the .cs extension).
It turned out that this setting will silently overrwrite the .cs file with an entity framework generated class.

.NET RSACryptoServiceProvider PEM + DER Support

Christian Etter — Wed, 03 Oct 2012 22:14:10 +0000

In .NET, RSACryptoServiceProvider greatly simplifies common tasks associated with public/private keys, such as signing of data and verifying a signature.
Unfortunately, there is only a single format (proprietary XML) available for importing and exporting public/private key data.
The two widely spread formats for key exchange, PEM and DER are not supported, which limits the usability of the class when working with different kinds of public key APIs.

There are a few workarounds though:

Write your own .NET code to support the desired formats, which might not be that simple, as the following blog post suggests.
Use a third party component, such as the PEMReader class of the Bouncy Castle Library (Java/C# port).
Leverage Microsoft Crypto API (CAPI), which provides support for opening and converting a variety of different public/private key file formats. Here I would like to demonstrate how this can be achieved by means of simple extension methods to the RSACryptoServiceProvider class.

Here is a sample RSA 1024 bit private key in PEM format:

This is the corresponding public key in textual PEM representation:

We cannot load these keys into the RSACryptoServiceProvider directly. However the class supports an import method which is compatible with CryptoAPI: ImportCspBlob().
With CAPI in return, we can do all the heavy lifting including format conversion which the .NET Framework does not support.
For public keys, the conversion process for PEM (string) key data requires the following steps:

Converting a public key in PEM string format into DER representation.

CryptStringToBinaryA( sPEM, (UInt32)sPEM.Length, CRYPT_STRING_FLAGS.CRYPT_STRING_BASE64HEADER, IntPtr.Zero, ref dwBinarySize, out dwSkip, out dwFlags ) )

Retrieve the key through a CERT_PUBLIC_KEY_INFO struct.

CryptDecodeObjectEx( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING, new IntPtr( (int)CRYPT_OUTPUT_TYPES.X509_PUBLIC_KEY_INFO ), DERData, (UInt32)DERData.Length, CRYPT_DECODE_FLAGS.CRYPT_DECODE_ALLOC_FLAG, IntPtr.Zero, ref pCertPublicKeyInfo, out dwCertPublicKeyInfoSize )

Convert the returned RSA key into a Diffie-Hellman Version 3 Public Key BLOBs or DSS Version 3 Public Key BLOBs struct:

CryptDecodeObjectEx( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING, new IntPtr( (int)CRYPT_OUTPUT_TYPES.RSA_CSP_PUBLICKEYBLOB ), RSAData, (UInt32)RSAData.Length, CRYPT_DECODE_FLAGS.CRYPT_DECODE_ALLOC_FLAG, IntPtr.Zero, ref pCertPublicKeyBlob, out dwCertPublicKeyBlobSize )

Call RSACryptoServiceProvider.ImportCspBlob() for the resulting binary data.

For private keys, the following steps are required:

Converting a private key in PEM string format into DER representation.

CryptStringToBinaryA( sPEM, (UInt32)sPEM.Length, CRYPT_STRING_FLAGS.CRYPT_STRING_BASE64HEADER, IntPtr.Zero, ref dwBinarySize, out dwSkip, out dwFlags ) )

For private keys, retrieve a pointer to an RSA private key BLOB directly:

CryptDecodeObjectEx( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING, new IntPtr( (int)CRYPT_OUTPUT_TYPES.PKCS_RSA_PRIVATE_KEY ), DERData, (UInt32)DERData.Length, CRYPT_DECODE_FLAGS.CRYPT_DECODE_ALLOC_FLAG, IntPtr.Zero, ref pRSAPrivateKeyBlob, out pRSAPrivateKeyBlobSize )

Call RSACryptoServiceProvider.ImportCspBlob() for the resulting binary data.

If you wrap the above logic into extension methods for RSACryptoServiceProvider, importing keys in PEM format and signing data can be done with a few lines of code:

// -----BEGIN RSA PRIVATE KEY-----...-----END RSA PRIVATE KEY-----
string sPrivateKeyPEM = File.ReadAllText( "PrivateKey.pem" );
 
using ( RSACryptoServiceProvider rsa = new RSACryptoServiceProvider() )
{
    rsa.PersistKeyInCsp = false;
    rsa.LoadPrivateKeyPEM( sPrivateKeyPEM );
    using ( SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider() )
        signature = rsa.SignData( dataToSign, sha1 );
}

Verifying the same signature also becomes easy:

// -----BEGIN PUBLIC KEY-----...-----END PUBLIC KEY-----
string sPublicKeyPEM = File.ReadAllText( "PublicKey.pem" );
 
using ( RSACryptoServiceProvider rsa = new RSACryptoServiceProvider() )
{
    rsa.PersistKeyInCsp = false;
    rsa.LoadPublicKeyPEM( sPublicKeyPEM );
    using ( SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider() )
        bVerifyResultOriginal = rsa.VerifyData( dataToSign, sha1, signature );
}

Here is a cut-and-paste solution based on an extension class to RSACryptoServiceProvider:

/*********************************************************************************
 * Copyright (c) 2013, Christian Etter info at christian-etter dot de
 * 
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 * 
 * 1. Redistributions of source code must retain the above copyright notice, this
 *    list of conditions and the following disclaimer. 
 * 2. Redistributions in binary form must reproduce the above copyright notice,
 *    this list of conditions and the following disclaimer in the documentation
 *    and/or other materials provided with the distribution. 
 * 
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
 * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
 * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *********************************************************************************/
 
using System;
using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Security.Cryptography;
 
/// Extension method for initializing a RSACryptoServiceProvider from PEM data string.
public static class RSACryptoServiceProviderExtension
{
    #region Methods
 
    /// Extension method which initializes an RSACryptoServiceProvider from a DER public key blob.
    public static void LoadPublicKeyDER( this RSACryptoServiceProvider provider, byte[] DERData )
    {
        byte[] RSAData = RSACryptoServiceProviderExtension.GetRSAFromDER( DERData );
        byte[] publicKeyBlob = RSACryptoServiceProviderExtension.GetPublicKeyBlobFromRSA( RSAData );
        provider.ImportCspBlob( publicKeyBlob );
    }
 
    /// Extension method which initializes an RSACryptoServiceProvider from a DER private key blob.
    public static void LoadPrivateKeyDER( this RSACryptoServiceProvider provider, byte[] DERData )
    {
        byte[] privateKeyBlob = RSACryptoServiceProviderExtension.GetPrivateKeyDER( DERData );
        provider.ImportCspBlob( privateKeyBlob );
    }
 
    /// Extension method which initializes an RSACryptoServiceProvider from a PEM public key string.
    public static void LoadPublicKeyPEM( this RSACryptoServiceProvider provider, string sPEM )
    {
        byte[] DERData = RSACryptoServiceProviderExtension.GetDERFromPEM( sPEM );
        RSACryptoServiceProviderExtension.LoadPublicKeyDER( provider, DERData );
    }
 
    /// Extension method which initializes an RSACryptoServiceProvider from a PEM private key string.
    public static void LoadPrivateKeyPEM( this RSACryptoServiceProvider provider, string sPEM )
    {
        byte[] DERData = RSACryptoServiceProviderExtension.GetDERFromPEM( sPEM );
        RSACryptoServiceProviderExtension.LoadPrivateKeyDER( provider, DERData );
    }
 
    /// Returns a public key blob from an RSA public key.
    internal static byte[] GetPublicKeyBlobFromRSA( byte[] RSAData )
    {
        byte[] data = null;
        UInt32 dwCertPublicKeyBlobSize = 0;
        if ( RSACryptoServiceProviderExtension.CryptDecodeObject( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING,
            new IntPtr( (int)CRYPT_OUTPUT_TYPES.RSA_CSP_PUBLICKEYBLOB ), RSAData, (UInt32)RSAData.Length, CRYPT_DECODE_FLAGS.NONE,
            data, ref dwCertPublicKeyBlobSize ) )
        {
            data = new byte[ dwCertPublicKeyBlobSize ];
            if ( !RSACryptoServiceProviderExtension.CryptDecodeObject( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING,
                new IntPtr( (int)CRYPT_OUTPUT_TYPES.RSA_CSP_PUBLICKEYBLOB ), RSAData, (UInt32)RSAData.Length, CRYPT_DECODE_FLAGS.NONE,
                data, ref dwCertPublicKeyBlobSize ) )
                throw new Win32Exception( Marshal.GetLastWin32Error() );
        }
        else
            throw new Win32Exception( Marshal.GetLastWin32Error() );
        return data;
    }
 
    /// Converts DER binary format to a CAPI CRYPT_PRIVATE_KEY_INFO structure.
    internal static byte[] GetPrivateKeyDER( byte[] DERData )
    {
        byte[] data = null;
        UInt32 dwRSAPrivateKeyBlobSize = 0;
        IntPtr pRSAPrivateKeyBlob = IntPtr.Zero;
        if ( RSACryptoServiceProviderExtension.CryptDecodeObject( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING, new IntPtr( (int)CRYPT_OUTPUT_TYPES.PKCS_RSA_PRIVATE_KEY ),
            DERData, (UInt32)DERData.Length, CRYPT_DECODE_FLAGS.NONE, data, ref dwRSAPrivateKeyBlobSize ) )
        {
            data = new byte[ dwRSAPrivateKeyBlobSize ];
            if ( !RSACryptoServiceProviderExtension.CryptDecodeObject( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING, new IntPtr( (int)CRYPT_OUTPUT_TYPES.PKCS_RSA_PRIVATE_KEY ),
                DERData, (UInt32)DERData.Length, CRYPT_DECODE_FLAGS.NONE, data, ref dwRSAPrivateKeyBlobSize ) )
                throw new Win32Exception( Marshal.GetLastWin32Error() );
        }
        else
            throw new Win32Exception( Marshal.GetLastWin32Error() );
        return data;
    }
 
    /// Converts DER binary format to a CAPI CERT_PUBLIC_KEY_INFO structure containing an RSA key.
    internal static byte[] GetRSAFromDER( byte[] DERData )
    {
        byte[] data = null;
        byte[] publicKey = null;
        CERT_PUBLIC_KEY_INFO info;
        UInt32 dwCertPublicKeyInfoSize = 0;
        IntPtr pCertPublicKeyInfo = IntPtr.Zero;
        if ( RSACryptoServiceProviderExtension.CryptDecodeObject( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING, new IntPtr( (int)CRYPT_OUTPUT_TYPES.X509_PUBLIC_KEY_INFO ),
            DERData, (UInt32)DERData.Length, CRYPT_DECODE_FLAGS.NONE, data, ref dwCertPublicKeyInfoSize ) )
        {
            data = new byte[ dwCertPublicKeyInfoSize ];
            if ( RSACryptoServiceProviderExtension.CryptDecodeObject( CRYPT_ENCODING_FLAGS.X509_ASN_ENCODING | CRYPT_ENCODING_FLAGS.PKCS_7_ASN_ENCODING, new IntPtr( (int)CRYPT_OUTPUT_TYPES.X509_PUBLIC_KEY_INFO ),
                DERData, (UInt32)DERData.Length, CRYPT_DECODE_FLAGS.NONE, data, ref dwCertPublicKeyInfoSize ) )
            {
                GCHandle handle = GCHandle.Alloc( data, GCHandleType.Pinned );
                try
                {
                    info = (CERT_PUBLIC_KEY_INFO)Marshal.PtrToStructure( handle.AddrOfPinnedObject(), typeof( CERT_PUBLIC_KEY_INFO ) );
                    publicKey = new byte[ info.PublicKey.cbData ];
                    Marshal.Copy( info.PublicKey.pbData, publicKey, 0, publicKey.Length );
                }
                finally
                {
                    handle.Free();
                }
            }
            else
                throw new Win32Exception( Marshal.GetLastWin32Error() );
        }
        else
            throw new Win32Exception( Marshal.GetLastWin32Error() );
        return publicKey;
    }
 
    /// Extracts the binary data from a PEM file.
    internal static byte[] GetDERFromPEM( string sPEM )
    {
        UInt32 dwSkip, dwFlags;
        UInt32 dwBinarySize = 0;
 
        if ( !RSACryptoServiceProviderExtension.CryptStringToBinary( sPEM, (UInt32)sPEM.Length, CRYPT_STRING_FLAGS.CRYPT_STRING_BASE64HEADER, null, ref dwBinarySize, out dwSkip, out dwFlags ) )
            throw new Win32Exception( Marshal.GetLastWin32Error() );
 
        byte[] decodedData = new byte[ dwBinarySize ];
        if ( !RSACryptoServiceProviderExtension.CryptStringToBinary( sPEM, (UInt32)sPEM.Length, CRYPT_STRING_FLAGS.CRYPT_STRING_BASE64HEADER, decodedData, ref dwBinarySize, out dwSkip, out dwFlags ) )
            throw new Win32Exception( Marshal.GetLastWin32Error() );
        return decodedData;
    }
 
    #endregion Methods
 
    #region P/Invoke Constants
 
    /// Enumeration derived from Crypto API.
    internal enum CRYPT_ACQUIRE_CONTEXT_FLAGS : uint
    {
        CRYPT_NEWKEYSET = 0x8,
        CRYPT_DELETEKEYSET = 0x10,
        CRYPT_MACHINE_KEYSET = 0x20,
        CRYPT_SILENT = 0x40,
        CRYPT_DEFAULT_CONTAINER_OPTIONAL = 0x80,
        CRYPT_VERIFYCONTEXT = 0xF0000000
    }
 
    /// Enumeration derived from Crypto API.
    internal enum CRYPT_PROVIDER_TYPE : uint
    {
        PROV_RSA_FULL = 1
    }
 
    /// Enumeration derived from Crypto API.
    internal enum CRYPT_DECODE_FLAGS : uint
    {
        NONE = 0,
        CRYPT_DECODE_ALLOC_FLAG = 0x8000
    }
 
    /// Enumeration derived from Crypto API.
    internal enum CRYPT_ENCODING_FLAGS : uint
    {
        PKCS_7_ASN_ENCODING = 0x00010000,
        X509_ASN_ENCODING = 0x00000001,
    }
 
    /// Enumeration derived from Crypto API.
    internal enum CRYPT_OUTPUT_TYPES : int
    {
        X509_PUBLIC_KEY_INFO = 8,
        RSA_CSP_PUBLICKEYBLOB = 19,
        PKCS_RSA_PRIVATE_KEY = 43,
        PKCS_PRIVATE_KEY_INFO = 44
    }
 
    /// Enumeration derived from Crypto API.
    internal enum CRYPT_STRING_FLAGS : uint
    {
        CRYPT_STRING_BASE64HEADER = 0,
        CRYPT_STRING_BASE64 = 1,
        CRYPT_STRING_BINARY = 2,
        CRYPT_STRING_BASE64REQUESTHEADER = 3,
        CRYPT_STRING_HEX = 4,
        CRYPT_STRING_HEXASCII = 5,
        CRYPT_STRING_BASE64_ANY = 6,
        CRYPT_STRING_ANY = 7,
        CRYPT_STRING_HEX_ANY = 8,
        CRYPT_STRING_BASE64X509CRLHEADER = 9,
        CRYPT_STRING_HEXADDR = 10,
        CRYPT_STRING_HEXASCIIADDR = 11,
        CRYPT_STRING_HEXRAW = 12,
        CRYPT_STRING_NOCRLF = 0x40000000,
        CRYPT_STRING_NOCR = 0x80000000
    }
 
    #endregion P/Invoke Constants
 
    #region P/Invoke Structures
 
    /// Structure from Crypto API.
    [StructLayout( LayoutKind.Sequential )]
    internal struct CRYPT_OBJID_BLOB
    {
        internal UInt32 cbData;
        internal IntPtr pbData;
    }
 
    /// Structure from Crypto API.
    [StructLayout( LayoutKind.Sequential )]
    internal struct CRYPT_ALGORITHM_IDENTIFIER
    {
        internal IntPtr pszObjId;
        internal CRYPT_OBJID_BLOB Parameters;
    }
 
    /// Structure from Crypto API.
    [StructLayout( LayoutKind.Sequential )]
    struct CRYPT_BIT_BLOB
    {
        internal UInt32 cbData;
        internal IntPtr pbData;
        internal UInt32 cUnusedBits;
    }
 
    /// Structure from Crypto API.
    [StructLayout( LayoutKind.Sequential )]
    struct CERT_PUBLIC_KEY_INFO
    {
        internal CRYPT_ALGORITHM_IDENTIFIER Algorithm;
        internal CRYPT_BIT_BLOB PublicKey;
    }
 
    #endregion P/Invoke Structures
 
    #region P/Invoke Functions
 
    /// Function for Crypto API.
    [DllImport( "advapi32.dll", SetLastError = true )]
    [return: MarshalAs( UnmanagedType.Bool )]
    internal static extern bool CryptDestroyKey( IntPtr hKey );
 
    /// Function for Crypto API.
    [DllImport( "advapi32.dll", SetLastError = true )]
    [return: MarshalAs( UnmanagedType.Bool )]
    internal static extern bool CryptImportKey( IntPtr hProv, byte[] pbKeyData, UInt32 dwDataLen, IntPtr hPubKey, UInt32 dwFlags, ref IntPtr hKey );
 
    /// Function for Crypto API.
    [DllImport( "advapi32.dll", SetLastError = true )]
    [return: MarshalAs( UnmanagedType.Bool )]
    internal static extern bool CryptReleaseContext( IntPtr hProv, Int32 dwFlags );
 
    /// Function for Crypto API.
    [DllImport( "advapi32.dll", CharSet = CharSet.Auto, SetLastError = true )]
    [return: MarshalAs( UnmanagedType.Bool )]
    internal static extern bool CryptAcquireContext( ref IntPtr hProv, string pszContainer, string pszProvider, CRYPT_PROVIDER_TYPE dwProvType, CRYPT_ACQUIRE_CONTEXT_FLAGS dwFlags );
 
    /// Function from Crypto API.
    [DllImport( "crypt32.dll", SetLastError = true, CharSet = CharSet.Auto )]
    [return: MarshalAs( UnmanagedType.Bool )]
    internal static extern bool CryptStringToBinary( string sPEM, UInt32 sPEMLength, CRYPT_STRING_FLAGS dwFlags, [Out] byte[] pbBinary, ref UInt32 pcbBinary, out UInt32 pdwSkip, out UInt32 pdwFlags );
 
    /// Function from Crypto API.
    [DllImport( "crypt32.dll", SetLastError = true )]
    [return: MarshalAs( UnmanagedType.Bool )]
    internal static extern bool CryptDecodeObjectEx( CRYPT_ENCODING_FLAGS dwCertEncodingType, IntPtr lpszStructType, byte[] pbEncoded, UInt32 cbEncoded, CRYPT_DECODE_FLAGS dwFlags, IntPtr pDecodePara, ref byte[] pvStructInfo, ref UInt32 pcbStructInfo );
 
    /// Function from Crypto API.
    [DllImport( "crypt32.dll", SetLastError = true )]
    [return: MarshalAs( UnmanagedType.Bool )]
    internal static extern bool CryptDecodeObject( CRYPT_ENCODING_FLAGS dwCertEncodingType, IntPtr lpszStructType, byte[] pbEncoded, UInt32 cbEncoded, CRYPT_DECODE_FLAGS flags, [In, Out] byte[] pvStructInfo, ref UInt32 cbStructInfo );
 
    #endregion P/Invoke Functions
}

I am attaching a Visual Studio 2012 solution containing a sample usage for PEM and DER encoded files. Click here to download.

Solution containing sample keys:

Test program output:

Visual Studio 2012 Upper Case Menu

Christian Etter — Sun, 30 Sep 2012 11:57:08 +0000

Visual Studio 2012 is out for two weeks now. The first thing that catches the eye is the new 2D flat chroma-reduced UI. Second is probably the all-uppercase menu. For those who don’t like it, there is a registry tweak to revert to mixed case writing.

Fortunately this is easy to achieve with a single value to be added to HKCU\Software\Microsoft\VisualStudio\11.0\General. Open the location in regedit.exe and add a DWORD value called SuppressUppercaseConversion which must be set to 1.

The result should look familiar again.

T-SQL Capitalize First Character in Each Word

Christian Etter — Thu, 26 Jul 2012 05:55:52 +0000

This is a small exercise in T-SQL for a common situation: It happens quite frequently that data is entered manually in slightly different ways regarding upper and lower case. Examples are “james smith”, “Gerald BLACK”, “MIKE O’NEILL” etc.

Normalizing such input can be done in a procedural way by iterating over each record with a couple of variables and two loops.

Is it possible to normalize such input using T-SQL in a non-procedural manner? It is, though I would leave it up to the reader to judge whether or not it is more efficient.

WITH CTE1 AS(
	SELECT 0 AS ID, N'章子怡' AS A UNION ALL
	SELECT 1, 'new york' UNION ALL
	SELECT 2, 'michael BOND' UNION ALL
	SELECT 3, 'DAN john o''keele' UNION ALL 
	SELECT 4, 'michaela dÜmpel-gerümpel'	UNION ALL
	SELECT 5, 'mieze katze luftMatratze' UNION ALL
	SELECT 6, 'gute grütze gedankenStütze'
), CTE2 AS (
	SELECT ROW_NUMBER() OVER( ORDER BY ID ) AS ID1, * FROM CTE1
), CTE3 AS (
	SELECT ID1, 1 AS ID2, SUBSTRING( A, 1, 1 ) AS C, A FROM CTE2
	UNION ALL 
	SELECT ID1, ID2 + 1, SUBSTRING( A, ID2 + 1, 1 ), A FROM CTE3 WHERE ID2 < LEN( A )
), CTE4 AS (
	SELECT A.ID1, A.ID2, A.A, CASE WHEN B.C IS NULL THEN UPPER( A.C ) ELSE LOWER( A.C ) END AS C FROM CTE3 AS A
		LEFT JOIN CTE3 AS B ON A.ID1 = B.ID1 AND A.ID2 - 1 = B.ID2 AND B.C NOT IN ( ' ', '.', ',', '[', '(', '''', '-' )
), CTE5 AS (
	SELECT CTE2.ID1, ( SELECT CTE4.C + '' FROM CTE4 WHERE CTE4.ID1 = CTE2.ID1 ORDER BY CTE4.ID2 FOR XML PATH(''), TYPE ).VALUE( 'text()[1]', 'NVARCHAR(MAX)' ) AS A
		FROM CTE2
)
SELECT * FROM CTE5 ORDER BY ID1 ASC OPTION( MAXRECURSION 32767 )

Here is the input:

0	章子怡
1	new york
2	michael BOND
3	DAN john o’keele
4	michaela dÜmpel-gerümpel
5	mieze katze luftMatratze
6	gute grütze gedankenStütze

This becomes:

1	章子怡
2	New York
3	Michael Bond
4	Dan John O’Keele
5	Michaela Dümpel-Gerümpel
6	Mieze Katze Luftmatratze
7	Gute Grütze Gedankenstütze

Obviously the above code won’t work if your name is MacGuyver, DiRosa or McDonalds. But it would not take much more than another CTE to fix such names as well as long as they adhere to a common capitalization scheme.

The Most Popular Photograph I Ever Took

Christian Etter — Sun, 03 Jun 2012 21:18:26 +0000

While cycling through Okinawa south of Naha, I came across a red traffic light. Looking to the left, there was a huge house built on an artificial tree. It did not strike me as that spectacular, but I figured since I had to wait I might as well take a snapshot.

This gigantic tree house restaurant can be found south of Naha on Okinawa, Japan. It is a restaurant called アジア (Asia). Shot with NIKON D80

I would have never expected this would turn out to be one of the most popular shots I ever took. It has been printed on 2 magazines.

T-SQL Word Randomizer – Does the Order of Letters in a Word Matter?

Christian Etter — Sat, 19 May 2012 21:37:14 +0000

Perhaps you have come across this paragraph before:

Acdrocing to a rcersaheer at Cdmiabrge Uterniisvy, it dens’ot mttear in waht odrer the letters in a word are, the only inapmtrot tinhg is that the frist and last lteetr be at the rghit plcae. The rset can be a ttoal mess and you can siltl raed it wtihuot plreobm. Tihs is baesuce the hmuan mind does not raed ervey lteetr by ilestf but the wrod as a wlohe.

When I saw this, I thought about how to write such a randomizer in T-SQL without using loops and variables:

WITH CTE1 AS(
	SELECT 'According to a researcher at Cambridge University, it doesn''t matter in what order the letters in a word are, the only important thing is that the first and last letter be at the right place. The rest can be a total mess and you can still read it without problem. This is because the human mind does not read every letter by itself but the word as a whole.' AS A
	UNION ALL
	SELECT 'Manche Leute sagen es spielt keine Rolle, in welcher Reihenfolge die Buchstaben in einem Wort angeordnet sind, solange der erste und der letzte Buchstabe nicht verändert wird.'
), CTE2 AS (
	SELECT ROW_NUMBER() OVER( ORDER BY LEFT( A, 0 ) ) AS ID1, * FROM CTE1
), CTE3 AS (
	SELECT ID1, 1 AS ID2, SUBSTRING( A, 1, 1 ) AS C, A FROM CTE2
	UNION ALL 
	SELECT ID1, ID2 + 1, SUBSTRING( A, ID2 + 1, 1 ), A FROM CTE3 WHERE ID2 < LEN( A )
), CTE4 AS (
	SELECT A.ID1, A.ID2, A.A, A.C, CASE WHEN B.C IS NOT NULL AND C.C IS NOT NULL THEN 0 ELSE 1 END AS ID3 FROM CTE3 AS A
		LEFT JOIN CTE3 AS B ON A.ID1 = B.ID1 AND A.ID2 - 1 = B.ID2 AND B.C NOT IN ( ' ', '.', ',', '[', '(', ']', ')' )
		LEFT JOIN CTE3 AS C ON A.ID1 = C.ID1 AND A.ID2 + 1 = C.ID2 AND C.C NOT IN ( ' ', '.', ',', '[', '(', ']', ')' )
), CTE5 AS (
	SELECT A.ID1, A.ID2, A.A, A.C, CASE A.ID3 WHEN 0 THEN ( SELECT MAX( B.ID2 ) + 1 FROM CTE4 AS B WHERE A.ID1 = B.ID1 AND B.ID2 < A.ID2 AND B.ID3 = 1 ) ELSE ID2 END AS ID3 FROM CTE4 AS A
), CTE6 AS (
	SELECT CTE2.ID1, ( SELECT CTE5.C + '' FROM CTE5 WHERE CTE5.ID1 = CTE2.ID1 ORDER BY CTE5.ID3, NEWID() FOR XML PATH(''), TYPE ).VALUE( 'text()[1]', 'NVARCHAR(MAX)' ) AS A
		FROM CTE2
)
SELECT * FROM CTE6 ORDER BY ID1 ASC OPTION( MAXRECURSION 32767 )

Here is a German example:

Mahnce Letue seagn es speilt kenie Rolle, in wcelher Renfhioglee die Bbetschuan in eniem Wort aegdnonret snid, solnage der erste und der ltteze Bsbctauhe nihct vedrreänt wrid.

Ok, this is not terribly efficient, but it proves that recursive CTEs have their right of existence at last ;-)