SisoDb – Using DbPipe to add compression

In v16.1.0 of SisoDb I’ve added a new interface SisoDb.IDbPipe which lets you hook into the process of writing and reading the string-data representing a document (default JSON) to and from the DB. You could use that for encryption, compression, ….? Now I will just show you how to add compression.

Lets implement a simple deflate compression using System.IO.Compression.


public class DeflatePipe : IDbPipe
{
    public string Writing(IStructureSchema structureSchema, string data)
    {
        return Compress(data);
    }

    public string Reading(IStructureSchema structureSchema, string data)
    {
        return Decompress(data);
    }

    private static string Compress(string j)
    {
        using (var source = new MemoryStream(Encoding.UTF8.GetBytes(j)))
        {
            source.Position = 0;
            using (var trg = new MemoryStream())
            {
                using (var c = new DeflateStream(trg, CompressionMode.Compress, false))
                {
                    source.CopyTo(c);
                }
                return Convert.ToBase64String(trg.ToArray());
            }
        }
    }

    private static string Decompress(string j)
    {
        using (var source = new MemoryStream(Convert.FromBase64String(j)))
        {
            source.Position = 0;
            using (var trg = new MemoryStream())
            {
                using (var c = new DeflateStream(source, CompressionMode.Decompress, false))
                {
                    c.CopyTo(trg);
                }
                return Encoding.UTF8.GetString(trg.ToArray());
            }
        }
    }
}

I don’t make use of the IStructureSchema but could easily use it to just use compression for a certain document by inspecting:


if(structureSchema.Name == "Customer") {}

or

if(structureSchema.Type.Type == typeof(Customer)) {}

Now lets hook the pipe in:

db.Pipe = new DeflatePipe();

I did one for LZ4 compression as well and lets just have a look at the result.


use [SisoDb.SampleApp]

---5000 items

-- Raw
--select sum(DATALENGTH(Json)) from dbo.CustomerStructure
--4883358 bytes

-- Deflate
--select sum(DATALENGTH(Json)) from dbo.CustomerStructure
--3678256 bytes

-- Lz4
--select sum(DATALENGTH(Json)) from dbo.CustomerStructure
--4845176 bytes

-- Diff
select 
	(4883358 - 3678256) TotalSaveDeflate, --1205102
	(4883358 - 3678256) / 5000 PerItemDeflate, --241
	(4883358 - 4845176) TotalSaveLZ4, --38182
	(4883358 - 4845176) / 5000 PerItemLZ4 --7

You should really evaluate both algorithm and the document type to see if you save anything.

//Daniel

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s