Benchmarks, SisoDb and RavenDb

This post is an extract from my recently posted post where I reply on criticism against SisoDb from a member in the RavenDb team. You can read it here: http://daniel.wertheim.se/2012/03/11/ranting-is-good-for-you/

First lets make it clear. I have never had any intentions creating a RavenDb vs SisoDb scene, these benchmarks are a result of Mr Itamar Syn-Hershkos criticism on twitter lately, where I feel he hasn’t got all information but mostly talking out of the old design of SisoDb.

Results

Using well known infrastructure in SQL Server, creating data type specific indexes and sets of data for simple key-values SisoDb is fast. And as it seems, actually faster than RavenDb in the measured scenarios below.

Note about deserialization of matches

NOTE! In medium and large set of data, when querying for Trips the result is much bigger that what RavenDb seems to return; where SisoDb returns all the actual matches, hence you can’t compare those times fairly. RavenDb’s 128 returned and deserialized records, vs SisoDb’s 16668 returned and deserialized records.

RavenDb – returning 128 matches

SisoDb – returning all 16668 matches

Memory consumption

Just looking at the client application that runs the tests. When running with Large sets 100.000 customers and 100.000 trips:

SisoDb 116Mb

RavenDb 1.16Gb

Differences

RavenDb and SisoDb are different. RavenDb has lots of concepts like dynamic vs static indexes, stale data, not returning all matching records etc. SisoDb aims at being Simple. E.g it does all work upfront and makes every leaf in your object hierarchy indexed. It doesn’t use any proxies etc. for change tracking etc. SisoDb is trying to be a fast, non magical, lightweight document-oriented provider over SQL-Server. Just take a look at the memory footprints above.

When using RavenDb in the tests below I have not forced wait for stale results in the timer that measures the operations. Hence the real insert time would actually be longer than indicated.

Both providers are used out of the box but with warm up behavior. Before timing any operations; inserts, queries and deletes have been executed so that each system gets a chance to create cache plans etc. Note that I’m not a skilled RavenDb user. I just used it out of the box. That is, no tweaking what so ever with static indexes etc. I couldn’t find any batch insert API in RavenDb; which SisoDb of course has.

Test machine

The machine is a simple laptop running both the server and client.
Windows 7 Ultimate, SQL2012
Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
8GB RAM
237Gb SSD with 190Gb free, Samsung PM810

Versions

RavenDb: v1.0.701
SisoDb: v10.4.2

Scenarios

The test code is available at GitHub. The scenarios hasn’t much to do with a real life situation and you should test it in your environment.

The test has three modes and each mode works with two different type of documents.

  • Small set of data (1.000 items)
  • Medium set of data (10.000 items)
  • Large set of data (100.000 items)

Test cases

public interface ITestCases
{
	void Warmup(
            Expression<Func<Customer, bool>> customerPredicate, 
            Expression<Func<Trip, bool>> tripPredicate);

	void BatchInsertCustomers(int numOfCustomers);
	void BatchInsertTrips(int numOfTrips);
	void SingleInsertCustomer();
	void SingleInsertTrip();

	long QueryCustomers(Expression<Func<Customer, bool>> predicate);
	long QueryTrips(Expression<Func<Trip, bool>> predicate);

	//Count methods only used after profiling to get number of items inserted
	long CountCustomers();
	long CountTrips();
}

Customer model

public class Customer
{
	public Guid Id { get; set; }
	public int CustomerNo { get; set; }
	public string Firstname { get; set; }
	public string Lastname { get; set; }
	public ShoppingIndexes ShoppingIndex { get; set; }
	public DateTime CustomerSince { get; set; }
	public Address BillingAddress { get; set; }
	public Address DeliveryAddress { get; set; }

	public Customer()
	{
		ShoppingIndex = ShoppingIndexes.Level0;
		BillingAddress = new Address();
		DeliveryAddress = new Address();
	}
}

public class Address
{
	public string Street { get; set; }
	public string Zip { get; set; }
	public string City { get; set; }
	public string Country { get; set; }
	public int AreaCode { get; set; }
}

public enum ShoppingIndexes
{
	Level0 = 0,
	Level1 = 10,
	Level2 = 20,
	Level3 = 30
}

Trip model

public class Trip
{
	public int Id { get; set; }
	public Transport Transport { get; set; }
	public Accommodation Accommodation { get; set; }
	public decimal Price { get; set; }
}

public class Transport
{
	public string DepartureCode { get; set; }
	public string DestinationCode { get; set; }
	public DateTime DepartureDate { get; set; }
	public int Duration { get; set; }
}

public class Accommodation
{
	public string HotelCode { get; set; }
	public string RoomCode { get; set; }
	public DateTime CheckinDate { get; set; }
	public int Duration { get; set; }
}

Customer query predicates
The queries for customers used in the tests.

//Small set
CustomerPredicate = c =>
	c.CustomerNo >= 500 && c.CustomerNo <= 550
	&& c.DeliveryAddress.Zip == "525",

//Medium set
CustomerPredicate = c =>
	c.CustomerNo >= 5000 && c.CustomerNo <= 5500
	&& c.DeliveryAddress.Zip == "5250",

//Large set
CustomerPredicate = c =>
	c.CustomerNo >= 50000 && c.CustomerNo <= 55000
	&& c.DeliveryAddress.Zip == "52500",

Trip query predicate
The query for trips used in the tests.

//Same for all test sets
var dateFrom = TestConstants.BaseLine.AddDays(10);
var dateTo = dateFrom.AddDays(5);

TripPredicate = t =>
	(t.Transport.DepartureDate >= dateFrom
	&& t.Transport.DepartureDate <= dateTo)
	&& t.Accommodation.Duration > 8

Screenshots of each testrun are inlcuded in the main branch at GitHub.

//Daniel

2 thoughts on “Benchmarks, SisoDb and RavenDb

  1. Pingback: Ranting is good for you « Daniel Wertheim

  2. Pingback: SisoDb v11.4.0 – with new named queries « Daniel Wertheim

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s