This is the last post of the series. We introduced a micro orm benchmark with few contestants and we implemented all required tests. In this post we analyze results.
Complete set of results
Let us start from a complete set of results (all numbers are miliseconds so for example 531 means 0.531 sec and 6502 means 6.502 sec):
| category | name | totalms |
| ADO.NET | TOP 1 | 144.0144 |
| Linq2Sql | TOP 1 | 526.0526 |
| EF ModelFirst NoTracking | TOP 1 | 531.0531 |
| EF ModelFirst Tracking | TOP 1 | 551.0551 |
| EF CodeFirst NoTracking | TOP 1 | 1063.1063 |
| EF CodeFirst Tracking | TOP 1 | 1069.1069 |
| nHibernate StateLess | TOP 1 | 654.0654 |
| nHibernate StateFull | TOP 1 | 658.0658 |
| ADO.NET | TOP 10 | 111.0111 |
| Linq2Sql | TOP 10 | 574.0574 |
| EF ModelFirst NoTracking | TOP 10 | 499.0499 |
| EF ModelFirst Tracking | TOP 10 | 627.0627 |
| EF CodeFirst NoTracking | TOP 10 | 1046.1046 |
| EF CodeFirst Tracking | TOP 10 | 1247.1247 |
| nHibernate StateLess | TOP 10 | 1194.1194 |
| nHibernate StateFull | TOP 10 | 1305.1305 |
| ADO.NET | TOP 100 | 270.027 |
| Linq2Sql | TOP 100 | 710.071 |
| EF ModelFirst NoTracking | TOP 100 | 675.0675 |
| EF ModelFirst Tracking | TOP 100 | 1596.1596 |
| EF CodeFirst NoTracking | TOP 100 | 1210.121 |
| EF CodeFirst Tracking | TOP 100 | 2836.2836 |
| nHibernate StateLess | TOP 100 | 5699.5699 |
| nHibernate StateFull | TOP 100 | 6502.6502 |
| ADO.NET | TOP 1000 | 1327.1327 |
| Linq2Sql | TOP 1000 | 2064.2064 |
| EF ModelFirst NoTracking | TOP 1000 | 1885.1885 |
| EF ModelFirst Tracking | TOP 1000 | 10700.0699 |
| EF CodeFirst NoTracking | TOP 1000 | 2365.2365 |
| EF CodeFirst Tracking | TOP 1000 | 18658.8657 |
| nHibernate StateLess | TOP 1000 | 51017.1012 |
| nHibernate StateFull | TOP 1000 | 58400.8395 |
| ADO.NET | TOP 10000 | 10878.0877 |
| Linq2Sql | TOP 10000 | 19527.9526 |
| EF ModelFirst NoTracking | TOP 10000 | 13496.3495 |
| EF ModelFirst Tracking | TOP 10000 | 122345.2333 |
| EF CodeFirst NoTracking | TOP 10000 | 13797.3796 |
| EF CodeFirst Tracking | TOP 10000 | 210554.0533 |
| nHibernate StateLess | TOP 10000 | 546423.6369 |
| nHibernate StateFull | TOP 10000 | 564736.468 |
The most interesting comparison to me was to see how retrieving data works for different sizes of datasets. Here are results presented on charts:
General observations:
- bare ADO.NET is of course always the fastest but for the largest dataset, EF with tracking turned off is almost as fast (which is amazing)
- Linq2Sql is outperformed by EF Model First No Tracking and, for the largest dataset, even EF Code First No Tracking is way faster than Linq2SQL
- EF Model First is always faster than EF Code First
- EF with tracking is always much slower than EF without tracking. For 1000 or more records tracking can probably hurt the performance badly if you are only reading the data
- nHibernate StateLess is always faster than nHibernate StateFull (which is expected)
- nHibernate is surprisingly slow when larger sets are retrieved (which is rather unexpected and frankly, I wonder if there is a way to speed it up as it looks rather disappointing)
Note that benchmaring ORMs should also involve inserts/updates/deletes in various scenarios. Please consider my experiments as valid only in this specific context (data retrieval).
I would also like to hear a comment or two about this. I am especially interested in the underperformance of nHibernate for large sets of data – I wonder what else could be done to speed it up.
3 comments:
Your experiment, while valid in some contexts, may be a bit wrong in others. Ayende Rahien spoke about micro benchmarks once and I tend to agree on many if his points.
However, you do have a point and it's very good to have hard data like yours to look. Thanks very much :)
@David: I know this Ayende's opinion and still believe that my approach reveals some interesting data. Note that not only I do bare SELECTs but also I "warm-up" every engine by skipping few initial measurements just to minimize the risk of any unwanted penalties coming from things like metadata initializing (which slow down the very first SELECT but do not slow down consecutive operations in the same process). My biggest concern is the unexpected slowness of nHibernate and unfortunately after consulting few fellow developers who use NH daily, I am still unable to correctly diagnose and fix the issue (if this IS an issue).
Wiktor, do your tests discount the startup time? Iow, does the test app do some EF call before meeasuring the particular test so that building the edmx or code first model into the DbContext is discounted? Thanks, Dave
Post a Comment