Framework comparison summary

To be able to size up an objective summing image I started to build an e-commerce website in all frameworks on the same model. These are my experiences after two weeks.

Full of surprises

I thought PERL would win gloriously on this test. The results was really impressive but Django did even better. I remember when I first saw the results I couldn’t believe them and run the test again. The other surprise was the really bad performance of PHP. Somehow I expect PHP would be the worst out of the candidates but I didn’t think that difference will be this big. That’s one of the reasons I didn’t test symfony 1.4 eventually. I didn’t see the point.

Installation & coding

Catalyst

In my humble opinion installing Catalyst was really troublesome. A lot of problems occured during the process. If you want to do it from ground zero you should be aware that it can take a lot of time and you should have the nerves to do it through. But once it is installed everything is fine. Developing is easy. Tremendous number of CPAN libraries are out to help you in your daily work. PERL has the benefit from being robust and flexible. You can do whatever you want and however you want almost on every abstraction level. Catalyst is a true alternative for every web developer who wants performance, flexibility, reliability and the power of PERL.

Django

Django installation was like a dream after Catalyst. It took 5 minutes, at most. Create the working site with the imported modules took another 5 minutes. Admin interface worked out of the box. Although I had some problems with imported models I can have them fixed quickly. If I mentioned the flexibility of Catalyst I have to write about the restrictions of Django. In contrast to PERL’s “There’s more than one way to do it” principle there is only one true way to do something in Django properly. Although it’s not really true after two weeks of developing I found myself to recreate algorithms, model definitions and processes to be more “Djangoish”. Whether it’s a good or bad thing everybody decides themselves. At the beginning I cursed a lot but I get used to think the Django way and I like to have standards. I have to admit I can develop functions (it’s important to emphasize not algoritmhs but functions) faster than I ever could in PHP.

Symfony

I’m working with symfony ever since 1.0 beta came out. I loved it especially after the home-made frameworks and giant if built index.phps. I made a whole bunch of sites (e-commerce, cms, business data processing etc.) based on symony 1.0, 1.2 and 1.4. I knew that this was not the most optimal system but learning curve is relatively short comparing to the aboves. And developing was quite fast. The advantage of PHP is beyond doubt when you have a company and have to hire people. Usually PHP developers are cheaper and easier to find than PERL or Python developers. But if I would make this benchmark earlier I would have changed most of my sites to Python or PERL and not made an effort to be able run Symfony sites (with couple of thousand visitors per day) on a single Amazon microinstance.

Documentation

There’s nothing really to complain about. All of them are very well documented. There are a lot of blogs and forums where we can find help if we stuck on a problem.

Plugins

I’m not really go deep in to this subject but as far as I can tell only symfony is the only conventional pluginnable system. The plugin philosophy shaped in accordance with the philosophy of the language. Out and away PERL has more a solution set for common problems not so much plugins. Python has some plugins but these are rather toolkits or full apps than pluggable moduls. I don’t want the judge any of them. I can live with all of them.

Numbers

And after all here are the magic numbers.

	Catalyst		Django		Symfony 1.2
Installation	2 hours		5 minutes		10 minutes
Import from DB	10 minutes		1 minute		1 minute
Requests/sec Higher is better
	Index	Show	Index	Show	Index	Show
AB 5 conc	93,7	122,76	123,42	158,78	24,29	23,77
AB 20 conc	94,39	122,18	118,64	167,52	23,81	23,6
AB 50 conc	94,51	120,24	116,92	160,34	23,75	23,67
Resource usage Lower is better
Memory	11,40%	11,40%	7,80%	6,10%	2,20% (per thread)	2,20% (per thread)
CPU	27,20%	25,10%	3,10%	2,20%	100,00%	100,00%