flex has a garbage collector but it’s a bit special on the way it handles memory so I thought I would layout some general rules on how to keep memory low.

first thing we need to understand is that any application we will build in our careers will be using some amount of memory, and because memory is not infinite we should keep our applications memory as low as possible.

I always say the best way to describe memory management is to “be smart” meaning, always have memory in mind, never let it go and by doing this your application will keep lean and thin all the time

by taking up too much memory you can get into allot of trouble, you can make your users hate the application because it is running slowly and making their computer run slowly. also, the application can simply crash.

I hear from programmers in my consulting that computers are getting more and more memory and you can see laptops coming with 4G of memory out of the box and so on, but actually we are also seeing a movement towards mobile devices such as mobile phones etc. these mobile devices do no allow us to use too much memory because it’s limited resource.

Memory leaks

memory leaks is a big issue in flex programming and it’s generally speaking divided into 2 parts, first when our application is using memory in parts we didn’t want it to use and the other part is when we didn’t handle the cleaning part efficiently enough.

Flash handles memory management through something called “references”, just to make things clear before we begin, the only way an object is cleaned by the garbage collector, is be not having any reference from that object to another object, that way you can make sure the object will be “picked up” by the garbage collector.

Event Listeners

Event listeners are by-far the most mis-used mis-understood part of flex applications, I have seen people/developers using those without even understanding what is actually happening behind the scenes and when (and if) those are cleaned up in memory.

I will try to explain
let’s take this peace of code

//objectX
objectY.addEventListener(some_event, some_function)

We have two objects (ObjectX, ObjectY), Object Y has a reference to a function in objectX just by passing the function, now, if your application has a reference to objectY, object X will not get cleaned because it has a reference to a function.

There is actually something called weak event listeners, by default the event listeners are “strong” meaning every event listener is a function, if you specify the event listener is “weak” then the event listener does not count as a reference.

Static Variables

Static variables are never cleaned in the entire session of a flex application, meaning if you set the variable’s value with an object, this object will never get cleaned.

If you want to make sure the object is cleaned you should null out the variable.

// static variable
public static var _foo:Foo;
//setting the value
_foo = new Foo();
//clean
_foo = null;

Dictionaries

Dictionaries are another big problem and source of misunderstanding in flex applications.

Dictionaries are key-value collections, meaning you have a collection that is built in the form of a key and a value attached to it.

You can use objects as your key and your value, for example let’s say we have two objects (ObjectX, ObjectY), you add an entry to the dictionary where ObjectX is the key and ObjectY is the value.

By default the values in dictionaries are “strong” and you can specify a “weak” key, by doing this, you can actually make sure that if nothing else has a reference to ObjectX, it will get cleaned by the garbage collection, BUT, ObjectY will not get cleaned even though the key for it got cleaned from the dictionary.

There are many questions on this issue from flex experts to adobe, those questions generally start with a WHYYYYYY????

I would also like to understand why but this is the situation and we have to deal with it, in the Flex-Show podcast Aaron says that if you iterate through the values of the dictionary, then all values that don’t have a key (it was cleaned remember?) gets cleaned immediately.

I never encountered this behavior but Aaron knows what he is talking about, so I trust him (Thanks Aaron).

Conclusion

I think those are the main three issues in flex applications, there are a few more but if you handle those three you should be fine and your application will actually be pretty lean (at least in memory management).

** this post was inspired by Aaron’s talk on the podcast, I enjoyed it so much I decided to make a post on the issue.

I have seen flex applications written in many ways, mxml files, as files, namespaces and more.

I guess there is no “right” way to do so, or at least not just one “right” way.

these past few months I have started working with mxml files in a way I think is very convenient, it keeps my mxml file very lean and clean, all of the view code is in a separate action-script file which inherits from a base class.

I want to show you this, first, this is the sample project, this is what it looks like in flash builder:

as you can see, there is the main application, an mxml component, the BaseViewHelper class and that is just about it.

now, let’s have a look at the BaseViewHelper class
note: this is a stripped down version of the class, it has many additions in my project, like a way to add validations, clean and dirty modes for the model and more, this is just to see the way I work, you can add your own additions to it.

package com.kensodev.core
{
	import mx.core.UIComponent;

	public class BaseViewHelper
	{
		private var _view:UIComponent;

		public function BaseViewHelper()
		{

		}

		public function init():void
		{

		}

		public function set view(v:UIComponent):void
		{
			_view = v;
		}

		public function get view():UIComponent
		{
			return _view;
		}
	}
}

as you can see, I kept it very simple, it has a view getter and setter to set the view it handles, it has an init function (for creation complete events).

now, let’s see an mxml component and how does the helper fits, but first let’s see a view helper which inherits from the BaseViewHelper class.

package com.kensodev.views.helpers
{
	import com.kensodev.core.BaseViewHelper;
	import com.kensodev.views.MyViewComponent;

	public class MyViewComponentHelper extends BaseViewHelper
	{
		public function MyViewComponentHelper()
		{
			super();
		}

		public function get myView():MyViewComponent
		{
			return MyViewComponent(this.view);
		}

		public override function init():void
		{
			//TODO Auto-generated method stub
			super.init();
		}

		public function myButton_clickHandler(event:MouseEvent):void
		{
			// TODO Auto-generated method stub
		}
	}
}

This is a helper file for the MyViewComponent, now, here’s the source code for implementing the helper inside each View component

<?xml version="1.0" encoding="utf-8"?>
<mx:Canvas xmlns:mx="http://www.adobe.com/2006/mxml"
		   width="400"
		   height="300"
		   creationComplete="helper.init()"
		   xmlns:helper="com.kensodev.views.helpers.*">

	<mx:Script>
		<![CDATA[

		]]>
	</mx:Script>

	<helper:MyViewComponentHelper view="{this}" id="helper" />

	<mx:Button id="myButton" click="helper.myButton_clickHandler(event)"

</mx:Canvas>

as you can see, the file is very clean, it has only data relevant to understand the structure of the component (which is all you need in mxml), all of the logic files “seats” behind the scenes.

there is a naming conventions for views and view helpers
its
view_name
view_nameHelper
this way, when you navigate your way through the code, you can always see what you are looking for with ease like so:

I hope you take inspiration with this way, I assure you I did, it’s a very useful way to manage large projects with many views and view helpers.

the source code of the example project is on git here.

http://github.com/KensoDev/view-helper-example

******

update:

after a discussion with @douglasknudsen I want to make things clear, this is not the ViewHelper implementation of cairngorm Mvc framework, it’s something you should (or can) write on your own just to make things organized and clean.

I am using it regardless to the MVC framework I am using which is RobotLegs B.T.W

This post, above anything else is a question and a try to spark a debate from my fellow developers and software craftsman out there.

Earlier this week I started working on a very interesting project for an international company creating test suites for devices and applications, the suite is cross platform and can run on any machine both windows and Linux and test any device, from mobile phones to DVD’s to diving sensors.

the client side is written in flex and pure AS3, the server and services are written in Java and Perl scripts.

What is my point?

My point is that this company has some pretty strong developers, all know a thing or 2 on software developement, both in Java and Flex.

Another thing, usually when I get invited to consult and develop for a company saying “we have a flex application” I do see an application but it’s horrible, un-scalable, mxml+code in the same file, services calls in mxml and more.

in this company, that wasn’t the case, the application is actually pretty solid, stable and gone over more then a couple of QA steps inside the company and clients.

You can imagine that the developers over there can contribute a thing or two to the community.

OK, so far so good, what am I saying here?

From conversations I had with the developers I found out something amazing, none of them has a twitter account, none of them has a blog and none, absolutely none contributed to an open source project, not even a line of code, not even opened a case in the WIKI.

so, what do you think?
Should all developers be active members of the community?
Should every developer have a profile and answer questions on the amazing StackOverflow.com?
Should every developer write a blog?

Love to hear your thoughts on this…

another thing, when you interview programmers, do you care about these issues?

But, that is not the point.

This week I was working with a company on a client side project, I had to email them the files every time and the .git folders inside the project were heavy and unnecessary.

If you worked with SVN in the future you probably know the export command, after searching for a bit in git help I found how you can do it in git with ease.

This is the command you should use

git checkout-index -a -f --prefix=/path/to/your/folder/

The ending forward slash is very important, do not forget to use it please.

a screenshot of my shell.

As a freelancer, all of the client’s communication and service is up to you, you need to provide them with the best platform to work on, with you, along side with the existing team inside the company (in-case you are working with them).

So, because I believe I have that kind of platform I thought I’d share (as I always do) with you, how I work with my clients, although there are tools and services in this post, it’s not about them, it’s about how you can use them, seeing the customer’s needs in the center of your concerns.

I as a freelancer very often work as a team leader inside a company, sometimes as the architect and sometimes as the development director, sometimes both and I even love working as a lead developer.

So, my service must be customizable around who I am at the moment and what does the client expect of me to be.

Let us begin…

SVN – Source control system

First, whether the company is working with a source control or not, I always bring my source control system with me, I work with SVN (or GIT).
I use beanstalk, you can read about it in a previous post I wrote here.

You probably ask yourself the same question everyone is asking me, why not use a regular SVN server hosted in a farm or something, well, my answer is always the same, beanstalk are giving me an all around solution for my needs for example:

TimeLine

All of the team commits can be seen inside a timeline very similar to facebook, to me as the leader or the architect it’s very easy to know in the morning what has been accomplished yesterday and by who.

Beanstalk svn timeline

As you can see, I can see very clearly who are the team members on this specific project, what are the change-sets they are responsible for and the file that have been changed during the work Yesterday or last month for that matter.

Also, I can use webhooks to do whatever I want, I can use email notifications for every commit (very useful) and many other more very useful features.

Project management system

I work with basecamp to manage all of the project aspects.

What do I manage:

• deadlines
• to-do’s
• messages
• chat
• files
• source

I’ll explain the source line

Every commit from beanstalk posts a message on the board at basecamp dashboard, meaning I can comment (beanstalk added a feature similar to that recently) on a team member commit and give him feedback on it.

Everything is kept inside the “project” context, I’ll five you an example, because at beanstalk you can see the source changes, very easily I can review the code in the web browser, that means when a team member writes bad code or code I expect to be cleaner or even if he didn’t comment his code enough ( I have clear rules on that) i simple message him the second I see it and he gets an email about it.

Another use I have for this feature is commenting out new features in the system, If I (or another team member) added a new code to the system, I usually post a comment to the commit and say what I added and what are the usage for it, this clarifies the usage for the team and there are no questions.

If the team is not clear how they can use a feature I added they go to the message and there they will see example source code and comments about that feature.

Basecamp project dashboard

In the screenshot above you can see a project dashboard. This is for a specific project, you as the owner of the account have a dashboard for all the project you are running in the system.

With the morning coffee, you can review what you have to do urgently and what can wait another day, making choices based on data is always better then choices based on your memory or anything else for that matter.

Backup

If I could create a shouting voice in this title I would, your client’s code, data, files and everything you have at your possession is very important you your client, so you should provide a descent backup service for him.

The client must know, no data is being lost, no matter if your laptop caught fire, stolen, broken or any other natural or synthetic disaster that can be caused in our daily crazy life on this world

I my self have a double backup.

I use a free software called syncback to backup all of the data to an external HD.

Also, I use a remote backup using Mozy.com.

So, every night, my computer backs up twice, once to a close location (HD) and one to a remote location (Cloud service).

Of course, using SVN is having sort of a backup for the code, but I so another backup just in case, of course for all the other stuff that are not code like images, documents or anything else.

Sync

Keeping in sync with your client is very important, I’m talking from the files point of view right now.
I use dropbox on my computer and have dropbox installed for every team member, this way if we have a design document or a mockup or anything else, the file is being updated for everyone at the same time.

So, every time you talk to your team or a member of your team, you are talking about the same thing, you actually look at the same file exactly, so if a team member has a comment he can write it down and everyone will see it the next time they open the file.

This is a very helpful feature dropbox has and I love using it.

Dropbox shared folder

Time tracking

I almost entirely work on an hourly basis with my clients, so I have to use a reliable system to track my time.

When working on an hourly basis you must be very careful not to work on a client X project on client’s Y dime.

For that, I use toggl.com, this service is another cloud service to track your time, the clear advantage of that service is that it has a desktop based widget, you begin to track time, if for some reason you left your computer during that time, the widget will inform you on that error when you get back.

That way, a client only pays for time you actually worked, he is more comfortable with working this way.

You can also set custom rates for each client, you can export reports in PDF format and more, the client has a public report, meaning he can see at real time what you are working on and how much time you spent on this mission.

Toggl client report

If you use the paid account those reports are available to the client, also you can brand it with your logo so when the client’s print it out it will look nice as well.

iPhone

I recently bought an iPhone, so here are some thought on how I use it to give a quality service.

Mail –> always available, whether I’m traveling or I’m out of my home office.

ToDo–> always synced

Calendatr -> synced

Dropbox -> synced

many many more feature such as to do lists (app store) and many more, I used blackberry and Nokia E71 before, I must say I really feel I am more productive when using the iPhone and more in touch with everything when I travel around.

iPhone

That’s it for now

part-2 coming soon…

I’d love to see your comments

It hit waves around twitter and facebook and got very popular over the weekend I wrote it.

I was also mentioned in their blog so here’s a link

Thank you guys for the mention and thank you for this really amazing product I use daily.

Whether you are using GIT or SVN, both self hosted or in the cloud.

Me, I’m a huge fan of cloud services and moving things and services into the cloud, managing my entire work (almost) in the cloud, from my email (GMail) to my backup service (Mozy) and now, SVN.

I will tell you about my solution in this post along with screenshots of what it looks like

Just to give you an idea, I’m talking about a solution on steroids, the sort of solutions that makes you feel like: “how the hell did I do this up until now”, this solution will work amazing for you if you are working in a team or working alone.

I do both, and I absolutely love this solution.

Like I said in the title, the solution is based on 2 cloud services

1. Beanstalkapp (link) – hosted SVN with an amazing web interface, user management and more
2. Basecamp (link) – hosted project management system from 37Signals.

The concept of the solution is simple, see what’s being done, by who, when and why, all this inside the context of a webpage with a link from the project management system back to the SVN and vice versa.

Also, you are able to track time being “wasted” on a mission, everything from within the comfort of the environment convenient to you.

You won’t need to change anything in your habits, if you are used to using a command line, you will use the same command line with some extra commands to track time and do other things.

So, let’s dig into just how you can do this:

First, open an account in beanstalk, the pricing range is convenient and I found myself rising from the trial period to my current account in no time.

Second, go and open an account in basecamp (this is not a must, just an extra).

In basecamp, find your account details and grab your API token (you can regenerate a new one at any time you want)

ok, we are set to go.

When you log in into beanstalk you will see the repositories tab on the top, when you click it, you can see all of your repositories.
it’s important to understand, each of your clients/users will see different things in this tab, meaning there’s a permission system, if you give access to a client he will see his projects only.

Now, click “Create Repository”
This screen will come up

Select a repository name (you will not be able to change this in the future so select it wisely)

After you give a name, you will be redirected to the next phase in the wizard, here, you can import your working copy (if you have one).
This is actually a very important feature for me, all of my SVN was self hosted and I wanted to keep the repo consistent and be able to revert back to older revisions of files and more.

After you import (or not) you will be redirected to giving permissions to clients/users.
you can skip this phase if you don’t need it or you don’t know yet, you can change the permissions at any time in the future.

You can also select (of course) the type of permissions the user has.

That’s it, your repository (first one) is setup.
now, you can click the activity page and you will see just a blank (or something else if you imported data) page.

This is your activity page and it will get filled with each commit user does.

On the right side, you can see a textInput where you can copy and paste he repository URL for this repository.
This url is the one you type into the SVN tool you are using, whether you are using the command line or Versions (mac) or even tortoise SVN (windows).

Cool, so let’s create a connection to basecamp now.

Let’s click the setup tab in the top.
We will get this screen

As you can see, basecamp is my option of choice but you you can select any service they have to offer, from twitter to freckle and many more.

What’s next.

type your basecamp url, paste in your API token.

After you type in the details and click next you will be able to select a project from you project management system.

After you select a project you’ll be able to select the category to which every commit message will be transmitted to.
I always select “code”

I’ll explain a bit.
When you commit a revision and type a message, this message will be posted to the “messages” section in basecamp.
The messages section contains every message users post, so the code revisions and every other message is in once place, for me this is amazing also because I can reply to a message.
When a programmer from you team posts a commit message you can reply, giving him anything from a feedback to rejects.

Also, you have a link from basecamp to the changeset in beanstalk.
you don’t have to exit, login and look for the latest change, it’s just a click of a button away.

Let’s see how it looks.
When I commit a revision it looks like this:

and this is where I go after I click the link inside basecamp

Also, you probably know that I do love to code .net and create amazing application with these technologies and frameworks ms are putting out into the market for years now.

Frankly, I’ve seen the myth’s about .net being written about, talked about and the buzz is out there. .net is being trashed talked at about every open source community out there. recently I started coding with Ruby On Rails and I was absolutely shocked by the myth’s that are going on in that online community of developers.

So, in this post I will take each point and act as a myth buster it’s going to be interesting…

Before I begin writing the myth’s and bursting them one at a time I want to say that in my opinion it is totally Microsoft’s blame, most of them come from earlier versions of .net when MS did a few (or more) mistakes and the myth’s stuck since then.

If I code asp.net my code will only fit IE and will not be in the new standards

You probably guessed it but this myth is not true, it exists since the beginning of time with .net 1.1 and the controls you dragged and dropped into the HTML which created ugly HTML.

Since then .net has grown, nowadays you can write you own HTML, every control that exists in web-Forms like the grid or data-list are now creating totally customizable code which you can write css for and have your own way of work with.

.net mvc did even better, you can write you own templates, even the default code generated by the templates is OK, it’s not great but it’s not ugly as well, comparing it to the code the Scaffolding feature of ROR created it’s pretty much the same stuff.

If you know what you want to create and you don’t use Controls when clean lean and mean HTML is needed your code will be clean, readable, validated and you’ll love it.

.net is SLOW

this one actually makes me laugh every time, .net code is compiled (if you want) and if you know what you are doing the code is super fast, super efficient and in tests I’ve made and in test @misfitgeek published on his blog .net wins in most cases.

You can argue both ways here really, you can say the tests were made on ms servers and not on linux but that doesn’t change the fact that even if there is a difference it won’t be noticeable to your client’s.

If you keep your code clean on server side, if you intelligently dividing you calls to the database so you won’t access it 200 time on page load (seen that happening I swear) then your code will be fast.

The ability to work side by side with SQL (all versions) is also an advantage with speed.

I have to work with an expensive IDE

Well, this is also a very ugly myth meant to discourage the use of .net. Today MS released an amazing IDE that is totally free of charge it is called Visual Web Developer Express and it is a free version of Visual Studio (with some features missing).

You can work with this IDE on small or on huge and complex websites, it does not limit you to the level that you can’t do something, some things are harder than with Visual Studio but it’s definitely possible.

I’ve used this IDE for about a year and I enjoyed working with it very much, when comparing it to other environments and languages which IDE’s cost hundreds of dollars it wins every time.

B.T.W, Visual studio and Visual Web Developer are in my opinion the total best IDE’s out there, the experience of development and debugging is absolutely amazing and easy.

I can’t be agile with .net

hmm… if someone will explain this myth to me I’ll think i’ll bust it in a second , you can, you can, we can, yes! we can!

you can be agile when developing .net just as much like when you are using ruby or php or any other language or framework.

with .net development is slow

HA, .net development is not slow for a very long time now, with .net MVC, nHibernate, code generation templates, LINQ you can be super fast, super efficient coder.

just take a look at the new MVC screencasts on the asp.net website by microsoft, sure, the screencasts never show the best practice and rush to coding just to explain the point but, you can code just as fast with Interfaces, TDD and every other best practice you may know.

Imagine yourself as a commander in the army, when you go to a mission you have to choose the soldiers carefully right?

right

well, net is now exactly like that, if you go to a mission and select the right troops your coding experience will be best, if you don’t you can expect problems, it’s the same in ROR, PHP and others…

Conclusion

There are more, some of them aren’t worth taking the time to answer some I don’t even remember but every myth can by busted especially with the rapid and ever developing .net.

if you have a comment, feel free…

Matt Cutts, probably the only guy in the world that can discuss the google algorithm finally breaks the silence about it and go out into the open and saying this…

DO NOT break you head struggling with getting your code to validate, I will exaplain some more about my opinion in this post

Well, I’m a clean code kinda guy, I like to write clean lean code both on server side and on client side, although I do not put all my efforts into getting that green light from the W3C validation.

WHY?

Well, it’s mainly because I believe (and now my beliefs are backed by Google) that valid code does not push you forward not in search results and not even in client flowing to your website.

I’ve seen websites in the last 10 months or so that are build with tables but validate, these sites had the W3C valid icon on the footer and that is just plain stupid.

WHY?

Because semantic code, not using tables for building the website, creating the proper tag for the right place and more are more important for me. (and for you)

Accesibility is a bigger issue in my opinion, I would love to see labels being coupled with a inputs and fieldset surrounding logical parts of your forms and more.

SO?

Well, first let’s say that I’m not saying you should not validate your code, you should pass it through the validator, even to just check that the images you have are with alt tags and that you don’t have duplicate tags and stuff like that.

Balancing…

You should really balance the amount of time you invest in getting your code to validate, take some of that time and put it into actually writing semantic code.

Think about the accesibility of your website.

What I do every single time is: I’m using firefox to disable the css and js of my website, I take the time to see whether my website still remains in logical structure that I intended it to be.

I check to see if I have the links in the right place, If I can navigate my way through the website using the keyboard.

When I have the demand, I also check the website blind-folded (literally), I check the website with a screen-reader and listen to it.

If I would have invested all of my time to see that green light I promise you some of these checkpoints and tests would have been unlooked.

I’d love to hear you opinion on this, feel free to comment or even reply to me via Twitter

Well, this was exactly the case with a client of mine, the client is a forex company, they had 2K concurrent users and they currently in the process of redesigning the client application and plan to grow up to 20K concurrent users.

This is a classic job for me, this is what I do, it’s a different company with different needs, so, in this post I will give you the solution I gave the company without revealing details about the company of course

What do you need to deal with?

As always, I started with a pen and a paper, sitting down with the company’s CTO, understanding the need of the software from the hardware.

This is a most important stage for me, learning to listen to the guys that hired you for the consulting is not always easy but let me tell you this, they know their product better then you, even if the design is not the best, even if it has flaws they know it better then you will ever do in the short time you are there for consulting.

OK, after that being said, let’s continue…

So, the software needs are like so:

• messaging from server to client
• latency is out of the question
• custom load balancing based on decision (20% load is OK or 80%???)
• amount of messages –> 2-4 messages a second per client

As you probably understand, these demands are huge for a 20K concurrent users application. I accepted the challenge and went to designing the solution.

planning

This application is based on high volumes, the volumes can vary at times. because this is a forex based application there are times that the messaging is dead meaning there is not much movement and there is time that the application should supply 2-4 messaged a second to each client.

And another surprise: the data is personal so you cannot use a shared cached object. delightful

Let’s attack it each section at a time shall we?

Messaging from client to server (push)

I chose to use MSMQ based application, basically you push messages into a queue of messages, the messages are then pulled from the queue by the appropriate application or in our case (as you will see next) appropriate server.

custom load balancing based on decision

Well, this is a tricky one, usually what you do in the client application is hard-code a server URI inside your code, this means if you want to make proper scaling you should create different versions of the application for each server right? WRONG!

The solution is like so…

Inside the server cluster there will be a “decision server”, this server will manager all the other servers in the cluster.
Each time a user is trying to connect, the application first tries to connect to that “decision server”, this server looks at it’s DB (you built) and sees the load on each server, the server then decides based on the allowed load which server he should lead you too and simply gives you the URI, the application takes this URI returned as a string and connects.

No Worries, I will explain more, but this is exciting ain’t it??

amount of messages –> 2-4 messages a second per client

Well, we will see about that in a second.

The solution (Explained)

So, let’s explain this for you.

The decision server

The decision server is just an HTTP service connected to an SQL server (can be express), this SQL server holds the servers it has in the cluster and a connection count.

The connection count is calculated to decide on the load, this is just simple math and based on experience with the system, the system can learn and the system admin can teach it.

If the server failed after 2000 connection then this is 100% load, the admin sets the system for 40% that, or whatever other number he wants.

hmm… interesting right.

Right, this is a custom service and so simple it’s unbelivable, now, you are probably wondering but this is one server handling all the load right?

Well, no, this server is kicked into action only the first time a user connects to the system per session, after the decision server returned the URI to the users, the user will never get to that server again unless he disconnects and tries to connect again.

The messaging server

The messaging server is based on WebOrb.net excellent messaging application, this application can conenct to a queue of messages, whether the queue is local o remote.

This is actually a pretty intuitive ready solution for you, you don’t have to custom program anthing, the WebOrb application is pretty solid and handlers a large amount of connections per server

The client

The client application is stupid, it has no logic to select the server, it connects to the server (decision) and get a URI, after that URI is in the application it holds it in cache or as a static variable so it won’t have to go back to the service for it.

Single point of failure?

As you notices (or not) there is a single point of failure in the system right?

Well yes, there is, the decision server is a single point of failure, if it fails to load or fails to return a URI the system won’t work.

To overcome this issue I simply added another server as a fallback, if you connect them with a load balance soltion you get also load control but that’s (at least for my solution) is an over kill.

Testing

When testing this cluster / server / client solution we got amazing results, the servers are at 20% load, the scaling is horizontal and you can connect another server any time you want thus creating another place for your users to connect to and instantly enabling more connections to your service.

conclusion

This was a very good project for me, I succeeded with my mission and I hope you took some info from what I did.

feel free to comment.

If you want to lear more about the series, please head here and read.

Ok, so this is the post about the first meeting.

Usually when you come to a consulting meeting you know where you are going, you know what the client wants and you can better prepare yourself for it.
This meeting was different, the client had many questions, he wasn’t sure what he wanted and how he wanted to do it.
Because the client is a startup company which deals with a lot of money, and has somewhere between 20-30K users, there is no rush in their behalf (unbelievable I know), they want it the best.

So, I had no preparations as to what the want, I knew the general idea, the have an html client, they want to transfer to flex. that’s it.

When I got there, we started talking.

The first question that popped into the room was the communication method they should use.
should they use AMF/xmlSocket/Http service or what….

So, lets break it down.

Each method has pro’s and con’s.
Flex has the advantage of not limiting you into a single method, you can use a couple or more methods of server side communication in a single flex application.
You will need a very good team of developers and a superman project manager, but it is totally not an impossible mission, it can and was already done before.

So, lets dig further in…

Before stepping into the pro’s and con’s lets handle what they want to achive (again, without compromising any data or even the company name).

They have users, the users are divided into groups, each group can contain X amount of users (x is a variable not a constant )
Above the group is an organization, an organization can have X amount of groups.

The data in the group is personal and cannot be compromised to other groups or users, the same with users of course, no data should leak between users, groups or organizations.

The calculated data is group specific, meaning each group has their financial data, the calculations are based on their data and their data only.

Calculations, ok we are getting there…
Question by me: Does the user/Group/Organization need to do an action in order of the data to be calculated?
Answer: no, the calculations are done periodically (1 time per second in average) and need to be sent to the client without him even knowing about it, the data should be updated on his screen.

OK, sound about right. a well known scenario, you we guys using webforms or any other request/response are probably scratching your head right now… don’t worry, its quite simple actually.

lets move forward please, lunch is almost here…

Question: what about other data on the screen, any other data needs this amount of crazy updating or should everything else be pretty standard?

The rest of the screen, containing somewhere between 10-20 parts should not be updated in this time periods, the other data is sometimes the result of a user action or a group action.

Question: should an action from a group be alerted to all the users inside it.
Answer: yes, sometimes the group did something that caused the users to loose/gain money so they should be alerted about it.

lunch

Client: the final thing we want to accomplish is the ability to reach every user/group/organization separately and send them admin messages about everything from a version update or a hosting change.

OK.

Great, I have my data, I asked my questions (I gave you a brief version of course). now let us talk.

1st – 1s calculated data.
Method of implementation.

Server – 1 calculating server based on Sql server 2005/2008 (we will not get into the configuration) a caching solution (sometimes the data is the same as before, why should the server sweat if the data has not changed).
2 – Messaging server connecting to that calculation server.
3. windows service using an xmlsocket connection based on clientIP, groupID, organizationID, when the data for an entity of those has changed you simply push the data to them, if the data has not changed you send out nothing, the GUI at the client will not change, the NET traffic will not overflow.

This way you keep it simple, you calculate the data on server 1, you send the data using server 2, connection to server 1 using a windows service which you can monitor performance.

Pro’s: very rapid communication, no overflowing the servers with requests, if the data has not changed nothing happens.
Just for a reference, the client implemented the same feature in their current application using a 1second JavaScript timer and a service.
Do you want to guess what happened? (hint: guess who crashed first, server or client)

Con’s scale up in this solution is not always easy and it is not done only in software, you need some hardware changing too, though in the beasts they have over there, they have time to worry about that.

2nd communication – client or group does something and data should change in the GUI.
For this solution I think the webOrb.net solution would be perfect, you get out of the box push in HTTP while the data changed, you don’t have to write any custom code for it.

The WebOrb.net is a solution that lets you use AMF3 or AMF0 or RTMP communication with a flex client. It is very robust, easily configured and even has a built in monitoring solution.

Pro’s: The weborb solution is hosted on top of IIS6/7 so you can configure it with flexibility of your choice.
you can write custom code, use the classes you already have in an asp.net application or generate a new application from scratch using the code generation solution.
scale up is very easy, using a load balancer and 2 servers you multiply the users that can use the system at once.
Clouding: you can pass this service into the cloud at AmazonEc2 or at any other clouding service and get out of the box scale up when needed.

Con’s: The push is not always in a heart bit, mainly when you are talking about 20-30K users.
as always in a 3rd party solution, you cannot actually do ANYTHING you want.

3rd Method: Admin Messages: because the messages are not meant to be urgent, (urgent messaging will be sent on the socket already connected) the most efficient thing to do is by using FileSystem on the server for every group, user, organization and have the flex client check it once every 24 hours using an httprequest to the file it is using.
That way, this solution is not using any database in real time and will not slow down the db which is well needed on other crucial actions.

That’s it….
That wasn’t all of the meeting, but that’s enough to chew on for this post, on the next post I will continue with the questions and answers that came up on the same meeting at this client.

Some of the subjects that we will discuss on that post:
1. Handling disconnect
2. Multilanguage
3. Skins
4. group work
5. svn
6. flex builder versions
7. graphic’s team brief
8. graphic artist–developer–project manager. why, how, when, who???

Actually this blog post is to answer a really nice guy from a forum (.Net) I run in Hebrew, but I thought it will be really a really nice blog post in general.

I get a lot of questions regarding the concept of using jQuery (in general) with .net and database data.

I actually blame Microsoft for having the tutorials in their website teaching only SqlDataSource which I never used and recommend that you don’t use too.

So, the majority of the questions I get are not at all about code, it’s about concept, how to communicate with data, how to retrieve data and more.

Ok, so this tutorial is all about jQuery, LINQ to Sql, and the jQuery tree.

The only difference is that I’m not going to display files in folders but pages from a Sql table, every page has a “ParentID” fields, so it will be very easy to understand.

This tutorial is also to clarify you can use the tree for any relational database or file data.

So,

Lets Start.

First, lets go and download the 2 packages.
jQuery
jQuery File Tree

We will open up a new website in visual studio.
Our files should look like this:

What do we have here?
Well, we have all the css from the jQuery file tree library, we have the Images also from this library, we have the js files from: jQuery, jQuery file tree and another js file which we created for our custom functions.

Alright, now let’s create our page:

We have all the script file and css files in our page, also, we wrote a div called “myFirstTree” in our page, this div will hold our tree.

What else do we need?

Well, like I said, we are displaying database data here, so lets create a very simple database with one table, also we will create a LINQ mapping file for it.
B.T.W, you can use any kind of data access, you can use nHibernate, or whatever you may find comfortable for your use.
I use linq for this example, don’t ask why, just felt like it

So, we have these fields:
pageID – Identity field
pageName
pageParentID

let’s create our data, just randomly fill our table with some data, and then we can go back and retrieve it from the client side using some jQuery.

So, we have our data, I don’t think i need to further explain this no?

Now, lets write some js code to retrieve the data from the server for us.

Our js code should look like this:

   1: $(document).ready(function() {

   2:     $('#myFirstTree').fileTree(

   3:     {

   4:         root: '0',

   5:         script: 'jQueryTreeHandler.ashx',

   6:         multiFolder: true,

   7:         loadMessage: 'Tree loading...'

   8:     }, function(file) {

   9:         alert(file);

  10:     });

  11: });

Let’s explain this code a bit shall we?

first, we chose our pre crated div “myFirstTree” and then we initialized the fileTree object with these parameters:

root: the first parameter that will be passed to our Handler

script: our path to the handler

multiFolder: whether our tree will be multi folder or not, when this is false, clicking each folder will collapse all the other folders in the tree.

loadingMessage: this message will be displayed when data is being retrieved from the server.

fileFunction: this function will be fired when a file from the tree is selected.

All pretty straight forward up until now, lets go on and create our handler:

the parameter we know is passed in is called “dir” so that is what we will use here.

So, this is what your handler code should look like:

   1: <%@ WebHandler Language="C#" Class="jQueryTreeHandler" %>

   2:

   3: using System;

   4: using System.Web;

   5: using System.Linq;

   6:

   7: public class jQueryTreeHandler : IHttpHandler {

   8:

   9:     public void ProcessRequest (HttpContext context) {

  10:         context.Response.ContentType = "text/plain";

  11:         int parentID = Convert.ToInt32(context.Request.Form["dir"]);

  12:

  13:         string list = "";

  14:         jQueryTreeDataContext db = new jQueryTreeDataContext();

  15:         var pages = from p in db.Pages

  16:                     where p.pageParentID == parentID

  17:                     select p;

  18:

  19:         foreach (Page item in pages)

  20:         {

  21:             list += String.Format("<li class=\"directory collapsed\"><a href=\"#\" rel=\"{0}\">{1}</a></li>", item.pageID.ToString(), item.pageName);

  22:         }

  23:

  24:         context.Response.Write(String.Format("<ul class=\"jqueryFileTree\" style=\"display: none;\">{0}</ul>", list));

  25:     }

  26:

  27:     public bool IsReusable {

  28:         get {

  29:             return false;

  30:         }

  31:     }

  32:

  33: }

Comment: I didn’t take care of a situation where a row has no children, it wasn’t necessary for my example. but in real life code you should, and if a row does not have any children you shouldn’t display a folder but a file instead.

and this is how our code looks like in the final result (Html output)

Comment: I had to tweak the jQuery file tree code in order for it to work with database data, normally it has a regullar expression validating the “rel” attribute actually contains data with “/”, that needed to be changes.

I uploaded the file to here for you, so download away…
jQuery file tree tweaked (269)