Monday, October 25, 2010

Detection of smoothness/Sluggish behavior of an UI operation via Auotmation

Hi, in this post i have tried to find out the solution of detecting the sluggish behavior while doing the operations on 3D objects via UI automation.
Before we start lets clear about what I meant by sluggish behavior - Sluggish behavior is something we found while dragging an object in an application from one point to another point and the dragging is not smooth OR mouse operation take some time to to show the effect in the application.
I thought of following steps to detect the above behavior visually via UI automation:
Steps (e.g. with dragging the object from one point to another point):
1. Set the position of mouse from and to Points for dragging.
2. Store the point of To (means end point) in above steps.
3. Store he color of the point in step2.
4. Perform the drag operation to end point.
5. Store the color of point on step 2 in a separate color variable.
6. Compare the Step 3 and Step 5 color values.
7. If Step 6 comparison result is same then there is sluggish behavior otherwise operation could be considered as smooth.

Example in C#:
Class Automation
{
//Set the cursor position
[DllImport("user32.dll", CharSet = CharSet.Auto)]
public static extern int SetCursorPos(int x, int y);

//Get the color
[DllImport("gdi32")]
public static extern uint GetPixel(IntPtr hDC, int XPos, int YPos);

[DllImport("User32.dll", CharSet = CharSet.Auto)]
public static extern IntPtr GetWindowDC(IntPtr hWnd);

//To perform Mouse drag
[DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
public static extern void mouse_event(long dwFlags, long dx, long dy, long cButtons, long dwExtraInfo);
//mouse button
public const UInt32 MOUSEEVENTF_MOVE = 0x00000001;
public const UInt32 MOUSEEVENTF_LEFTDOWN = 0x0002;
public const UInt32 MOUSEEVENTF_LEFTUP = 0x0004;

static void Main(string[] args)
{
//Set the start point on object to be drag
System.Drawing.Point startPoint = new Point(10, 10);
//Set the end point of drag to.
System.Drawing.Point endPoint = new Point(150, 100);

//Get the color of end point
IntPtr dc = GetWindowDC(IntPtr.Zero);
long colorBefore = GetPixel(dc, endPoint.X, endPoint.Y);

//Perform drag operation using mouse
SetCursorPos(startPoint.X, startPoint.Y);
System.Threading.Thread.Sleep(100);
mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0); //hold
SetCursorPos(endPoint.X, endPoint.Y);
mouse_event(MOUSEEVENTF_MOVE, 0, 0, 0, 0); //Drag
mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, 0); //Release

//Get the color of end point for comparison
long colorAfter = GetPixel(dc, endPoint.X, endPoint.Y);

if (colorAfter != colorBefore)
{
Console.WriteLine("Drag operation is SMOOTH");
}
else
{
//Sluggish behavior
Console.WriteLine("Drag operation was not SMOOTH");
}

}
}

Sunday, October 3, 2010

Comparing images based on objects contained within an image

Hi, in this post I am writing the algoritham to compare images which is much better than tha conventional image comprison.

Convetional Way of comparison:
An image has a background and foreground. Generally we do image comparison to find out similarity or difference by comparing images pixel by pixel from (0,0) position to end of the last pixel of both the images. Also, in such kind of comparison there is one more restriction of images should be of same size.

With my appraoch:
With this appraoch we would be able to compare two images which are of different size and foreground objects are different only in form of displacement.
The restriction in this approach is that the foreground objects in an image should have same morphology as in the target image. The only difference expected between two images is size and displacement of foreground objects.

High level algorithm :
1. Accept two input image which are the candidate for comparison process
2. Extract all the objects from image-1 and image-2.
3. Crop those images from image-1 and image-2.
4. Resize all the objects in image-1 to make it of same size as in image-2 for better comparison result.
5. Compare all the objects in image-1 with that of image-2 to find out best match and output the difference and similarity.
I. Compare objects by conventional method of pixel by pixel comparison OR/AND
II. Compare objects by subset identification comparison method. (Note: in subset comparison we try to find out if one object is a part of the other object)

flowchart of the image comparison algorithm :