Gridder released!

Gridder has come to its first release!

Gridder is an open-source project consisting of a suite of portlets which are designed to enable users to submit jobs in a grid environment, without the need of an extensive technical background. It comes with a full manual and it also includes documentation produced during development, which addresses topics related to grid and cluster set up and configuration. Gridder seamlessly integrates with OGCE, as it was the focus during the whole development process.

Gridder was developed at a privately-held software development company in Argentina, where I work, and sponsored by a national scientific promotion agency.

Briefly, the Gridder project consisted of setting up a grid and a cluster from scratch, researching and applying that knowledge to some concrete development. That development was a set of portlets designed to simplify job submission tasks with end-user friendliness in mind, and a well defined separation of concerns from the user point of view (namely tasks and jobs), and from the technical point of view (namely the architecture and design). While there is a lot of improvement that can be made to them, you may find these portlets and the ideas they present useful.

Comments and suggestions are greatly appreciated.

Many people have helped with this project by replying to emails I have been sending to mailing lists during the last year. To all of them, and to the whole grid and cluster community, the most sincere thanks.

Gridder is also now open to adding members interested in actively participating in the project.

If you would like to take a look at Gridder, please visit the Gridder web site at http://gridder.sourceforge.net and the SourceForge project site at http://sourceforge.net/projects/gridder

Array of pointers or pointer to array?

Dynamic memory and pointers are core subjects when working with C. Sometimes, doubts that seem quite basic at first come to you. I recently had to determine whether a statement reading 'type *name[length]' declared an array of length pointers, or a pointer to an array of size length. The former, as intuition dictates, is the right answer.

The catch here is that modifier precedence matters. The array modifier has higher precedence than the pointer modifier. Thus, [ ] is applied to name and an array is declared. This results in an array of size length.

The declaration could be modified with the use of parenthesis, to be 'type (*name)[length]'. In this case, the pointer modifier is applied first, and therefore a pointer is declared.

There is a quite good explanation at
http://msdn2.microsoft.com/en-us/library/1x82y1z4(VS.80).aspx which I will try to summarize in the following lines.

*) Start with the identifier
*) Go to the right first: brackets have higher precedence that asterisks.
*) Whenever you hit a parenthesis, solve the parenthesis as a unit.
When you have to solve, for instance, the type of an argument, start
from the innermost parenthesis (this is from the second article I mention below).
*) Last, apply the type specifier.

Then, for 'type *name[length]':

The identifier name is declared as
(right) an array of size length of
(left) pointers to
(type specifier) type

For 'type (*name)[length]':

The identifier name is declared as
(right, hit parenthesis, solve as a unit) a pointer to
(right) an array of size length of
(type specifier) type

Parenthesis (not shown in my example) for specifying functions are subject to the same rules as brackets, have the same precedence and associate from left to right with them (that is, the one most to the left is interpreted first).

There another article worth reading at
http://www.codeproject.com/cpp/complex_declarations.asp

From the last article, I would like to highlight the recommendation of always writing the asterisk next to the identifier and not to the type. If you write 'int* p, q' you could be mislead to think that both p and q are pointers, however only p is. If you write 'int *p, q', there's no such confusion. For both p and q to be pointers, you would have to write 'int *p, *q'.

Grid computing - A definition

As I mentioned on a previous post, one of the areas in which I have been working on lately is that of Grid computing. But what is Grid computing? As a colleague at work once said - and I believe he was not the first I heard that though from - being able to state a definition of something is a big step in understanding that something. I will try to reproduce the definition in the following lines.

So, here it goes: A Grid is a set of several resources that can be used in the form of a single, powerful resource, so that it does not matter where exactly the power that is used comes from, only that it is there as part of that very powerful and big resource. Note that I use the word resource and not computer, I do that because with this technology other elements, such as sensors, may be shared. In a Grid, physical location of the resources does not matter, nor does it matter that those resources be heterogeneous.

The last two aspects I mentioned seem to be what distinguishes a Grid from a Cluster. In a Cluster, usually all resources are homogeneous, e.g. all have the same operating system. This is not a restriction in a Grid. Also, a Cluster is usually placed as a whole inside an administrative domain, i.e. managed by a single authority (I would like to get further explanation on this), whereas a Grid can expand accross different administrative domains. In practice, what all this means is that a set of computers from anywhere in the world, belonging to different people, and running different operating systems, can have its power combined and added up into a Grid structure - and you cannot have that with a Cluster. In addition to all that, it can be noted that in a Cluster environment there is a master, whereas in a Grid environment all resources are peers.

I was just re-reading a definition from GridCafé (link some lines below) and note that it states that a Grid is aimed at sharing computing power and data storage capacity. This sets me to think if anything else could be shared. It also mentions that it is mainly directed to sharing over the Internet, though I think uses could be found for any internet (lower case i).

In order for a Grid to work, a middleware is used. A middleware (yet another definition I will intend to give here) is a layer of software placed above the operating system level, but below the application level, that makes something heterogeneous seem homogeneous and provides services to make use of it.

The most known middleware software available for grids is the Globus Toolkit (from now on, Globus). Globus is an application that provides a set of services that solve common tasks needed in a Grid environment, such as execution management - e.g. job submission -, data management - e.g. file transfer -, security-related tasks, and information services - e.g. resource monitoring and discovery -. It exposes those services as APIs that can be used from higher level applications.

Grid computing - for the sake of formality - could be defined as the use of grids.

There is a great place for Grid newbies called GridCafe, it can be accessed at http://gridcafe.web.cern.ch/gridcafe/

A last remark, that would deserve its own post: what can a Grid be used for? Well, I was reading also about this from one of IBM redpapers at http://www.redbooks.ibm.com/ called "Fundamentals of Grid Computing" (specific link is http://www.redbooks.ibm.com/abstracts/redp3613.html). I will mention only one aspect here: resources are usually dimensioned on a most-demanding case basis, for each organizational unit - think of them as being silos -. But that capacity is not used at its peak all the time in every organization unit. What if you could take advantage of idle capacity in one place, when at another place you are running out of resources? Well, your overall need for resources (in total) would decrease.

Visual Basic Fusion - Interfacing VB6 and .NET

This came to me in a MSDN Flash email. VB6 applications can actually use, with severe restrictions, .NET classes. The original article, in Spanish, can be found at: http://www.microsoft.com/spanish/msdn/latam/articulos/2007_06_18/default.aspx

VB6 has class modules, but does not support a true object model. Thus, .NET classes that want to be used from VB6 have restrictions.

Briefly:

1) There must exist a default (no parameter) constructor.
2) Probably the most difficult to bypass: data types are different in VB6 and .NET.
3) Static methods cannot be accessed.
4) Overloaded methods are arbitrarily appended numbers to their names so that they can be told from each other, when using .NET classes from VB6.
5) The assembly must be available in the Global Assembly Cache (GAC).

The solution proposed in the article consists of writing .NET wrapper classes for all .NET classes not abiding by those restrictions, and exposing these wrappers in turn to VB6. Therefore, such a wrapper class will have to add a default constructor if there is not one available (and defer true object creation adding an initializer method that takes the parameters originally intended for the constructor if needed), add methods that in turn call static methods so as for the call to seem to VB6 as being made from an instance of the object, add methods that rename overloaded methods to appropriate - and different - names, and over all, convert between data types as needed. I tend to think all methods will end up returning quite primitive data types, because if you have object composition in your .NET object model, how can you tell VB6 what the object you are returning from a method - supposing you did so - is mapped to in VB6? There is the possibility of using the Object data type, which according to the article is automatically converted between VB6 and VB.NET, still the question of whether this would do remains for me.

When these restriction-compliant classes are available in the GAC, they must be registered as COM objects so that they can be seen from VB6. This is done with the regasm command.

All in all, it does not seem like a simple task, but it could be useful in some cases when there is a legacy application and some very specific feature from .NET is needed.

Making UIDs and GIDs equal for a user having accounts on several computers

Some time ago, I had some trouble when mounting a directory from one linux computer onto another. I needed to simulate a shared file system so I could get Globus GRAM PBS jobmanager to work on my installation, composed of the two Linux nodes forming a cluster. As I did not have such a file system, I managed to get things to work by having the same user on both computers and exposing the .globus directory from the user's home directory in one of the nodes using Samba, and mounting that directory on the other node.

The thing is, when you have a cluster it makes sense that the home directory be shared (eg. using NFS, though in this case I used Samba). It is also usually the case that the same /etc/passwd is used, or NIS or LDAP based solutions are set up.

When the /etc/passwd file is shared, or in some other way the UID for each user accross the various computers is the same, sharing a directory as I wanted to do works fine. The problem I had was that I had set up a user on both computers for my tests, but the UID assigned to them was different on each one. So, when I mounted the directory I exposed on one computer onto the other, listing the files would show me the wrong owner on the latter. Actually, it would show me the name of the local user mapped to the UID that my test user had on the computer that exposed the directory - that is, the other computer.

Because of that problem, I had to resort to making the UID and GID for my test user equal on both computers. To achieve that, I found this interesting link: http://docsrv.sco.com:507/en/OSAdminG/uaT.chguid.html

Briefly, I first changed the GID for my test user to a new group id, so as for this GID to be the same on both computers (depending on the scenario, it may suffice with issuing the GID change command on just one computer, if the GID used on the other computer is available). To do this I used the command 'groupmod -g newGID groupName'. Before doing this, the previous GID number must be noted down somewhere as it will be needed later.

It was then necessary to update the GID of all files that previously referenced the now obsolete group number. This is done by finding all the files that reference the old GID, and issuing a chgrp command on them, executing ' find base_dir -group oldGID -print | xargs -t chgrp groupName'. If the user only had files inside the home dir, setting base_dir to that home directory should be enough.

Then, the user id must be updated. Again, depending on the scenario this may need to be done on one or both computers. The command ' usermod -g groupName -u newUID userName' changes the user id and sets its primary group membership to the appropriate group. The old UID must be noted down for use with the next command to be executed.

As with the GID change, some files will now be orphan. You can spot an orphan file because when you list it it will show the UID, i.e. the number, instead of the user name (the same applies to groups). For files inside the user's home directory, the UID will be updated automatically. For other files, it will be necessary to issue a 'find base_dir -user oldUID -print | xargs -t chown userName' command, which will find all files referencing the old UID and change their ownership to the user, thus effective updating the UID.

Now, when mounting the directory with 'mount -t smbfs -o username=userName,password=password,workgroup=WORKGROUP,uid=userName,gid=groupName //shared_dir local_dir' the right user and group will be shown on the computer in which local_dir resides. The uid and gid options are needed if the mount command is issued as a user other than userName, so that mounted files are owned by the user used for the tests.

Deleting Tomcat work directory using WST in Eclipse

Today I was struggling to get a Tomcat server to launch. I had performed a heavy update to the project and even though apparently everything was OK, I kept getting an InvalidClassException in the initNonProxy method of ObjectStreamClass.

I rebuilt the project (maven eclipse), refreshed it, cleaned it, nothing seemed to work.

Finally, I deleted Tomcat work directory and everything became fine again.

The Tomcat work directory is located in the .metadata\.plugins\org.eclipse.wst.server.core directory of the workspace, I just deleted a directory which name started with tmp (in my case, it was tmp3). After launching Tomcat again from Eclipse, the directory is recreated, but no leftovers from previous executions get in the middle.

Hello, World!

Hi all,

This is my first post in my newly created blog. I intend this blog to serve as a way of organizing the knowledge I have gained in the latest times while working on a variety of projects. I hope it will be better than leaving scattered notes everywhere, plus it's pubilc so I can share it with others, and it's more easily searchable.

As time permits, I want to transfer interesting notes and tips into this space, also adding new ones as I keep gaining more experience.