Anil Saha

The agile experience

2010-03-18T01:44:00.000-07:00

There are hundreds and thousands of literature available on internet about agile methodology. So, I’ll not dive myself to explain what it is and what it is not. Rather, this blog entry is purely my own experience on agile methodology. Couple of years ago I joined Financial Market System Development team of Standard Chartered Bank in Singapore and I was introduced with agile process. After working with traditional waterfall model for last few years, agile was like a fresh air to breath. I was so excited that I named my blog as “agilej” and in my dream I was about open a fast food chain in India with a name of agile food (read fast food!!).

Jokes apart, it was quite interesting to work with a model which has some pragmatic ideas to overcome the drawbacks of waterfall model. But, as we know, nothing is perfect in this world. If there are something wrong in waterfall, then of course it has good credentials as well, otherwise the majority of IT industry would not have opted for it. At the same time, agile is not Mr. Perfect. It is very young approach and there are so many speculations about it. The choice of the approach completely depends on what kind project we are doing, how much funding we have and how many good and smart developers are with you. Those factors drive the decisions to go with waterfall way or agile way.

Before I start, here are list of abbreviations I’m using in the rest of the article:-

IT :- Information technology
BA :- Business Analyst
RA :- Requirement Analysis
BRD :- Business Requirement Document
FS :- Functional Specification. It is enhance version of BRD, typically written by BA.
DD :- Design Document
HLD :- High Level Design
LLD :- Low Level Design
CR :- Change Request
UML :- Unified Modeling Language
CI :- Continuous Integration
TDD :-Test Driven Development

Age old waterfall model is mature enough and there are some visible pitfalls. I’ll address on by one.

The complex team structure is the main bottleneck. It opens lot of loopholes which can be political instruments within the project teams. Normally, in technical side, we’ve team lead, module lead, technical analyst and many more jargons you can imagine. The developer is the poorest guy in the team, who is treated like code monkey and although they are doing one of the important job, they hardly get the due respect, either in terms of pay or appreciation.
The mode of communication is liner in waterfall model. BA is the middle man between Developers and Business. Developers hardly get the opportunity to interact with business people. That is a huge gap in the process. The job of BA is very critical, who translate the business requirement into something which the user want to do by click of the mouse. Unfortunately, the translation is done correctly in many cases. I’m not saying that BA is incompetent, but the communication will be much more productive if you include developers in all the decision making process. Business people may need some functionality which may be technically challenging; the developers are better person to understand than the BA.
What is the difference between project manager and project lead? If you google it, you will get some theoretical answers. But practically are those roles are not identical? That is also one of the problems: too many managers. And here agile is very lean.
There is no doubt that documentation is as important as the binaries. But, the waterfall model put so much stress on documentation, that we are drifted away from the real software product. Documents like BRD, FS, DD (some even go for HLD, LLD) can be done in smart and lean way. We do not need so much garbage. UML is the best tool to document all the necessary artifacts. That will reduce hundreds pages of FS into few sheets of use cases, activity diagrams, which are easier to understand.
The initial phases of waterfall model viz. RA and design are treated like honeymoon period of the project. People just talk and talk and write something on the functional specification, which changes after one week. Fortunately some organizations do some static prototyping of the delivery, but most company depends on the abstractions of BRD or FS. It is assumed that other parties involved in the project will understand that collection of sheets. And honestly speaking that happens very rarely.

Now, enough shit about waterfall model. Time to give some claps for the agile process:-

In a typical waterfall model, the business gets the taste of the software in UAT, where user will test whether the product is acceptable or not. I worked with a bank before with a huge retail banking project. It took almost a year for RA, six months of design and six months of development. So, a requirement which was quoted by user one year ago in the BRD, he can see it after twelve months. And hardly there are cases that they will be happy. In today’s financial word, one year is a huge time. So, if the user is not happy with UAT (which eventually happens!!), a lot of extra pain is gifted to the project team. Agile is more pragmatic in this regard. In agile practice, there no clear separation RA, design and development. Unlike waterfall, agile deliver a workable running piece of software in an iteration. Iteration could be of one or two weeks, depending on the complexity of the delivery. So, after user came up with a requirement, the BA and developer can deliver that in iteration. Business can test it and give their feedback. BA and Developers work together to implement that. So, this cycle will reduce the chances of getting surprised.
CI is very powerful feature of agile. Unlike waterfall, where a separate team is created to look after build and deployment process, agile automate it by using CI products like atlassian bamboo and many other. As I told, agile deliver in iteration and believes in software as evolving process, we need to deliver very quickly. We can configure the frequency of build and deployment
Agile gives importance to people over process .Unlike waterfall, developers get the deserved respect. They are part of the important decision making process and can talk with the business people directly, if needed.
Nice collaborating platforms like Mingle or JIRA are very powerful to show the current status of the project and who is supposed to do what. Every stakeholder of the project and login to system and can see what is on his plate.
Agile community vividly speaks to replace heavyweight BRD/FS by UML diagrams and unit tests. Tools like Enterprise architect are very powerful which can be used by any stake holder of the project. Also, TDD approach is very practical where the unit tests encapsulate the little tiny functions. There are powerful tools available for code coverage, static code analysis to produce nice and healthy code.

Some people in IT community see agile as a revolution in software development process, something like October revolution. Lots of people are coming up with some new ideas to shape agile process. Some says, we don’t need any testers, unit tests are enough; some says don’t do documentation at all. In http://www.infoq.com you can see a lot of blogs, articles.
The bottom line is waterfall will coexist with agile. May be new development projects will give a try to the agile methodology, but maintenance and enhancement projects will keep on using waterfall model. In my opinion use agile process if and only if all the stakeholders of the project understand what it is and what is expected out of it. Don’t just jump into agile, because your friend in other company is doing it. More and more organizations are going to adopt agile in future, but we can not erase waterfall model immediately. Big firms like financial institutes, which are globally distributed, have offshore onsite model, may find it difficult to switch to agile. Although there are agile models for distributed systems, but I know how much effective they are. Let see how the agile process is shaping up.

Apache HTTP Server Installation and Configuration on Red Hat Linux

2009-11-12T18:41:00.000-08:00

In the early days of my career I had the impression that apache http server is a part of apache tomcat server. But those are completely two different products.
I’m very much familiar with tomcat, but never had any chance to install and configure apache http server from scratch. In my current project, we wanted to secure our application by switching to https from http. The application is running on Jboss. So, there are two choices: - either configure Jboss for https or use apache http server to do that job. If the apache http server is configured to https, it will redirect the requests to Jboss. So, we opted for the second option:- to use http server.It was not an easy task for me, but finally I managed to accomplish it with lot of googling and
trial and error methods. In this blog entry, I‘ll share my experience.

Apache HTTP Server Installation

First we need to install apache http server. If you are installing on windows, it is very easy. Just few clicks and you are done. But, unfortunately the actual pain will start once you change the httpd.conf file for SSL configuration. The apache http service will not start with some stupid message and you are stuck. So, my advice is to download the apache http binaries for Linux or Solaris boxes. I did on Red Hat Linux box, so I’ll note down the steps:-

Download latest version of binaries (I used httpd-2.0.63.tar.gz) from apache http website.
Copy it in /tmp directory.
Extract it using the command tar -xvzf httpd-2.0.63.tar.gz
A directory httpd-2.0.63 will be created.
Go to the directory /tmp/httpd-2.0.63
Execute the below commands step by step:-

1) make clean

2) ./configure --enable-rewrite --enable-ssl --with-ssl

3) make

4) make install

Note that you need root access to execute the above commands. The above process will create an Apache server in /usr/local/apache2 directory.

So, apache is ready with SSL and rewrite modules.

Private Key and Certificate Generation

Now, we need the keys and certificates for the SSL.JDK ships with a utility called keytool which is used to create and manage encryption keys and certificates. We need to do the following:-

Use the keytool utility to create a RSA key pair (public key and private key). Here are the sample command :-

[root@box sctool]# cd /tmp/jks/

[root@box jks]# /usr/java/jdk1.6.0_16/bin/keytool -genkeypair -alias myapp.alias -dname

"CN=myboxname.com, OU=Department, O=XYZ Bank, L=Singapore, ST=Singapore, C=SG" -keystore

mykeystore.jks

Enter keystore password:

Re-enter new password:

Enter key password for

(RETURN if same as keystore password):

[root@box jks]# ls

mykeystore.jks

[root@box jks]# /usr/java/jdk1.6.0_16/bin/keytool -list -keystore mykeystore.jks

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

myapp.alias, Nov 3, 2009, PrivateKeyEntry,

Certificate fingerprint (MD5): AD:85:96:0F:33:BB:35:BF:5B:0D:C9:92:21:1F:A2:61

Please note the name myboxname.com . It must exactly match with the host name of the box on which the application will be running. In, my case, myboxname.com is the hostname of the box on which JBoss server is running. The application URL will be something like https://myboxname.com/myappname. So, give extra attention while creating key pairs.

We need to generate a csr file from the public key.

            [root@box jks]# /usr/java/jdk1.6.0_16/bin/keytool -certreq -alias myapp.alias -file myapp.csr -keystore             mykeystore.jks
            Enter keystore password:
            [root@box jks]# ls
            myapp.csr mykeystore.jks
            [root@box jks]#
            The myapp.csr looks similar to below
            [root@box jks]# more myapp.csr

            -----BEGIN NEW CERTIFICATE REQUEST-----             MIICejCCAjgCAQAwdTELMAkGA1UEBhMCU0cxEjAQBgNVBAgTCVNpbmdhcG9yZTESMBAGA1UEBxMJ U2luZ2Fwb3JlMREwDwYDVQQKEwhYWVogQmFuazETMBEGA1UECxMKRGVwYXJ0bWVudDEWMBQGA1UEAxMNb
XlhcHBuYW1lLmNvbTCCAbgwggEsBgcqhkjOOAQBMIIBHwKBgQD9f1OBHXUSKVLfSpwu7OTn
9hG3UjzvRADDHj+AtlEmaUVdQCJR+1k9jVj6v8X1ujD2y5tVbNeBO4AdNG/yZmC3a5lQpaSfn+gE
exAiwk+7qdf+t8Yb+DtX58aophUPBPuD9tPFHsMCNVQTWhaRMvZ1864rYdcq7/IiAxmd0UgBxwIV
AJdgUI8VIwvMspK5gqLrhAvwWBz1AoGBAPfhoIXWmz3ey7yrXDa4V7l5lK+7+jrqgvlXTAs9B4Jn
UVlXjrrUWU/mcQcQgYC0SRZxI+hMKBYTt88JMozIpuE8FnqLVHyNKOCjrh4rs6Z1kW6jfwv6ITVi
8ftiegEkO8yk8b6oUZCJqIPf4VrlnwaSi2ZegHtVJWQBTDv+z0kqA4GFAAKBgQCtlvqZD0z9Jvjn
ShxfqpnXooCmP0ncVEog/yGYJ3irytzI0EERTKxeYq6tUCwUWlr1xq1worOp94m/ETJFJYPkAUkA
qaLorCStyhsD+tS+jmFa2nr1ywAErcZJ7qTDoYPWLXRB8jpo3fwzfSx66roSwPwarrWP1ZLP93+H
w9LZq6AAMAsGByqGSM44BAMFAAMvADAsAhQ50JZ11KPJ0S0ZN4BAF08btQfGYAIUNVFJCuiomeR3
99xnK/vS9tZ5/uE=

            -----END NEW CERTIFICATE REQUEST-----
            [root@box jks]#

      Submit the myapp.csr file to the certificate issuer. If it is an internet application, you need to get certificate from venders like VeriSign or any other venders. If it is intranet application, there might that some security department in your organization who can issue a certificate for you. Or else you can create your own using openssl.

The private key is also needed in RSA format, not in PKCS#8 format.

We need to extract the private key from mykeystore.jks file against the alias myapp.alias.Please have a look into the below link on how to extract the private key http://conshell.net/wiki/index.php/Keytool_to_OpenSSL_Conversion_tips There is a free tool to extract the private key. http://yellowcat1.free.fr/index_ktl.html. Use java Web start to launch the application

Screen shots to extract the private key

We need the private key in RSA format, because apache does not understand PKCS#8 PEM format. So, finally, we get three files out of the above exercise:-
            1. root.cer
          2. myapp.cer
            3. myapp.key

Configuring /usr/local/apache2

1. Create www directory under /usr/local/apache2
2. Create myapp directory under /usr/local/apache2/www.
3. Create ssl.crt directory under /usr/local/apache2.
4. Copy the 3 files in /usr/local/apache2/ssl.crt directory.
                        (1)root.cer
                        (2)myapp.cer
                        (3)myapp.key
5. Copy mod_jk.so to /usr/local/apache2/modules.
6. Copy workers.properties to /usr/local/apache2/conf. The content is
                 worker.list=jboss_ajp13
                 worker.jboss_ajp13.port=8009
                 worker.jboss_ajp13.host= myboxname.com
                 worker.jboss_ajp13.type=ajp13

7. Append the below entries /usr/local/apache2/conf httpd.conf file
                 NameVirtualHost *:80
                 NameVirtualHost *:443

                 <VirtualHost *:443>
                     ServerName myboxname.com
                     SSLEngine on
                     SSLOptions +StdEnvVars +ExportCertData +StrictRequire
                     SSLCertificateFile /usr/local/apache2/ssl.crt/ myapp.cer
                     SSLCertificateKeyFile /usr/local/apache2/ssl.crt/myapp.key
                     SSLCertificateChainFile /usr/local/apache2/ssl.crt/root.cer
                     DocumentRoot /usr/local/apache2/www/myapp
                     ErrorLog /usr/local/apache2/logs/myapp-error_log
                     CustomLog /usr/local/apache2/logs/myapp-access_log common
                     <IfModule mod_jk.c>
                             JkMount /* jboss_ajp13
                     </IfModule>
                 </VirtualHost>

                 <VirtualHost *:80 >
                     RewriteEngine On
                     RewriteCond %{HTTPS} off
                     RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}myappname
                     DocumentRoot /usr/local/apache2/www/myapp
                     ServerName myboxname.com
                     ErrorLog /usr/local/apache2/logs/xyz-error_log
                     CustomLog /usr/local/apache2/logs/xyz-access_log common
                 </VirtualHost>

8.Restart apache2
       /usr/local/apache2/bin/apachectl restart

Conclusion
So, now we are ready to go. Apache http server is running on port 80 and Jboss is running on 8080. So, all requests to apache will be forwarded to jboss application server.

https://myboxname.com/myappname

Passing oracle.sql.ARRAY to a Stored Procedure

2009-05-28T19:04:00.000-07:00

The scope of this post is to share my experience to pass an oracle ARRAY to a stored procedure. It was a bit painful exercise and I will be happy if it can help my fellow developers.

Our objective was to do a bulk insert into Oracle table. We are using spring/hibernate configuration to persist in oracle. But there was a situation where we need to persist millions of records at one shot and hibernate was bit slow for that. So, our DBA wrote a stored procedure to do a bulk insert.

But we need to populate the oracle ARRAY with all records to be inserted using java code. I faced two issues

Getting native oracle connection from a wrapped connection. In general, the application server (Jboss, Geronimo for example) or web server (like Tomcat) uses javax.sql.DataSource interface to wrap the native database connection.
Constructing two dimensional ARRAY and STRUCT

Oracle Related stuff

Suppose, we have a database table called TABLE_TEST

CREATE TABLE "Schema"."TABLE_TEST"
( "ID" NUMBER(10,0) NOT NULL,
"NAME" VARCHAR2 NOT NUL,
"ATTRIBUTE1" VARCHAR2,
"ATTRIBUTE2" VARCHAR2,
CONSTRAINT "PK_ TABLE_TEST" PRIMARY KEY ("ID")
)

Create a type

create or replace type T_TYPE is object (
NAME VARCHAR2(100),
ATTRIBUTE1 VARCHAR2(100),
ATTRIBUTE2 VARCHAR2(100)
)

Create a type TB_T_TYPE of type T_TYPE
create or replace type TB_T_TYPE is table of T_TYPE;

Stored procedure
It generate the Id from the sequence

create or replace procedure POPULATE_ TABLE_TEST (example TB_T_TYPE) as
type t_tb_seq is table of number;
l_t_seq t_tb_seq := t_tb_seq();
begin
for i in 1.. example.count loop
l_t_seq.extend;
select s_ TABLE_TEST.nextval into l_t_seq(i) from dual;
end loop;
forall i in 1.. example.count
insert into TABLE_TEST(id,name,attribute1,attribute2)
values (l_t_seq(i),
treat(example(i) as T_TYPE).NAME,
treat(example(i) as T_TYPE).ATTRIBUTE1,
treat(example(i) as T_TYPE).ATTRIBUTE2
);
end;

Java code

applicationContext

<org.springframework.jdbc.core.JdbcTemplate">

OracleStoredProcedure

import oracle.sql.ARRAY;
import oracle.sql.ArrayDescriptor;
import oracle.sql.STRUCT;
import oracle.sql.StructDescriptor;

public class OracleStoredProcedure {
private JdbcTemplate jdbcTemplate;
private final String ORACLE_STRUCT = "T_TYPE ";
private final String ORACLE_ARRAY = "TB_T_TYPE";
private static Log log = LogFactory.getLog(OracleStoredProcedure.class);

public void setJdbcTemplate(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}

public void insertAll(List records) throws DaoException {
Connection conn=null;
StructDescriptor structDescriptor=null;
ArrayDescriptor arrayDescriptor=null;
int iSize = records.size();
Object[] arrObj =null;
Object[][] recObj =null;
try {
//For Tomcat/JBoss
conn=jdbcTemplate.getDataSource().getConnection();
log.debug("got connection from jdbcTemplate ");
structDescriptor = StructDescriptor.createDescriptor(ORACLE_STRUCT, conn.getMetaData().getConnection());
arrayDescriptor = ArrayDescriptor.createDescriptor(ORACLE_ARRAY, conn.getMetaData().getConnection());

arrObj = new Object[iSize];
recObj = new Object[iSize][5];

//Structuring obj and arrays

for (int j = 0; j < iSize ;j++){

TableTest ob= records.get(j);

recObj[j][0]=ob.getName();

recObj[j][1]=ob.getAttribute1();

recObj[j][2]=ob.getAttribute2();

arrObj[j] = new STRUCT(structDescriptor, conn.getMetaData().getConnection(), recObj[j]);

}

ARRAY arr = new ARRAY(arrayDescriptor, conn.getMetaData().getConnection(), recObj);

PreparedStatement preparedStatement=conn.prepareStatement("{call POPULATE_ TABLE_TEST (?)}");

preparedStatement.setArray(1, arr);

preparedStatement.execute();

log.debug("Stored procedure POPULATE_ TABLE_TEST is executed");

}catch (Exception e) {

throw new DaoException("Error while doing bulk insert using POPULATE_ TABLE_TEST Stored procedure", e);

} finally{

try {

if (conn != null){

conn.close();

log.debug("connection from jdbcTemplate is closed");

}

} catch (SQLException e2) {

log.debug("problem while closing connection");

}

My presentation on EJB3.0

2009-01-07T03:35:00.000-08:00

Long long days ago(more than 2 years!!) I did a presentation on EJB3.0 for Singapore Java user group. Chirstopher hosted the slides on the web. While browsing, I found it Here is the link.
EJB3.0

Software Architecture Documentation.Part-I

2008-12-27T18:43:00.000-08:00

In my current project I’ve been assigned the task to make a nice-and-easy-to-understand software architecture document. The system which we are developing is brand new one and it connects 95% of the products available in our bank. As a follower of agile methodology, we need to be very precise with our document. Unlike waterfall model, we did not want to over blot by High Level Design (HLD), Low Level design (LLD) documents. Business Requirement Documents (BRD) by the Business Analyst (BA) is the source of our document. So, the scope of the document was to replace 100 pages of FS by some concise but meaningful document. As a matter of fact, UML is the language of choice for our document, because its n-dimensional graphical representation explains the scope of the proposed system with details workflow and business processes. Also, all parties involved in the project, viz. Business, BA, Developer, Architect, Manager can participate to make it robust and powerful. From the development team I get the chance to work with Use cases and activity diagrams and it is very interesting job with a bit of pain. But as they say “No pain, No gain”, I welcome this pain to have a wonderful experience.

As we know architecture is the blue print of a proposed system, either for software or construction or any other industry. So, documenting the architecture is a very crucial task. Multiple parties involved in the project have different expectations from the document. Business people don’t care about the jargons of UML. They just want to click on the screen and get the business logic to be executed in the expected manner as agreed in BRD.BA wants to make sure that the activity diagrams rightly explain the workflow, so on and so forth. Based on his previous experience my boss advised me to focus on the below UML diagrams:-

Context of the Project. It specifies what the proposed system is supposed to do. How is going make the difference with the existing process
Domain Model. It is the repository of all vocabularies used in the project. If any new jargon is discovered, this model needs to be updated as well.
Data Model. It is basically the various database tables and/or external downloads from other system.
Business Process. An array of activities which is system driven and user don’t participate with it explicitly. For example, we need to load all the trades and bonds before the user start clicking on the system. An loading the trades and bonds consists of several steps which we call business process
Use cases. It is explicitly user driven activity. For example, users added a new product in the system or send a e-mail to some customer, that is an use case. I’ll explain it more details in the later part of this post
So on and so forth

What is a use case? After asking google, I got millions of records within a fraction of second. But more I dig into the links, the more I get diverted from my objective. It’s seems to be like modern art, which can be explained from different angles. For some people, it may be just a few lines of scratches on a paper, but could be an interesting fact. Anyway, wikipedia [http://en.wikipedia.org/wiki/Use_case] defined it with proper attributes and let me share what I understood.

A use case is a collection of activities which will be performed on the proposed application. It must be explained with goals, summary, pre-conditions, post-conditions, exception handlings, notes, issues (if anything is note clear from BA/ Business side), so on and so forth. It should be assisted with activity diagram(s) to explain how the use case will be initiated and the state of the system will be modified after the use case is performed. For example currency code maintenance can be thought as a use case which will do a typical CRUD (Create, Read, Update and Delete) operation. ISO currency code is basic entity in bank, so maintenance of it is very common. Initially I made four different use cases for each CRUD operations. But that was not a practical and reasonable approach. A use case can be compared as a Java class which contains variables, object references, methods, constructors etc. If you just think of variable, it is just an attribute which is useless unless it is associated with a class or interface. Same is true for a use case. It is not just an UML diagram which explains the click event by the user on the screen, but an entire process to describe how the system will behave after an action is taken by the user and we need to capture the behaviour of the system from all possible angles. That makes a use case. So, if we describe the CRUD operation on currency code with all possible attributes, I think we can create a use case. But, there might be some room for more improvement and only experience and teach us how to do it.

My Singnature in Agilemanifesto

2008-12-25T21:11:00.000-08:00

http://agilemanifesto.org/sign/display.cgi?ms=000000135

shall vs will

2008-11-21T21:32:00.000-08:00

There is a funny difference between the age old words ‘shall’ and ‘will’! No, I’m not talking about grammar. I was involved with an internet banking project for a local Singapore bank and got a chance to work with a Functional Requirement Specification (FRS) document for a while. Then I came to know the subtle difference between shall and will. According to Wikipedia ”Most requirement specifications use the word shall to denote something that is required, while reserving the will for a statement of fact … requirements with ‘shall’ are the mandatory requirements, meaning, ‘must’, or ‘have to’.”

So, if the project is supposed to be delivered by some software firm, the business analyst guys have to pay extra attention to shall and will!!

It is quite interesting fact and also a crap at the same time. May be that is one of the reason, agile methodologies talk vividly to replace heavy-weight FRS documents by some understandable UML diagrams. It will save both time and cost and developers can focus on the delivery rather than playing politics with the weapon of shall and will.

kick off...

2008-11-21T19:10:00.000-08:00

Finally, I kick off my blog! It is quite amazing to see how the blogging community is growing and contributing a lot to the human knowledge base. I personally benefited from a lot of excellent blogs to understand some complex concept or to crack a problem. Also, sometimes it provides links which you will get after a lot of googling. It’s has been quite a long in IT industry and I’m quite fortunate to work some of the excellent projects on which I contributed a lot. So, I thought why not to start a blog and let share with others about my experiences?So, here it is.