How to Compile HADOOP in Windows

One of my projects requires some research on perfomance of application capable of processing very large dataset for storing historical data of manufacturing process.

One of the application that is subject of my research is an open-source HADOOP. I’ve decided to test its perfomance in windows environment that sits on top of CYGWIN framework.

There are many trivial issues that can be resolved easily at first, until I arrived at a rather hard issue, i.e. that I can’t run tasktracker because of these error(s):

2012-09-21 15:24:19,161 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: \tmp\hadoop-xxx\mapred\local\ttprivate to 0700

After spending some time for finding information about this issue on the net, I’ve arrived at the site that suggest some changes in the source code of this application, especially inside setPermission method of hadoop’s FileUtil.java.

Thus, it requires the ability to perform successful compilation of hadoop’s original source, which I think it should be an easy task. But I realize, there are no easy things in open source world 🙂

After setting up my netbeans projects for hadoop source compilation and resolving minor issues, I came across another show stopper at compilation time as follows:

java.io.IOException: Cannot run program “autoreconf” (in directory “D:\Projects\Hadoop\src\native”): CreateProcess error=2, The system cannot find the file specified

As far as I know, the “autoreconf” is part of command usually runs in unix or linux operating system. Since cygwin framework is a sort of emulator of this kind of environment, I decided to give it a try inside its bash shell, and the command is indeed recognized.

So, how about if I run the compilation process inside cygwin’s bash shell ? An interesting alternative, so first, I have to correctly installed the Apache’s ant java compiler utility and tested it inside cygwin shell.

Well, it turns out that the error is still the same.

The next things I try to do is to eliminate the compiling steps that requires “autoreconf” or “configure” command, and this proves to be rather complex. After performing some experiment, I decided to stop this activities, because it requires in-depth knowledge about structure of build.xml mechanism. It will be days before I came up with tangible results if I persist using this method.

Actually this compilation step is not required because native interface files already shipped in the binary version, and I just want to perform changes inside the java modules.

I again try to find whether there is a solution about this issue in the net, but seems I got no luck this time.

Then come this idea, what if I perform substitution of “autoreconf” command with some dummy or arbitrary *.EXE that does nothing other that to tell the compiler that it can be run successfully ?

And this proves to be an effective one because I can now perform build without any show stopper anymore:

Image

Image

Advertisements

3 Responses to “How to Compile HADOOP in Windows”

  1. Sayantan Says:

    How to do “Then come this idea, what if I perform substitution of “autoreconf” command with some dummy or arbitrary *.EXE that does nothing other that to tell the compiler that it can be run successfully “?

  2. eka s Says:

    the compiler failed because it tries to execute autoreconf assuming there’s an “autoreconf.exe” in the computer, which don’t exist, so it is by replacing any occurences of “autoreconf” inside build.xml (i.e by replacing <exec executable="autoreconf" … with some other dummy executable name) that basically does nothing and returns successful status

  3. Sayantan Says:

    Thanks eka!!! However, I solved it by removing the native core config task dependency with the hadoop core task in the build XML.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: