Environment: Cloudera Cent OS 6.2 (CDH 5.0).
When I run a FileSystemCat example, I found there is something that needed to be configured before you compile the .class file and run it on Hadoop.
1. Source file should be compiled to “.class” file to be run on Hadoop.
My source file is FileSystemCat.java. But when I run “javac FileSystemCat.java”, there is a lot of errors, such as the followings.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
FileSystemCat.java:4: error: package org.apache.hadoop.conf does not exist import org.apache.hadoop.conf.Configuration; ^ FileSystemCat.java:5: error: package org.apache.hadoop.fs does not exist import org.apache.hadoop.fs.FileSystem; ^ FileSystemCat.java:6: error: package org.apache.hadoop.fs does not exist import org.apache.hadoop.fs.Path; ^ FileSystemCat.java:7: error: package org.apache.hadoop.io does not exist import org.apache.hadoop.io.IOUtils; ^ FileSystemCat.java:13: error: cannot find symbol Configuration conf = new Configuration(); ^ symbol: class Configuration location: class FileSystemCat FileSystemCat.java:13: error: cannot find symbol Configuration conf = new Configuration(); ^ symbol: class Configuration location: class FileSystemCat FileSystemCat.java:14: error: cannot find symbol FileSystem fs = FileSystem.get(URI.create(uri), conf); ^ symbol: class FileSystem location: class FileSystemCat FileSystemCat.java:14: error: cannot find symbol FileSystem fs = FileSystem.get(URI.create(uri), conf); ^ symbol: variable FileSystem location: class FileSystemCat FileSystemCat.java:17: error: cannot find symbol in = fs.open(new Path(uri)); ^ symbol: class Path location: class FileSystemCat FileSystemCat.java:18: error: cannot find symbol IOUtils.copyBytes(in, System.out, 4096, false); ^ symbol: variable IOUtils location: class FileSystemCat FileSystemCat.java:20: error: cannot find symbol IOUtils.closeStream(in); ^ symbol: variable IOUtils location: class FileSystemCat 11 errors |
This is because we did not specify the classpath. We can use the following command to compile it.
|
1 |
javac -classpath /opt/cloudera/parcels/CDH/lib/hadoop/hadoop-common-2.3.0-cdh5.0.0.jar FileSystemCat.java |
For convenience, we can export the classpath, such as
|
1 |
export CLASSPATH=.:your_hadoop_path/hadoop-common-0.21.0.jar:$CLASSPATH |
2. Adding the path to Hadoop configuration file.
When I run the program like the following:
|
1 |
hadoop FileSystemCat hdfs://localhost/user/cloudera/habit.txt |
There would be an error like:
|
1 |
Error: Could not find or load main class FileSystemCat |
We need to add the path to file hadoop-env.sh. The location of this file is “/etc/hadoop/conf” on my machine.
In this file, you need to export your path of the class file to HADOOP_CLASSPATH, like:
|
1 |
export HADOOP_CLASSPATH=/home/cloudera/workspace/FileSystem/bin |
After that, you can run your program successfully.