I had problems using Hibench 2.2 in new hadoop cluster, yarn.
when I used prepare.sh to make data used in wordcount and sort, DATASIZE and NUM_MAPS in configure.sh aren't recognized correctly.
So I edited some lines which looks like old code, after that, it was executed correctly.
it is original source code
#line: 39
# generate data$HADOOP_EXECUTABLE jar $HADOOP_EXAMPLES_JAR randomtextwriter \-D test.randomtextwrite.bytes_per_map=$((${DATASIZE} / ${NUM_MAPS})) \-D test.randomtextwrite.maps_per_host=${NUM_MAPS} \$COMPRESS_OPT \$INPUT_HDFS
wordcount/bin/prepare.sh
and i changed it to
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-examples*.jar randomtextwriter \
$COMPRESS_OPT \
-D mapreduce.randomtextwriter.totalbytes=${DATASIZE} \
-D mapreduce.randomtextwriter.bytespermap=$((${DATASIZE} / ${NUM_MAPS})) \
$INPUT_HDFS
Naturally, test.randomtextwrite.maps_per_host can be changed to mapreduce.randomtextwriter.mapsperhost.
I hope that this post will help some people using hibench in yarn cluster.
'Programming > Java' 카테고리의 다른 글
hadoop mapreduce.task.io.sort.mb 에 대해 (0) | 2014.12.12 |
---|---|
HiBench를 새로운 hadoop인 yarn에서 실행할때 일부 밴치마크가 제대로 작동하지 않는 버그 (1) | 2014.04.16 |
Hadoop common version(v2.1.0 beta) compile 하기 (0) | 2013.08.23 |
[Java]File Class를 사용해서 Directory 검색하기 (0) | 2011.02.15 |
[Java]Apache POI로 Java에서 Excel 내용 읽기 (0) | 2010.11.16 |