'2014/09'에 해당되는 글 2

  1. 2014.09.21 hadoop 2.4.1 on mac
  2. 2014.09.05 mysql 설치

hadoop 2.4.1 on mac

1. java 1.6 에서 namenode start 가 안될 때

2014-09-21 15:30:34,943 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join

java.lang.InternalError: Can't connect to window server - not enough permissions.

        at java.lang.ClassLoader$NativeLibrary.load(Native Method)

        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1833)

        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1730)

        at java.lang.Runtime.loadLibrary0(Runtime.java:823)

        at java.lang.System.loadLibrary(System.java:1044)

        at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.awt.Toolkit.loadLibraries(Toolkit.java:1605)

        at java.awt.Toolkit.<clinit>(Toolkit.java:1627)

        at sun.awt.AppContext$2.run(AppContext.java:240)

        at sun.awt.AppContext$2.run(AppContext.java:226)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.awt.AppContext.initMainAppContext(AppContext.java:226)

        at sun.awt.AppContext.access$200(AppContext.java:112)

        at sun.awt.AppContext$3.run(AppContext.java:306)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.awt.AppContext.getAppContext(AppContext.java:287)

        at com.sun.jmx.trace.Trace.out(Trace.java:180)

        at com.sun.jmx.trace.Trace.isSelected(Trace.java:88)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.isTraceOn(DefaultMBeanServerInterceptor.java:1830)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:929)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:916)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)

        at com.sun.jmx.mbeanserver.JmxMBeanServer$2.run(JmxMBeanServer.java:1195)

        at java.security.AccessController.doPrivileged(Native Method)

        at com.sun.jmx.mbeanserver.JmxMBeanServer.initialize(JmxMBeanServer.java:1193)



JmxMBeanServer.initialize 가 안되서 발생하는 것으로

JVM option 에 다음과 같이 추가 (.bashrc 같은 곳에)

## java process not to show

export JAVA_TOOL_OPTIONS='-Djava.awt.headless=true'


2. How to start hadoop

ssh 설정

http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/SingleCluster.html


hdfs permission guide

http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html

permission on local file system and HDFS

http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/SecureMode.html


3. historyserver  가 start 되지 않을 때

seoeun@seair:/usr/lib/hadoop$ sudo -u mapred sbin/mr-jobhistory-daemon.sh start historyserver

chown: /usr/lib/hadoop-2.4.1/logs: Operation not permitted

--> mr-jobhistory-daemon.sh 에서  chown 하는 부분을 수정. (hadoop-daemon.sh 참조)

if [ ! -w "$HADOOP_LOG_DIR" ] ; then

echo '-w HADOOP_LOG_DIR'

mkdir -p "$HADOOP_MAPRED_LOG_DIR"

chown $HADOOP_MAPRED_IDENT_STRING $HADOOP_MAPRED_LOG_DIR

fi


* inode / 에 대한 permission이 없다고 하면,
/tmp/ 를 chmod 777 로 설정되었는 지 확인. (permission on local, hdfs)

4. job 이 submit 은 되어서 ACCEPTED는 되었으나 schedule 되지 않을 때
yarn.log를 잘 살펴보면 unhealthy node에 대한 log가 있다면,
014-09-22 13:43:24,802 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Node 192.168.31.244:62938 reported UNHEALTHY with details: 1/1 local-dirs turned bad: /HDFS-2.4.1/yarn/nm-local-di

/HDFS-2.4.1/yarn에 대해서 yarn user에 대한 permission을 체크. 


5. 일반 user로 job 실행이 안될때

Permission denied: user=seoeun, access=EXECUTE, inode="/tmp/hadoop-yarn/staging":mapred:hadoop:drwxrwx---

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)

yarn.app.mapreduce.am.staging-dir 에대한 execute 권한이 없을 경우, user를 supergroup에 포함시킨다. 예를 들면 hadoop.


6. nohub 때문에 process 가 detach 안될 때.

start-yarn.sh 를 타고 들어가서 nohub을 삭제


7. Error starting ResourceManager

org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator not found 

capacity-scheduler.xml

<name>yarn.scheduler.capacity.resource-calculator</name>

<value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>

8. Error starting NodeManager

FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager

java.lang.IllegalArgumentException: The ServiceName: mapreduce.shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers

yarn-site.xml

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

신고

'개발' 카테고리의 다른 글

mysql dump  (0) 2014.10.22
network trouble shooting  (0) 2014.10.01
hadoop 2.4.1 on mac  (0) 2014.09.21
mysql 설치  (0) 2014.09.05
Oozie logfilter  (0) 2014.04.28
OAuth  (0) 2008.11.26
TRACKBACK 0 COMMENT 0

mysql 설치

1. remote connection이 안될 때

/etc/my.cnf  에서 bind 확인


2. password 로 접속이 안될 때.

select * from mysql.user 로 확인.

mysql.user 테이블에 user를 insert 하고 password 를 셋팅한다.

select user, host from mysql.user;

CREATE USER 'root'@'%' IDENTIFIED BY 'your-passwd'; 

flush privileges;

SET PASSWORD FOR 'root'@'%' = PASSWORD('your-passwd');


http://dev.mysql.com/doc/refman/5.1/en/set-password.html

신고

'개발' 카테고리의 다른 글

network trouble shooting  (0) 2014.10.01
hadoop 2.4.1 on mac  (0) 2014.09.21
mysql 설치  (0) 2014.09.05
Oozie logfilter  (0) 2014.04.28
OAuth  (0) 2008.11.26
Maven Repository  (0) 2008.11.07
TRACKBACK 0 COMMENT 0