<div dir="ltr"><div><div><div>Hi All,<br><br>I have posted this to to the hadoop group however there is one doubt which I think could be answered here.<br></div>I have set two application containers which will have the IP's are 10.0.3.200 and 10.0.3.201.<br>
</div>When I start and hadoop data node on 10.0.3.201 I see the logs having a trace of 10.0.3.148 which I have never created, I am not sure how this IP got created.<br></div><div>I don't see any container with 10.0.3.148 from the lxc web panel, wondering how this IP is getting created as I suspect this is causing the issue with hadoop setup.<br>
</div><div>Any inputs would be appreciated, let me know if you need more information from my side.<br><br></div><div>Thanks,<br>Vicky<br></div><div><div><div><div><div><br><div class="gmail_quote">---------- Forwarded message ----------<br>
From: <b class="gmail_sendername">Vicky Kak</b> <span dir="ltr"><<a href="mailto:vicky.kak@gmail.com">vicky.kak@gmail.com</a>></span><br>Date: Mon, Mar 24, 2014 at 3:21 PM<br>Subject: Setting Hadoop on LinuxContainers Fails.<br>
To: <a href="mailto:user@hadoop.apache.org">user@hadoop.apache.org</a><br><br><br><div dir="ltr"><div><div><div><div><div>Hi All,<br><br></div>I am using linuxcontainer(<a href="http://linuxcontainers.org/" target="_blank">http://linuxcontainers.org/</a>) for configuring the hadoop cluster for the testing.<br>
</div>I have create two linux application containers which are called hadoop1/hadoop2. The IP's associated with the hadoop1 is 10.0.3.200 and with hadoop2 is 10.0.3.201.<br>
<br></div>I am able to start the Namenode on 10.0.3.200 but when i try to start the DataNode on 10.0.3.201 I see the following error at 10.0.3.201<br><br>****************************************************************************************<br>
$ hdfs datanode<br>14/03/24 09:30:57 INFO datanode.DataNode: STARTUP_MSG: <br>/************************************************************<br>STARTUP_MSG: Starting DataNode<br>STARTUP_MSG: host = Hadoop2/<a href="http://10.0.3.148" target="_blank">10.0.3.148</a><br>
STARTUP_MSG: args = []<br>STARTUP_MSG: version = 2.2.0<br>STARTUP_MSG: classpath = /home/ubuntu/Installed/hadoop-2.2.0/etc/hadoop:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/servlet-api-2.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-el-1.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/log4j-1.2.17.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jersey-server-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/guava-11.0.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-codec-1.4.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/paranamer-2.3.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jersey-core-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jersey-json-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jettison-1.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/activation-1.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/xz-1.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-math-2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-lang-2.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/asm-3.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/junit-4.8.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-cli-1.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jsch-0.1.42.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-io-2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jsp-api-2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jetty-6.1.26.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-digester-1.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/commons-net-3.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/lib/xmlenc-0.52.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/hadoop-nfs-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/asm-3.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/home/ubuntu/Installed/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/home/ubuntu/Installed/hadoop-2.2.0/contrib/capacity-scheduler/*.jar<br>
STARTUP_MSG: build = <a href="https://svn.apache.org/repos/asf/hadoop/common" target="_blank">https://svn.apache.org/repos/asf/hadoop/common</a> -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z<br>STARTUP_MSG: java = 1.7.0<br>
************************************************************/<br>14/03/24 09:30:57 INFO datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]<br>14/03/24 09:30:57 WARN common.Util: Path /home/ubuntu/dallaybatta-data/hdfs/datanode should be specified as a URI in configuration files. Please update hdfs configuration.<br>
14/03/24 09:30:58 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties<br>14/03/24 09:30:58 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).<br>14/03/24 09:30:58 INFO impl.MetricsSystemImpl: DataNode metrics system started<br>
14/03/24 09:30:58 INFO datanode.DataNode: Configured hostname is Hadoop2<br>14/03/24 09:30:58 INFO datanode.DataNode: Opened streaming server at /<a href="http://0.0.0.0:50010" target="_blank">0.0.0.0:50010</a><br>14/03/24 09:30:58 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s<br>
14/03/24 09:30:58 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog<br>14/03/24 09:30:58 INFO http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)<br>
14/03/24 09:30:58 INFO http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode<br>14/03/24 09:30:58 INFO http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs<br>
14/03/24 09:30:58 INFO http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static<br>14/03/24 09:30:58 INFO datanode.DataNode: Opened info server at localhost:50075<br>
14/03/24 09:30:58 INFO datanode.DataNode: dfs.webhdfs.enabled = false<br>14/03/24 09:30:58 INFO http.HttpServer: Jetty bound to port 50075<br>14/03/24 09:30:58 INFO mortbay.log: jetty-6.1.26<br>14/03/24 09:30:59 INFO mortbay.log: Started SelectChannelConnector@localhost:50075<br>
14/03/24 09:30:59 INFO ipc.Server: Starting Socket Reader #1 for port 50020<br>14/03/24 09:30:59 INFO datanode.DataNode: Opened IPC server at /<a href="http://0.0.0.0:50020" target="_blank">0.0.0.0:50020</a><br>14/03/24 09:30:59 INFO datanode.DataNode: Refresh request received for nameservices: null<br>
14/03/24 09:30:59 INFO datanode.DataNode: Starting BPOfferServices for nameservices: <default><br>14/03/24 09:30:59 WARN common.Util: Path /home/ubuntu/dallaybatta-data/hdfs/datanode should be specified as a URI in configuration files. Please update hdfs configuration.<br>
14/03/24 09:30:59 INFO datanode.DataNode: Block pool <registering> (storage id unknown) service to /<a href="http://10.0.3.200:9000" target="_blank">10.0.3.200:9000</a> starting to offer service<br>14/03/24 09:30:59 INFO ipc.Server: IPC Server Responder: starting<br>
14/03/24 09:30:59 INFO ipc.Server: IPC Server listener on 50020: starting<br>14/03/24 09:30:59 INFO common.Storage: Lock on /home/ubuntu/dallaybatta-data/hdfs/datanode/in_use.lock acquired by nodename 2618@Hadoop2<br>14/03/24 09:31:00 INFO common.Storage: Locking is disabled<br>
14/03/24 09:31:00 INFO datanode.DataNode: Setting up storage: nsid=1367523242;bpid=BP-1489452897-10.0.3.253-1395650301038;lv=-47;nsInfo=lv=-47;cid=CID-b9e031fa-ebeb-4d52-9ead-4e65f49246ce;nsid=1367523242;c=0;bpid=BP-1489452897-10.0.3.253-1395650301038<br>
14/03/24 09:31:00 INFO impl.FsDatasetImpl: Added volume - /home/ubuntu/dallaybatta-data/hdfs/datanode/current<br>14/03/24 09:31:00 INFO impl.FsDatasetImpl: Registered FSDatasetState MBean<br>14/03/24 09:31:00 INFO datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1395674259100 with interval 21600000<br>
14/03/24 09:31:00 INFO impl.FsDatasetImpl: Adding block pool BP-1489452897-10.0.3.253-1395650301038<br>14/03/24 09:31:00 INFO impl.FsDatasetImpl: Scanning block pool BP-1489452897-10.0.3.253-1395650301038 on volume /home/ubuntu/dallaybatta-data/hdfs/datanode/current...<br>
14/03/24 09:31:00 INFO impl.FsDatasetImpl: Time taken to scan block pool BP-1489452897-10.0.3.253-1395650301038 on /home/ubuntu/dallaybatta-data/hdfs/datanode/current: 11ms<br>14/03/24 09:31:00 INFO impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1489452897-10.0.3.253-1395650301038: 13ms<br>
14/03/24 09:31:00 INFO impl.FsDatasetImpl: Adding replicas to map for block pool BP-1489452897-10.0.3.253-1395650301038 on volume /home/ubuntu/dallaybatta-data/hdfs/datanode/current...<br>14/03/24 09:31:00 INFO impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1489452897-10.0.3.253-1395650301038 on volume /home/ubuntu/dallaybatta-data/hdfs/datanode/current: 0ms<br>
14/03/24 09:31:00 INFO impl.FsDatasetImpl: Total time to add all replicas to map: 1ms<br>14/03/24 09:31:00 INFO datanode.DataNode: Block pool BP-1489452897-10.0.3.253-1395650301038 (storage id DS-1380795562-10.0.3.201-50010-1395650455122) service to /<a href="http://10.0.3.200:9000" target="_blank">10.0.3.200:9000</a> beginning handshake with NN<br>
14/03/24 09:31:00 FATAL datanode.DataNode: Initialization failed for block pool Block pool BP-1489452897-10.0.3.253-1395650301038 (storage id DS-1380795562-10.0.3.201-50010-1395650455122) service to /<a href="http://10.0.3.200:9000" target="_blank">10.0.3.200:9000</a><br>
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1380795562-10.0.3.201-50010-1395650455122, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-b9e031fa-ebeb-4d52-9ead-4e65f49246ce;nsid=1367523242;c=0)<br>
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:739)<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3929)<br> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:948)<br>
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)<br> at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24079)<br>
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)<br> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)<br> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)<br>
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)<br> at java.security.AccessController.doPrivileged(Native Method)<br> at javax.security.auth.Subject.doAs(Subject.java:415)<br> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)<br>
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)<br><br> at org.apache.hadoop.ipc.Client.call(Client.java:1347)<br> at org.apache.hadoop.ipc.Client.call(Client.java:1300)<br> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)<br>
at $Proxy9.registerDatanode(Unknown Source)<br> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>
at java.lang.reflect.Method.invoke(Method.java:601)<br> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)<br> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)<br>
at $Proxy9.registerDatanode(Unknown Source)<br> at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)<br> at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)<br>
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)<br> at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)<br> at java.lang.Thread.run(Thread.java:722)<br>
14/03/24 09:31:00 WARN datanode.DataNode: Ending block pool service for: Block pool BP-1489452897-10.0.3.253-1395650301038 (storage id DS-1380795562-10.0.3.201-50010-1395650455122) service to /<a href="http://10.0.3.200:9000" target="_blank">10.0.3.200:9000</a><br>
14/03/24 09:31:00 INFO datanode.DataNode: Removed Block pool BP-1489452897-10.0.3.253-1395650301038 (storage id DS-1380795562-10.0.3.201-50010-1395650455122)<br>14/03/24 09:31:00 INFO datanode.DataBlockScanner: Removed bpid=BP-1489452897-10.0.3.253-1395650301038 from blockPoolScannerMap<br>
14/03/24 09:31:00 INFO impl.FsDatasetImpl: Removing block pool BP-1489452897-10.0.3.253-1395650301038<br>14/03/24 09:31:02 WARN datanode.DataNode: Exiting Datanode<br>14/03/24 09:31:02 INFO util.ExitUtil: Exiting with status 0<br>
14/03/24 09:31:02 INFO datanode.DataNode: SHUTDOWN_MSG: <span style="background-color:rgb(255,0,0)"><span><span style="background-color:rgb(204,204,204)"></span></span></span><br>/************************************************************<br>
<span style="background-color:rgb(255,0,0)"><b>SHUTDOWN_MSG: Shutting down DataNode at Hadoop2/<a href="http://10.0.3.148" target="_blank">10.0.3.148</a></b></span><br>************************************************************/<br>
<br>
****************************************************************************************<br><br><br></div>And here is the corresponding error coming at NameNode( 10.0.3.201)<br><br>****************************************************************************************<br>
14/03/24 09:31:00 WARN blockmanagement.DatanodeManager: Unresolved datanode registration from 10.0.3.201<br>14/03/24 09:31:00 ERROR security.UserGroupInformation: PriviledgedActionException as:ubuntu (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1380795562-10.0.3.201-50010-1395650455122, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-b9e031fa-ebeb-4d52-9ead-4e65f49246ce;nsid=1367523242;c=0)<br>
14/03/24 09:31:00 INFO ipc.Server: IPC Server handler 3 on 9000, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.registerDatanode from <a href="http://10.0.3.201:60951" target="_blank">10.0.3.201:60951</a> Call#1 Retry#0: error: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1380795562-10.0.3.201-50010-1395650455122, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-b9e031fa-ebeb-4d52-9ead-4e65f49246ce;nsid=1367523242;c=0)<br>
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1380795562-10.0.3.201-50010-1395650455122, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-b9e031fa-ebeb-4d52-9ead-4e65f49246ce;nsid=1367523242;c=0)<br>
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:739)<br> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3929)<br> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:948)<br>
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)<br> at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24079)<br>
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)<br> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)<br> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)<br>
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)<br> at java.security.AccessController.doPrivileged(Native Method)<br> at javax.security.auth.Subject.doAs(Subject.java:415)<br> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)<br>
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)<br>****************************************************************************************<br><br></div><div>I don't know from where <span style="background-color:rgb(255,0,0)"><b>10.0.3.148 </b></span><br>
ip is coming yet, could be due to some lxc configurations. What can be interpreted from the hadoop error information?<br><br></div><div>Let me know if you need more info about my environment to provide some insights.<br>
<br></div><div>Regards,<br>Vicky<br></div><div><br></div><div><br></div><br></div>
</div><br></div></div></div></div></div></div>