Monday, September 9, 2013

hadoop on cloudera quickstart vm test example 01 wordcount

It's not easy to install hadoop and related items like hdfs hive and so on of your own, and it is more difficult to config them after installation.

Thanks to cloudera, we can test hadoop with its integrated tool kit (Cloudera QuickStart VM). it provides vmware, kvm and virtualbox edition to download. Everything is configured and you can test without any difficulty.

in this video, I show a example hot to do wordcount in hadoop. The youtube link is:

Steps not included in the video:
1: download vmware player or virtualbox and install;
2: download Cloudera QuickStart VM from cloudera(the link may change all the time, so you can google keyword "Cloudera QuickStart VM" to download).

The example of the wordcount test is:

1: install wget on centos server
sudo yum -y install wget
2: create test dir in /home/cloudera
mkdir /home/cloudera/test
cd /home/cloudera/test
3: create test txt file
echo "what can I do with hadoop on hadoop server or hive server" > test1.txt
4: put the txt file to hdfs
hdfs dfs -mkdir /user/cloudera/input
hdfs dfs -put /home/cloudera/test/test1.txt /user/cloudera/input/
5: go to  /usr/lib/hadoop-mapreduce/
cd /usr/lib/hadoop-mapreduce/
6: run the job
hadoop jar hadoop-mapreduce-examples.jar wordcount /user/cloudera/input/test1.txt /user/cloudera/output
7: check what are there in the output
hdfs dfs -ls /user/cloudera/output/
8: reat the output file
hdfs dfs -cat /user/cloudera/output/part-r-00000


  1. This is one of the most incredible blogs on hadoop. Ive read in a very long time. The amount of information in here is stunning, like you practically wrote the book on the subject. Your blog is great for anyone who wants to understand this subject more. Great stuff; please keep it up!
    Hadoop Training in hyderabad

  2. Actually, you have explained the technology to the fullest. Thanks for sharing the information you have got. It helped me a lot. I experimented your thoughts in my training program.

    Hadoop Training Chennai
    Hadoop Training in Chennai
    Big Data Training in Chennai

  3. Cloud is one of the tremendous technology that any company in this world would rely on(cloud computing training chennai). Using this technology many tough tasks can be accomplished easily in no time. Your content are also explaining the same(Cloud computing training centers in chennai). Thanks for sharing this in here. You are running a great blog, keep up this good work.

  4. Well Said. The content provided is true up to my knowledge. This made me to understand the concepts very clear. Thanks for sharing this wonderful information in here. Keep blogging article like this. I have bookmarked this page for future reference as well.

    Hadoop Training Chennai | Big Data Training | JAVA Course in Chennai

  5. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.

    Best Java Training Institute Chennai

  6. I am happy to this post..Interesting post! Thanks for writing it.What's wrong with this kind of post exactly? It follows your previous guideline for post length as well as vpc interview questions

  7. Thanks a lot very much for the high quality and results-oriented help. I won’t think twice to endorse your blog post to anybody who wants and needs support about this area.
    digital marketing training in marathahalli

    digital marketing training in rajajinagar

    Digital Marketing Training in online

    full stack developer training in pune

    full stack developer training in annanagar

  8. Existing without the answers to the difficulties you’ve sorted out through this guide is a critical case, as well as the kind which could have badly affected my entire career if I had not discovered your website.
    full stack developer training in tambaram

    full stack developer training in velachery

  9. Nice Blog, When i was read this blog i learnt new things & its truly have well stuff related to developing technology, Thank you for sharing this blog. Digital Marketing Training in Mumbai

  10. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.

    Digital Marketing Training in Mumbai

    Six Sigma Training in Dubai

    Six Sigma Abu Dhabi

  11. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Python training in btm
    Python training in marathahalli
    AWS Training in chennai

  12. I think you have a long story to share and i am glad after long time finally you cam and shared your experience.
    Blueprism training in Pune

    Blueprism online training

    Blue Prism Training in Pune

  13. Thanks for the informative article. This is one of the best resources I have found in quite some time. Nicely written and great info. I really cannot thank you enough for sharing.
    Data Science Training in Chennai
    Data science training in bangalore
    Data science online training
    Data science training in pune

  14. This blog gives very important info about SAS Thanks for sharing.Very well written information. Many

    SAS training

  15. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Good discussion. Thank you.
    Six Sigma Training in Abu Dhabi
    Six Sigma Training in Dammam
    Six Sigma Training in Riyadh

  16. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    java training in chennai | java training in bangalore

    java online training | java training in pune

  17. The knowledge of technology you have been sharing thorough this post is very much helpful to develop new idea. here by i also want to share this.

    angularjs-Training in velachery

    angularjs Training in bangalore

    angularjs Training in bangalore

    angularjs Training in btm

    angularjs Training in electronic-city

  18. I simply wanted to thank you so much again. I am not sure the things that I might have gone through without the type of hints revealed by you regarding that situation.
    safety course in chennai

  19. Appreciation for really being thoughtful and also for deciding on certain marvelous guides most people really want to be aware of.

    Cloud Training
    Software Testing Training
    Tableau Training in Chennai
    QlikView Training in Chennai
    Microstrategy Training in Chennai

  20. This blog is full of Innovative ideas.surely i will look into this insight.please add more information's like this soon.
    AWS Course in Anna Nagar
    Best AWS Training Institute in Anna nagar
    AWS Courses in T nagar
    AWS Training Institutes in T nagar

  21. I am really enjoying reading your well-written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
    RPA Training in Chennai
    RPA Training Institute in Chennai
    Robotic Process Automation Training
    Robotics courses in bangalore
    RPA classes in bangalore

  22. Do you mind if I quote a couple of your posts as long as I provide credit and sources back to your blog?
    fire and safety course in chennai

  23. Amazing article. Your blog helped me to improve myself in many ways thanks for sharing this kind of wonderful informative blogs in live. I have bookmarked more article from this website. Such a nice blog you are providing ! Kindly Visit Us @ Tours and Travels in Madurai | Best Travels in Madurai | Madurai Travels

  24. Nice tips. Very innovative... Your post shows all your effort and great experience towards your work Your Information is Great if mastered very well.
    Microsoft Azure online training
    Selenium online training
    Java online training
    Java Script online training
    Share Point online training

  25. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information. 
    devops online training

    aws online training

    data science with python online training

    data science online training

    rpa online training