Monday, September 9, 2013

hadoop on cloudera quickstart vm test example 01 wordcount

It's not easy to install hadoop and related items like hdfs hive and so on of your own, and it is more difficult to config them after installation.

Thanks to cloudera, we can test hadoop with its integrated tool kit (Cloudera QuickStart VM). it provides vmware, kvm and virtualbox edition to download. Everything is configured and you can test without any difficulty.

in this video, I show a example hot to do wordcount in hadoop. The youtube link is:


Steps not included in the video:
1: download vmware player or virtualbox and install;
2: download Cloudera QuickStart VM from cloudera(the link may change all the time, so you can google keyword "Cloudera QuickStart VM" to download).


The example of the wordcount test is:

1: install wget on centos server
sudo yum -y install wget
2: create test dir in /home/cloudera
mkdir /home/cloudera/test
cd /home/cloudera/test
3: create test txt file
echo "what can I do with hadoop on hadoop server or hive server" > test1.txt
4: put the txt file to hdfs
hdfs dfs -mkdir /user/cloudera/input
hdfs dfs -put /home/cloudera/test/test1.txt /user/cloudera/input/
5: go to  /usr/lib/hadoop-mapreduce/
cd /usr/lib/hadoop-mapreduce/
6: run the job
hadoop jar hadoop-mapreduce-examples.jar wordcount /user/cloudera/input/test1.txt /user/cloudera/output
7: check what are there in the output
hdfs dfs -ls /user/cloudera/output/
8: reat the output file
hdfs dfs -cat /user/cloudera/output/part-r-00000

28 comments:

  1. This is one of the most incredible blogs on hadoop. Ive read in a very long time. The amount of information in here is stunning, like you practically wrote the book on the subject. Your blog is great for anyone who wants to understand this subject more. Great stuff; please keep it up!
    Hadoop Training in hyderabad

    ReplyDelete
  2. Cloud is one of the tremendous technology that any company in this world would rely on(cloud computing training chennai). Using this technology many tough tasks can be accomplished easily in no time. Your content are also explaining the same(Cloud computing training centers in chennai). Thanks for sharing this in here. You are running a great blog, keep up this good work.

    ReplyDelete
  3. I am happy to this post..Interesting post! Thanks for writing it.What's wrong with this kind of post exactly? It follows your previous guideline for post length as well as clarity.aws vpc interview questions

    ReplyDelete
  4. Thanks a lot very much for the high quality and results-oriented help. I won’t think twice to endorse your blog post to anybody who wants and needs support about this area.
    digital marketing training in marathahalli

    digital marketing training in rajajinagar

    Digital Marketing Training in online


    full stack developer training in pune


    full stack developer training in annanagar

    ReplyDelete
  5. Existing without the answers to the difficulties you’ve sorted out through this guide is a critical case, as well as the kind which could have badly affected my entire career if I had not discovered your website.
    full stack developer training in tambaram

    full stack developer training in velachery



    ReplyDelete
  6. Nice Blog, When i was read this blog i learnt new things & its truly have well stuff related to developing technology, Thank you for sharing this blog. Digital Marketing Training in Mumbai

    ReplyDelete
  7. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Python training in btm
    Python training in marathahalli
    AWS Training in chennai

    ReplyDelete
  8. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Good discussion. Thank you.
    Anexas
    Six Sigma Training in Abu Dhabi
    Six Sigma Training in Dammam
    Six Sigma Training in Riyadh

    ReplyDelete
  9. Appreciation for really being thoughtful and also for deciding on certain marvelous guides most people really want to be aware of.

    Cloud Training
    Software Testing Training
    Tableau Training in Chennai
    QlikView Training in Chennai
    Microstrategy Training in Chennai

    ReplyDelete
  10. Very interesting information that you have shared with us.i have personally thank you for sharing your ideas with us.
    android development course in bangalore
    Android Training in Thirumangalam
    Android Training in Amjikarai
    Android Training in Padur

    ReplyDelete
  11. This blog is full of Innovative ideas.surely i will look into this insight.please add more information's like this soon.
    AWS Course in Anna Nagar
    Best AWS Training Institute in Anna nagar
    AWS Courses in T nagar
    AWS Training Institutes in T nagar

    ReplyDelete
  12. Do you mind if I quote a couple of your posts as long as I provide credit and sources back to your blog?
    fire and safety course in chennai

    ReplyDelete
  13. Nice tips. Very innovative... Your post shows all your effort and great experience towards your work Your Information is Great if mastered very well.
    Microsoft Azure online training
    Selenium online training
    Java online training
    Java Script online training
    Share Point online training

    ReplyDelete

  14. Thank you for sharing the article. The data that you provided in the blog is informative and

    effective. Best Devops Training Institute

    ReplyDelete
  15. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.

    Digital marketing course

    ReplyDelete
  16. Very informative post. Social media will give the loads of opportunities to grow your business and helps you attract the targeted audiences (both B2B and B2C). Sharing more infographic content on social media will give more visibility.Thanks for sharing...keep update...
    Very informative post. Social media will give the loads of opportunities to grow your business and helps you attract the targeted audiences (both B2B and B2C). Sharing more infographic content on social media will give more visibility.Thanks for sharing...keep update...
    very nice to read this page
    Ai & Artificial Intelligence Course in Chennai
    PHP Training in Chennai
    Ethical Hacking Course in Chennai Blue Prism Training in Chennai
    UiPath Training in Chennai

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Very good information which is very useful for the readers....thanks for sharing it and do share more posts like this.

    Data Science Training in Gurgaon
    Bigdata Hadoop Training in Gurgaon

    ReplyDelete
  19. cyber security training london- Leading CompTIA Cybersecurity Analyst CySA+ Training provider in london. We provide all CompTIA courses. 100% Hands on practical.

    ReplyDelete