Why I'm still wrong when I try to using nltk into my python work?
I use the solution here http://stackoverflow.com/questions/6811549/how-can-i-include-a-python-package-with-hadoop-streaming-job

But I still got wrong :


stderr logs
Traceback (most recent call last):
  File "/tmp/hadoop-root/mapred/local/taskTracker/root/jobcache/job_201205290007_0024/attempt_201205290007_0024_m_000000_2/work/./m5.py", line 6, in <module>
    yaml = importer.load_module('yaml')
zipimport.ZipImportError: can't find module 'yaml'
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
	at org.apache.hadoop.mapred.Child.main(Child.java:211)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
log4j:WARN Please initialize the log4j system properly.



Do you have the same problem?
my code is simple:

    #!/usr/bin/env python
    import zipimport
    import sys
    importer = zipimport.zipimporter('nltkandyaml.zip')
    yaml = importer.load_module('yaml')
    nltk = importer.load_module('nltk')

    input_text = []
    for line in sys.stdin:
     	    line = line.strip()
  	    words = line.split()
 	    for word in words:
		    input_text.append(word)

    output = nltk.FreqDist(input_text).keys()
    for word in output:
	    print ("%s\t%s" % (word, 1))

my command is clear:

    bin/hadoop jar /stream-path/*-stream-.zip
    -input /
    -output /
    -mapper map.py
    -reducer reduce.py
    -file /../map.py
    -file /../reduce.py
    -file /../nltkandyaml.mod
Why I'm still wrong when I try to using nltk into my python work?
===

I use the solution here http://stackoverflow.com/questions/6811549/how-can-i-include-a-python-package-with-hadoop-streaming-job

But I still got wrong :


stderr logs
Traceback (most recent call last):
  File "/tmp/hadoop-root/mapred/local/taskTracker/root/jobcache/job_201205290007_0024/attempt_201205290007_0024_m_000000_2/work/./m5.py", line 6, in <module>
    yaml = importer.load_module('yaml')
zipimport.ZipImportError: can't find module 'yaml'
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
	at org.apache.hadoop.mapred.Child.main(Child.java:211)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
log4j:WARN Please initialize the log4j system properly.



Do you have the same problem?
my code is simple:

    #!/usr/bin/env python
    import zipimport
    import sys
    importer = zipimport.zipimporter('nltkandyaml.zip')
    yaml = importer.load_module('yaml')
    nltk = importer.load_module('nltk')

    input_text = []
    for line in sys.stdin:
     	    line = line.strip()
  	    words = line.split()
 	    for word in words:
		    input_text.append(word)

    output = nltk.FreqDist(input_text).keys()
    for word in output:
	    print ("%s\t%s" % (word, 1))

my command is clear:

    bin/hadoop jar /stream-path/*-stream-.zip
    -input /
    -output /
    -mapper map.py
    -reducer reduce.py
    -file /../map.py
    -file /../reduce.py
    -file /../nltkandyaml.mod