blog darkness...

words, music, food

tools.DistCp: Exception encountered java.lang.IllegalArgumentException: n must be positive

When using distcp on hadoop, I sometimes get the following error:

tools.DistCp: Exception encountered java.lang.IllegalArgumentException: n must be positive

Which is slightly misleading.  Internet searches (Stack Overflow) make it seem like this can be transient or when you have an empty directory (n = 0).  In my case, the error was because the number of nested directories in my recursive copy was too large for distcp to handle

To get around this, I need to list out my first level of subdirectories and wrap my distcp around a for loop to distcp each top level directory individually like so:

for i in `hdfs dfs -ls hdfs://PATH/TO/DIR/| awk '{print$8}'| awk -F'/' '{print$6}'`; do distcp hdfs://PATH/TO/DIR/$i hdfs://PATH/TO/DEST/$i; done

blog comments powered by Disqus