Copying lots of small files

25 12 2013

So I have this situation where I need to copy lots of small files (10s of thousands of files each around 100KB) across linux machines. The problem with scp is that it is slow (guess it has something to do with having to setup and tear down a connection for each file). Another option – bbcp. Not sure if this can solve the problem as it is more tuned for large files being split across multiple streams in parallel.

Solution? tar, pipe the output to ssh on the remote machine and untar 🙂

Say you want to transfer the contents of dir1 from machine1 to dir1 on machine2

on machine1/dir1 :~$ 
cd dir1
tar czf - * | ssh machine2 "cd /dir2/; tar xzf -"

Much faster 🙂

Advertisements

Actions

Information

One response

21 08 2014
Vuong

add “./” will help not copying the “.”

tar czf – ./* | ssh machine2 “cd /dir2/; tar xzf -“

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: