Copying lots of small files

25 12 2013

So I have this situation where I need to copy lots of small files (10s of thousands of files each around 100KB) across linux machines. The problem with scp is that it is slow (guess it has something to do with having to setup and tear down a connection for each file). Another option – bbcp. Not sure if this can solve the problem as it is more tuned for large files being split across multiple streams in parallel.

Solution? tar, pipe the output to ssh on the remote machine and untar 🙂

Say you want to transfer the contents of dir1 from machine1 to dir1 on machine2

on machine1/dir1 :~$ 
cd dir1
tar czf - * | ssh machine2 "cd /dir2/; tar xzf -"

Much faster 🙂


Transit directions in Windows Phone 7

27 11 2011

I got a Mango phone. I still feel like biting into it… so juicy and cute 🙂 Bing maps looks amazing! But the problem (other than the fact that you need to touch the screen for “turn by turn” navigation) is that there is no public transit directions… yet.

There are a lot of third party apps of course that do query Bing transit. One of the good ones is Mango Transit (Free). The problem with this app – like many others, is that your destination is not searched for. It almost seems like all these apps expect you to provide a full address. No way I will do that. That is a search engine’s job 🙂

So I actually started to build my own Bing Maps app for WP7. The problem was that the results were pathetic for businesses (restaurants, stores etc). They were in a different continent – and were spelled wrong. I guess you need to use some other API to search those, but anyway, that is not the point of this post.

The solution is actually quite simple. Install the transit app (Mango Transit (free)). The native Bing Maps client on the phone searches for businesses very well. So, search there. Once you pick your destination, swipe right, (about>reviews>apps). Your installed Mango Transit should be listed there. CLICK ON IT! 🙂

Now this only works when you need directions from your current location to somewhere else. The main advantage of course is GPS. It does not work if you want transit directions from two other points in the world. But then again, you can directly hit on IE (make sure you set the Desktop version as the preferred site) – you wont use GPS anyway. … I know I know I’m just making excuses.


UPDATE: Forget everything I just said. The author of the app (yes Jeff himself!), just commented on this article to let me know that he had updated Mango Transit. Silly me, I should have checked it out after it was automatically updated a few days ago. In any case, the app now does EVERYTHING. It even searches for places based on partial name matches, amazingly shows you pins at all appropriate locations (unlike a list of places on a separate screen that you need to click), AND continues to provide transit directions. Awesome, just awesome! Thank you so much Jeff!!! 🙂

Debug WiFi Problems

21 11 2011

Vistumber is a cool tool. It lets you view all available wifi networks in range. Also shows wireless channel, type (a/b/g/n), encryption and signal strength.

Surround Sound With HDMI

11 07 2011

So, I am working with a Dell XPS 1640 laptop. ATI Mobility Radeon HD 3670… and I wanted to connect this to an Onkyo receiver to get 5.1 sound through an HDMI cable. If you have a similar necessity, read on to learn how I managed to get things working.
The first thing I tried after following many forums was to update my drivers. Running Win7 x64, the official Dell drivers are quite old. Oh, BTW, HDMI is from the graphics card, so its the ATI driver that you need to update. This also means that the graphics card handles the audio out.. so no, the sound driver may be different and may not play a role. I tried updating with both the old Dell website version and the latest drivers from ATI. But, when I go to Control Panel>Sound and look at properties of the HDMI connection, I see only stereo. (PS: you may need to connect your laptop with a HDMI cable to something before the properties and stuff get activated). Bing told me through the official ATI MOBILITY Radeon webpage that the card supports Stereo in general, but 5.1 only for AC3 encoded files that use DTS or Dolby…. hmmmm…. Well, as a good step, after installing the latest ATI drivers, I installed Realtek HDMI audio drivers. No, I dont see Realtek listed as a manufacturer anywhere, but lots of people used it and hey, it works. So just do it.

Interestingly, although a simple surround sound test file only played in stereo (i.e. the rear left speaker stuff is output on the front left… and the center speaker is heard on both the front left and right), a file that had AC3 DTS sound played just fine in 5.1 stereo…. OK, so what this means is that unlike regular analog audio cables – where the final signal is simply sound waves for vibrating a carbon diaphragm, HDMI actually carries ENCODED data… heck AC3 directly even!! The receiver/amplifier decodes this stuff. Of course, the HDMI cable can also carry uncompressed PCM (Pulse Code Modulation) data – and if it does so, then decoding for audio in this case was carried out in software – which looked at the stereo capability and gave only two channels.

Cool! So the first solution was to simply convert a few files I had to AC3 DTS. (Coz another file with just AC3 worked in stereo only). Problem? VirtualDub and VirtualDubMod were troublesome. Moreover, I did not want to spend a loooong time decoding/encoding again. Then the nicest thing happened. Well, lots of nice things, but here’s what finally worked.

What we want to do is output ALL audio as AC3 onto the HDMI. That way, there is no stereo conversion, all the 5.1 or higher data is sent in encoded format to the receiver – which then does whatever you paid for it to do. So, install Klite Mega Codec pack (I used x86 on an x64 machine). Now, by default, all audio decoding happens with ffdshow audio decoder. So open the options for that. And in Output, (Output format for uncompressed or decoded streams), enable AC3. THATS IT! Of course, select to directly pass-through all that DTS, Dolby etc stuff.

In your Onkyo receiver, if its shows you PCM as the incoming data stream, it should switch to Dolby or DTS now when you actually play a file. In my case, AAC was still in stereo through Windows Media Player, but Media Player Classic worked fine for everything. No other tweaks. No adding output filters as AC3Filter… nothing. 🙂

PS: Another solution – that is not as invasive as installing the entire klite codec pack – is to simply install AC3Filter from That thing claims to be able to output any sound from your computer in AC3 format. But you may have to do some extra stuff (Like add it as a post-processing renderer in your media player etc)


? in filename? Blasphemy!

12 01 2011

It all started when I got a few files from a friend. He copied them over to my external HDD from linux. I guess linux allows question-mark (?) in file names and so there were a few files with that. The problem was that they wouldn’t get copied to my Windows PC because Windows doesn’t allow ? as part of file names. The worst part was that I could not even rename it – which actually makes sense. The solution? Fire-up your handy virtual machine running linux, connect the external HDD to that, grep for all file names containing ?. Why grep? Coz ? is also a wildcard and so find may not work the way you want. Well, there should be some way to specify the question mark character itself, but I was too sleepy. Anyway, find|grep ? should do the trick. Rename them in linux. Happy Windows.

Live Mesh

12 01 2011

So I decided to try out Windows Live Essentials. Live Messenger is really cool – its got facebook integration etc. Problems include not being able to “like” facebook messages, and you cannot play facebook hosted videos directly (youtube, dailymotion and I guess everything else works fine). Nice things include an awesome picture viewer, and really neat interface – along with integrated chat and Live mail. But no gtalk integration – so I cannot switch over just yet – although I really want to leave Gtalk’s old and stale interface behind.

Anyway, the other thing was Live Mesh 2011. Now I guess it started off as a sync utility, but for Vista and Win 7, it gives ANY windows machine remote access capabilities. That is awesome!! Install it on your machine, login via a live account and thats all the configuration. Go to any other machine, login to and your machine should be listed there. Through the browser, you can login and do stuff. The web interface idea is similar to LogMeIn.

The only problem is that the whole thing works on ActiveX on IE. So you cannot use it on a different browser and you also need to install Live essentials / Mesh on the remote system from which you are trying to connect 😦

Maybe silverlight instead?

Copying lots of files

12 01 2011

So, I had to copy a lot of files from my external HDD to my local HDD. Lot = 1.2 million files. The thing was in some folder hierarchy and obviously I wanted to preserve all that stuff. Problems

  1. Not safe to “move” because in case my experiments go awry, I still want the data to stay on the external HDD. So only looking at copy options.
  2. Default right click copy-paste was kind of slow. Or maybe it isn’t, but lets just assume it is.
  3. move and copy will on Windows will not work for directories

Solution – XCOPY
Create a folder INTO which you want the contents to come. cd into that folder. Then type

xcopy <path to directory whose contents will get copied. NOTE contents within this dir will be copied> /e /d

Notice the two switches e and d. e forces all directories that do not exist to be created – even if its empty on the external HDD. d is a trick 🙂 In case you decide to stop the process and carry on later, just Ctrl-C. Later, run the same command again – with /e and /d and copying will resume exactly where it left off. Dont worry about Ctrl-C cutting a large file and that file getting partially copied. It will be deleted on Ctrl-C and will resume from the beginning next time.

EDIT: If you have a LOT of files, it takes a LOT of time to resume coz it needs to check a LOT of files.