Copying lots of small files

So I have this situation where I need to copy lots of small files (10s of thousands of files each around 100KB) across linux machines. The problem with scp is that it is slow (guess it has something to do with having to setup and tear down a connection for each file). Another option – bbcp. Not sure if this can solve the problem as it is more tuned for large files being split across multiple streams in parallel.

Solution? tar, pipe the output to ssh on the remote machine and untar 🙂

Say you want to transfer the contents of dir1 from machine1 to dir1 on machine2

on machine1/dir1 :~$ 
cd dir1
tar czf - * | ssh machine2 "cd /dir2/; tar xzf -"

Much faster 🙂

Transit directions in Windows Phone 7

I got a Mango phone. I still feel like biting into it… so juicy and cute 🙂 Bing maps looks amazing! But the problem (other than the fact that you need to touch the screen for “turn by turn” navigation) is that there is no public transit directions… yet.

There are a lot of third party apps of course that do query Bing transit. One of the good ones is Mango Transit (Free). The problem with this app – like many others, is that your destination is not searched for. It almost seems like all these apps expect you to provide a full address. No way I will do that. That is a search engine’s job 🙂

So I actually started to build my own Bing Maps app for WP7. The problem was that the results were pathetic for businesses (restaurants, stores etc). They were in a different continent – and were spelled wrong. I guess you need to use some other API to search those, but anyway, that is not the point of this post.

The solution is actually quite simple. Install the transit app (Mango Transit (free)). The native Bing Maps client on the phone searches for businesses very well. So, search there. Once you pick your destination, swipe right, (about>reviews>apps). Your installed Mango Transit should be listed there. CLICK ON IT! 🙂

Now this only works when you need directions from your current location to somewhere else. The main advantage of course is GPS. It does not work if you want transit directions from two other points in the world. But then again, you can directly hit maps.bing.com on IE (make sure you set the Desktop version as the preferred site) – you wont use GPS anyway. … I know I know I’m just making excuses.

 

UPDATE: Forget everything I just said. The author of the app (yes Jeff himself!), just commented on this article to let me know that he had updated Mango Transit. Silly me, I should have checked it out after it was automatically updated a few days ago. In any case, the app now does EVERYTHING. It even searches for places based on partial name matches, amazingly shows you pins at all appropriate locations (unlike a list of places on a separate screen that you need to click), AND continues to provide transit directions. Awesome, just awesome! Thank you so much Jeff!!! 🙂

Debug WiFi Problems

Vistumber http://www.vistumbler.net/ is a cool tool. It lets you view all available wifi networks in range. Also shows wireless channel, type (a/b/g/n), encryption and signal strength.

Surround Sound With HDMI

So, I am working with a Dell XPS 1640 laptop. ATI Mobility Radeon HD 3670… and I wanted to connect this to an Onkyo receiver to get 5.1 sound through an HDMI cable. If you have a similar necessity, read on to learn how I managed to get things working.
The first thing I tried after following many forums was to update my drivers. Running Win7 x64, the official Dell drivers are quite old. Oh, BTW, HDMI is from the graphics card, so its the ATI driver that you need to update. This also means that the graphics card handles the audio out.. so no, the sound driver may be different and may not play a role. I tried updating with both the old Dell website version and the latest drivers from ATI. But, when I go to Control Panel>Sound and look at properties of the HDMI connection, I see only stereo. (PS: you may need to connect your laptop with a HDMI cable to something before the properties and stuff get activated). Bing told me through the official ATI MOBILITY Radeon webpage that the card supports Stereo in general, but 5.1 only for AC3 encoded files that use DTS or Dolby…. hmmmm…. Well, as a good step, after installing the latest ATI drivers, I installed Realtek HDMI audio drivers. No, I dont see Realtek listed as a manufacturer anywhere, but lots of people used it and hey, it works. So just do it.

Interestingly, although a simple surround sound test file only played in stereo (i.e. the rear left speaker stuff is output on the front left… and the center speaker is heard on both the front left and right), a file that had AC3 DTS sound played just fine in 5.1 stereo…. OK, so what this means is that unlike regular analog audio cables – where the final signal is simply sound waves for vibrating a carbon diaphragm, HDMI actually carries ENCODED data… heck AC3 directly even!! The receiver/amplifier decodes this stuff. Of course, the HDMI cable can also carry uncompressed PCM (Pulse Code Modulation) data – and if it does so, then decoding for audio in this case was carried out in software – which looked at the stereo capability and gave only two channels.

Cool! So the first solution was to simply convert a few files I had to AC3 DTS. (Coz another file with just AC3 worked in stereo only). Problem? VirtualDub and VirtualDubMod were troublesome. Moreover, I did not want to spend a loooong time decoding/encoding again. Then the nicest thing happened. Well, lots of nice things, but here’s what finally worked.

What we want to do is output ALL audio as AC3 onto the HDMI. That way, there is no stereo conversion, all the 5.1 or higher data is sent in encoded format to the receiver – which then does whatever you paid for it to do. So, install Klite Mega Codec pack (I used x86 on an x64 machine). Now, by default, all audio decoding happens with ffdshow audio decoder. So open the options for that. And in Output, (Output format for uncompressed or decoded streams), enable AC3. THATS IT! Of course, select to directly pass-through all that DTS, Dolby etc stuff.

In your Onkyo receiver, if its shows you PCM as the incoming data stream, it should switch to Dolby or DTS now when you actually play a file. In my case, AAC was still in stereo through Windows Media Player, but Media Player Classic worked fine for everything. No other tweaks. No adding output filters as AC3Filter… nothing. 🙂

PS: Another solution – that is not as invasive as installing the entire klite codec pack – is to simply install AC3Filter from ac3filter.net. That thing claims to be able to output any sound from your computer in AC3 format. But you may have to do some extra stuff (Like add it as a post-processing renderer in your media player etc)

Yay!

? in filename? Blasphemy!

It all started when I got a few files from a friend. He copied them over to my external HDD from linux. I guess linux allows question-mark (?) in file names and so there were a few files with that. The problem was that they wouldn’t get copied to my Windows PC because Windows doesn’t allow ? as part of file names. The worst part was that I could not even rename it – which actually makes sense. The solution? Fire-up your handy virtual machine running linux, connect the external HDD to that, grep for all file names containing ?. Why grep? Coz ? is also a wildcard and so find may not work the way you want. Well, there should be some way to specify the question mark character itself, but I was too sleepy. Anyway, find|grep ? should do the trick. Rename them in linux. Happy Windows.

Live Mesh

So I decided to try out Windows Live Essentials. Live Messenger is really cool – its got facebook integration etc. Problems include not being able to “like” facebook messages, and you cannot play facebook hosted videos directly (youtube, dailymotion and I guess everything else works fine). Nice things include an awesome picture viewer, and really neat interface – along with integrated chat and Live mail. But no gtalk integration – so I cannot switch over just yet – although I really want to leave Gtalk’s old and stale interface behind.

Anyway, the other thing was Live Mesh 2011. Now I guess it started off as a sync utility, but for Vista and Win 7, it gives ANY windows machine remote access capabilities. That is awesome!! Install it on your machine, login via a live account and thats all the configuration. Go to any other machine, login to devices.live.com and your machine should be listed there. Through the browser, you can login and do stuff. The web interface idea is similar to LogMeIn.

The only problem is that the whole thing works on ActiveX on IE. So you cannot use it on a different browser and you also need to install Live essentials / Mesh on the remote system from which you are trying to connect 😦

Maybe silverlight instead?

Copying lots of files

So, I had to copy a lot of files from my external HDD to my local HDD. Lot = 1.2 million files. The thing was in some folder hierarchy and obviously I wanted to preserve all that stuff. Problems

1. Not safe to “move” because in case my experiments go awry, I still want the data to stay on the external HDD. So only looking at copy options.
2. Default right click copy-paste was kind of slow. Or maybe it isn’t, but lets just assume it is.
3. move and copy will on Windows will not work for directories

Solution – XCOPY
Create a folder INTO which you want the contents to come. cd into that folder. Then type

xcopy <path to directory whose contents will get copied. NOTE contents within this dir will be copied> /e /d

Notice the two switches e and d. e forces all directories that do not exist to be created – even if its empty on the external HDD. d is a trick 🙂 In case you decide to stop the process and carry on later, just Ctrl-C. Later, run the same command again – with /e and /d and copying will resume exactly where it left off. Dont worry about Ctrl-C cutting a large file and that file getting partially copied. It will be deleted on Ctrl-C and will resume from the beginning next time.

EDIT: If you have a LOT of files, it takes a LOT of time to resume coz it needs to check a LOT of files.

Minimize Skype but dont miss a missed call

In the good old days, Skype would minimize to the system tray (that area with the clock and volume control n stuff) and NOT take up space on the taskbar (that area with the start button and which lists all open windows). But the first version of Skype for Windows 7 did not do that. They initially argued that staying on the taskbar all the time was THE thing to do – as ordained by M$. Then they changed it.

So now after unticking a checkbox in Skype options, when you minimize it, it goes to the system tray and not the taskbar. Problem – if it was on the taskbar and you got a missed call, you could see a cute little orange thinge overlaid. But skype in the system tray can get hidden. It does have the capability to show you the cute little orange thinge though!

So, right click on the taskbar, properties, “Notificiation Area”-customize – “Skype – Show icon and notifications”. If you think that makes the system tray waaaaay tooo big, on windows 7, you can remove the Action Center – “Only show notifications”. Seriously, how many times do you REALLY need and use the action center?

LogMeIn

I’ve tried other things…

Windows Remote Desktop – Only works if your target machine is running a “professional” version of Windows
VNC – You need to install it yourself, and also configure port forwarding on your router. NAT may be a pain.
SSH – Only for Linux. Also needs to be configured beforehand. NAT problems, port forwarding. Remote computer needs to have X capability.
Screen sharing just doesnt do it and there are many many variants of VNC.

Of course, all this is for a graphical interface. I hate command lines. So, LogMeIn. It seems really nice. Advantages

1. You need to install an app on your target machine. But all port forwarding if any is handled behind the scenes (I am guessing there is something else at work here)
2. Its FREE
3. You can login to a website and then login to any of your registered systems that are ON.
4. No app needed on the remote computer. It uses Java or something else. Of course there are standalone versions of VNC available too.

Problem –
Me being paranoid, I dont really know if they are logging into my machine when I’m not looking and doing the most horrendous of things – rating all my music 5 stars 😦

Outlook Add-in

This is kind of old, but I guess it never got onto this blog. So, I was playing around with the new Office and there was this problem on Win-7. Older versions of Outlook used to display a small yellow envelope in the system tray when I had new mail. But they decided to merge that into the Taskbar icon which is always there anyway. The only problem is that the notification feature does not work with IMAP inboxes… and of course IMAP is the way to go if you like to organize all your mail into >100 folders and you don’t want to do that twice! So there I was, all helpless and unproductive coz I didn’t know when I had new mail!

So, I tried to fix it myself. Fire up Visual Studio and there is actually an Outlook plugin that you can create. Now being M$ and all, every time you run the code in VS for debugging, it starts up Outlook and lets you test it out :). All I needed to do was to monitor each of my IMAP inboxes for new mail. This was quite a long time ago, so a lot of the details are hazy (of course that means it fits perfectly with this blog’s lofty incomplete ideals). I think I looked at just the unread count numbers. There were events fired automatically when you manually set an old message to unread – so no problem there. Of course if you do decide to use this, and happen to use POP3 too, the POP3 notification icon will overlap and ruin your tiny icon – so disable that in Outlook and track the POP3 mailbox yourself too.

Unfortunately, like all good experimental code, it doesn’t work on some occasions. I think its got something to do with my addin losing connection with the mailbox when I hibernate/restart etc. Of course the nice thing is that you can enable those translucent purplish-blue popups for new mails as events for IMAP too. That means I pretty much catch all my new mail in-time and can even delete stuff from that purplish-blue thinge. Meh.