Copying lots of small files

25 12 2013

So I have this situation where I need to copy lots of small files (10s of thousands of files each around 100KB) across linux machines. The problem with scp is that it is slow (guess it has something to do with having to setup and tear down a connection for each file). Another option – bbcp. Not sure if this can solve the problem as it is more tuned for large files being split across multiple streams in parallel.

Solution? tar, pipe the output to ssh on the remote machine and untar 🙂

Say you want to transfer the contents of dir1 from machine1 to dir1 on machine2

on machine1/dir1 :~$ 
cd dir1
tar czf - * | ssh machine2 "cd /dir2/; tar xzf -"

Much faster 🙂

Advertisements




Debug WiFi Problems

21 11 2011

Vistumber http://www.vistumbler.net/ is a cool tool. It lets you view all available wifi networks in range. Also shows wireless channel, type (a/b/g/n), encryption and signal strength.





Surround Sound With HDMI

11 07 2011

So, I am working with a Dell XPS 1640 laptop. ATI Mobility Radeon HD 3670… and I wanted to connect this to an Onkyo receiver to get 5.1 sound through an HDMI cable. If you have a similar necessity, read on to learn how I managed to get things working.
The first thing I tried after following many forums was to update my drivers. Running Win7 x64, the official Dell drivers are quite old. Oh, BTW, HDMI is from the graphics card, so its the ATI driver that you need to update. This also means that the graphics card handles the audio out.. so no, the sound driver may be different and may not play a role. I tried updating with both the old Dell website version and the latest drivers from ATI. But, when I go to Control Panel>Sound and look at properties of the HDMI connection, I see only stereo. (PS: you may need to connect your laptop with a HDMI cable to something before the properties and stuff get activated). Bing told me through the official ATI MOBILITY Radeon webpage that the card supports Stereo in general, but 5.1 only for AC3 encoded files that use DTS or Dolby…. hmmmm…. Well, as a good step, after installing the latest ATI drivers, I installed Realtek HDMI audio drivers. No, I dont see Realtek listed as a manufacturer anywhere, but lots of people used it and hey, it works. So just do it.

Interestingly, although a simple surround sound test file only played in stereo (i.e. the rear left speaker stuff is output on the front left… and the center speaker is heard on both the front left and right), a file that had AC3 DTS sound played just fine in 5.1 stereo…. OK, so what this means is that unlike regular analog audio cables – where the final signal is simply sound waves for vibrating a carbon diaphragm, HDMI actually carries ENCODED data… heck AC3 directly even!! The receiver/amplifier decodes this stuff. Of course, the HDMI cable can also carry uncompressed PCM (Pulse Code Modulation) data – and if it does so, then decoding for audio in this case was carried out in software – which looked at the stereo capability and gave only two channels.

Cool! So the first solution was to simply convert a few files I had to AC3 DTS. (Coz another file with just AC3 worked in stereo only). Problem? VirtualDub and VirtualDubMod were troublesome. Moreover, I did not want to spend a loooong time decoding/encoding again. Then the nicest thing happened. Well, lots of nice things, but here’s what finally worked.

What we want to do is output ALL audio as AC3 onto the HDMI. That way, there is no stereo conversion, all the 5.1 or higher data is sent in encoded format to the receiver – which then does whatever you paid for it to do. So, install Klite Mega Codec pack (I used x86 on an x64 machine). Now, by default, all audio decoding happens with ffdshow audio decoder. So open the options for that. And in Output, (Output format for uncompressed or decoded streams), enable AC3. THATS IT! Of course, select to directly pass-through all that DTS, Dolby etc stuff.

In your Onkyo receiver, if its shows you PCM as the incoming data stream, it should switch to Dolby or DTS now when you actually play a file. In my case, AAC was still in stereo through Windows Media Player, but Media Player Classic worked fine for everything. No other tweaks. No adding output filters as AC3Filter… nothing. 🙂

PS: Another solution – that is not as invasive as installing the entire klite codec pack – is to simply install AC3Filter from ac3filter.net. That thing claims to be able to output any sound from your computer in AC3 format. But you may have to do some extra stuff (Like add it as a post-processing renderer in your media player etc)

Yay!





? in filename? Blasphemy!

12 01 2011

It all started when I got a few files from a friend. He copied them over to my external HDD from linux. I guess linux allows question-mark (?) in file names and so there were a few files with that. The problem was that they wouldn’t get copied to my Windows PC because Windows doesn’t allow ? as part of file names. The worst part was that I could not even rename it – which actually makes sense. The solution? Fire-up your handy virtual machine running linux, connect the external HDD to that, grep for all file names containing ?. Why grep? Coz ? is also a wildcard and so find may not work the way you want. Well, there should be some way to specify the question mark character itself, but I was too sleepy. Anyway, find|grep ? should do the trick. Rename them in linux. Happy Windows.





Live Mesh

12 01 2011

So I decided to try out Windows Live Essentials. Live Messenger is really cool – its got facebook integration etc. Problems include not being able to “like” facebook messages, and you cannot play facebook hosted videos directly (youtube, dailymotion and I guess everything else works fine). Nice things include an awesome picture viewer, and really neat interface – along with integrated chat and Live mail. But no gtalk integration – so I cannot switch over just yet – although I really want to leave Gtalk’s old and stale interface behind.

Anyway, the other thing was Live Mesh 2011. Now I guess it started off as a sync utility, but for Vista and Win 7, it gives ANY windows machine remote access capabilities. That is awesome!! Install it on your machine, login via a live account and thats all the configuration. Go to any other machine, login to devices.live.com and your machine should be listed there. Through the browser, you can login and do stuff. The web interface idea is similar to LogMeIn.

The only problem is that the whole thing works on ActiveX on IE. So you cannot use it on a different browser and you also need to install Live essentials / Mesh on the remote system from which you are trying to connect 😦

Maybe silverlight instead?





Copying lots of files

12 01 2011

So, I had to copy a lot of files from my external HDD to my local HDD. Lot = 1.2 million files. The thing was in some folder hierarchy and obviously I wanted to preserve all that stuff. Problems

  1. Not safe to “move” because in case my experiments go awry, I still want the data to stay on the external HDD. So only looking at copy options.
  2. Default right click copy-paste was kind of slow. Or maybe it isn’t, but lets just assume it is.
  3. move and copy will on Windows will not work for directories

Solution – XCOPY
Create a folder INTO which you want the contents to come. cd into that folder. Then type

xcopy <path to directory whose contents will get copied. NOTE contents within this dir will be copied> /e /d

Notice the two switches e and d. e forces all directories that do not exist to be created – even if its empty on the external HDD. d is a trick 🙂 In case you decide to stop the process and carry on later, just Ctrl-C. Later, run the same command again – with /e and /d and copying will resume exactly where it left off. Dont worry about Ctrl-C cutting a large file and that file getting partially copied. It will be deleted on Ctrl-C and will resume from the beginning next time.

EDIT: If you have a LOT of files, it takes a LOT of time to resume coz it needs to check a LOT of files.





Minimize Skype but dont miss a missed call

28 12 2010

In the good old days, Skype would minimize to the system tray (that area with the clock and volume control n stuff) and NOT take up space on the taskbar (that area with the start button and which lists all open windows). But the first version of Skype for Windows 7 did not do that. They initially argued that staying on the taskbar all the time was THE thing to do – as ordained by M$. Then they changed it.

So now after unticking a checkbox in Skype options, when you minimize it, it goes to the system tray and not the taskbar. Problem – if it was on the taskbar and you got a missed call, you could see a cute little orange thinge overlaid. But skype in the system tray can get hidden. It does have the capability to show you the cute little orange thinge though!

So, right click on the taskbar, properties, “Notificiation Area”-customize – “Skype – Show icon and notifications”. If you think that makes the system tray waaaaay tooo big, on windows 7, you can remove the Action Center – “Only show notifications”. Seriously, how many times do you REALLY need and use the action center?