New Timing on CUDA-C SEBS

Last summer, I attended the GPU Technology Conference (GTC2012) in San Jose, CA (click here for the home page of the conference). NVIDIA provided some of us with a GeForce GTX480 with 1.5GB memory. This GPU has 480 cores on it. It took me a while before I can install it on a system and it took me some more time before I can recompile CUDA-C implementatoin of SEBS algorithm on GTX480 and get some new timing. (sorry I was really busy).

Remember that previously I was able to get maximum 380 speed up on Tesla M1060 installed on a Dell Server with an extension which the GPU were installed on the extension (click here). This meant a 10 days calculation would be cut down into less than hour. (assuming that we are not bounded by reading and writing to hard disk, which we already reached that limit). This time I was able to get 554 times speed up on GTX 480. Look at the graph your self.

The New GTX 690 have 3072 computing cores on it. Well, I am hoping to be able to get one of these cards. If I get one of these new cards, I will recompile the code without changing anything and run it on GTX 690 and see how much more speedup do I get. I think I will need to increase the size of the problem to really get this beast busy computing. Let’s see. So, NVIDIA, when are you giving out some GTX 690s?

Installing CUDA Under Ubuntu 10.04

Well, I wanted to write something on this topic; but noticed that almost what I wanted to write is already available here.

So enjoy it.

 

 

@ Attribute in MAC OSX and files being in use showing in gray

It happens sometimes that you want to open a file and the finder tells you: Oopsy, can’t do that. This file is being used. (well, of course in much more professional wording). You wonder, what program is using it? and why the file is shown in gray? You might restart your system saying that whatever that program using this file was, is not going to be running anymore. After restart, the file is still gray and if you try to open it, you will get the same message.

Well, it might be many thing. But one sometimes some files are assigned some attributes and all these problems are generated. Here is how to fix it. First open a terminal by typing terminal in spotlight on top right corner of your screen and pushing enter.

Once the terminal opened switch to the folder were your gray file is. One easy way is to type “cd” in the terminal, then push space and then drag the folder containing your gray file and drop it in the terminal. The full path will be pasted and then you can push enter.

Now type

 ls -la filename 

You will see in the first column of output some stuff like “-rw-r–r–” (it might differ from this). Check if your file has “@” as its attributes, such as “-rw-r–r–@”. Well if it has that you can see what extra attributes are added to the file by typing:

 xattr filename 

it will show you some stuff such as “com.apple.FinderInfo” or some other stuff. If you decided to remove this attribute, then type:

 xattr -d TheAttributes Filename 

“TheAttributes” in above command is what you got from “xattr filename”.

Hope this helps.

Opening Multiple Instances of Firefox

Previously, I wrote how to open multiple instances of Safari and Firefox (here). However, the procedure won’t work for the newer versions of Firefox. So, we need a workaround.

Here is the solution. Save the following script to the location of your choice.

#!/bin/bash
export ProfDIR="/Users/$USER/Library/App*/Firefox"
export BINDIR="/Applications/Firefox.app/Contents/MacOS"
#Creating a Randomly named profile
RandomNumber=$RANDOM
$BINDIR/firefox-bin -CreateProfile $RandomNumber
#Copying the default profile over
cp -R $ProfDIR/Profiles/*.default/* $ProfDIR/Profiles/*.$RandomNumber/
# Running the new instance of the Firefox
$BINDIR/firefox-bin -p $RandomNumber
#Removing the created Profile
rm -rf $ProfDIR/Profiles/*.$RandomNumber
#Removing the profile name
head -n 7 $ProfDIR/profiles.ini > tmp
cat tmp > $ProfDIR/profiles.ini
rm -rf tmp

Make sure that you give the executing permission to the script with chmod command. Then, if you call the script above, it will create a new instances of FireFox for you.let’s say we have stored the script in ~/myscripts/firefox. Then all you need to do is to type:

~/myscripts/firefox &

Remember, this assumes the default profile is the only profile present and you are going to start the Firefox with the default profile. So, if you have multiple profile already created then make sure you adjust the code in “Removing the profile name” section. As Always, use at your own risk rules apply.

Hope this will help.

Slicing ArcGIS ASC files into tiles

There are times that you want to slice an ArcGIS RASTER file stored in ASC format into some smaller tiles. This could be due to lack of enough memory, or if you want to perform an operation, such as calculating area solar radiation, on a smaller region of your data set.

One solution is to mask a portion of your data. However, if you want to systematically break your data into bunch of smaller non-overlapping  tiles, it is better to have a program that does it for you automatically. You can still do this manually, probably you need a spreadsheet program, to calculate the coordinates of the lower left corner of all the tiles, and you have to adjust for the number of rows and columns that you need to add to make the ncols and nrows divisible by the number of tiles that you want in each directions. Then you need to go to ArcGIS, and key in all those numbers one by one.

Let say you want to break your data into a 3 by 2 tiles. There are 6 tiles in total. But what if there are 30 by 20 tiles. there are 600 tiles now. Well, now you need a script.

The script is called slice_asc_Platforms, where platforms is mac, win, or linux. The command prompt tool is easy to use:

 slice_asc_[mac,win,linux] InputFile nH nV Output 

where:

• “InputFile” is, well, the  the input file, i.e. ArcGIS ASC RASTER file,
• nH is the number of tiles in the horizontal direction, i.e. longitudinal,
• nV is the number of tiles in the vertical direction, i.e. latitudinal,
• and “Output” is used in naming the output files.

The out files are named using “Output_hXXXXX_vYYYYY.asc” where XXXXX and YYYYY are the tile number in horizontal and vertical. So the tile (3,4) would be Output_h00003_v00004.asc and so on. The tiles are numbered from left to right and from top to bottom.

If there is a tile that does not contain any data, i.e. all the values are set to NO_DATA field, this tile is skipped and it is not written to the hard drive.

So, where to get the tool? for Mac click here, for linux click here, and for windows click here. As usual, I have not tested the windows version, but the linux and mac versions are tested. There is a cygwin version that can be downloaded here and it is tested.

NOTE: NO_DATA value must be an integer value.

Mounting a Folder through a firewall using SSH tunneling

Quite often it happens that you want to access a station located behind a firewall and use the file and documents that you have stored there. Particularly, if you are a grad student, you want to connect to your university computer, edit your code, or do whatever that you are doing from a remote location, most possibly from your home.

One method is to just ssh to the firewall and from there ssh to your system and enable the X11 forwarding on both of the ssh connection, using -Y or -X. Then you can launch the editor of your choice, or the program that you need to execute and start working remotely. However, depending on how busy the firewall is, and how good your Internet connection is, you might experience different performance and quality. Quite often it is horrible performance.

You can also use different applications such as VNC or Remote Desktop connections. But these all have a big major problem; most of the time, they are blocked on the firewall. One good alternative is using the TeamViewer. It is easy to install, easy to use, and did I mentioned it is also free! (click here.) But again, the performance you get depends on the network connection and your bandwidth.

The better solution is to mount the remote folder as one of your folders and use the editor on your own computer to edit the file. The only time that you need to pass data over the network is when you need to save the file or read the files.

Previously, I mentioned how to do this on MAC systems using MacFusion, (click here). But if you are outside the university perimeter, you are blocked by the firewall and you are not able to mount the require remote folder locally. Here, it is discussed how to use SSH and SSHFS to accomplish this task.

First you need to have ssh installed. Fortunately it is already installed on MAC and most of the Linux systems. Then you need to install SSHFS. On MAC type “sudo port install sshfs” and on Ubuntu Linux type “sudo apt-get install sshfs”. you are ready.

Now you need to make a tunnel between your machine and the end station passing through the firewall. Just type:

ssh -f UserName@FireWall -L PortNumber:EndStation:22 -N

choose a port number that is not being used on your system. You can check the already used one by typing “netstat”.

You will be asked for your password on the firewall. After the tunneling has been established you need to type

 sshfs -p PortNumber username@127.0.0.1:RemoteLocation LocalMountPoint 

remember to use the same PortNumber as the one used in the previous command. RemoteLocation is the folder on the end station that you want to mount, eg. /home/username, and the LocalMountPoint is the address of the local folder that you want it to be mounted to your end station.

Now, the remote location is mounted on your system as if it was one of the folder on your own system. Just execute what ever program that you want and point it to use the files in that folder.

enjoy.

Download MOD13A2, Convert, Graph NDVI using NCL scripts.

It’s been a while since I have started using the NCAR Command Language (NCL). To my opinion it is indeed a great tool to process data in geoscience and automate batch processing.

Quite recently, couple of my friends asked me if I can assist them with processing some data from MODIS. Even on LinkedIn, one person from Brazil was interested in the same procedure. After sending the scripts to all these people separately, I told myself, why not just upload the code here and let every one using it.

Here is what this script does:

1. It downloads the MOD13A2 data for a given tile and year. You can read the information about the MODDownload script separately here.
2. This script retrieves the NDVI data out of all of the MODIS files downloaded (23 per year) and scales the data by the proper scale factor and store the results in a NetCDF file with (time,lat,lon) dimensions. So it is ready for further time series analysis.
3. It graphs all the time series using NCAR graphics generating PS or PNG files.

You can download the script from here. After downloading edit PrepTile and set the tile number and the year of your choice, (currently it is set to download tile h09v05 for the year 2008). Set the locations were you want the files to be stored and then run the script.

Feel free to make any changes that you like to the code in order to make it more suitable for your own needs, and if you thought the changes that you made, can be helpful to others, you can let me know and I will include it in the file for future distribution.

What you need to run the script?

1. I have tested the code under Linux and Mac; however, there shouldn’t be any problem on cygwin on windows machine.
2. You need to have NCL installed properly and in the PATH. You can get NCL from here.
3. You must have wget installed on the system. On ubuntu you just type “sudo apt-get install wget” and on mac you type “sudo port install wget”, (well, if you have mac port installed, you can get mac port from here).
4. To generate PNG graphics, you need ImageMagick. If you don’t have ImageMagick you can only generate PS graphics using this script. (although if you change the script you might be able to generate other types of graphics, but currently I just made it that way). You can get ImageMagick from here.

Ok, hope you like the script and enjoy using NCL.