Harmonic ANalysis of Time Series (HANTS)

What is It?

When a periodic time dependent data set, such as NDVI, is decomposed into sum of sinusoidal functions, the procedure is called Harmonic ANalysis of Time series (HANTS) or Fourier Analysis. At the end, one can obtain phase and amplitudes, which can be used for further processing and analyzing of the data set.

Why HANTS?

Moving data from time or space into frequency domain may be done for various reasons. The main advantage of this is that it becomes much easier to study how the data is varying based on different frequencies. However, my focus here is on the three applications:

  1. Smoothing the data and removing the outliers,
  2. Gap filling or obtaining the missing data,
  3. Data compression.

How to perform HANTS?

The goal in this step is to find a set of coefficients which shows the contribution of each sine or cosine function in reconstructing the original data set. There are various method to obtain this. Here, the coefficients are obtained through least square curve fitting method, based on the algorithm/software provided on:

http://gdsc.nlr.nl/gdsc/en/tools/hants.

The NLR HANTS, freely available to download from above link, calculates the Fourier series coefficients while filtering out the outliers. This software originally was developed to remove the cloud effect in various remote sensing applications and perform temporal interpolation to obtain a gapless data set when there is no satellite image or product available. However, it can be also used to perform data compression.

The software is about 2MB and freely available to download. However, the software is available in executable (binary) format. As more and more students, scientists, and researchers are using MATLAB these days, it was decided to also provide the MATLAB version of the main function, i.e. the HATNS itself.

MATLAB version of HANTS?

The MATLAB version of HANTS is the exact translation of the original FORTRAN code developed by prof. Wout Verhoef to MATLAB m-file. The only difference is that the sort and matrix inversion available in MATLAB are used instead of the ranking and the inversion algorithm used in the FORTRAN code. A sample output of the MATLAB version is shown below.

Testing the MATLAB version of HANTS algorithm

Smoothing a data sets and removing the outliers

There are four MATLAB files available:

  1. HANTS.m: which calculates amplitude, phase, and the smoothed data for a single time series.
  2. ApplyHANTS.m: which calls HANTS for each pixel of a data set assuming the input data set has the dimension of [time,lat,lon].
  3. ReconHANTSData.m: receives the amplitude and phase and reconstructs the smoothed time series.
  4. ReconstructImage.m: receives the amplitude and phase and reconstructs the spatial data set assuming that both phase and amplitude have the dimension of [nf+1,lat,lon], where nf is the number of frequencies above zero frequency, one of the inputs of HANTS.m.

The figure below shows smoothed and gap filled land surface temperature provided through MODIS products.

Data Compression:

Once it is decided to use the smoothed data set, it is possible to perform data compression. The entire data sets can be reconstructed by knowing only the phase and the amplitudes. In above figure only three frequencies above zero frequency was used. This means to reconstruct the red line, i.e. the data that we want to use, only 4 amplitudes and 3 phase are needed, i.e. 7 numbers, while the entire time series consist of 366 numbers for one year. This means a compression rate of 52.3. This can be very beneficial for spatial data. As an example a MODIS tile with 1200X1200 pixels and one year of daily data consists of 527.04 data entry. If this many elements are about to be stored as floating points one needs 2011MB, whereas storing only the phases and amplitudes requires only 38 MB. This is a huge reduction of data volume. Yet, it is still possible to use one of the compression programs (gzip, rar) to further compress the file.

NOTE: In order to obtain the MATLAB version of HANTS algorithm, you can either download it from here or send me an e-mail.

Advertisements

46 Responses to Harmonic ANalysis of Time Series (HANTS)

  1. basha says:

    Dear Sir, I am Md basha from India, i want to apply Harmonics analysis for my time series temperature data to find out the amplitude and phase for Diurnal semi diurnal variation. please send me the program for Harmonic analysis.

    • mabouali says:

      Thanks for your interest in HANTS implementation on MATLAB. I have attached the program to the separate e-mail. Please feel free to contact me if there was any questions.

  2. Silvia Alfieri says:

    Dear Sirs I’m interested in matlab code of HANTS algoritmh. I’m working on daily time series of Modis land surface temperature. Thank you in advance. Silvia Alfieri

  3. Long Yang says:

    Hi,could you send me the program (matlab version )please?I want to analyze the diurnal cycles of rainfall in my research. Thanks !

  4. Hamid says:

    Dear Mohammad aziz
    I am very glad find you again in the same related topic to my work. I also used HANTS a lot during my research, I am wondering if it possible extend the number of image which HANTS can handles to for example 3*12*744 or not, i mean is it any restriction about number of images. Another hints for my experiment is that in binary version of HANTS, we select NOF (no of ferequency) and based on that more relevant amplitude calculate and used to reconstruct smooth image. I think if it is possible to do it for each pixel separately it would be give more accurate results.

    Thanks In advance
    Hamid Ghafarian

    • mabouali says:

      Dear Hamid,

      It is nice to hear from you again. I am happy that you find MATLAB implementation of HANTS interesting. This MATLAB version, which I will send you in separate e-mail, does not have any restriction on the number of images that can be used. If you can hold the entire data set in the memory, it shouldn’t be any problem. Though, I have not test it really for longer than 366 images per year.

      I do agree that if you provide more frequencies, i.e. bigger NoF, the time series is reconstructed much more exactly. This means that the reconstructed harmonic functions will resemble the original time series better. In fact if you have 366 data entry in one time series, and you choose NoF to be 366 then the reconstruction would be exact; hence, no smoothing at all.

      However, the point of using this implementation of HANTS is that your data set has some outliers. Therefore, you do not want to completely reconstruct the data sets. And you want to get rid of those outliers. At the end, it might take couple of try and error to find out the optimized number for NoF. (Since this is MATLAB and looping in MATLAB is slow; therefore, it is suggested to apply it first on small sub set of the image).

      Current implementation of HANTS on MATLAB, accepts only one NoF for the entire image. So, no changes in that aspect. But including the varying NoF across the image isn’t that hard. Feel free to make changes to the code. And If you wanted we can include that modified version also in the distribution.

      Once again, I am so happy to hear from you again. Keep in touch my friend.

      regards,

  5. vincent says:

    Hi, could you send the matlab files ?
    (HANTS.m)

    Thank you for your work

  6. Jagat says:

    hi Hamid, I am looking for the matlab algorithm of HANSTS. May I get this. thank you in advance.

  7. Salvatore Falanga Bolognesi says:

    Hi, could you send the matlab files
    (HANTS.m;ApplyHANTS.m;ReconHANTSData.m;ReconstructImage.m)?
    Thank you in advance. Salvatore Falanga Bolognesi

  8. Neha Sharma says:

    Hi, I am working with MODIS NDVI 8-day composite, can you suggest how to select parameters in HANTS

    • mabouali says:

      In HANTS.m all the input variables are defined. I copy paste it here for further information

      % Inputs:
      % ni = nr. of images (total number of actual samples of the time
      % series)
      % nb = length of the base period, measured in virtual samples
      % (days, dekads, months, etc.)
      % nf = number of frequencies to be considered above the zero frequency
      % y = array of input sample values (e.g. NDVI values)
      % ts = array of size ni of time sample indicators
      % (indicates virtual sample number relative to the base period);
      % numbers in array ts maybe greater than nb
      % If no aux file is used (no time samples), we assume ts(i)= i,
      % where i=1, …, ni
      % HiLo = 2-character string indicating rejection of high or low outliers
      % low = valid range minimum
      % high = valid range maximum (values outside the valid range are rejeced
      % right away)
      % fet = fit error tolerance (points deviating more than fet from curve
      % fit are rejected)
      % dod = degree of overdeterminedness (iteration stops if number of
      % points reaches the minimum required for curve fitting, plus
      % dod). This is a safety measure
      % delta = small positive number (e.g. 0.1) to suppress high amplitudes
      %

  9. Bill Wang says:

    Dear Mohammad aziz,

    Hi, I am working on the time series processing and notice this HANTS page. May I have the matlab files you mentioned? Thanks in advance!

    I think one critical issue in HANTS is the selection of the harmonic periods. For 10-day NDVI time series, the pre-defined harmonic periods (36, 18 and 12) may be optimal, but for other time series, it may not be. I am thinking to use FFT to identify some “characteristic” frequency of my time series and use those periods rather than the priori set periods to perform HANTS. Do you think there will be any problem doing this? Thanks for your time reading and thinking about it.

    Regards,

  10. Neha Sharma says:

    Hi! thanx for your reply….
    whenever i use hants for smoothing the NDVI curve, it makes the NDVI curve of crops smooth but it deteriorate the curve of forest area…why it is so????

  11. Neha Sharma says:

    hi,
    HANTS creates the amplitude and phase images…..can u please explain what does this phase images reprsents?? and what is the meaning of phase offset….plz explain in detail..

  12. Benoit P. says:

    Hi,
    I work actually on ndvi time series from SPOT-VGT.
    Could you send me the matlab files please ?
    Thanks for your work

  13. Suru says:

    Dear Sirs,
    I am working on harmonic analysis of precipitation over Southwest. I am interested in in your matlab code (HANTS). Can you please send me this code on my email.

    Thank you in advance.
    Suru

  14. Mouhamad Issa says:

    Dear sir,
    Could you please send me the MATLAB version of HANTS algorithm?
    Regads
    Mouhamad

  15. Bill Chen says:

    Dear Sir,
    I am a student learning to process the SPOT-VGT NDVI recently. I am wondering if you could kindly share with me the MATLAB version of the HANTS algorithm? I think it will greatly help me learn about the time series analyses. Thanks in advance for your kind help!

  16. Tasneem Ahmed says:

    Dear Sir,
    I am a student learning to calculate the Harmonic Analysis of MODIS 8-Day Best data using Vegetation Indices recently. I am wondering if you could kindly share with me the MATLAB version of the HANTS algorithm? I think it will greatly help me learn about the time series analysis. Thanks in advance for your kind help!

  17. Irene Garonna says:

    Dear Sir, I would also like to receive a copy of the HANTS matlab file if possible, for use in NDVI timeseries analysis. Also, thank you very much for sharing this post and for such a useful contribution! All the best.

  18. Ryne C. Phillips says:

    Dear Sir,
    I am a Civil Engineering graduate student. I would like to use the MATLAB codes to perform the HANTS algorithm on daily MODIS LST data. Could you please email me the MATLAB codes for the HANTS algorithm? Your help will be greatly appreciated.

    Thanks,

    Ryne C. Phillips

  19. Ghufran says:

    Hi,
    How can i obtain predictions from this library. For instance, say i run the harmonics analysis for 1:n to obtain the amplitude and phase parameters. But now, i want to get the reconstructed values for n+1:N. Is it possible?

    • mabouali says:

      Well this approach assumes periodic behavior. So, if you applied hants on n points and n is your base period, then the value at n+1 is equal to the value at point 1.
      In general if your period is T then the value at point i is equal to the value at point i+T

  20. jihyun says:

    Can I get your harmonic analysis tool? I want to apply for my research about diurnal cycle. Thank you in advance.

  21. liugh says:

    Dear mabouali,
    Thank you very much for your matlab codes to perform the HANTS algorithm. Now I use these codes and meet with some difficulties. I calculate the Harmonic Analysis of GIMMS 15-Day NDVI data recently. I would like to reconstruct the daily NDVI based on 24 NDVI values in one year at each grid. First I applied the ApplyHants.m to get the smoothed data which is based on 24 NDVI values. But I do not know how to transfer the 24 smoothed NDVI data (2 data in each month, so 24 NDVI data in one year) to 365 smoothed data (365 days in one year). Could you please explain it to me? Your help will be greatly appreciated.

    Best wishes
    liugh

  22. Choi, Chul Hyun says:

    Dear mabouali,
    I’m using HANTS(http://gdsc.nlr.nl/gdsc/en/tools/hants) now.. But output file has no extension(aaa_pif, aaaA001_bin, aaa_psf, aaa_pof…). How can I read this file..?
    Are there some programs to read this file? then, Could you send me the program by e-mail..please.
    Your help will be greatly appreciated.

  23. Revati says:

    Dear Sir,
    Greetings,
    Got a useful information regarding HANTS from your post and has helped me in my work. I m working with MODIS 16 bit signed data with 8 day temporal interval. I amusing a Hants software for windows, in which it shows error phase file does not exist.Please tell me how to rectify it. I have tried many time n have cross checked my inputs parameters.
    Thanking You
    Revati M

  24. Srinivas says:

    Is there an R option to implement this?

  25. kumar says:

    Hello,
    I would like to analysis of diurnal cycle of rainfall.Could i get first harmonic analysis code?

  26. Dear M. Abouali,

    Thank you for sharing your program, it seem to be a very powerful program as I try it on my own data, and it can plot a function very close to my data. Right now, I am learning how to use it to determine phase and amplitudes of a time series data because I need that for my work. From your examples (Hi, Lo and none), there are arrays representing amplitudes and phases given by ‘amp_??’ and ‘phi_??’, I try to understand what it is about and please correct me if I am wrong.

    1. Are the amplitudes given by the sum of the amp_?? for example, sum(amp_Hi) = 107.828, which is roughly equal to the max(yr_Hi).
    2. Can you tell me how to select the value of nf?
    3. I cannot understand how to use the phase array, for example, phi_Hi =
    0
    180.924
    319.48
    157.083
    But from the graph the phase of this time series seem to be within 40 (scale of the x-axis), and I don’t understand how this number related to the phase.

  27. Calvin Cheung says:

    Hi there,

    I am interested to know how to obtain phase and amplitudes from your program, based on your
    example, which array represent the phase and amplitudes?

    Regards,

    Calvin

    • mabouali says:

      If you look at the function definition:

      function [amp,phi,yr]=HANTS(ni,nb,nf,y,ts,HiLo,low,high,fet,dod,delta)

      the first one is amplitudes and the second one is phase.

  28. End says:

    Dear Mabouali,

    Would you kind enough to send me a Matlab version of HANTS which is used to determine high frequency signal with reasonalbe amplitude of magnetometer data? (140 data points of 1 day at a given longitude: Totally I have 72 longitude and 365 days)

  29. Vincent says:

    Hi Abouali,
    Please provide an example of how you did the gap filling using HANTS similar to the second graph above. I have data for a years period with gaps (NaNs). I need to estimate the phi and amp from the same data then gap fill it. Will appreciate.

    Thanks
    Vincent

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s