How to count how many x are greater than (x>0.2, 03 and 0.4) in a specific time?

Question

Ara on 14 Mar 2013

0
Link

Direct link to this question

https://au.mathworks.com/matlabcentral/answers/67125-how-to-count-how-many-x-are-greater-than-x-0-2-03-and-0-4-in-a-specific-time

Edited: Ara on 27 Sep 2020

Hi Everybody,

I need to know how many (x>0.2,0.3&0.4)occurred at each hour. First, count it to make me able tabulate it (i.e. in time between 5-6, we observed 5 event(x>0.2) and so on) and then plot it as a bar. Please assume, x and y are defined as a column data in .xls format.

x             y
0.05          10 
0.1           10.1 (1 min)
0.02          10.2
0.2           10.3
0.012         10.4 
.             . 
.             . 
.             .
0.2           11

Thank you in advance,

6 Comments
Show 4 older commentsHide 4 older comments

Ara on 18 Mar 2013

@ Walter: Exactly. As you said, It should be like 10.01 and 10.10. And the signal varied usually during an hour. It means between each minutes or even more minutes hard to find (x>0.2,0.3 or 0.4).

Image Analyst on 18 Mar 2013

You'll have to read them in as text then and parse them into two separate numbers.

Sign in to comment.

Sign in to answer this question.

Answer 1

Cedric on 18 Mar 2013

4
Link

Direct link to this answer

https://au.mathworks.com/matlabcentral/answers/67125-how-to-count-how-many-x-are-greater-than-x-0-2-03-and-0-4-in-a-specific-time#answer_79009

Edited: Cedric on 18 Mar 2013

Open in MATLAB Online

The following could be a solution..

 >> x = rand(1,1e3)/2 ;             % Fake x, for the example. 
 >> y = sort(rand(1,1e3)*23.99) ;   % Fake y, for the example.

Build a cell array of distributions

 >> bins  = 0.05 : 0.1 : 0.45 ;              
 >> dists = arrayfun(@(h) hist(x(floor(y)==h), bins), 0:23, ...
                     'UniformOutput', false) ;

Test..

 >> size(dists)
 ans =
     1    24
 >> dists{1}                      % Distribution for the period [0-1h[.
 ans =
     6     6     9    12     6
 >> dists{12}                     % Distribution for the period [11-12h[.
 ans =
     3    12     4     7     5

so for the period [11-12h[, 3 values are in the range [0,0.1[, 12 in the range [0.1, 0.2[, etc..

Note: you can achieve the same thing using a more basic approach, that you would have to put in a loop..

 >> id_hr = y >= 11 & y < 12 ;    % Index of y values (hrs) in the range 11-12h.
 >> sum(x(id_hr) < 0.1)           % Count # of corresponding x values below 0.1.
 ans =
      3

here we find the 3 that we found above as 1st element of dists{12}.

6 Comments
Show 4 older commentsHide 4 older comments

Cedric on 18 Mar 2013

Edited: Cedric on 18 Mar 2013

Open in MATLAB Online

It seems that I brought a few simplifications while you were testing my first version; sorry for this. You made just one little mistake actually, with respect to what I had done, when you defined bins: the way you wrote it means 0.2 to 0.4 by steps of 0.3, which is not what you want. Also, note that HIST takes the center of bins and not left boundaries, and that boundaries inclusion might not be exactly what you want. At this point, I guess that it is better to avoid using HIST if you want to have a total control over boundaries, and to build your own hist function.

The following should work (that I updated according to the simplification that I brought in my edited answer):

 s4   = rand(1, 1e3)/2 ;            % Fake x, for the example. 
 time = sort(rand(1,1e3)*23.99) ;   % Fake y, for the example.
 % Build an anonymous function called myHist which builds a vector of 
 % three counts, for ranges ]0.2,0.3], ]0.3,0.4], and >0.4.
 myHist = @(v) [sum(v>0.2 & v<=0.3), sum(v>0.3 & v<=0.4), sum(v>0.4)] ;
 % Build a cell array of distributions
 dists = arrayfun(@(h) myHist(s4(floor(time)==h)), 0:23, ...
                  'UniformOutput', false) ;

If you evaluate

bar([0.25,0.35,0.45], dists{2}, 1)

you get a barplot for content of cell #2 which represents the distribution during the period [1-2h[. Now what do you want to plot against time? If it is the evolution of the # of occurrences of values in the range ]0.2,0.3] for example, I would say that the simplest is to convert to cell array into a matrix, which makes indexing simpler:

 >> dists_mat = cell2mat(dists.')
 dists_mat =
   8     7
  12     6
  11     7
   4     6
   7     0
   8     4
   9     5
   7    10
  13     8
  13    10
   5     4
   7     7
   7     4
   9    12
  11     8
  10    10
  12     7
   5    12
  10     9
   9     6
   5     8
  10    15
  14     5
  10     7

and here you can plot/bar/etc each column as a function of e.g. the hour

plot(0:23, dists_mat(:,1)) ;

Ara on 18 Mar 2013

Thank you VERY much. Let me play around with the plot and if you do not mind I will ask any question that I faced with.

Ara on 27 Sep 2020

Edited: Ara on 27 Sep 2020

Hi Cedric,

You wrote a FOR loop in my code. How to get rid of the loop and read data for one day of interest only without going to a circles again and again.

I would like to write my CV in Latex. Can you help me or do you have a template of it so that I can use it?

If I find the answer of this question, I will be the happiest woman in this World.

Please tell me how to correct it?

Best,

Ara

Sign in to comment.

Answer 2

Cedric on 18 Mar 2013

3
Link

Direct link to this answer

https://au.mathworks.com/matlabcentral/answers/67125-how-to-count-how-many-x-are-greater-than-x-0-2-03-and-0-4-in-a-specific-time#answer_79066

Open in MATLAB Online

You're welcome!

Well, at this point you should build a test dataset just to check, e.g.

 >> s4   = [0.21, 0.21, 0.21, 0.32, 0.45, 0.45, 0.25] ;
 >> time = [1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  2] ;

so there should be [3, 1, 2] associated with period [1-2h[, and [1, 0, 0] associated with period [2-3h[.

 >>  myHist = @(v) [sum(v>0.2 & v<=0.3), sum(v>0.3 & v<=0.4), sum(v>0.4)] ;
 >> dists = arrayfun(@(h) myHist(s4(floor(time)==h)), 0:23, ...
                  'UniformOutput', false) ;
 >> dists{1}          % [0-1h[
 ans =
     0     0     0
 >> dists{2}          % [1-2h[
 ans =
     3     1     2
 >> dists{3}          % [2-3h[
 ans =
     1     0     0

It seems to be working. Then, using your dataset, you can check manually what happens with the period 5-6h. I show you here with the period 1-2h:

 >> id_time = time >= 1 & time < 2            % Flag relevant times.
 id_time =
     1     1     1     1     1     1     0

this is a vector of logicals that we use then for indexing s4

 >> s4_selection = s4(id_time)
 s4_selection =
    0.2100    0.2100    0.2100    0.3200    0.4500    0.4500

as you can see, these are the elements of s4 that correspond to a time in the range 1-2h[. Now we test myHist on this set of values

 >> myHist(s4_selection)
 ans =
     3     1     2

It seems to be working.. let's check manually that the first count is correct

 >> s4_selection > 0.2 & s4_selection <= 0.3
 ans =
     1     1     1     0     0     0

again, vector of logicals flagging relevant elements; counting them just means summing the vector of logicals (there is a conversion to numeric)

 >> sum(s4_selection > 0.2 & s4_selection <= 0.3)
 ans =
     3

So it's working. Now ARRAYFUN repeats the same kind of operations with the small difference that instead of checking whether time is greater than h and smaller than h+1 for h=0:23, we test whether floor(time)==h.

Hope it helps; let me know if you are having troubles using this material!

15 Comments
Show 13 older commentsHide 13 older comments

Cedric on 19 Mar 2013

Edited: Cedric on 19 Mar 2013

Open in MATLAB Online

No need for another question/answer; I added a second answer yesterday by mistake actually (was tired ;-)), but I should just have gone on using comments under my first answer.

1. I have no idea; the only thing that I can say is that the code would work as well if it is working now, as the FLOOR operation reduces any positive floating point number to the nearest integer below. So whether you have 10, 10.1, 10.2, 10.3, 10.4, etc, or 10, 10.2, 10.4, 10.6, etc, doesn't matter: they will all be floor'ed to 10 (which is what you want as you are producing an hourly statistics). But my guess is that you would have just half the data points, which might "screw" a bit your hourly distributions if you have as little as 29 points/hour and not all of them are >0.2.

2. I am not sure that I understand what you want to do, so let's discuss an example: assume that you have s5, a vector of satellite IDs..

 >> time = [1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  2] ;
 >> s4   = [0.21, 0.21, 0.21, 0.32, 0.45, 0.45, 0.25] ;
 >> s5   = [3,    5 ,   3,    6,    6,    2,    5] ;

and you want to get the IDs of all satellites that gave a measure during each hour. You can get this with:

 >> sats = arrayfun(@(h) unique(s5(floor(time)==h)), 0:23, ...
                    'UniformOutput', false) ;

Let's check the output:

 >> sats{1}
 ans =
   Empty matrix: 0-by-1
 >> sats{2}
 ans =
     2     3     5     6
 >> sats{3}
 ans =
     5

It seems to be fine. Here we are doing something simpler than before; for each hour h in 0:23 we get all satellite IDs that produced a measurement during [h-h+1[. As some satellites can produce multiple measures during an hour (I guess), we ask for a list of unique IDs.

Again, you can perform one step manually to understand..

 >> id_time = floor(time) == 1      % Flag relevant times.
 id_time =
     1     1     1     1     1     1     0

(this time I used floor() instead of time >= 1 & time <2, just for the example)

 >> s5_selection = s5(id_time)      % All satellite IDs relevant to 
                                    % period [1-2h[.
 s5_selection =
     3     5     3     6     6     2

we just want a lit of unique IDs actually, so..

 >> unique(s5_selection)
 ans =
     2     3     5     6

which is what we got in sats{2}.

Cedric on 23 Mar 2013

Open in MATLAB Online

Hi Ara, no problem. Looks like you want night time indeed! The simplest solution is probably to use what we've done already leading to two cell arrays: dists and sats, and to take all the cells but cells # 11 to 22. You could for example define

 >> dists_NT = dists([1:10, 23, 24]) ;       % Suffix _NT for Night Time. 
 >> sats_NT  = sats([1:10, 23, 24]) ;

Be aware though that these reduced arrays have 12 cells, and cells #11 and #12 correspond to periods [22-23h[ and [23-24h[, so you cannot use the index anymore to obtain directly the hour at the end of the period (which you can do on dists and sats as e.g. dists{24} contains the distribution for period [23-24h[).

Another solution if you really didn't want to get results for the period 10h-22h, would be to update a little calls to ARRAYFUN and replace 0:23 with [0:9,22,23]. For example:

 >> dists = arrayfun(@(h) myHist(s4(floor(time)==h)), [0:9,22,23], ...
                     'UniformOutput', false) ;

A third solution would be to reduce the data almost from the beginning. Say you have your three vectors S4, Time, and PRN as defined below

        S4        Time        PRN
        0.5       2           28
        0.1       6           2
        0.3       10          3
        0.2       14          17   
        0.6       18          8  
        0.4       22          1

You can locate Time elements that are in the range that you want as follows (I use the suffix _NT for Night Time);

 >> id_NT = Time <= 10 & Time >= 22
 id_NT =
     1
     1
     1
     0
     0
     1

you see that it is a vector of logicals that are false for Time = 14h or 18h. Then you can use this vector for logical indexing all vectors of data:

 >> S4_NT   = S4(id_NT) ;
 >> Time_NT = Time(id_NT) ;
 >> PRN_NT  = PRN(id_NT) ;

You could then use these reduces vectors in your computations.

As you can see, there are several ways to proceed; the difficulty is actually to understand what happens the size of reduced output and how to index it properly. If you just want to sum up values or compute a mean, there is no problem, but if you want to access the cell that corresponds to the period [22-23h[, you have to be clear about how the index was shifted when part of the data was discarded.

Ara on 23 Mar 2013

Open in MATLAB Online

Thank you, you are so helpful. Just, Could you please again look at this last version of the code to make me sure is the correct one?

clear all;clc;close all;
data=xlsread('1608_1.xls');
data_filterr=find(data(:,25)>60);
data_filtered=data(data_filterr,:);
elev_cutof20=find(data_filtered(:,6)>=15);
data_cutoff15=data_filtered(elev_cutof20,:);
r=data_cutoff15(:,2);
time=((r./3600)-24*1);
s4r=data_cutoff15(:,8);
s4cor=data_cutoff15(:,9);
s4=sqrt(s4r.^2-s4cor.^2);
PRN=data_cutoff15(:,3);
s4 = rand(1,1e3)/2 ;             % Fake x, for the example. 
time = sort(rand(1,1e3)*23.99) ;   % Fake y, for the example.
%Build a cell array of distributions
   bins  = 0.05 : 0.1 : 0.45 ;              
  %  dists = arrayfun(@(h) hist(x(floor(y)==h), bins), 0:23, ...
  %                      'UniformOutput', false) ;
    dists_mat2 = zeros(24, 3) ;
   for h = 0 : 23
      hId = time >= h & time < h+1 ;
      dists_mat2(h+1, 1) = sum(s4(hId)>0.2 & s4(hId)<=0.3) ;
      dists_mat2(h+1, 2) = sum(s4(hId)>0.3 & s4(hId)<=0.4) ;
      dists_mat2(h+1, 3) = sum(s4(hId)>0.4) ;    
   end
   myHist = @(v) [sum(v>0.2 & v<=0.3), sum(v>0.3 & v<=0.4), sum(v>0.4)] ;
    dists = arrayfun(@(h) myHist(s4(floor(time)==h)), 0:23, ...
        'UniformOutput', false) ;
dists_NT = dists([1:10, 23, 24]) ;       % Suffix _NT for Night Time. 
 sats_NT  = sats([1:10, 23, 24]) ;
   dists = arrayfun(@(h) myHist(s4(floor(time)==h)), [0:9,22,23], ...
                       'UniformOutput', false) ;
  %    sats = arrayfun(@(h) unique(PRN(floor(time)==h)),[0:9,22,23], ...
  %                     'UniformOutput', false) ;
        dists_mat = cell2mat(dists.');
   figure;
        bar([0.25,0.35,0.45], dists{2}, 1);
    figure;plot(0:9, dists_mat(:,1)) ;           
     hold on; plot(0:9, dists_mat(:,2)) ;
     hold on;Plot(0:9, dists_mat(:,3)) ;

Cedric on 23 Mar 2013

Edited: Cedric on 23 Mar 2013

Open in MATLAB Online

I cannot check/guarantee that it is working, because it's a lengthy process that requires you to experiment with sets of known values (so you can check "by hand" that computations are correct), then maybe with larger sets of random values (based on relevant distributions), and finally with your dataset. The code that you pasted contains a bit of all the examples that we discussed, so you might want to eliminate duplicates..

The only thing that I can do at this point is to format it a little better and to add comments (after that I'm off for the night!).

 s4 = rand(1,1e3)/2 ;               % Fake s4, for the example. 
 time = sort(rand(1,1e3)*23.99) ;   % Fake time, for the example.
 % -- This was the first example, using bins that are not exactly
 %    what you need.
 % bins  = 0.05 : 0.1 : 0.45 ;              
 % dists = arrayfun(@(h) hist(x(floor(y)==h), bins), 0:23, ...
 %                      'UniformOutput', false) ;
 % -- This is an example showing how to proceed with a FOR loop instead 
 %    of anonymous functions and ARRAYFUN. This was just in case you would
 %    not understand the latter two mentioned mechanism/function.
 % dists_mat2 = zeros(24, 3) ;
 % for h = 0 : 23
 %     hId = time >= h & time < h+1 ;
 %     dists_mat2(h+1, 1) = sum(s4(hId)>0.2 & s4(hId)<=0.3) ;
 %     dists_mat2(h+1, 2) = sum(s4(hId)>0.3 & s4(hId)<=0.4) ;
 %     dists_mat2(h+1, 3) = sum(s4(hId)>0.4) ;    
 %  end
 % -- This is the example that computes distributions for all hours in
 %    the range 0:23.
 myHist = @(v) [sum(v>0.2 & v<=0.3), sum(v>0.3 & v<=0.4), sum(v>0.4)] ;
 dists = arrayfun(@(h) myHist(s4(floor(time)==h)), 0:23, ...
                  'UniformOutput', false) ;
 % -- Here you should put code for computing sats!
 % ...
 % -- This shows you how to build smaller cell arrays, with a selection
 %    of cells that correspond to your time range. Note that this could
 %    NOT work because you do not compute sats in the code above, so I
 %    commented out the line computing sats_NT and you'll have to add the 
 %    code for defining it above.
 dists_NT = dists([1:10, 23, 24]) ;       % Suffix _NT for Night Time. 
 % sats_NT  = sats([1:10, 23, 24]) ;
 % -- This is an example that shows you how to compute only the 
 %    distributions that correspond to the relevant time range.
 %    It should output the same result as dists_NT above.
 dists = arrayfun(@(h) myHist(s4(floor(time)==h)), [0:9,22,23], ...
                       'UniformOutput', false) ;
 % sats = arrayfun(@(h) unique(PRN(floor(time)==h)),[0:9,22,23], ...
 %                     'UniformOutput', false) ;
 % See, dists and dists_NT are equal: the evaluation of the following 
 % returns 1 (true).
 isequal(dists, dists_NT)
 % -- Here you seem to display dists (more precisely counts) for the 
 %    period [1-2h[.
 figure;
 bar([0.25,0.35,0.45], dists{2}, 1);
 % -- Here you seem to display the evolution over the first 10 hours of
 %    3 counts/bins. 
 %  Note: you made a mistake; the first argument when you call plot has 
 %    10 elements and the second had 12 elements as dists_mat is a 12x3
 %    array (1:10 and 23, 24 means 12 times). I brought a little 
 %    correction for selecting only rows 1:10 of dists_mat.
 %    You also need to execute "hold on" only once after the figure is 
 %    created; there is no need to repeat it after each plot. I added a
 %    few colors, a grid, and a legend, just for the example.
 dists_mat = cell2mat(dists.');
 figure;
 plot(0:9, dists_mat(1:10,1), 'g') ;           
 hold on; 
 plot(0:9, dists_mat(1:10,2), 'b') ;
 plot(0:9, dists_mat(1:10,3), 'r') ;
 grid on ;
 legend('bin ]0.2,0.3]', 'bin ]0.3,0.4]', 'bin ]0.4,\infty[') ;

Ara on 26 Mar 2013

Open in MATLAB Online

Hi Cedric, sorry again bothering you(feeling ashamed). Just one more question regarding the number of satellites, I made a table as below so want to describe but I am not sure all the (9 PRNs) during 006-007 LT observed s4>0.2 simultaneously or just give the number of PRNs not necessarily greater than 0.2? In other words, if during each hrs all satellites observed simultaneously the event (s4>0.2)so the result would be very interesting for my work.However, you assumed some example and then check it out to let me understand but in that case all s4 assumed greater than 0.2. Perhaps, another Hist should be define? I do not know so please you tell me. Here is one line of the table.

Date  Time(hrs)  0.2<S4≤0.3  0.3<S4≤ 0.4  S4>0.4   PRNs
1Nov2010  006-007  11    3    0       3
                                                                6
                                                                14
                                                                16
                                                                19
                                                                20 
                                                                23
                                                                24    
                                                                31 
sats = arrayfun(@(h) unique(PRN(floor(time)==h)),[0:9,22,23], ...
                     'UniformOutput', false) ;

Ara on 28 Mar 2013

I emailed you. Thanks.

Ara on 22 Sep 2020

Edited: Ara on 27 Sep 2020

Hi Dr. Cedric Wannaz,

Thank you for providing useful explanation and the best solution back in 2013.

I still read through your code, programming and all you tought me about MATLAB and it is very useful so thanks again for being helpful all the time. However, I wish to contact you as I amfacing some questions about MATLAB programming and I need to send it to your email address. I wrote to your email address as you previuosly provided for me but I guess you no longer check your email!!!

I need to install Mozilla File the one that you have asked me to use in Australia. How can I install that file again?

I have some more questions and I wish to email it to you to discuss more details about MATLAB solution and I need to send the code to you but I could not find your emal address in your profile. Would you mind to provide your email address in your profile or here so that I can send my questions along with my code to you? You have my email address so you can write back to me in any of my email.

Thanks.

Ara

Sign in to comment.

How to count how many x are greater than (x>0.2, 03 and 0.4) in a specific time?

6 Comments
Show 4 older commentsHide 4 older comments

Accepted Answer

6 Comments
Show 4 older commentsHide 4 older comments

More Answers (1)

15 Comments
Show 13 older commentsHide 13 older comments

See Also

Categories

Tags

Community Treasure Hunt

How to count how many x are greater than (x>0.2, 03 and 0.4) in a specific time?

6 Comments Show 4 older commentsHide 4 older comments

Accepted Answer

6 Comments Show 4 older commentsHide 4 older comments

More Answers (1)

15 Comments Show 13 older commentsHide 13 older comments

See Also

Categories

Tags

Community Treasure Hunt

6 Comments
Show 4 older commentsHide 4 older comments

6 Comments
Show 4 older commentsHide 4 older comments

15 Comments
Show 13 older commentsHide 13 older comments