Skip to content
Snippets Groups Projects
  • Lawrence Crowl's avatar
    bff0e529
    Add a contrib script for comparing the performance of two sets of · bff0e529
    Lawrence Crowl authored
    compiler runs.
    
    Usage documentation is in the script.
    
    The script produces output of the form:
    
    $ compare_two_ftime_report_sets "Log0/*perf" "Log3/*perf" 
    
    Arithmetic sample for timevar log files
    "Log0/*perf"
    and selecting lines containing "TOTAL" with desired confidence 95 is 
    trial count is 4, mean is 443.022 (95% confidence in 440.234 to 445.811),
    std.deviation is 1.75264, std.error is 0.876322
    
    Arithmetic sample for timevar log files
    "Log3/*perf"
    and selecting lines containing "TOTAL" with desired confidence 95 is 
    trial count is 4, mean is 441.302 (95% confidence in 436.671 to 445.934),
    std.deviation is 2.91098, std.error is 1.45549
    
    The first sample appears to be 0.39% larger,
    with 60% confidence of being larger.
    To reach 95% confidence, you need roughly 14 trials,
    assuming the standard deviation is stable, which is iffy.
    
    Tested on x86_64 builds.
    
    
    Index: contrib/ChangeLog
    
    2012-11-05  Lawrence Crowl  <crowl@google.com>
    
    	* compare_two_ftime_report_sets: New.
    
    From-SVN: r193277
    bff0e529
    History
    Add a contrib script for comparing the performance of two sets of
    Lawrence Crowl authored
    compiler runs.
    
    Usage documentation is in the script.
    
    The script produces output of the form:
    
    $ compare_two_ftime_report_sets "Log0/*perf" "Log3/*perf" 
    
    Arithmetic sample for timevar log files
    "Log0/*perf"
    and selecting lines containing "TOTAL" with desired confidence 95 is 
    trial count is 4, mean is 443.022 (95% confidence in 440.234 to 445.811),
    std.deviation is 1.75264, std.error is 0.876322
    
    Arithmetic sample for timevar log files
    "Log3/*perf"
    and selecting lines containing "TOTAL" with desired confidence 95 is 
    trial count is 4, mean is 441.302 (95% confidence in 436.671 to 445.934),
    std.deviation is 2.91098, std.error is 1.45549
    
    The first sample appears to be 0.39% larger,
    with 60% confidence of being larger.
    To reach 95% confidence, you need roughly 14 trials,
    assuming the standard deviation is stable, which is iffy.
    
    Tested on x86_64 builds.
    
    
    Index: contrib/ChangeLog
    
    2012-11-05  Lawrence Crowl  <crowl@google.com>
    
    	* compare_two_ftime_report_sets: New.
    
    From-SVN: r193277