Statistics and Data Analysis - Maple Programming Help

Online Help

All Products    Maple    MapleSim


Home : Support : Online Help : System : Information : Updates : Maple 2019 : updates/Maple2019/Statistics

Statistics and Data Analysis

 

Least Trimmed Squares Regression

Correlogram

Detrend

Difference

More Updates

Least Trimmed Squares Regression

The LeastTrimmedSquares command computes least trimmed squares regression for some data.

withStatistics:

In this example, we have 1000 data points. There is a single independent variable, x, with values uniformly distributed between 0 and 10. The dependent variable is a linear function of the independent variable plus some additive noise, y=5 x + 10 + noise, where the noise is from a probability distribution known to have severe outliers - the Cauchy distribution, with location parameter 0 and scale parameter 5.

xSampleUniform0,10,1000:

noiseSampleCauchy0,1,1000:

y  5*x + noise +~ 10: 

Here we see all data points:

pp  PointPloty, xcoords = x,size=0.7 , golden: pp;

Linear least squares regression will be severely affected by the outliers.

ls_regression_result  Fita * X + b, x, y, X;

ls_regression_result3.44970682383807X+10.6816568681413

(1.1)

ls_deviation_from_model  coeffls_regression_result, X, 1  52 + coeffls_regression_result, X, 0  102;

ls_deviation_from_model2.86806501793850

(1.2)

Least trimmed squares regression gets much closer to the true line without noise.

lts_regression_result  LeastTrimmedSquaresx, y, X;

lts_regression_result5.03537530551575X+9.82475419561272

(1.3)

lts_deviation_from_model  coefflts_regression_result, X, 1  52 + coefflts_regression_result, X, 0  102;

lts_deviation_from_model0.0319625041956780

(1.4)

The result is even better if we include 900 out of the 1000 points, instead of the default of a little over 500.

lts_900_regression_result  LeastTrimmedSquaresx, y, X, include=900;

lts_900_regression_result5.00862730339998X+10.0156318668695

(1.5)

lts_900_deviation_from_model  coefflts_900_regression_result, X, 1  52 + coefflts_900_regression_result, X, 0  102;

lts_900_deviation_from_model0.000318785625780919

(1.6)

The other robust regression method, implemented in the RepeatedMedianEstimator procedure, also gets a good result.

rme_regression_result  RepeatedMedianEstimatorx, y, X;

rme_regression_result10.0306661686300+5.00564125476873X

(1.7)

rme_deviation_from_model  coeffrme_regression_result, X, 1  52+ coeffrme_regression_result, X, 0  102; 

rme_deviation_from_model0.000972237653807886

(1.8)

In order to visualize these results, we show the same point plot as before, including the four regression lines. The three regression lines from robust methods cannot be distinguished, but the least squares method is clearly off. We zoom in on the vertical range that includes most points.

plotsdisplaypp, plotls_regression_result, lts_regression_result, lts_900_regression_result, rme_regression_result, X=0..10, legend=Least squares, Least trimmed squares, Least trimmed squares (900 points), Repeated median estimator, view=0..10, 10..110,size=0.7 , golden;

 

Correlogram

The Correlogram command computes autocorrelations of a data set and displays the result as a column plot with dashed lines indicating the lower and upper 95% confidence bands for the normal distribution N(0,1/L), where L is the size of the sample 'X', and a caption reporting how many of the displayed columns lie outside of the bands of plus or minus 2, 3, and 4 standard deviations respectively. AutoCorrelationPlot is an alias for the Correlogram command.

CorrelogramImportdatasets/sunspots.csv,base=datadir,output=Matrix265..310, 2

Detrend

The Detrend command removes any trend from a set of data.

restart:

withStatistics:

For example, specify some data:

dataMatrix0,1.8,1,0.7,2.5,2.8,4,4.2,6.2,3

01.810.72.52.844.26.23

(3.1)

Fit a linear model to the data:

lmLinearFita+bt,data,t

lm1.49598376946009+0.366429281218947t

(3.2)

It can be observed that from the plot of the data and the linear model that there is some upward trend. The Detrend command removes any trend from the data.

detrend_dataDetrenddata

0.304016230539913671.1624130506790330.387943027492546831.23829910566412730.7678453130175553

(3.3)

This can be observed in the following plot:

plots:-display     ScatterPlotdata,color=Black,legend=Original Data,symbol=solidcircle,symbolsize=15,     plotlm,color=Black,legend=Trend,linestyle=dash,     ScatterPlotdata..,1,detrend_data,color=Red,legend=Detrended Data,symbol=diamond,symbolsize=15,     plotMeandetrend_data,color=Red,legend=Mean of Detrended Data,linestyle=dot,view=mindata..,10.1..maxdata..,1+0.1,default,size=0.5 , golden

 

Detrend has also been added as an option to several routines in SignalProcessing including SignalPlot, Periodogram, and Spectrogram.

Difference

The Difference command computes lagged differences between elements in a data set.

withStatistics:

Define some data:

xseqi2,i=1..10

149162536496481100

(4.1)

Differencex

35791113151719

(4.2)

More Updates

Display of Data Structures

The default display of data structures such as rtables (matrices, vectors, and arrays) as well as data frames and data series has been changed. Previously a summary would be shown for the data structure, but now the display shows the header of the data (by default the first 10 rows and 10 columns) and the size of the data structure.

Matrix(20,20,rand(1..10));

_rtable18446883718659727478

(5.1.1)

For more details, see the User Interface updates page.

DataFrames and DataSeries

Several commands have been updated to support DataFrames and DataSeries, including remove, select and selectremove. Other new commands such as Detrend and Difference also support DataFrames and DataSeries.

Biplot

The Biplot command has a new option, components, which specifies the principal components used in the biplot.

DataSummary, FivePointSummary, FrequencyTable

The DataSummary, FivePointSummary, and FrequencyTable commands  have a new option, tableweights, which specifies the relative column widths in the displayed embedded table.