DataFrames in Statistics - Maple Programming Help

Home : Support : Online Help : Statistics and Data Analysis : Statistics Package : Statistics/DataFrames

DataFrames in Statistics

Description

 • This help page describes how to use Statistics commands on DataFrame objects, and other spreadsheet-type data in matrices, sometimes called Matrix data sets.
 • Many of the data sets you might encounter are two-dimensional in nature. They will have information about a number of items or events; for each item or event, the same properties are known. Such data sets can easily be represented in a DataFrame by having each row correspond to an item and each column to one property of all these items. This is how you would typically store such data in a spreadsheet. You can also store such data in a Matrix, as long as you keep track of labels for the rows and columns yourself.
 • Many commands in the Statistics package can be used with this type of data:
 – The following computational commands can be run on DataFrame objects (or Matrices). They are computed per column and the results are returned in a DataSeries object. The labels for the DataSeries are the column labels of the DataFrame. Alternatively, they are computed per column of a Matrix and the results are returned in a row Vector.

 – The following visualization commands, listed on the Statistics Visualization help page, also accept DataFrame objects. Generally, the row and column labels are used to label data points and data sets, respectively, as appropriate.

 – Statistics[SplitByColumn] and Statistics[Join] split Matrices into submatrices and join them back together.
 – DataFrame/Aggregate does similar things for DataFrame objects.
 • Additional examples are found in the Statistics with DataFrames example worksheet.

Examples

 > $\mathrm{with}\left(\mathrm{Statistics}\right):$

We construct a DataFrame with housing data. The first column has number of bedrooms, the second has the area in square feet, the third has price.

 > $\mathrm{bedrooms}≔⟨3,4,2,4,3,2,2,3,4,4,2,4,4,3,3⟩$
 ${{\mathrm{_rtable}}}_{{18446883940617134078}}$ (1)
 > $\mathrm{area}≔⟨1130,1123,1049,1527,907,580,878,1075,1040,1295,1100,995,908,853,856⟩$
 ${{\mathrm{_rtable}}}_{{18446883940499542014}}$ (2)
 > $\mathrm{price}≔⟨114700,125200,81600,127400,88500,59500,96500,113300,104400,136600,80100,128000,115700,94700,89400⟩$
 ${{\mathrm{_rtable}}}_{{18446883940499543814}}$ (3)
 > $\mathrm{HouseSalesData}≔\mathrm{DataFrame}\left(\left[\mathrm{bedrooms},\mathrm{area},\mathrm{price}\right],\mathrm{columns}=\left[\mathrm{Bedrooms},\mathrm{Area},\mathrm{Price}\right]\right)$
 ${\mathrm{DataFrame}}{}\left({{\mathrm{_rtable}}}_{{18446883940499536582}}{,}{\mathrm{rows}}{=}\left[{1}{,}{2}{,}{3}{,}{4}{,}{5}{,}{6}{,}{7}{,}{8}{,}{9}{,}{10}{,}{11}{,}{12}{,}{13}{,}{14}{,}{15}\right]{,}{\mathrm{columns}}{=}\left[{\mathrm{Bedrooms}}{,}{\mathrm{Area}}{,}{\mathrm{Price}}\right]\right)$ (4)

We can determine the average number of bedrooms, average area, and average price with just the Mean command.

 > $\mathrm{Mean}\left(\mathrm{HouseSalesData}\right)$
 ${\mathrm{DataSeries}}{}\left(\left[\begin{array}{ccc}3.1333333333333333& 1021.0666666666667& 1.0370666666666667{10}^{5}\end{array}\right]{,}{\mathrm{labels}}{=}\left[{\mathrm{Bedrooms}}{,}{\mathrm{Area}}{,}{\mathrm{Price}}\right]{,}{\mathrm{datatype}}{=}{{\mathrm{float}}}_{{8}}\right)$ (5)

We can also determine the standard error for this mean.

 > $\mathrm{StandardError}\left(\mathrm{Mean},\mathrm{HouseSalesData}\right)$
 ${\mathrm{DataSeries}}{}\left(\left[\begin{array}{ccc}0.21528865819918697& 56.08322613730644& 5615.399461401755\end{array}\right]{,}{\mathrm{labels}}{=}\left[{\mathrm{Bedrooms}}{,}{\mathrm{Area}}{,}{\mathrm{Price}}\right]{,}{\mathrm{datatype}}{=}{{\mathrm{float}}}_{{8}}\right)$ (6)

Or the 30th percentile for each column.

 > $\mathrm{Percentile}\left(\mathrm{HouseSalesData},30\right)$
 ${\mathrm{DataSeries}}{}\left(\left[\begin{array}{ccc}2.9333333333333327& 905.0666666666666& 89340.0\end{array}\right]{,}{\mathrm{labels}}{=}\left[{\mathrm{Bedrooms}}{,}{\mathrm{Area}}{,}{\mathrm{Price}}\right]{,}{\mathrm{datatype}}{=}{{\mathrm{float}}}_{{8}}\right)$ (7)

The GridPlot command can display scatter plots of pairs of columns.

 > $\mathrm{GridPlot}\left(\mathrm{HouseSalesData}\right)$
 ${"Tabulate"}$ (8)

 $\mathrm{Bedrooms}$ $\mathrm{Area}$ $\mathrm{Price}$

We can use the lower diagonal entries to display the values for the correlation.

 > $\mathrm{GridPlot}\left(\mathrm{HouseSalesData},\mathrm{lower}=\mathrm{Correlation}\right)$
 ${"Tabulate0"}$ (9)

 $\mathrm{Bedrooms}$ $0.488597380223581068$ $\mathrm{Area}$ $0.835789649694966386$ $0.704398937794964319$ $\mathrm{Price}$

We can determine the average area and price for subgroups of sales defined by number of bedrooms. (The Aggregate command is part of the DataFrame object, not the Statistics package, so it is not available for Matrices.)

 > $\mathrm{Aggregate}\left(\mathrm{HouseSalesData},\mathrm{Bedrooms}\right)$
 ${\mathrm{DataFrame}}{}\left(\left[\begin{array}{ccc}2& 901.75& 79425.0\\ 3& 964.2& 1.001200{10}^{5}\\ 4& 1148.0& 1.2288333333333333{10}^{5}\end{array}\right]{,}{\mathrm{rows}}{=}\left[{1}{,}{2}{,}{3}\right]{,}{\mathrm{columns}}{=}\left[{\mathrm{Bedrooms}}{,}{\mathrm{Area}}{,}{\mathrm{Price}}\right]\right)$ (10)

To create a box plot of prices for each number of bedrooms requires a little more effort.

 > $\mathrm{split}≔\mathrm{SplitByColumn}\left(\mathrm{HouseSalesData},\mathrm{Bedrooms}\right)$
 ${\mathrm{split}}{≔}\left[\left[\begin{array}{cccc}{}& {\mathrm{Bedrooms}}& {\mathrm{Area}}& {\mathrm{Price}}\\ {3}& {2}& {1049}& {81600}\\ {6}& {2}& {580}& {59500}\\ {7}& {2}& {878}& {96500}\\ {11}& {2}& {1100}& {80100}\end{array}\right]{,}\left[\begin{array}{cccc}{}& {\mathrm{Bedrooms}}& {\mathrm{Area}}& {\mathrm{Price}}\\ {1}& {3}& {1130}& {114700}\\ {5}& {3}& {907}& {88500}\\ {8}& {3}& {1075}& {113300}\\ {14}& {3}& {853}& {94700}\\ {15}& {3}& {856}& {89400}\end{array}\right]{,}\left[\begin{array}{cccc}{}& {\mathrm{Bedrooms}}& {\mathrm{Area}}& {\mathrm{Price}}\\ {2}& {4}& {1123}& {125200}\\ {4}& {4}& {1527}& {127400}\\ {9}& {4}& {1040}& {104400}\\ {10}& {4}& {1295}& {136600}\\ {12}& {4}& {995}& {128000}\\ {13}& {4}& {908}& {115700}\end{array}\right]\right]$ (11)
 > $\mathrm{price_split}≔\mathrm{map}\left(\mathrm{df}↦\mathrm{convert}\left(\mathrm{df}\left[\mathrm{Price}\right],\mathrm{Vector}\right),\mathrm{split}\right)$
 $\left[\left[\begin{array}{rrrr}81600& 59500& 96500& 80100\end{array}\right]{,}\left[\begin{array}{rrrrr}114700& 88500& 113300& 94700& 89400\end{array}\right]{,}\left[\begin{array}{rrrrrr}125200& 127400& 104400& 136600& 128000& 115700\end{array}\right]\right]$ (12)
 > $\mathrm{BoxPlot}\left(\mathrm{price_split},\mathrm{datasetlabels}=\left[2,3,4\right]\right)$

Most of the things mentioned above can be done with a Matrix, too. Consider the following examples.

 > $\mathrm{HSD_Matrix}≔\mathrm{convert}\left(\mathrm{HouseSalesData},\mathrm{Matrix}\right)$
 ${{\mathrm{_rtable}}}_{{18446883940452906518}}$ (13)
 > $\mathrm{Mean}\left(\mathrm{HSD_Matrix}\right)$
 $\left[\begin{array}{ccc}3.1333333333333333& 1021.0666666666667& 1.0370666666666667{10}^{5}\end{array}\right]$ (14)
 > $\mathrm{Percentile}\left(\mathrm{HSD_Matrix},30\right)$
 $\left[\begin{array}{ccc}2.9333333333333327& 905.0666666666666& 89340.0\end{array}\right]$ (15)

Some commands have calling sequences where one of the arguments is compared to the data; this is the case for the second argument of AbsoluteDeviation and for the origin parameter of Moment. In these cases, it typically does not make much sense to use the same value for each column, so Maple supports using a list or Vector of values instead. These commands do not yet work directly with DataFrame objects.

 > $\mathrm{AbsoluteDeviation}\left(\mathrm{HSD_Matrix},\left[3,1000,100000\right]\right)$
 $\left[\begin{array}{ccc}0.6666666666666666& 157.46666666666667& 18333.333333333332\end{array}\right]$ (16)
 > $\mathrm{StandardError}\left(\mathrm{Moment},\mathrm{HSD_Matrix},1,\mathrm{origin}=\left[3,1000,100000\right]\right)$
 $\left[\begin{array}{ccc}0.2079886036764016& 54.181543939829766& 5424.991278368138\end{array}\right]$ (17)
 >