accumulatePackage version 0.9.3.
Use citation('accumulate') to cite the package.
Accumulate is a package for grouped aggregation, where
the groups can be dynamically collapsed into larger groups. When this
collapsing takes place and how collapsing takes place is
user-defined.
The latest CRAN release can be installed as follows.
install.packages("accumulate")Next, the package can be loaded. You can use
packageVersion (from base R) to check which version you
have installed.
We will use a built-in dataset as example.
> data(producers)
> head(producers)
sbi size industrial trade other other_income total
1 3410 8 151722 2135 0 -1775 152082
2 2840 7 50816 NA 158 949 59876
3 2752 5 4336 NA 0 36 4959
4 3120 6 18508 NA 0 80 20682
5 2524 7 21071 0 0 442 21513
6 3410 6 24220 1069 0 239 25528This synthetic dataset contains information on various sources of
turnover from producers, that are labeled with an economic activity
classification (sbi) and a size class
(0-9).
We wish to find a group mean by sbi x size. However, we
demand that the group has at least five records, otherwise we combine
the size classes of a single sbi group. This can be done as
follows.
> a <- accumulate(producers
+ , collapse = sbi*size ~ sbi
+ , test = min_records(5)
+ , fun = mean, na.rm=TRUE)
> head(round(a))
sbi size level industrial trade other other_income total
1 3410 8 1 364397 2859 33 353 546117
2 2840 7 0 23160 823 49 329 25812
3 2752 5 NA NA NA NA NA NA
4 3120 6 0 20710 504 112 200 21702
5 2524 7 0 27954 1268 55 456 30468
6 3410 6 1 364397 2859 33 353 546117The accumulate function does the following:
sbi and size
occurring in the data, it checks whether test is satisfied.
Here, it tests whether there are at least five records.
level is set to 0
(no collapsing took place).sbi as grouping variable for the current combination of
sbi and size. Then, if there are enough
records, the mean is computed for each variable and the output variable
level is set to 1 (first level of collapsing has been
used).NA for the current sbi and
size combination.Explicitly, for this example we see that for
(sbi,size)==(2752,5) no satisfactory group of records was
found under the current collapsing scheme. Therefore the
level variable equals NA and all aggregated
variables are missing as well. For (sbi,size)==(2840,7)
there are sufficient records, and since level=0 no
collapsing was necessary. For the group (sbi,size)=(3410,8)
there were not enough records to compute a mean, but taking all records
in sbi==3410 gave enough records. This is signified by
level=1, meaning that one collapsing step has taken place
(from sbi x size to sbi).
Let us see how we specified this call to accumulate
target groups ~ collapsing scheme. The output is always at
the level of the target groups. The collapsing scheme determines which
records are used to compute a value for the target groups if the
test is not satisfied.test is a function that
should accept any subset of records of producers and return
TRUE or FALSE. In this case we used the
convenience function min_records(5) provided by
accumulate. The function min_records() creates
a testing function for us that we can pass as testing function.fun is the aggregation function
that will be applied to each group.Observe that the accumulate function is similar to R’s built-in
aggregate function (this is by design). There is a second
function called cumulate that has an interface that is
similar to dplyr::summarise.
> a <- cumulate(producers, collapse = sbi*size ~ sbi
+ , test = function(d) nrow(d) >= 5
+ , mu_industrial = mean(industrial, na.rm=TRUE)
+ , sd_industrial = sd(industrial, na.rm=TRUE))
> head(round(a))
sbi size level mu_industrial sd_industrial
1 3410 8 1 364397 535446
2 2840 7 0 23160 13937
3 2752 5 NA NA NA
4 3120 6 0 20710 21151
5 2524 7 0 27954 15089
6 3410 6 1 364397 535446Notice that here, we wrote our own test function.
(sbi, size) could not be
computed, even when collapsing to sbi? (You need to run the
code and investigate the output).?mean on how to compute
trimmed means.A collapsing scheme can be defined in a data frame or with a formula of the form
target grouping ~ collapse1 + collapse2 + ... + collapseNHere, the target grouping is a variable or product of
variables. Each collapse term is also a variable or product
of variables. Each subsequent term defines the next collapsing step. Let
us show the idea with a more involved example.
The sbi variable in the producers dataset
encodes a hierarchical classification where longer digit sequences
indicate higher level of detail. Hence we can collapse to lower levels
of detail by deleting digits at the end. Let us enrich the
producers dataset with extra grouping levels.
> producers$sbi3 <- substr(producers$sbi,1,3)
> producers$sbi2 <- substr(producers$sbi,1,2)
> head(producers,3)
sbi size industrial trade other other_income total sbi3 sbi2
1 3410 8 151722 2135 0 -1775 152082 341 34
2 2840 7 50816 NA 158 949 59876 284 28
3 2752 5 4336 NA 0 36 4959 275 27We can now use a more involved collapsing scheme as follows.
> a <- accumulate(producers, collapse = sbi*size ~ sbi + sbi3 + sbi2
+ , test = min_records(5), fun = mean, na.rm=TRUE)
> head(round(a))
sbi size level industrial trade other other_income total
1 3410 8 1 364397 2859 33 353 546117
2 2840 7 0 23160 823 49 329 25812
3 2752 5 2 19526 39 52 151 20603
4 3120 6 0 20710 504 112 200 21702
5 2524 7 0 27954 1268 55 456 30468
6 3410 6 1 364397 2859 33 353 546117For (sbi,size) == (2752,5) we have 2 levels of
collapsing. In other words, for that aggregate, all records in
sbi3 == 275 were used.
trade and
total using the cumulate function under the
same collapsing scheme as defined above.(sbi,size) have been
collapsed to level 0, 1, 2, or 3. Tabulate them.sbi code and compute the means of all variables.Collapsing schemes can be represented in data frames that have the form
[target group, parent of target group, parent of parent of target group,...].The package comes with a helper function that creates such a scheme from hierarchical classifications that are encoded as digits.
For the sbi example we can do the following to derive a
collapsing scheme.
> sbi <- unique(producers$sbi)
> csh <- csh_from_digits(sbi)
> names(csh)[1] <- "sbi"
> head(csh)
sbi A1 A2 A3 A4
1 3410 3410 341 34 3
2 2840 2840 284 28 2
3 2752 2752 275 27 2
4 3120 3120 312 31 3
5 2524 2524 252 25 2
6 2875 2875 287 28 2Here, the column sbi denotes the original (maximally)
5-digit codes, A1 the 4-digit codes, and so on. It is
important that the name of the first column matches a column in the data
to be agregated. Both cumlate and accumulate
accept such a data frame as an argument. Here is an example with
cumulate.
> a <- cumulate(producers, collapse = csh, test = function(d) nrow(d) >= 5
+ , mu_total = mean(total, na.rm=TRUE)
+ , sd_total = sd(total, na.rm=TRUE))
> head(a)
sbi level mu_total sd_total
1 3410 0 546117.22 844001.47
2 2840 0 31265.28 35053.37
3 2752 2 20603.08 31286.51
4 3120 0 26548.61 26784.60
5 2524 0 23434.68 18022.32
7 2875 0 15962.24 9640.14In this representation is is not possible to use multiple grouping variables, unless you combine multiple grouping variables into a single one, for example by pasting them together.
The advantage of this representation is that it allows users to externally define a (manually edited) collapsing scheme.
csh to compute the median of all numerical
variables of the producers dataset with
accumulate (hint: you need to remove the size
variable).There are several options to define test on groups of records:
min_records(), min_complete(), or
frac_complete().from_validator() function.Let us look at a small example for each case. For comparison we will always test that there are a minimum of five records.
> # load the data again to loose columns 'sbi2' and 'sbi3' and work
> # with the original data.
> data(producers)
> # 1. using a helper function
> a <- accumulate(producers, collapse = sbi*size ~ sbi
+ , test = min_records(5)
+ , fun = mean, na.rm=TRUE)
> # 2. using a 'validator' object
> rules <- validate::validator(nrow(.) >= 5)
> a <- accumulate(producers, collapse = sbi*size ~ sbi
+ , test = from_validator(rules)
+ , fun = mean, na.rm=TRUE)
> # 3. using a custom function
> a <- accumulate(producers, collapse=sbi*size ~ sbi
+ , test = function(d) nrow(d) >= 5
+ , fun = mean, na.rm=TRUE)An aggregate may be something more complex than a scalar. The
accumulate package also supports complex aggregates such as
linear models.
> a <- cumulate(producers, collapse = sbi*size ~ sbi
+ , test = min_complete(5, c("other_income","trade"))
+ , model = lm(other_income ~ trade)
+ , mean_other = mean(other_income, na.rm=TRUE))
> head(a)
sbi size level model mean_other
1 3410 8 NA <logical> NA
2 2840 7 1 <lm> 249.3333
3 2752 5 NA <logical> NA
4 3120 6 0 <lm> 199.8889
5 2524 7 0 <lm> 456.2500
6 3410 6 NA <logical> NAHere, we demand that there are at least five records available for estimating the model.
The linear models are stored in a list of type
object_list. Subsets or individual elements can be accessed
as usual with data frames.
> a$model[[1]]
[1] NA
> a$model[[2]]
Call:
lm(formula = other_income ~ trade)
Coefficients:
(Intercept) trade
221.59429 0.06937 If you write your own test function from scratch, it is easy to
overlook some edge cases like the occurrence of missing data, a column
that is completely NA, or receiving zero records. The
function smoke_test() accepts a data set and a test
function and runs the test function on several common edge cases based
on the dataset. It does not check whether the test function
works as expected, but it checks that the output is TRUE or
FALSE in all cases and reports errors, warnings and mesages
if they occur.
As an example we construct a test function that checks whether one of the variables has sufficient non-zero values.
> my_test <- function(d) sum(other != 0) > 3
> smoke_test(producers, my_test)
Test with full dataset raised issues.
ERR: object 'other' not foundOops, we forgot to refer to the data set. Let’s try it again.
> my_test <- function(d) sum(d$other != 0) > 3
> smoke_test(producers, my_test)
Test with full dataset raised issues.
NA detected in output (must be TRUE or FALSE)
Test with first record and other is NA raised issues.
NA detected in output (must be TRUE or FALSE)
Test with first record and all values NA raised issues.
NA detected in output (must be TRUE or FALSE)
Test with full dataset and sbi is NA for all records raised issues.
NA detected in output (must be TRUE or FALSE)
Test with full dataset and size is NA for all records raised issues.
NA detected in output (must be TRUE or FALSE)
Test with full dataset and industrial is NA for all records raised issues.
NA detected in output (must be TRUE or FALSE)
Test with full dataset and trade is NA for all records raised issues.
NA detected in output (must be TRUE or FALSE)
Test with full dataset and other is NA for all records raised issues.
NA detected in output (must be TRUE or FALSE)
Test with full dataset and other_income is NA for all records raised issues.
NA detected in output (must be TRUE or FALSE)
Test with full dataset and total is NA for all records raised issues.
NA detected in output (must be TRUE or FALSE)Our function is not robust against occurrence of NA.
Here’s a third attempt.
sbi*size ~ sbi1 + sbi2 as collapsing scheme. Make sure
there are at least 10 records in each group.industrial and
total, but demand that there are not more than 20% zeros in
other. Use csh as collapsing scheme.