Chapter 1 Regression and the Normal Distribution

Chapter description

Regression analysis is a statistical method that is widely used in many fields of study, with actuarial science being no exception. This chapter introduces the role of the normal distribution in regression and the use of logarithmic transformations in specifying regression relationships.

1.1 Fitting a normal distribution

In this section, you learn how to:

Calculate and interpret two basic summary statistics
Fit a data set to a normal curve
Calculate probabilities under a standard normal curve

1.1.1 Video

Video Overhead Details

Hide

A Details. Description of the data

To illustrate a data set that can be analyzed using regression methods, we consider some data included in Galton’s 1885 paper. These data include the heights of 928 adult children (child_ht), together with an index of their parents’ height (parent_ht). Here, all female heights were multiplied by 1.08, and the index was created by taking the average of the father’s height and rescaled mother’s height. Galton was aware that the parents’ and the adult child’s height could each be adequately approximated by a normal curve. In developing regression analysis, he provided a single model for the joint distribution of heights.

heights <- read.csv("CSVData\\galton_height.csv", header = TRUE)
#heights <- read.csv("https://assets.datacamp.com/production/repositories/2610/datasets/c85ede6c205d22049e766bd08956b225c576255b/galton_height.csv", header = TRUE)
plot(jitter(heights$parent_ht),jitter(heights$child_ht), ylim = c(60,80), xlim = c(60,80),
     ylab = "height of child", xlab = "height of parents")
abline(lm(heights$child_ht~heights$parent_ht))
abline(0,1,col = "red")

Hide

B Details. Read and examine data structure

The data has already been read into a dataset called heights. Examine the structure of the data with the function str() and use the head() command to looks at the first few records.

heights <- read.csv("CSVData\\galton_height.csv",header = TRUE)
#heights <- read.csv("https://assets.datacamp.com/production/repositories/2610/datasets/c85ede6c205d22049e766bd08956b225c576255b/galton_height.csv", header = TRUE)
str(heights)
head(heights)

'data.frame':   928 obs. of  2 variables:
 $ child_ht : num  72.2 73.2 73.2 73.2 68.2 ...
 $ parent_ht: num  74.5 74.5 74.5 74.5 73.5 73.5 73.5 73.5 73.5 73.5 ...
  child_ht parent_ht
1     72.2      74.5
2     73.2      74.5
3     73.2      74.5
4     73.2      74.5
5     68.2      73.5
6     69.2      73.5

Hide

C Details. Summary stats for parents’ height

Next, examine the distribution of the child’s height and then examine the distribution of the parents height.

ht_par <- heights$parent_ht
hist(ht_par)

mean(ht_par)
sd(ht_par)

[1] 69.26293
[1] 1.912274

Hide

D. Fit a normal curve to parents’ height details

(mparent <- mean(ht_par))
(sdparent <- sd(ht_par))
x <- seq(60, 80,by = 0.1)
hist(ht_par, freq = FALSE)
lines(x, dnorm(x, mean = mparent, sd = sdparent), col = "blue")

[1] 69.26293
[1] 1.912274

Hide

E Details. Use the normal approximation to determine the probability of the height of tall parents

TallHeight <- 72
pnorm(TallHeight, mean = mparent, sd = sdparent)
pnorm(72, mean = mean(ht_par), sd = sd(ht_par))
(StdUnitsTallHeight <- (TallHeight - mparent)/sdparent)
pnorm(StdUnitsTallHeight, mean = 0, sd = 1)

[1] 0.9238302
[1] 0.9238302
[1] 1.431317
[1] 0.9238302

1.1.2 Exercise. Fitting Galton’s height data

Assignment Text

The Galton data has already been read into a dataframe called heights. These data include the heights of 928 adult children child_ht, together with an index of their parents’ height parent_ht. The video explored the distribution of the parents’ height; in this assignment, we investigate the distribution of the heights of the adult children.

Instructions

Define the height of an adult child as a global variable
Use the function mean() to calculate the mean and the function sd() to calculate the standard deviation
Use the normal approximation and the function pnorm() determine the probability that an adult child’s height is less than 72 inches

1.1.3 Exercise. Visualizing child’s height distribution

Assignment Text

As in the prior exercise, from the Galton dataset heights, the heights of 928 adult children have been used to create a global variable called ht_child. We also have basic summary statistics, the mean height mchild and the standard deviation of heights in sdchild. In this exercise, we explore the fit of the normal curve to this distribution.

Instructions

To visualize the distribution, use the function hist() to calculate the histogram. Use the freq = FALSE option to give a histogram with proportions instead of counts.
Use the function seq() to determine a sequence that can be used for plotting. Then, with the function lines(), superimpose a normal curve on the histogram
Determine the probability that a child’s height is greater than 72 inches

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6IiNoZWlnaHRzIDwtIHJlYWQuY3N2KFwiQ1NWRGF0YVxcXFxnYWx0b25faGVpZ2h0LmNzdlwiLGhlYWRlciA9IFRSVUUpXG5oZWlnaHRzIDwtIHJlYWQuY3N2KFwiaHR0cHM6Ly9hc3NldHMuZGF0YWNhbXAuY29tL3Byb2R1Y3Rpb24vcmVwb3NpdG9yaWVzLzI2MTAvZGF0YXNldHMvYzg1ZWRlNmMyMDVkMjIwNDllNzY2YmQwODk1NmIyMjVjNTc2MjU1Yi9nYWx0b25faGVpZ2h0LmNzdlwiLCBoZWFkZXIgPSBUUlVFKVxuaHRfY2hpbGQgPC0gaGVpZ2h0cyRjaGlsZF9odFxubWNoaWxkIDwtIG1lYW4oaHRfY2hpbGQpXG5zZGNoaWxkIDwtIHNkKGh0X2NoaWxkKSIsInNhbXBsZSI6IiNWaXN1YWxpemUgdGhlIERpc3RyaWJ1dGlvblxuX19fKF9fXywgZnJlcSA9IEZBTFNFKVxuXG4jRGV0ZXJtaW5lIGEgc2VxdWVuY2UuIFRoZW4sIGdyYXBoIGEgaGlzdG9ncmFtIHdpdGggYSBub3JtYWwgY3VydmUgc3VwZXJpbXBvc2VkXG54IDwtIHNlcSg2MCwgODAsYnkgPSAwLjEpXG5fX18oeCwgZG5vcm0oeCxtZWFuID0gbWNoaWxkLCBzZCA9IHNkY2hpbGQpLCBjb2wgPSBcImJsdWVcIilcblxuIyBEZXRlcm1pbmUgdGhlIHByb2JhYmlsaXR5IHRoYXQgYSBjaGlsZCdzIGhlaWdodCBpcyBncmVhdGVyIHRoYW4gNzJcbnByb2IgPC0gMSAtIFxucHJvYiIsInNvbHV0aW9uIjoiaGlzdChodF9jaGlsZCwgZnJlcSA9IEZBTFNFKVxueCA8LSBzZXEoNjAsIDgwLGJ5ID0gMC4xKVxubGluZXMoeCwgZG5vcm0oeCwgbWVhbiA9IG1jaGlsZCwgc2QgPSBzZGNoaWxkKSwgY29sID0gXCJibHVlXCIpXG5wcm9iIDwtIDEgLSBwbm9ybSg3MiwgbWVhbiA9IG1jaGlsZCAsIHNkID0gc2RjaGlsZClcbnByb2IiLCJzY3QiOiJleCgpICU+JSBjaGVja19mdW5jdGlvbihcImhpc3RcIixub3RfY2FsbGVkX21zZz1cIlVzZSB0aGUgaGlzdCBjb21tYW5kIHRvIGNyZWF0ZSBhIGhpc3RvZ3JhbSBvZiB0aGUgY2hpbGRyZW4ncyBoZWlnaHRzLlwiKSAlPiUge1xuICBjaGVja19hcmcoLiwgXCJ4XCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHdoYXQgZGF0YSB0byBtYWtlIGEgaGlzdG9ncmFtIGZyb20/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiTWFrZSBzdXJlIHRvIGNyZWF0ZSBhIGhpc3RvZ3JhbSBvZiB0aGUgY2hpbGRyZW4ncyBoZWlnaHRzLlwiKVxuICBjaGVja19hcmcoLiwgXCJmcmVxXCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHRoYXQgeW91IHdvdWxkIGxpa2UgYSBkZW5zaXR5IGhpc3RvZ3JhbT9cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJNYWtlIHN1cmUgdG8gY3JlYXRlIGEgZGVuc2l0eSBoaXN0b2dyYW0gaW5zdGVhZCBvZiBhIGZyZXF1ZW5jeSBoaXN0b2dyYW0uXCIpXG59XG5leCgpICU+JSBjaGVja19mdW5jdGlvbihcImxpbmVzXCIsbm90X2NhbGxlZF9tc2c9XCJQbGVhc2UgdXNlIHRoZSBsaW5lcyBmdW5jdGlvbiB0byBvdmVybGF5IGEgbm9ybWFsIGN1cnZlIG9uIHlvdXIgaGlzdG9ncmFtXCIpICU+JSB7XG4gIGNoZWNrX2FyZyguLCBcInhcIixhcmdfbm90X3NwZWNpZmllZF9tc2c9XCJIYXZlIHlvdSBzcGVjaWZpZWQgdGhlIGluZGVwZW5kYW50IHZhcmlhYmxlIGNvcnJlY3RseT9cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJSZW1lYmVyIHRoYXQgd2UgYXJlIGludGVyZXN0ZWQgaW4gaG93IHByb2JhYmlsaXR5IGNoYW5nZXMgYXMgb3VyIHZhbHVlIG9mIHggY2hhbmdlcyFcIilcbiAgY2hlY2tfYXJnKC4sIFwieVwiLGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkhhdmUgeW91IHNwZWNpZmllZCB0aGUgZGVwZW5kYW50IHZhcmlhYmxlIGNvcnJlY3RseT9cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJSZW1lYmVyIHRoYXQgd2UgYXJlIGludGVyZXN0ZWQgaW4gaG93IHByb2JhYmlsaXR5IGNoYW5nZXMgYXMgb3VyIHZhbHVlIG9mIHggY2hhbmdlcyFcIilcbn1cbmV4KCkgJT4lIGNoZWNrX29iamVjdChcInByb2JcIiwgdW5kZWZpbmVkX21zZz1cIk1ha2Ugc3VyZSB0byBhc3NpZ24gdGhlIHByb2JhYmlsaXR5IG9mIGEgY2hpbGQncyBoZWlnaHQgYmVpbmcgZ3JlYXRlciB0aGFuIDcyIGluY2hlcyB0byBwcm9iLlwiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIk1ha2Ugc3VyZSB0byBmaW5kIHRoZSBwcm9iYWJpbGl0eSBvZiBhIGNoaWxkJ3MgaGVpZ2h0IGJlaW5nIEdSRUFURVIgdGhhbiA3MiBpbmNoZXMuXCIpXG5zdWNjZXNzX21zZyhcIkV4Y2VsbGVudCEgVmlzdWFsaXppbmcgYSBkaXN0cmlidXRpb24sIGVzcGVjaWFsbHkgd2l0aCByZWZlcmVuY2UgdG8gYSBub3JtYWwsIGlzIGltcG9ydGFudCBmb3IgY29tbXVuaWNhdGluZyByZXN1bHRzIG9mIHlvdXIgYW5hbHlzaXMuXCIpIiwiaGludCI6IlVzZSB0aGUgZnVuY3Rpb24gPGEgaHJlZj1cImh0dHBzOi8vd3d3LnJkb2N1bWVudGF0aW9uLm9yZy9wYWNrYWdlcy9zdGF0cy92ZXJzaW9ucy8zLjUuMC90b3BpY3MvTm9ybWFsL1wiPmRub3JtKCk8L2E+IHRvIGNhbGN1bGF0ZSB0aGUgbm9ybWFsIGRlbnNpdHksIHNpbWlsYXIgdG8gdGhlIGN1bXVsYXRpdmUgcHJvYmFiaWxpdGVzIHRoYXQgeW91IGNhbGN1bGF0ZWQgdXNpbmcgPGEgaHJlZj1cImh0dHBzOi8vd3d3LnJkb2N1bWVudGF0aW9uLm9yZy9wYWNrYWdlcy9zdGF0cy92ZXJzaW9ucy8zLjUuMC90b3BpY3MvTm9ybWFsL1wiPnBub3JtKCk8L2E+PC9wPlxuXG48cD5UbyBjYWxjdWxhdGUgcHJvYmFiaWxpdGllcyBncmVhdGVyIHRoYXQgYW4gYW1vdW50LCBzaW1wbHkgdXNlIDEgbWludXMgdGhlIGN1bXVsYXRpdmUgcHJvYmFiaWxpdHkifQ==

1.2 Visualizing distributions

In this section, you learn how to:

Calculate and interpret distributions using histograms
Calculate and interpret distributions using density plots

1.2.1 Video

Video Overhead Details

Hide

Details. Data descriptions

For our first look at an insurance data set, we consider data from Rempala and Derrig (2005). They considered claims arising from automobile bodily injury insurance coverages. These are amounts incurred for outpatient medical treatments that arise from automobile accidents, typically sprains, broken collarbones and the like. The data consists of a sample of 272 claims from Massachusetts that were closed in 2001 (by “closed,” we mean that the claim is settled and no additional liabilities can arise from the same accident). Rempala and Derrig were interested in developing procedures for handling mixtures of “typical” claims and others from providers who reported claims fraudulently. For this sample, we consider only those typical claims, ignoring the potentially fraudulent ones.

# Reformat Data Set
injury <- read.csv("CSVData\\MassBodilyInjury.csv",header = TRUE)
str(injury)
head(injury)
#  PICK THE SUBSET OF THE DATA CORRESPONDING TO PROVIDER A
injury2 <- subset(injury, providerA ! = 0 )
injury2$claims <- 1000*injury2$claims
injury2$logclaims <- log(injury2$claims)
injury3 <- injury2[c("claims","logclaims")]
#write.csv(injury3,"CSVData\\MassBI.csv",row.names = FALSE)

Hide

A Details. Bring in Data, Introduce Logarithmic Claims

injury <- read.csv("CSVData\\MassBI.csv",header = TRUE)
#  CHECK THE NAMES, DIMENSION IN THE FILE AND LIST THE FIRST 8 OBSERVATIONS  ;
str(injury)
head(injury)
attach(injury)
claims <- injury$claims
par(mfrow = c(1, 2))
hist(claims)
hist(logclaims)

'data.frame':   272 obs. of  2 variables:
 $ claims   : int  45 47 70 75 77 92 117 117 140 145 ...
 $ logclaims: num  3.81 3.85 4.25 4.32 4.34 ...
  claims logclaims
1     45  3.806662
2     47  3.850148
3     70  4.248495
4     75  4.317488
5     77  4.343805
6     92  4.521789

Hide

B Details. Show how to get a finer grid for histograms

par(mfrow = c(1, 2))
hist(logclaims)
hist(logclaims,breaks = 15)

Hide

C Details. Introduce the density plot

par(mfrow = c(1, 2))
plot(density(logclaims))
hist(logclaims, breaks = 15,freq = FALSE)
lines(density(logclaims))

1.2.2 Exercise. Visualizing bodily injury claims with density plots

Assignment Text

In the prior video, you learned about the Massachusetts bodily injury dataset. This dataframe, injury, has been read in and the global variable claims has been created. This assignment reviews the hist() function for visualizing distributions and allows you to explore density plotting, a smoothed version of the histogram.

Instructions

Use the function log() to create the logarithmic version of the claims variable
Calculate a histogram of logarithmic with 40 bins using an option in the hist() function, breaks =.
Create a density plot of logarithmic claims using the functions plot() and density().
Repeat the density plot, this time using a more refined bandwidth equal to 0.03. Use an option in the density() function, bw =.

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6IiNpbmp1cnkgPC0gcmVhZC5jc3YoXCJDU1ZEYXRhXFxcXE1hc3NCSS5jc3ZcIixoZWFkZXIgPSBUUlVFKVxuaW5qdXJ5IDwtIHJlYWQuY3N2KFwiaHR0cHM6Ly9hc3NldHMuZGF0YWNhbXAuY29tL3Byb2R1Y3Rpb24vcmVwb3NpdG9yaWVzLzI2MTAvZGF0YXNldHMvOGNjYTE5ZDA1MDNmY2Y2ZTlkMzBkOWNiOTEyZGU1YmE5NWVjYjljMS9NYXNzQkkuY3N2XCIsIGhlYWRlciA9IFRSVUUpXG5jbGFpbXMgPC0gaW5qdXJ5JGNsYWltcyIsInNhbXBsZSI6IiNDcmVhdGUgdGhlIGxvZ2FyaXRobWljIGNsYWltcyB2YXJpYWJsZVxubG9nY2xhaW1zIDwtIF9fX1xuXG4jQ3JlYXRlIGEgaGlzdG9ncmFtIHVzaW5zIDQwIGJpbnNcbl9fXyhsb2djbGFpbXMsIGJyZWFrcyA9IDQwLGZyZXEgPSBGQUxTRSlcbmJveCgpXG5cbiMgQ3JlYXRlIGEgZGVuc2l0eSBwbG90IG9mIGxvZ2FyaXRobWljIGNsYWltc1xucGxvdChfX18obG9nY2xhaW1zKSlcblxuIyBDcmVhdGUgYSBkZW5zaXR5IHBsb3Qgb2YgbG9nYXJpdGhtaWMgY2xhaW1zIHdpdGggYSBiYW5kd2lkdGggb2YgMC4wM1xuX19fIiwic29sdXRpb24iOiJsb2djbGFpbXMgPC0gbG9nKGNsYWltcylcbmhpc3QobG9nY2xhaW1zICwgYnJlYWtzID0gNDAsZnJlcSA9IEZBTFNFKVxuYm94KClcbnBsb3QoZGVuc2l0eShsb2djbGFpbXMpKVxucGxvdChkZW5zaXR5KGxvZ2NsYWltcywgYncgPSAwLjAzKSkiLCJzY3QiOiJleCgpICU+JSBjaGVja19vYmplY3QoXCJsb2djbGFpbXNcIix1bmRlZmluZWRfbXNnPVwiTWFrZSBzdXJlIHRvIGFzc2lnbiB0aGUgbG9nIG9mIHRoZSBjbGFpbXMgZGF0YSB0byBgbG9nY2xhaW1zYC5cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2cgPSBcIllvdSBtYWRlIGFuIGVycm9yIGluIHRoZSBkZWZpbml0aW9uIG9mIHRoZSBsb2dhcml0aG1pYyBjbGFpbXMuIENoZWNrIG91dCB0aGUgZGVmaW5pdGlvbiBvZiB0aGUgbG9nKCkgZnVuY3Rpb24uXCIpXG5leCgpICU+JSBjaGVja19mdW5jdGlvbihcImhpc3RcIixub3RfY2FsbGVkX21zZz1cIk1ha2Ugc3VyZSB0byB1c2UgYGhpc3RgIHRvIGNyZWF0ZSBhIGhpc3RvZ3JhbS5cIikgJT4lIHtcbiAgY2hlY2tfYXJnKC4sIFwieFwiLGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkRpZCB5b3UgYXNzaWduIHggdG8gYmUgdGhlIGxvZyBvZiBvdXIgY2xhaW1zIGRhdGE/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiUGxlYXNlIGNyZWF0ZSBhIGhpc3RvZ3JhbSBvZiBsb2djbGFpbXMuXCIpXG4gIGNoZWNrX2FyZyguLCBcImZyZXFcIixhcmdfbm90X3NwZWNpZmllZF9tc2c9XCJEaWQgeW91IHNwZWNpZnkgdGhhdCB3ZSB3b3VsZCBsaWtlIGEgZGVuc2l0eSBoaXN0b2dyYW0/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiUGxlYXNlIGNyZWF0ZSBhIGRlbnNpdHkgaGlzdG9ncmFtIGluc3RlYWQgb2YgYSBmcmVxdWVuY3kgaGlzdG9ncmFtLlwiKVxuICBjaGVja19hcmcoLiwgXCJicmVha3NcIixhcmdfbm90X3NwZWNpZmllZF9tc2c9XCJEaWQgeW91IHNwZWNpZnkgdGhlIG51bWJlciBvZiBicmVha3Mgd2Ugd291bGQgbGlrZSBpbiBvdXIgaGlzdG9ncmFtP1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIk1ha2Ugc3VyZSB0byBzZXQgYnJlYWtzIGVxdWFsIHRvIDQwIVwiKVxufVxuZXgoKSAlPiUgY2hlY2tfZnVuY3Rpb24oXCJib3hcIixub3RfY2FsbGVkX21zZz1cIk1ha2Ugc3VyZSB0byBjYWxsIGBib3hgIGluIG9yZGVyIHRvIGNyZWF0ZSBhIGRlY29yYXRpdmUgYm94IGFyb3VuZCBvdXIgaGlzdG9ncmFtIVwiKVxuZXgoKSAlPiUgY2hlY2tfZnVuY3Rpb24oXCJwbG90XCIsaW5kZXg9MSxub3RfY2FsbGVkX21zZz1cIkhhdmUgeW91IHBsb3R0ZWQgdGhlIGRlbnNpdHkgb2YgdGhlIGxvZyBvZiBjbGFpbXM/XCIpICU+JSBjaGVja19hcmcoLiwgXCJ4XCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHRoZSBkYXRhIHdlIHNob3VsZCB1c2UgdG8gY3JlYXRlIHRoZSBwbG90P1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIlVzZSB0aGUgZGVuc2l0eSBmdW5jdGlvbiB0byBwbG90IHRoZSBkZW5zaXR5IG9mIGxvZ2NsYWltcy5cIilcbmV4KCkgJT4lIGNoZWNrX2Z1bmN0aW9uKFwicGxvdFwiLGluZGV4PTIsbm90X2NhbGxlZF9tc2c9XCJDcmVhdGUgYW5vdGhlciBwbG90IHVzaW5nIGBwbG90YCB0aGF0IGRpc3BsYXlzIHRoZSBkZW5zaXR5IG9mIGxvZ2FyaXRobWljIGNsYWltcyB3aXRoIGEgYmlud2lkdGNoIG9mIDAuMDMuXCIpICU+JVxuICBjaGVja19hcmcoLiwgXCJ4XCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHRoZSBkYXRhIHdlIHNob3VsZCB1c2UgdG8gY3JlYXRlIHRoZSBwbG90P1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIlVzZSB0aGUgZGVuc2l0eSBmdW5jdGlvbiB0byBwbG90IHRoZSBkZW5zaXR5IG9mIGxvZ2NsYWltcy4gQWxzbyByZW1lbWJlciB0aGF0IGBid2AgaXMgYW4gYXJndW1lbnQgb2YgZGVuc2l0eSwgbm90IHBsb3QuXCIpXG5zdWNjZXNzX21zZyhcIkV4Y2VsbGVudCEgVmlzdWFsaXppbmcgdGhlIGRpc3RyaWJ1dGlvbiBpcyBpbXBvcnRhbnQgYW5kIHNtb290aGluZyB0ZWNobmlxdWVzIGFsbG93IHZpZXdlcnMgdG8gc2VlIGltcG9ydGFudCBwYXR0ZXJucyB3aXRob3V0IGJlaW5nIGRpc3RyYWN0ZWQgYnkgcmFuZG9tIGZsdWN0YXRpb25zLlwiKSIsImhpbnQiOiJSZW1lbWJlciB0aGF0IHlvdSBjYW4gbmVzdCBmdW5jdGlvbnMgdG8gY3JlYXRlIGEgZGVzaXJlZCBvdXRjb21lISBJbiB0aGlzIGV4YW1wbGUsIGl0IGlzIHVzZWZ1bCB0byBuZXN0IDxjb2RlPmRlbnNpdHkoKTwvY29kZT4gaW5zaWRlIG9mIDxjb2RlPnBsb3QoKTwvY29kZT4gdG8gY3JlYXRlICBkZW5zaXR5IHBsb3QhPC9wPlxuXG48cD5iYW5kd2lkdGggb3IgPGNvZGU+Ync8L2NvZGU+IGlzIGFuIGFyZ3VtZW50IG9mIDxjb2RlPmRlbnNpdHkoKTwvY29kZT4sIHNvIGluIG9yZGVyIHRvIGNoYW5nZSB0aGUgZGVmYXVsdCB2YWx1ZSwgeW91IG11c3Qgc3BlY2lmeSBpdCBpbnNpZGUgb2YgeW91ciA8Y29kZT5kZW5zaXR5KCk8L2NvZGU+IGNhbGwuIn0=

1.3 Summarizing distributions

In this section, you learn how to:

Calculate and interpret basic summary statistics
Calculate and interpret distributions using boxplots
Calculate and interpret distributions using qq plots

1.3.1 Video

Video Overhead Details

Hide

A Details. Summary statistics

injury <- read.csv("CSVData\\MassBI.csv",header = TRUE)
#injury <- read.csv("https://assets.datacamp.com/production/repositories/2610/datasets/8cca19d0503fcf6e9d30d9cb912de5ba95ecb9c1/MassBI.csv", header = TRUE)
attach(injury)

# SUMMARY STATISTICS
summary(injury)
sd(claims);sd(logclaims)
length(claims)

     claims          logclaims     
 Min.   :   45.0   Min.   : 3.807  
 1st Qu.:  892.5   1st Qu.: 6.794  
 Median : 2210.0   Median : 7.701  
 Mean   : 2697.7   Mean   : 7.388  
 3rd Qu.: 3215.0   3rd Qu.: 8.076  
 Max.   :50000.0   Max.   :10.820  
[1] 3944.445
[1] 1.10093
[1] 272

Hide

B Details. Boxplot

# BASIC BOXPLOT
boxplot(logclaims)

quantile(logclaims, probs = 0.75)

# BOXPLOT WITH ANNOTATION
boxplot(logclaims, main = "Boxplot of logclaims")
text(1,  7.6,  "median", cex = 0.7)
text(1,  6.55, "25th percentile", cex = 0.7)
text(1,  7.95, "75th percentile", cex = 0.7)
arrows(1.05, 4.9, 1.05, 3.6, col = "blue", code = 3, angle = 20, length = 0.1)
text(1.1,  4.4, "outliers", cex = 0.7)
text(1.1, 10.9, "outlier",  cex = 0.7)

     75% 
8.075579

Hide

C Details. QQ Plot

summary(injury)
quantile(claims, probs = 0.75)
quantile(logclaims, probs = 0.75)
log(quantile(claims, probs = 0.75))
qnorm(p = 0.75, mean = mean(logclaims), sd = sd(logclaims))
(qnorm(p = 0.75, mean = mean(logclaims), sd = sd(logclaims)) -mean(logclaims)) /
       sd(logclaims)
qnorm(p = 0.75, mean = 0, sd = 1)

#  QUANTILE - QUANTILE PLOT
qqnorm(logclaims)
qqline(logclaims)

     claims          logclaims     
 Min.   :   45.0   Min.   : 3.807  
 1st Qu.:  892.5   1st Qu.: 6.794  
 Median : 2210.0   Median : 7.701  
 Mean   : 2697.7   Mean   : 7.388  
 3rd Qu.: 3215.0   3rd Qu.: 8.076  
 Max.   :50000.0   Max.   :10.820  
 75% 
3215 
     75% 
8.075579 
     75% 
8.075583 
[1] 8.131056
[1] 0.6744898
[1] 0.6744898

1.3.2 Exercise. Summarizing bodily injury claims with box and qq plots

Assignment Text

The Massachusetts bodily injury data has already been read and used to create the global variable claims representing bodily injury claims. The previous video showed how to present the distribution of logarithmic claims which appeared to be approximately normally distributed. However, users are not really interested in log dollars but want to know about a unit of measurement that is more intuitive, such as dollars.

So this assignment is based on claims, not the logarithmic version. You will use the functions boxplot() and qqnorm() to visualize the distribution through boxplots and quantile-quantile, or qq-, plots. But, because we are working with such a skewed distribution, do not be surprised that it is difficult to interpret these results readily.

Instructions

Produce a box plot for claims
Determine the 25th empirical percentile for claims using the quantile() function.
Determine the 25th percentile for claims based on a normal distribution using the qnorm() function.
Produce a normal qq plot for claims using the function qqnorm(). The qqline() function is handy for producing a reference line.

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6IiNpbmp1cnkgPC0gcmVhZC5jc3YoXCJDU1ZEYXRhXFxcXE1hc3NCSS5jc3ZcIiwgaGVhZGVyID0gVFJVRSlcbmluanVyeSA8LSByZWFkLmNzdihcImh0dHBzOi8vYXNzZXRzLmRhdGFjYW1wLmNvbS9wcm9kdWN0aW9uL3JlcG9zaXRvcmllcy8yNjEwL2RhdGFzZXRzLzhjY2ExOWQwNTAzZmNmNmU5ZDMwZDljYjkxMmRlNWJhOTVlY2I5YzEvTWFzc0JJLmNzdlwiLCBoZWFkZXIgPSBUUlVFKVxuY2xhaW1zIDwtIGluanVyeSRjbGFpbXMiLCJzYW1wbGUiOiIjUHJvZHVjZSBhIGJveCBwbG90IGZvciBjbGFpbXNcbl9fXyhjbGFpbXMpXG5cbiNEZXRlcm1pbmUgdGhlIDI1dGggZW1waXJpY2FsIHBlcmNlbnRpbGUgZm9yIGNsYWltc1xucTI1IDwtIF9fXyhjbGFpbXMsIHByb2JzID0gX19fKVxucTI1XG5cbiNEZXRlcm1pbmUgdGhlIDI1dGggcGVyY2VudGlsZSBmb3IgY2xhaW1zIGJhc2VkIG9uIGEgbm9ybWFsIGRpc3RyaWJ1dGlvblxucW4yNSA8LSBfX18ocCA9IF9fXywgbWVhbiA9IG1lYW4oY2xhaW1zKSwgc2QgPSBzZChjbGFpbXMpKVxucW4yNVxuXG4jUHJvZHVjZSBhIG5vcm1hbCBxcSBwbG90IGZvciBjbGFpbXNcbl9fXyhjbGFpbXMpXG5fX18oY2xhaW1zKSIsInNvbHV0aW9uIjoiIyBTb2x1dGlvblxuYm94cGxvdChjbGFpbXMpXG5xMjUgPC0gcXVhbnRpbGUoY2xhaW1zLCBwcm9icyA9IDAuMjUpXG5xMjVcbnFuMjUgPC0gcW5vcm0ocCA9IDAuMjUsIG1lYW4gPSBtZWFuKGNsYWltcyksIHNkID0gc2QoY2xhaW1zKSlcbnFuMjVcbnFxbm9ybShjbGFpbXMpXG5xcWxpbmUoY2xhaW1zKSAgICIsInNjdCI6ImV4KCkgJT4lIGNoZWNrX2Z1bmN0aW9uKFwiYm94cGxvdFwiLG5vdF9jYWxsZWRfbXNnPVwiTWFrZSBzdXJlIHRvIHVzZSBgYm94cGxvdGAgdG8gY3JlYXRlIGEgdmlzdWFsIHJlcHJlc2VudGF0aW9uIG9mIHRoZSBkYXRhLlwiKSAlPiUgY2hlY2tfYXJnKC4sIFwieFwiLCBhcmdfbm90X3NwZWNpZmllZF9tc2c9XCJIYXZlIHlvdSBzcGVjaWZpZWQgdGhlIGRhdGEgUiBzaG91bGQgdXNlIHRvIGNyZWF0ZSB0aGUgYm94cGxvdD9cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJQbGVhc2UgY3JlYXRlIGEgYm94cGxvdCBvZiBgY2xhaW1zYC5cIilcbmV4KCkgJT4lIGNoZWNrX2Z1bmN0aW9uKFwicXVhbnRpbGVcIixub3RfY2FsbGVkX21zZz1cIkhhdmUgeW91IGNhbGxlZCBgcXVhbnRpbGVgIHRvIGZpbmQgdGhlIGVtcGlyaWNhbCBwZXJjZW50aWxlP1wiKSAlPiUge1xuICBjaGVja19hcmcoLiwgXCJwcm9ic1wiLGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkhhdmUgeW91IHNwZWNpZmllZCB3aGF0IGVtcGlyaWNhbCBwZXJjZW50aWxlIHdlIHdvdWxkIGxpa2UgdG8gZmluZD9cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJJZiB3ZSB3YW50IHRvIGZpbmQgdGhlIFl0aCBwZXJjZW50aWxlLCBtYWtlIHN1cmUgdG8gc2V0IHByb2JzIGVxdWFsIHRvIFkgaW4gZGVjaW1hbCBmb3JtYXQuXCIpXG4gIGNoZWNrX2FyZyguLCBcInhcIixhcmdfbm90X3NwZWNpZmllZF9tc2c9XCJIYXZlIHlvdSBzcGVjaWZpZWQgd2hpY2ggZGF0YSBSIHNob3VsZCB1c2UgdG8gZmluZCB0aGUgMjV0aCBlbXBpcmljYWwgcGVyY2VudGlsZT9cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJNYWtlIHN1cmUgdG8gZmluZCB0aGUgMjV0aCBlbXBpcmljYWwgcGVyY2VudGlsZSBvZiBgY2xhaW1zYC5cIilcbn1cbmV4KCkgJT4lIGNoZWNrX29iamVjdChcInEyNVwiLHVuZGVmaW5lZF9tc2c9XCJNYWtlIHN1cmUgdG8gYXNzaWduIHRoZSAyNXRoIHF1YW50aWxlIHRvIGBxMjVcIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJNYWtlIHN1cmUgdG8gZmluZCB0aGUgMjV0aCBxdWFudGlsZSBvZiBjbGFpbXMuXCIpXG5leCgpICU+JSBjaGVja19vYmplY3QoXCJxbjI1XCIsdW5kZWZpbmVkX21zZz1cIk1ha2Ugc3VyZSB0byBhc3NpZ24gdGhlIG5vcm1hbCB2YWx1ZSBhc3NvY2lhdGVkIHdpdGggdGhlIDI1dGggcGVyY2VudGlsZSB0byBxbjI1XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiV2UgY2FuIHVzZSBhIG5vcm1hbCBhcHByb3hpbWF0aW9uIHRvIHNvbHZlIGZvciBxbjI1IHVzaW5nIGBxbm9ybWAuXCIpXG5leCgpICU+JSBjaGVja19mdW5jdGlvbihcInFxbm9ybVwiLG5vdF9jYWxsZWRfbXNnPVwiSGF2ZSB5b3UgY2FsbGVkIGBxcW5vcm1gIHRvIGNyZWF0ZSBhIG5vcm1hbCBxcSBwbG90P1wiKSAlPiUgY2hlY2tfYXJnKC4sIFwieVwiLGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkhhdmUgeW91IHNwZWNpZmllZCB3aGljaCBkYXRhIFIgc2hvdWxkIHVzZSB0byBjcmVhdGUgdGhlIHFxIHBsb3Q/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiTWFrZSBzdXJlIHRoYXQgeW91IGFyZSBjcmVhdGluZyBhIHFxLXBsb3QgZm9yIGBjbGFpbXNgLlwiKVxuZXgoKSAlPiUgY2hlY2tfZnVuY3Rpb24oXCJxcWxpbmVcIixub3RfY2FsbGVkX21zZz1cIkhhdmUgeW91IGNhbGxlZCBgcXFsaW5lYCB0byBvdmVybGF5IGEgbm9ybWFsIGxpbmUgb24geW91ciBxcSBwbG90P1wiKSAlPiUgY2hlY2tfYXJnKC4sIFwieVwiLGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkhhdmUgeW91IHNwZWNpZmllZCB3aGljaCBkYXRhIFIgc2hvdWxkIHVzZSB0byBjcmVhdGUgYW5kIHBsb3QgdGhlIGxpbmU/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiTWFrZSBzdXJlIHRoYXQgeW91IGFyZSBhZGRpbmcgYSBxcS1saW5lIGZvciBgY2xhaW1zYC5cIilcbnN1Y2Nlc3NfbXNnKFwiQ29uZ3JhdHVsYXRpb25zIG9uIGxlYXJuaW5nIGFib3V0IGJveCBhbmQgcXEgcGxvdHMuIEFsdGhvdWdoIHlvdSBhcmUgdW5saWtlbHkgdG8gc2hvdyB0aGVzZSBwbG90cyB0byBjb25zdW1lcnMgb2YgeW91ciBhbmFseXNpcywgeW91IHdpbGwgZmluZCB0aGVtIHVzZWZ1bCB0b29scyBhcyB3ZSBleHBsb3JlIG11bHRpdmFyaWF0ZSBhc3BlY3RzIG9mIGRhdGEuXCIpIiwiaGludCI6Ik5vdGUgdGhhdCA8YSBocmVmPVwiaHR0cHM6Ly93d3cucmRvY3VtZW50YXRpb24ub3JnL3BhY2thZ2VzL3N0YXRzL3ZlcnNpb25zLzMuNS4wL3RvcGljcy9Ob3JtYWwvXCI+cW5vcm0oKTwvYT4gKG9uZSA8Y29kZT5xPC9jb2RlPikgaXMgZm9yIGEgbm9ybWFsIHF1YW50aWxlIGFuZCA8YSBocmVmPVwiaHR0cHM6Ly93d3cucmRvY3VtZW50YXRpb24ub3JnL3BhY2thZ2VzL3N0YXRzL3ZlcnNpb25zLzMuNS4wL3RvcGljcy9xcW5vcm0vXCI+cXFub3JtKCk8L2E+LiAodHdvIDxjb2RlPnE8L2NvZGU+JiMzOTtzISkgaXMgZm9yIHRoZSBub3JtYWwgcXEgcGxvdCJ9

1.3.3 Exercise. Effects on distributions of removing the largest claim

Assignment Text

The Massachusetts bodily injury dataframe injury has been read in; our focus is on the claims variable in that dataset.

In the previous exercise, we learned that the Massachusetts bodily injury claims distribution was not even close to approximately normal (as evidenced by the box and qq- plots). Non-normality may be induced by skewness (that we will handle via transformations in the next section). But, seeming non-normality can also be induced by one or two very large observations (called an outlier later in the course). So, this exercise examines the effects on the distribution of removing the largest claims.

Instructions

Use the function tail() to examine the injury dataset and identify the largest claim
Use the function subset() to create a subset omitting the largest claim
Compare the summary statistics of the omitted claim distribution to the full distribution
Compare the two distributions visually via histograms plotted next to another. par(mfrow = c(1, 2)) is used to organize the plots you create. Do not alter this code.

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6IiNpbmp1cnkgPC0gcmVhZC5jc3YoXCJDU1ZEYXRhXFxcXE1hc3NCSS5jc3ZcIiwgaGVhZGVyID0gVFJVRSlcbmluanVyeSA8LSByZWFkLmNzdihcImh0dHBzOi8vYXNzZXRzLmRhdGFjYW1wLmNvbS9wcm9kdWN0aW9uL3JlcG9zaXRvcmllcy8yNjEwL2RhdGFzZXRzLzhjY2ExOWQwNTAzZmNmNmU5ZDMwZDljYjkxMmRlNWJhOTVlY2I5YzEvTWFzc0JJLmNzdlwiLCBoZWFkZXIgPSBUUlVFKVxuY2xhaW1zIDwtIGluanVyeSRjbGFpbXMiLCJzYW1wbGUiOiIjIEV4YW1pbmUgdGhlIHRhaWwgb2YgdGhlIGBpbmp1cnlgIGRhdGFzZXRcbnRhaWwoX19fKVxuXG4jIENyZWF0ZSBhIHN1YnNldCBvbWl0dGluZyB0aGUgbGFyZ2VzdCBjbGFpbVxuaW5qdXJ5MiA8LSBzdWJzZXQoaW5qdXJ5LCBfX18pXG5cbiMgQ29tcGFyZSB0aGUgc3VtbWFyeSBzdGF0aXN0aWNzIG9mIHRoZSBvbWl0dGVkIGNsYWltIGRpc3RyaWJ1dGlvbiB0byB0aGUgZnVsbCBkaXN0cmlidXRpb25cbnN1bW1hcnkoX19fKVxuc3VtbWFyeShpbmp1cnkyKVxuXG4jIENvbXBhcmUgdGhlIHR3byBkaXN0cmlidXRpb25zIHZpc3VhbGx5IHZpYSBoaXN0b2dyYW1zIHBsb3R0ZWQgbmV4dCB0byBhbm90aGVyXG5wYXIobWZyb3cgPSBjKDEsIDIpKVxuaGlzdChfX18sIGZyZXEgPSBGQUxTRSwgIG1haW4gPSBcIkZ1bGwgRGF0YVwiKVxuaGlzdChfX18sIGZyZXEgPSBGQUxTRSwgIG1haW4gPSBcIkxhcmdlc3QgQ2xhaW0gT21pdHRlZFwiKSIsInNvbHV0aW9uIjoidGFpbChpbmp1cnkpXG5pbmp1cnkyIDwtIHN1YnNldChpbmp1cnksIGNsYWltcyA8IDI1MDAwIClcbnN1bW1hcnkoaW5qdXJ5KVxuc3VtbWFyeShpbmp1cnkyKVxucGFyKG1mcm93ID0gYygxLCAyKSlcbmhpc3QoY2xhaW1zLCBmcmVxID0gRkFMU0UsICBtYWluID0gXCJGdWxsIERhdGFcIilcbmhpc3QoaW5qdXJ5MiRjbGFpbXMsIGZyZXEgPSBGQUxTRSwgIG1haW4gPSBcIkxhcmdlc3QgQ2xhaW0gT21pdHRlZFwiKSIsInNjdCI6ImV4KCkgJT4lIGNoZWNrX2Z1bmN0aW9uKFwidGFpbFwiLG5vdF9jYWxsZWRfbXNnPVwiSGF2ZSB5b3UgdXNlZCBgdGFpbGAgdG8gdmlldyB0aGUgbGFzdCA2IG9ic2VydmF0aW9ucyBvZiB0aGUgZGF0YT9cIikgJT4lIGNoZWNrX2FyZyguLCBcInhcIixhcmdfbm90X3NwZWNpZmllZF9tc2c9XCJIYXZlIHlvdSBzcGVjaWZpZWQgd2hpY2ggZGF0YWZyYW1lIHlvdSB3b3VsZCBsaWtlIHRvIHZpZXc/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiTWFrZSBzdXJlIHRvIHVzZSB0YWlsIHRvIHNlZSB0aGUgbGFzIDYgZW50cmllcyBpbiBgaW5qdXJ5YC5cIilcbmV4KCkgJT4lIGNoZWNrX2Z1bmN0aW9uKFwic3Vic2V0XCIsbm90X2NhbGxlZF9tc2c9XCJIYXZlIHlvdSBjYWxsZWQgYHN1YnNldGAgdG8gY3JlYXRlIGEgc3Vic2V0IG9mIG91ciBvcmlnaW5hbCBkYXRhP1wiKSAlPiUgY2hlY2tfYXJnKC4sIFwieFwiLGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkhhdmUgeW91IHNwZWNpZmllZCB0aGUgb3JpZ2luYWwgZGF0YSB5b3Ugd291bGQgbGlrZSB0byB0YWtlIGEgc3Vic2V0IGZyb20/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiTWFrZSBzdXJlIHRvIGNyZWF0ZSBhIHN1YnNldCBvZiBgaW5qdXJ5YC5cIilcbmV4KCkgJT4lIGNoZWNrX29iamVjdChcImluanVyeTJcIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJNYWtlIHN1cmUgdGhhdCBgaW5qdXJ5MmAgaXMgdGhlIHNhbWUgYXMgYGluanVyeWAgYnV0IHdpdGhvdXQgdGhlIGxhcmdlc3QgY2xhaW0uIFRyeSBhbmQgdGhpbmsgb2YgY3JlYXRpdmUgd2F5cyB0byByZW1vdmUgdGhhdCBvYnNlcnZhdGlvbiBmcm9tIHRoZSBkYXRhIVwiKVxuZXgoKSAlPiUgY2hlY2tfZnVuY3Rpb24oXCJzdW1tYXJ5XCIsaW5kZXg9MSxub3RfY2FsbGVkX21zZz1cIkhhdmUgeW91IGNhbGxlZCBgc3VtbWFyeWAgdG8gdmlldyBiYXNpYyBzdGF0aXN0aWNzIGFib3V0IHRoZSBkYXRhP1wiKSAlPiUgY2hlY2tfYXJnKC4sIFwib2JqZWN0XCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHdoaWNoIGRhdGEgeW91IHdvdWxkIGxpa2UgYSBzdW1tYXJ5IG9mP1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIk1ha2Ugc3VyZSB0byBnZXQgc3VtbWFyeSBzdGF0aXN0aWNzIG9mIGBpbmp1cnlgLlwiKVxuZXgoKSAlPiUgY2hlY2tfZnVuY3Rpb24oXCJzdW1tYXJ5XCIsaW5kZXg9Mixub3RfY2FsbGVkX21zZz1cIkhhdmUgeW91IGNhbGxlZCBgc3VtbWFyeWAgdG8gdmlldyBiYXNpYyBzdGF0aXN0aWNzIGFib3V0IHRoZSBkYXRhP1wiKSAlPiUgY2hlY2tfYXJnKC4sIFwib2JqZWN0XCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHdoaWNoIGRhdGEgeW91IHdvdWxkIGxpa2UgYSBzdW1tYXJ5IG9mP1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIk1ha2Ugc3VyZSB0byBnZXQgc3VtbWFyeSBzdGF0aXN0aWNzIG9mIGBpbmp1cnkyYC5cIilcbmV4KCkgJT4lIGNoZWNrX2Z1bmN0aW9uKFwicGFyXCIsbm90X2NhbGxlZF9tc2c9XCJIYXZlIHlvdSBjYWxsZWQgYHBhcmAgdG8gY2hhbmdlIHRoZSBncmFwaGljYWwgcGFyYW1ldGVycz9cIikgJT4lIGNoZWNrX2FyZyguLCBcIm1mcm93XCIsIGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkhhdmUgeW91IHNwZWNpZmllZCB0aGUgbnVtYmVyIG9mIHJvd3MvY29sdW1ucyBmb3IgeW91ciBwbG90IHRvIGhhdmU/XCIpICU+JSBjaGVja19lcXVhbChpbmNvcnJlY3RfbXNnPVwiUGxlYXNlIGRvbnQgY2hhbmdlIHRoaXMgcGFydC4gaXQgc2hvdWxkIHJlYWQgYHBhcihtZnJvdz1jKDIsMSkpYFwiKVxuZXgoKSAlPiUgY2hlY2tfZnVuY3Rpb24oXCJoaXN0XCIsaW5kZXg9MSxub3RfY2FsbGVkX21zZz1cIkhhdmUgeW91IGNhbGxlZCBgaGlzdGAgdG8gY3JlYXRlIGEgaGlzdG9ncmFtIG9mIHRoZSBkYXRhP1wiKSAlPiUge1xuICBjaGVja19hcmcoLiwgXCJ4XCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHdoaWNoIGRhdGEgc2hvdWxkIGJlIHVzZWQgdG8gY3JlYXRlIGEgaGlzdG9ncmFtP1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIkNyZWF0ZSB0aGUgZmlyc3QgaGlzdG9ncmFtIHVzaW5nIGFsbCBvZiB0aGUgb2JzZXJ2ZWQgY2xhaW1zLlwiKVxuICBjaGVja19hcmcoLiwgXCJmcmVxXCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHRoYXQgd2Ugd291bGQgbGlrZSBhIGRlbnNpdHkgaGlzdG9ncmFtP1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIk1ha2Ugc3VyZSB0byBjcmVhdGUgYSBkZW5zaXR5IGhpc3RvZ3JhbSBpbnN0ZWFkIG9mIGEgZnJlcXVlbmN5IGhpc3RvZ3JhbS5cIilcbn1cbmV4KCkgJT4lIGNoZWNrX2Z1bmN0aW9uKFwiaGlzdFwiLGluZGV4PTIsbm90X2NhbGxlZF9tc2c9XCJIYXZlIHlvdSBjYWxsZWQgYGhpc3RgIHRvIGNyZWF0ZSBhbm90aGVyIGhpc3RvZ3JhbSBvZiB0aGUgZGF0YT9cIikgJT4lIHtcbiAgY2hlY2tfYXJnKC4sIFwieFwiLGFyZ19ub3Rfc3BlY2lmaWVkX21zZz1cIkhhdmUgeW91IHNwZWNpZmllZCB3aGljaCBkYXRhIHNob3VsZCBiZSB1c2VkIHRvIGNyZWF0ZSBhIGhpc3RvZ3JhbT9cIikgJT4lIGNoZWNrX2VxdWFsKGluY29ycmVjdF9tc2c9XCJNYWtlIHN1cmUgdG8gY3JlYXRlIHRoZSBzZWNvbmQgaGlzdG9ncmFtIGJhc2VkIG9uIGNsYWltcyB3aXRoIHRoZSBsYXJnZXN0IG9uZSByZW1vdmVkLlwiKVxuICBjaGVja19hcmcoLiwgXCJmcmVxXCIsYXJnX25vdF9zcGVjaWZpZWRfbXNnPVwiSGF2ZSB5b3Ugc3BlY2lmaWVkIHRoYXQgd2Ugd291bGQgbGlrZSBhIGRlbnNpdHkgaGlzdG9ncmFtP1wiKSAlPiUgY2hlY2tfZXF1YWwoaW5jb3JyZWN0X21zZz1cIk1ha2Ugc3VyZSB0byBjcmVhdGUgYSBkZW5zaXR5IGhpc3RvZ3JhbSBpbnN0ZWFkIG9mIGEgZnJlcXVlbmN5IGhpc3RvZ3JhbS5cIilcbn1cbnN1Y2Nlc3NfbXNnKFwiQ29uZ3JhdHVsYXRpb25zISBUaGUgZ29hbCBvZiBwcmVkaWN0aXZlIG1vZGVsaW5nIGlzIHRvIGRpc2NvdmVyIHBhdHRlcm5zIGluIHRoZSBkYXRhLiBIb3dldmVyLCBzb21ldGltZXMgc2VlbWluZyAncGF0dGVybnMnIGFyZSB0aGUgcmVzdWx0IG9mIG9uZSBvciB0d28gdW51c3VhbCBvYnNlcnZhdGlvbnMuIFVudXN1YWwgb2JzZXJ2YXRpb25zIG1heSBiZSBkdWUgdG8gaW5jb3JyZWN0IGRhdGEgZ2F0aGVyaW5nIHByb2NlZHVyZXMgb3IganVzdCBkdWUgdG8gd2lsZCBmbHVjdHVhdGlvbnMgaW4gYSBwcm9jZXNzIG9mIGludGVyZXN0IGJ1dCBhcmUgY29tbW9uIGluIHByZWRpY3RpdmUgbW9kZWxpbmcuXCIpIiwiaGludCI6IkZvciB0aGlzIGRhdGEgc2V0LCB0aGUgW3N1YnNldCgpXSBhcmd1bWVudCA8Y29kZT5jbGFpbXMgJmx0OyAyNTAwMDwvY29kZT4gd2lsbCBrZWVwIGFsbCBidXQgdGhlIGxhcmdlc3QgY2xhaW0ifQ==

1.4 Transformations

In this exercise, you learn how to:

Symmetrize a skewed distribution using a logarithmic transformation

1.4.1 Video

Video Overhead Details

Hide

A Details. Simulate a moderately skewed distribution, with transforms

#  FIGURE 1.7 - SIMULATE CHI-SQUARE, CREATE 3 TRANSFORMATIONS
set.seed(1237)                  # set the seed of the random number generator
                                # allows us to replicate results
X1 <- 10000*rchisq(500, df = 2) # generate variables randomly from a skewed distribution
X2 <- X1^(0.5)                  # square root transform, could also use sqrt(X1)
X3 <- log(X1)                   # logarithmic transform
X4 <- -1/X1                     # negative reciprocal transform

Hide

B Details. Visualize the distributions

par(mfrow = c(2, 2), cex = .75, mar = c(3,5,1.5,0))
hist(X1, freq = FALSE,  nclass = 16, main = "", xlab = "", ylab = "", 
      las = 1, yaxt = "n",xlim = c(0,200000),ylim = c(0,.00005))
axis(2, at = seq(0,.00005,.00001),las = 1, cex = .3, 
  labels = c("0", "0.00001", "0.00002","0.00003", "0.00004", "0.00005"))
mtext("Density", side = 2, at = .000055, las = 1, cex = .75)
mtext("y", side = 1, cex = .75, line = 2)

par(mar = c(3,4,1.5,0.2))
hist(X2, freq = FALSE,  nclass = 16, main = "", xlab = "", ylab = "", 
      las = 1,xlim = c(0,400), ylim = c(0,.008))
mtext("Density", side = 2, at = .0088, las = 1, cex = .75)
mtext("Square root of y", side = 1, cex = .75, line = 2)

par(mar = c(3.2,5,1,0))
hist(X3,  freq = FALSE,  nclass = 16, main = "", xlab = "", ylab = "", las = 1, ylim = c(0,.4))
mtext("Density", side = 2, at = .44, las = 1, cex = .75)
mtext("Logarithmic y", side = 1, cex = .75, line = 2)

par(mar = c(3.2,4,1,0.2))
hist(X4, freq = FALSE,  nclass = 16, main = "",xlab = "", ylab = "", las = 1, ylim = c(0,100))
mtext("Density", side = 2, at = 110, las = 1, cex = .75)
mtext("Negative reciprocal of y", side = 1, cex = .75, line = 2)

1.4.2 Exercise. Distribution of transformed bodily injury claims

Assignment Text

We have now examined the distributions of bodily injury claims and its logarithmic version. Grudgingly, we have concluded that to fit a normal curve the logarithmic version of claims is a better choice (again, we really do not like log dollars but you’ll get used to it in this course). But, why logarithmic and not some other transformations?

A partial response to this question will appear in later chapters when we describe interpretation of regression coefficients. Another partial response is that the log transform seems to work well with skewed insurance data sets, as we demonstrate visually in this exercise.

Instructions

Use the code par(mfrow = c(2, 2)) so that four graphs appear in a 2 by 2 matrix format for easy comparisons. Plot the density() of

claims
square root of claims
logarithmic claims
negative reciprocal of claims