Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Community Proposals
Community Proposals
tag:snake search within a tag
answers:0 unanswered questions
user:xxxx search by author id
score:0.5 posts with 0.5+ score
"snake oil" exact phrase
votes:4 posts with 4+ votes
created:<1w created < 1 week ago
post_type:xxxx type of post
Search help
Notifications
Mark all as read See all your notifications »
Q&A

Can the relative magnitude of standardized coefficients indicate variable importance?

+3
−0

Comparing the standardized coefficients (resulting from linear regression) across different samples can be problematic (because the sample variances can be influenced by measurement error).

But can the standardized coefficients of two different variables be compared in one sample - to ascertain which variable has a 'larger effect' ?

Is it a reasonable way to infer the larger effect - even if an imperfect one?

History

0 comment threads

2 answers

You are accessing this answer with a direct link, so it's being shown above all other answers regardless of its score. You can return to the normal view.

+1
−0

Why, yes. If one has a linear regression $\mathbf{y} = \mathbf{X} \mathbf{\beta} + \mathbf{\epsilon}$ and models the errors as i.i.d Gaussians with known standard deviation $\sigma$ then the likelihood function for the "true" coefficients will be a multivariate Gaussian with that standard deviation.

The covariance matrix that determines this multivariate Gaussian will be the same as the covariance matrix of the columns of $X$. This is because we are orthogonally projecting the spherical Gaussian distribution surrounding $y$ onto the hyperplane defined by $X$.

Of course for less than 30 data points we don't know the right standard deviation and so we have to use a T-distribution for our likelihood function.

As for the OP's particular question of how to test whether one coefficient is greater than another: for large samples, all we need to do is run our linear regression, then run it again with the two coefficients "pegged" to equal each other (by merging two columns of the $\mathbf{X}$ matrix), and look at the relative increase in the standard deviation of our error estimate; that gives us our test statistic which we can plug into the one-tailed Z-test. (Briefly, this computes how far, in units of standard deviations of the error, the line $\beta_1=\beta_2$ is from the estimated parameters.)

The same approach works for testing whether one coefficient is greater or less than any linear combination of the other coefficients.

History

1 comment thread

Why 30? Normally for estimating the mean of a normal (or "Gaussian") population, I would use a t-dist... (3 comments)
+1
−0

This can be hazardous if two predictors are correlated.

For example, suppose one predictor is the amount of money a person spends on one kind of staple groceries and another is how much the same person spends on another kind of staple groceries. A person with a dozen dependents spends a lot on both and a person with no dependents not much, so the two are highly correlated. Then it may be that neither variable is statistically significant after the other is taken into account, but either is statistically significant when considered alone.

Another situation is that neither is statistically significant when considered alone, but either is statistically significant when considered after the other. That happens, for example, when the difference between the two is what is highly correlated with the response variable. For example, it may be impossible to predict whether a person is male or female based on their height at age 13, and similarly for their height at age 14, but the difference between those two heights can predict the gender most of the time.

Predictors may be completely uncorrelated in cases where the experiment is designed that way and the value of each predictor is fixed by design. In that case, statistical significance of each predictor can be assessed separately.

However, statistical significance may be very different from importance. Suppose it were found that people with left-wing political view prefer certain colors and right-wingers others (purely hypothetical; I know of no data on this), and the difference is statistically significant. It may nonetheless be a very tiny difference (e.g. $52\%$ of left-wingers like green, as do $52.1\%$ of right-wingers), of no practical importance. Statistical significance just means one can be confident that the difference exists in the population from which the sample was taken, because the sample consisted of a very very large number of independent observations.

Now recall what $\text{“ }R^2\text{ ”}$ is:

\begin{align} R^2 &= \frac{\sum(\text{fitted value minus sample mean value})^2}{\sum(\text{observed value minus sample mean value})^2} \\[8pt] & = \frac{\text{explained variability}}{\text{total variability}}. \end{align}

(In linear regression with only one predictor (which is not relevant here because there is only one predictor), this is just the square of the correlation – hence the name. With multiple predictors, $\text{“ }R^2\text{ ”}$ is still defined that way but there is no such thing as $\text{“ }R\text{ ”}.$) Sometimes it is of interest to look at how much $R^2$ is increased by adding a particular predictor to the model after the others are taken into account.

Finally, I must tell you about a potentially very unpleasant fact: How to reduce this particular question to a math problem, amenable to solution by mathematics, is the subject of much uncertainty and disagreement even after very many people have devoted careers spanning decades to the matter and thousands of research papers have addressed it. And the answer actually depends on the purpose for which the results are to be used.

Consult textbooks on two topics: Applied linear regression and design of experiments.

History

0 comment threads

Sign up to answer this question »