DATA ANALYTICS QUESTIONS PART-2

The previous question contained Exploratory Data Analysis and Data and Sampling Distribution topics. Check it out here. This blog will cover questions on Statistical Testing and Significance Testing.

Q1. WHAT IS A/B TESTING?

In simple words, A/B test is an experiment with 2 (or more) groups to establish which of the 2 (treatments, products, procedure tec.) is superior. Often one of the two treatments is the standard existing treatment( or no treatment).

Q2. WHAT IS HYPOTHESIS TESTING? WHAT IS NULL AND ALTERNATIVE HYPOTHESIS IN HYPOTHESIS TEST?

Hypothesis test help you learn whether random chance might be responsible for an observed effect.

The null hypothesis is a general statement or default position that there is no relationship between two measured phenomena, or no association among groups. In other words, it is a basic assumption or made based on domain or problem knowledge.

The alternative hypothesis is the hypothesis used in hypothesis testing that is contrary to the null hypothesis. It is usually taken to be that the observations are the result of a real effect (with some amount of chance variation superposed).

Example: Null = “B is not X% greater than A,” alternative = “B is X% greater than A”.

Taken together Nulla and Alternative Hypotheses are mutually exhaustive.

See more: https://towardsdatascience.com/hypothesis-testing-in-machine-learning-using-python-a0dc89e169ce

Q3. Why do we need a hypothesis? Why not just look at the outcome of the experiment and go with whichever treatment does better?

The answer lies in the tendency of the human mind to underestimate the scope of natural random behavior.

1. The failure to anticipate extreme events.
2. The tendency to misinterpret random events as having patterns of some significance.

Statistical hypothesis testing was invented as a way to protect researchers from being fooled by random chance.

Q4. WHAT IS THE DIFFERENCE BETWEEN A\B TESTING AND HYPOTHESIS TESTING?

A/B testing is more like, which out of ‘a’ and ‘b’ is better (in terms of profit or views or anything), Hypothesis test determines whether a random chance is actually responsible for an observed fact.

Q5. WHAT IS RESAMPLING?

Resampling in statistics means to repeatedly sample values from observed data, with a general goal of assessing random variability in a statistic. It is of 2 types: Bootstrap and permutation.

Q6. WHAT IS PERMUTATION TEST?

In a permutation procedure, two or more samples are involved, typically the groups in an A/B or other hypothesis test. The permutation procedure is as follows:

1. Combine the results from the different groups in a single data set.

2. Shuffle the combined data, then randomly draw (without replacing) a

resample of the same size as group A.

3. From the remaining data, randomly draw (without replacing) a resample

of the same size as group B.

4. Do the same for groups C, D, and so on.

5. Whatever statistic or estimate was calculated for the original samples (e.g., difference in group proportions), calculate it now for the resamples, and record; this constitutes one permutation iteration.

6. Repeat the previous steps R times to yield a permutation distribution of the test statistic.

Now go back to the observed difference between groups and compare it to the set of permuted differences. If the observed difference lies well within the set of permuted differences, then we have not proven anything — the observed difference is within the range of what chance might produce. However, if the observed difference lies outside most of the permutation distribution, then we conclude that chance is not responsible. In technical terms, the difference is statistically significant.

Q7. WHAT ARE THE ADVANTAGES OF RESAMPLING?

1. It can be applied to all sorts of data, whether binary or numeric.
2. The size of samples can be different or the same
3. The assumption that data is normally distributed can be overlooked.

Q8. WHAT DOES “STATISTIC SIGNIFICANCE” MEAN?

Statistical significance helps to quantify whether a result is likely due to chance or some factor of interest. In simple words, it helps you identify whether a finding is real or you just got lucky(or unlucky).

See more about the concept -https://hbr.org/2016/02/a-refresher-on-statistical-significance

Q9. WHAT IS P-VALUE?

P-value measures statistical significance. It is the frequency with which the model produces a more extreme result than the observed value.

If the p-value is 0.308, it means that we would expect to achieve the same result by random chance over 30% of the time. The lesser the p-value, more stronger the surprise (or more evidence in favor of alternate hypothesis).

A beautiful intuitive understanding of p-values: https://www.youtube.com/watch?v=9jW9G8MO4PQ

Q10. WHAT ARE TYPE 1 AND TYPE 2 ERRORS?

Type 1 — When you mistakenly conclude an effect is real when it is really just due to chance

Type 2 — When you mistakenly conclude that an effect is not real i.e., due to chance, when it really is real.

Example for Type 1 and Type 2 errors

Q11. WHAT IS t-TEST?

It is a test to measure statistical significance among 2 groups(for more than 2 we use ANOVA). The data should follow normal distribution and may have an unknown variance. The smaller the t-value, the more similarity exists between the two sample sets.

Read more: https://www.investopedia.com/terms/t/t-test.asp

Q12. WHAT IS ANOVA?

For comparing more than 2 groups, each with numeric data, the procedure that tests for statistical significance among these different groups is called Analysis of Variance or ANOVA.

ANOVA is calculated by the f-statistics, it is based on the ratio of the variance across group means (i.e., the treatment effect) to the variance due to residual error. The higher the more significant the result.

Read more https://www.investopedia.com/terms/a/anova.asp

Calculating f-statistics: https://www.youtube.com/watch?v=EFdlFoHI_0I

Q13. WHAT IS THE DECOMPOSITION OF VARIANCE?

For any observed data value within a data set, we can break it down into the grand average, the treatment effect, and the residual error. We call this a “decomposition of variance.”

Q14. WHAT IS A 2 WAY ANOVA?

A two-way ANOVA test is a statistical test used to determine the effect of two nominal predictor variables on a continuous outcome variable.

Q15. WHAT IS THE CHI-SQUARED TEST?

A chi-squared test is of 2 types-

1. To test how well the sample data matches the population. This kind of test is called the goodness of fit test.
2. To test how much 2 variables are related to each other, using a contingency table.

The chi-square test follows the chi-squared distribution. The appropriate chi-squared distribution is determined by the degrees of freedom as

d.o.f.=(m-1)x(n-1),

where m,n represent the rows and columns of the contingency table.

A lower chi-squared value means the variables are highly correlated. If the chi-squared value is high, we need to look up for chi-square critical value, if the observed chi-squared value is more than the critical value, we can conclude that there is a difference between the groups.

Read more: https://www.statisticshowto.com/probability-and-statistics/chi-square/

Q16. WHAT IS MULTI-ARM BANDIT ALGORITHM?

Multi-arm bandit is an approach to testing, that allows explicit optimization and more rapid decision making than conventional statistical procedures.

Traditional A/B tests envision a random sampling process, which can lead to excessive exposure to the inferior treatment. Multi-arm bandits, in contrast, alter the sampling process to incorporate information learned during the experiment and reduce the frequency of the inferior treatment. They also facilitate the efficient treatment of more than two treatments.

References: O’rielly’s Practical Statistics for Data Scientist, Investopedia, Statistics How to