Retrieving the source code

While there's quite a lot of ways to retrieve the source code, I chose to directly use the inspect module, which provides access to the stack of frames (function calls) and their corresponding source code.

stack = inspect.stack()
frame_info = stack[1]
 
frame = frame_info.frame
file = inspect.getsourcefile(frame)
position = frame_info.positions
 
# get source code
lines = map(
    lambda line: linecache.getline(file, line),
    range(position.lineno, position.end_lineno + 1),
)
lines = list(lines)
if len(lines) > 1:
    lines[0] = lines[0][position.col_offset :]
    lines[-1] = lines[-1][: position.end_col_offset]
else:
    # if only one line, slice once since col_offset and end_col_offset
    # are relative to the same line
    lines[0] = lines[0][position.col_offset : position.end_col_offset]
src = "".join(lines)

After retrieving the corresponding source code, we can parse it into an AST using python's built-in ast module.

tree = ast.parse(src)
expr = ast_tree.body[0].value

After this, how we process the expression depends on how this function is called. Here, I only considered when function is called directly macro(...) and when called with subscript macro[...].

if isinstance(expr, ast.Call):
    return expr.args
elif isinstance(expr, ast.Subscript):
    return expr.slice
else:
    raise ValueError("Could not get arguments")

Once we have the arguments, we can process them as we want.

Putting everything together, we can create a macro decorator that transforms normal functions into macros.

@enum.unique
class MacroLevel(enum.Enum):
    AST = 1
    STR = 2
 
 
def macro(*, level: MacroLevel = MacroLevel.AST):
    R = TypeVar("R")
 
    def decorator(func: Callable[..., R]) -> Callable[..., R]:
        def wrapper(*_, **__):
            stack = inspect.stack()
            frame_info = stack[1]
            args_ast = _get_args_ast(frame_info)
 
            # fmt: off
            args = pipe(
                level,
                match(
                    case(MacroLevel.AST) >> args_ast,
                    case(MacroLevel.STR) >> matchV(args_ast)(
                        case(lambda x: isinstance(x, list)) >> (lambda x: [ast.unparse(arg) for arg in x]),
                        default >> ast.unparse,
                    ),
                )
            )
            # fmt: on
            return func(args, frame_info.frame)
 
        return cast(Callable[..., R], wrapper)
 
    return decorator

where _get_args_ast is the function that retrieves the arguments' AST.

Here, I also implement a MacroLevel which controls how those args are passed to the function.

Get the typings correct

While the implementation above works, it's annoying when editor's type check and syntax highlighting keeps complaining. To fix this, I create a PlaceHolder class, which utilizes dunder methods in Python to allow basically any operations.

class __Placeholder:
    def __add__(self, _):
        return self
 
    def __sub__(self, _):
        return self
 
    def __mul__(self, _):
        return self
    # ...

Example usage of the macro

After completing the macro, it's tempting to implement some powerful features with it. Here, I'm inspired by the MacroPy lib to create a similar shorthand for lambda expression (but more powerful as it allows to specify arbitrary arguments through _1, _2, etc.)

class __f:
    @staticmethod
    @macro(level=MacroLevel.AST)
    def __getitem__(args_src: ast.expr, frame: FrameType):
        placeholders = set()
        for node in ast.walk(args_src):
            if isinstance(node, ast.Name) and node.id in [
                "_",
                *(f"_{i}" for i in range(1, 10)),
            ]:
                placeholders.add(node.id)
 
        if "_" in placeholders and placeholders - {"_"}:
            raise ValueError(
                "A quick lambda should use either _ or _1, _2, ..., but not both."
            )
 
        if "_" in placeholders:
            args = ["_"]
        else:
            args = sorted(placeholders, key=lambda x: int(x[1:]))
 
        args = map(lambda x: ast.arg(arg=x, annotation=None), args)
        ast_func = ast.Lambda(
            args=ast.arguments(
                list(args), [], kwonlyargs=[], kw_defaults=[], defaults=[]
            ),
            body=args_src,
        )
        return eval(
            ast.unparse(ast_func),
            frame.f_globals,
            frame.f_locals,
        )
 
 
f = __f()
_: Any = __Placeholder()
_1, _2, _3, _4, _5, _6, _7, _8, _9 = [_] * 9

For this project, I chose to use Next.js, TailwindCSS, and MDX. Next.js is mostly for routing, SSR, and SSG. Tailwind allows me to style my components quickly and efficiently. MDX is great for adding custom content that is hard to achieve with Markdown.

Design

I followed DaisyUI's semantic color system for my blog's design, with the following colors:

• Primary
• Secondary
• Accent
• Neutral
• Info
• Success
• Warning
• Error

Features

Dark mode

Implemented through media queries and TailwindCSS.

However, the dark mode color scheme is not as good as the light mode, so I still need to improve it in the future.

Syntax highlighting

Implemented with Shiki and rehype-pretty-code

def some_function():
    print("This is a test function")

Table of contents

Implemented by finding all the headings in the post and generating a list of links. (source code)

Latex Support

Implemented with KaTeX

Other features

• RSS feed
• SEO
• Sitemap
• Posts search by categories and tags
• Pagination and infinite scroll
• GitHub flavored markdown

Definitions

Data

facts or measurements that are collect, analyze, and interpret —— raw material of statistical analysis can be quantitative and qualitative

• Cross-Sectional - collected at the same time
• Longitudinal - time-series collected over several time periods

Data Set - all the data, across observations and variables

Variable

may assume different values

• Discrete - can be counted —— takes on integer values
• Continuous - if interval is between a and b, variable can take any value in between

Constant - remains the same

Elements

an element is a unit of data, which is represented as a set of attributes or measurements —— entities on which data are collected

Observation - set of values for the variables for an element set of measurements for a single data entity

Population

all entities of interest in an investigative study

Population parameters - characteristics or quantities of the population

Probability - when we draw a sample for a population with known parameters

Measurement of Variables

• Nominal Level - sometimes called “categorical”
• Ordinal Level - order is meaningful
• Interval Level - interval between the values is fixed
• Ratio Level - zero has meaning

Sample - subset of the population

Sample Statistics - characteristics or qualities of the sample

Statistics

• Descriptive - describe data that have been collected
• Inferential - use data that have been collected to generalize or make inference based on it

Univariate distributions - explore how the collection of values for each variable is distributed across an array of possible values

use tables and graphs to capture the information

Distributions

• axes
• step size
• shape
• range / spread
• center
• unusual / gaps and outliers

Normal Distribution

• symmetry
• median = mean
• bell shape

Center

• Normal Distribution
  • $\Mu$ = population mean = $\sum x_i / N$
  • $\bar{x}$ = sample mean = $\sum x_i / n$
• Skewed Data
  • Median / exact middle / 50% percentile / middle quartile

Unusual

• Normal Distribution
  • Gap
  • Outliers - 3 standard deviations away from the mean - 68% / 95% / 99%
• Skewed Data
  • Outliers - $Q_{75} + 1.5 \times IQR$ and $Q_{25} - 1.5 \times IQR$

Spread

• Normal Distribution

  $$\sigma=\sqrt{\frac{\sum(x_i-M)^2}{N}}$$ $$s=\sqrt{\frac{\sum(x_i-M)^2}{n-1}}$$ $$range = max - min$$

• Skew Data

  • IQR - difference between the third and first quartile = $Q_{75}-Q_{25}$

Shape

• normal
• uniform
• skew
  • less on left = negative skew / left skew
  • less on right = positive skew / right skew

Percentile

percentile - way of measuring relative position within a data set

The raw score where a certain percentage of score fell below that number

percentile rank - relative position within a given dataset

$Rank = \dfrac{\text{percentile}}{100} * (\text{number of items} +1)$

Modal

unimodal - have one mode

symmetric unimodal - mean = median = mode

bimodal - have two mode / peak

Non-skewed normal-like

center - mean

spread - standard deviation - on average how far the data is from the mean

median, IQR - Resistant Statistics

mean, standard deviation - Non-Resistant Statistics

spread ≈ variability ≈ dispersion ≈ heterogeneity ≈ unpredictability

range ≈ IQR ≈ standard deviation ≈ variance

Skewness

• When a distribution is positively skewed, the skewness is positive.
• When a distribution is negatively skewed, the skewness is negative.
• When a distribution symmetric, the skewness is 0.
• When you have 2 or more distributions, and you want to compare skewness, you use skewness ratio.
• By convention when the skewness ratio exceed 2 we consider the distribution highly skewed.

Transformations

Linear Transformation

Center

Spread

Shape

Z-score

Nonlinear Transformation

• retains the order
• changes the relative distances between the values in the distribution, and, in doing so, impacts spread and shape

Unit 2: Bivariate Data

Bivariate Data

• two variables
• usually comparing at least interval type data
• does one variable relate to another, if so, how?
  • shape: linear / non-linear
  • direction: postiive, negative
  • strength: $r = \frac{\sum z_x\times z_y}{N-1}$
• correlation $\neq$ causation

1. obtain data - 2 variables (above nominal)
2. scatterplot
3. scale your data, fit a regression line
4. look at correlation
5. describe shape, direction, and strength

Correlation Coefficient

• $\geq0.7$ - strong
• $[0.4,0.7)$ - moderate
• $<0.4$ - weak

Linear Regression

line of best fit is obtained by minimizing the square of deviations to all points on the scatter plot

deviation also called residual

$r^2$ Interpretation

$r^2=p\%$ means that the independent variable accounts for $p\%$ of the change in dependent variable

Unit 3: Probability

Definitions

simple experiment - any action (flipping a coin, choosing a card) that leads to one of several possible outcomes

events - observable outcomes

If we only have two outcomes that are mutually exclusive, they are called complements.

Suppose $E_i$ are independent of each other

Conditional Probability

• two events not independent of each other
• determining probability of an event happening given that another has happened

The Law of Total Probability

Bayes’ Theorem

Binomial Distribution

B - binary

I - independent

N - fixed number of trials

S - probability success for each trial remains the same

• k - number of successes
• n - number of trials
• p - probability of success

10% Rule

The 10 percent rule is used to approximate the independence of trials where sampling is taken without replacement, If the sample size is less than 10% of the population size, then the trial size can be treated as independent even if it’s not.

a rule of thumb: n needs to be so large that the expected number of successes and failures are both at least 10

Fundamental Counting Principle

states that if one event can occur in $m$ ways and a second event can occur in $n$ ways for each at the occurrences of the first event then the first event and second event can occur in $m \times n$ ways

Permutation

ordered sequences of objects in which each possible object occurs at most once but not all objects need to be used. The total number of permutations when selecting k objects from a total of n choice is:

Normal Distribution

Simple Random Sample

sample is chosen from a given in such a way as to ensure that person (or thing) in a population has an equal of independent chance of being picked for a sample

Central Limit Theorem

Given a population of values with no specify distribution and a sample size of $N$ that is sufficiently large, the sampling distribution of means for samples drawn from this population with replacement can be described as follows:

1. Its shape is approximately normal
2. Its mean $\mu_{\bar x}=\mu$
3. Its standard deviation $\sigma_{\bar x}=\dfrac{\sigma}{\sqrt{N}}$ , also called the standard error

Large Sample Condition:

• original distribution is approximately normal
• sample size greater than 30

Population Proportion

population proportion - $P$

sample proportion - $\hat p$

Sampling Distribution of a Sample Proportion

Choose a simple random sample of size $n$ from a population of size $N$, with a proportion of $p$ successes. Then:

• the mean of the sampling distribution is equal to $p$ $\mu_{\hat p}=p$
• standard deviation of the sampling distribution of $\hat p$ $\sigma_{\hat p}=\sqrt{\dfrac{p(1-p)}{n}}$ as long as $n \leq \dfrac{1}{10} N$
• as $n$ increases the sampling distribution approaches normal as long as $np \geq 10 \ \text{and} \ n(1-p) \geq 10$

Sampling Distribution of $\bar x$

Suppose $\bar x$ is the mean of a simple random sample of size $n$ drawn from a large population $N$ with mean $\mu$ and a standard deviation of $\sigma$. As long as $n \leq \dfrac{1}{10} N$

• $\mu_{\bar x}= \mu$
• $\sigma_{\bar x} = \dfrac{\sigma}{\sqrt{n}}$

Point estimate is the statistic calculated from your sample.

1. don’t expect point estimator $\bar x$ to be the mean
2. should be part of the normal distribution
3. $95\%$ of the data is within 2 standard deviation of the mean
4. Whenever $\bar x$ is within 10 points of $\mu$, $\mu$ is within 10 points of $\bar x$, this happens in about $95\%$ of all samples

Definition of Confidence Interval, Margin of Error, Confidence Level

A $C\%$ confidence interval gives an interval of plausible values for a parameter

Confidence Level

The confidence level $C$ gives the overall success rate of the method for calculating the confidence interval. That is in $C\%$ of all possible samples of a particular size $n$. This method would yield an interval that captures the true parameter value.

Interval yields plausible values.

Generally we chose a confidence level of $90\%$. $95\%$ is most common.

confidence interval for $p$:

confidence interval for means:

T Distribution

• As degree of freedom increase, approaches standard normal curve
• symmetric mean centered around normal
• standard deviation changes
• bell curve
• unimodal
• $\text{degree of freedom}=n-1$

Significance Test

Null and Alternative Hypothesis

$\mathcal{H}_0$ - a claim

$\mathcal{H}_a$ - alternative hypothesis, which means $\mathcal{H}_0$ is false

Parameter of interest $p$ is the true proportion of successes

Statistical test weigh evidence against a claim and an alternate claim

• Step 1: state hypothesis, hypothesis should express suspicions we have before we see the data
  • one-sided $>$ or $<$
  • two-sided $\neq$
  $$\mathcal{H}_0: p = \text{value} \ \ \text{or} \ \ \mathcal{H}_a:p < (> \text{or} \neq) \ \text{value}$$ $$\mathcal{H}_0: \mu = \text{value} \ \ \text{or} \ \ \mathcal{H}_a:\mu < (> \text{or} \neq) \ \text{value}$$
• Step 2: Does the data give convincing evidence against the null?
  • Reject the null
  • Fail to reject the null

P-Value

The probability computed assuming $\mathcal{H}_0$ is true that the statistic (such as $\hat p$ or $\bar x$) would take a value as extreme as or more extreme than the one actually observed in the direction of $\mathcal{H}_a$.

Small p-values are evidence against $\mathcal{H}_0$ observed result is unlikely to occur when $\mathcal{H}_0$ is true.

If the p-value is smaller than $\alpha$ we say result are statistically significant at a level of $\underline \alpha$ (significant level), we reject the $\mathcal{H}_0$ and conclude convincing evidence in favor of $\mathcal{H}_a$.

Levels of $\alpha$: $0.01,0.05, 0.1$

Test about a population proportion (1-prop z-test)

1. state what parameter of interest is
2. check conditions
   • random - SRS / well-designed condition
   • independence - if not replacing $n\leq \frac{1}{10}N$
   • normal - $np \geq 10 \ \text{and} \ n(1-p) \geq 10$
3. perform the test
   • test statistic: $\text{normalcdf}(\text{z-score},\text{upper/lower bound}, mean, \text{standard deviation})$
   • p-value
4. conclude
   • compare p-value to $\alpha$
   • reject or fail to reject in context

1 sample t-test

1. State parameter of interest
2. Check conditions
   • random - SRS / well-designed condition
   • independent - 10% or less of N is sampling if without replacement
   • normal - $\geq30$ / population distribution is normal
3. Hypothesis: $\mathcal{H}_0 \ \text{and} \ \mathcal{H}_a$
4. perform the test
   • test statistic:
   $$\frac{\text{stats} - \text{parameter}}{\text{standard error}}=\frac{\bar x - \mu_0}{s_x / \sqrt{n}}=t$$
5. Conclusion

Power

The power of a test against a specific alternative is the probability that the test will reject $\mathcal{H}_0$ at a chosen significance level $\alpha$ when the specified alternative value of the parameter is true.

1. get value according to $\alpha$ through $\text{invNorm}$
2. find position of the value in the actual distribution

Effect

Effect size is a quantitative measure of the strength or magnitude of a phenomenon. It tells how large the difference is between groups or how strong the relationship is between variables independent of sample size.

Cohen’s d - means

Cohen’s h

possible values: $-\pi \ \text{and} \ \pi$

Comparing 2 means

Sampling distribution of $\bar x_1 - \bar x_2$

choose an SRS of size $n_1$ from a population with mean $\mu_1$ and standard deviation $\sigma_1$ and an SRS of size $n_2$ from a population with mean $\mu_2$ and standard deviation $\sigma_2$

• shape - approximately normal if sample size is large enough
• center - $\bar x = \bar x_1 - \bar x_2 = \mu_1 - \mu_2$
• spread - $s = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$

Confidence Interval

2 samples t-test

• $\mathcal{H}_0: \mu_1 - \mu_2 = 0 \ \ \text{or} \ \ \mu_1=\mu_2$
• $\mathcal{H}_a: \mu_1 - \mu_2 > (< \text{or} \neq) \ 0 \ \ \text{or} \ \ \mu_1> (< \text{or} \neq) \ \mu_2$
• $\text {df}=n_1+n_2-2$

Comparing 2 proportions

Sampling Distribution for $\hat p_1- \hat p_2$

• center: $\mu_{\hat p_1- \hat p_2}=p_1- p_2$
• shape: approximately normal if the following conditions are met $$n_1p_1\geq10 \ \text{and} \ n_1(1-p_1)\geq10 \\ n_2p_2\geq10 \ \text{and} \ n_2(1-p_2)\geq10$$
• spread: $\sqrt{\frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2}}$
• independent: no more than $10 \%$ of population for either $n_1 \ \text{or} \ n_2$

Confidence Interval

2 prop z-test

• $\mathcal{H}_0: p_1-p_2=0$
• $\mathcal{H}_a: p_1-p_2 > (< \text{or} \neq) \ 0$

• compare $p_v$ to an $\alpha$

Nonparametric Inference Techniques

Everything thus far with inference relies on explicit assumptions about population, distributions, and parameters

• Today focus on how to work with categorical data
• Chi-square methods are relatively free from assumptions

Chi-Square Distribution

Given a normal distribution with a mean of $\mu$ and a variance of $\sigma^2$, we select a random $X$ from population.

The distributions of $z^2$ values could be obtained.

• $\mu = \text{df}$
• $\sigma = \sqrt{2\cdot \text{df}}$
• $\text{mode} = \text{df} - 2$

pchisq("x^2value", df, lower.tail = F)

Chi-Square Goodness of Fit Test

Statement

• $\mathcal{H}_0: \text{The percentage distributions is same as given}$
• $\mathcal{H}_0: \text{value doesn't match}$

Collect random sample of size $N$, how unlikely is it for the observed value to differ as much as they do from the expected values if $\mathcal{H}_0$ is true?

Conditions

• Cells must mutually exclusive and exhaustive
• Observations need to be independent
• Expected frequency value must be at least 5

• $\text{df} = k - 1 = \text{categories} - 1$

type = c(<categories...>)
freq = c(<actual_values...>)
prob = c(<expected_prop...>)
chisq.test(freq, p = prob)

The Chi-Square Test of Independence

Enable us to determine whether two categorical variables are related

Statement

• $\mathcal{H_0}: \text{two variables are mutually independent}$
• $\mathcal{H_a}: \text{two variables are not mutually independent}$

Conditions

Same as Chi-Square Goodness of Fit Test

• Individual observations are independent of each other
• $\min(E_{ij})\geq5$

• $\text{df} = (\text{row}-1)\cdot(\text{column}-1)$
• $E_{ij}=\text{(row total)} \cdot \text{(column total)} \div \text{(sample total)}$
• $E_{ij} \ \text{is assuming that independence is true}$

The Chi-Square Test for Homogeneity

Determine whether different populations or groups have the same distribution of a categorical variable $\text{CV}$.

Statement

• $\mathcal{H}_0: \text{The distribution of the CV are the same across all groups}$
• $\mathcal{H}_a: \text{The distribution of the CV are not the same across all groups}$

Conditions

Same as Chi-Square Goodness of Fit Test

Inferential Statistics for Linear Regression

If data are a random sample from a larger population, we need statistical inference to answer following questions:

• Is there a relationship between x and y in the population or could the pattern we see using the sample data happen by chance?
• In the population, how much will the predicted value of y changes for every increase of 1 unit of x?
• What is the marginal of error for the estimate?

Sampling Distribution of a Slope

Choose a Simple Random Sample of $n$ observations $(x,y)$ from a population of size $N$

• $\mu_b = \beta$
• $\sigma_b = \dfrac{\sigma}{\sigma_x \sqrt n} \ \ \text{as long as} \ \ n \leq \dfrac1{10} N$
• The distribution of y for each value of x follows a normal distribution

Conditions:

• Linear - look at the scatter plot and plot of the residual
• Independent
• Normal - for any value of $(x,y)$, varies according to a normal distribution
  • look at the residuals - box plot, hist, stem&leaf plot
• Equal Standard Deviation - the standard deviation of y is the same for all values of x
  • look at a scatterplot of residuals
• Random - data from a well-designed experiment or random sample

Confidence Interval

Significance Test

Hypothesis

• $\mathcal{H}_0: \Beta = \Beta_0$
• $\mathcal{H}_a: \Beta > (< \text{or} \neq) \ \Beta_0$

Test Statistics

Conclusion

If $\Beta_0 = 0$ determines if the value of y is impacted by the value of x