a

WS2425/Data Science/Cheat Sheet Mockup.md

@@ -0,0 +1,400 @@
+![[lecture_02.pdf]]
+
+
+
+1. **Definition of Data:**
+   - Data is information collected, stored, or processed. It is ubiquitous and can be measured or categorized.
+
+2. **Data Basics:**
+   - **Basic Population:** Entire group of interest (e.g., all students).
+   - **Sample:** Subset of the population (e.g., students in a lecture).
+   - **Statistical Unit:** Individual data point (e.g., one student).
+   - **Variable:** Characteristics of units (e.g., name, population size).
+   - **Value:** Specific value of a variable (e.g., "123456" for "MatrNr").
+
+3. **Data Categories:**
+   - **Structured vs. Unstructured:**
+     - **Structured:** Organized data with a predefined format (e.g., tables).
+     - **Unstructured:** No traditional format (e.g., text, images).
+   - **Discrete vs. Continuous:**
+     - **Discrete:** Countable values (e.g., grades).
+     - **Continuous:** Any value within a range (e.g., temperature).
+   - **Levels of Measurement:**
+     - **Nominal:** Labels without order (e.g., colors).
+     - **Ordinal:** Ordered labels (e.g., school grades).
+     - **Interval:** Ordered with equal intervals (e.g., Celsius).
+     - **Ratio:** Interval with a true zero (e.g., weight).
+   - **Qualitative vs. Quantitative:**
+     - **Qualitative:** Categorical (e.g., gender).
+     - **Quantitative:** Numerical (e.g., height).
+
+![[lecture_03.pdf]]
+
+
+1. **Primary vs. Secondary Data:**
+   - **Primary Data:** Collected directly for a specific purpose (e.g., surveys, experiments).
+   - **Secondary Data:** Existing data from other sources (e.g., books, journals).
+
+2. **Ways to Obtain Data:**
+   - **Capturing Data:** Collecting through sensors, observations, or experiments.
+   - **Retrieving Data:** Accessing from databases, APIs, or open data sources.
+   - **Collecting Data:** Scraping from websites or logs when direct access isn't available.
+
+3. **Databases:**
+   - **Relational Databases:** Use SQL for structured data but have limitations with big data.
+   - **NoSQL Databases:** Handle unstructured or semi-structured data, offering flexibility and scalability.
+   - **Document-Oriented Databases:** Store data in formats like JSON, ideal for e-commerce and IoT.
+
+4. **APIs:**
+   - REST-APIs enable communication between systems, using HTTP methods (GET, POST, PUT, DELETE).
+   - Often require authentication (e.g., API keys) and provide data in JSON/XML formats.
+
+5. **Data Scraping:**
+   - Extracting data from websites or logs when APIs aren't available.
+   - Legal and ethical considerations must be addressed.
+
+
+![[lecture_04.pdf]]
+
+
+1. **Data Protection and Anonymization:**
+   - **GDPR Compliance:** Personal data must be protected, and usage requires consent.
+   - **Anonymization:** Removing personal identifiers to prevent individual identification.
+   - **Pseudonymization:** Using non-unique identifiers, requiring additional info for identification.
+   - **Hashing:** Converting data to fixed-size values (e.g., SHA-256) for privacy.
+
+2. **Statistical Basics:**
+   - **Descriptive Statistics:** Summarizes data (e.g., mean, median).
+   - **Exploratory Data Analysis:** Identifies patterns and outliers.
+   - **Inferential Statistics:** Draws conclusions about populations from samples.
+
+3. **Frequencies and Histograms:**
+   - **Frequencies:** Count of occurrences of each value.
+   - **Absolute vs. Relative Frequencies:** Raw counts vs. proportions.
+   - **Histograms:** Visual representation of data distribution across classes.
+
+4. **Empirical Distribution Function (EDF):**
+   - Plots cumulative frequencies to show data distribution over a range.
+
+5. **Data Visualization:**
+   - **Pie Charts:** Effective for showing proportions of categorical data.
+   - **Bar Charts:** Compare frequencies across categories.
+   - **Histograms:** Display distribution of continuous data.
+
+![[lecture_05.pdf]]
+
+
+1. **Central Tendencies:**
+   - **Mode:** The most frequently occurring value in a dataset.
+   - **Median:** The middle value when data is ordered, dividing the dataset into two equal halves.
+   - **Mean:** The average value, calculated by summing all observations and dividing by the number of observations.
+
+2. **Statistical Dispersion:**
+   - **Range:** The difference between the maximum and minimum values.
+   - **Interquartile Range (IQR):** The difference between the third quartile (Q3) and first quartile (Q1), representing the middle 50% of the data.
+   - **Variance and Standard Deviation:** Measures of spread, with variance being the average squared deviation from the mean and standard deviation the square root of variance.
+
+3. **Data Visualization:**
+   - **Histograms:** Display the distribution of continuous data across classes.
+   - **Box Plots:** Show the five-number summary (minimum, Q1, median, Q3, maximum) and identify outliers.
+
+4. **Outliers:**
+   - Defined as data points falling outside the range of [Q1 - 1.5*IQR, Q3 + 1.5*IQR].
+   - Can indicate errors, unusual observations, or novel data points.
+
+![[lecture_06.pdf]]
+
+
+1. **Empirical Variance Calculation**:
+   - **Data**: Daily temperatures (°C): 11.2, 13.3, 14.1, 13.7, 12.2, 11.3, 9.9
+   - **Mean (x̄)**: 12.24
+   - **Sum of squared deviations**: 14.16
+   - **Empirical Variance (s̃²)**: \( $\frac{14.16}{7}$ = 2.02 \)
+   - **Empirical Standard Deviation (s̃)**: \( $\sqrt{2.02} \approx 1.42$ \)
+
+2. **Contingency Table Analysis**:
+   - **Variables**: Growth (shrinking, growing, growing strongly) and Location (North, South)
+   - **Absolute Frequencies**:
+     - North: 29 (growing), 13 (growing strongly), 7 (shrinking)
+     - South: 13 (growing), 19 (growing strongly), 2 (shrinking)
+   - **Chi-squared (χ²) Test**:
+     - Calculated χ²: 7.53
+   - **Corrected Pearson Contingency Coefficient (K*P)**:
+     - \( K*P = $\sqrt{\frac{7.53}{7.53 + 83}} \times \sqrt{\frac{2}{2}} = 0.41$ \)
+     - Interpretation: Weak to medium correlation between location and growth.
+
+3. **Correlation Coefficient**:
+   - **Pearson Correlation Coefficient (rXY)** for population and area:
+     - Calculated rXY: 0.70
+     - Interpretation: Strong positive correlation.
+
+4. **Key Concepts**:
+   - **Empirical Variance**: Measures data spread around the mean.
+   - **Contingency Tables**: Used for nominal/ordinal data to assess associations.
+   - **Pearson Correlation**: Measures linear correlation between metric variables, ranging from -1 to 1.
+
+![[lecture_07.pdf]]
+
+
+
+1. **Probability Theory Basics**:
+   - **Sample Space (Ω)**: Set of all possible outcomes.
+   - **Event**: Subset of the sample space.
+   - **Probability Axioms** (Kolmogorov):
+     1. \( P(A) $\geq 0$ \)
+     2. \( P(Ω) = 1 \)
+     3. Additivity for disjoint events.
+
+2. **Conditional Probability**:
+   - **Definition**: \( P(A|B) = $\frac{P(A \cap B)}{P(B)}$ \)
+   - **Independence**: Events A and B are independent if \( P(A \cap B) = P(A)P(B) \).
+
+3. **Bayes' Theorem**:
+   - **Formula**: \( P(B|A) = $\frac{P(A|B)P(B)}{P(A)}$ \)
+   - **Example**: 
+     - \( P(B|A) = $\frac{0.9 \times 0.01}{0.9 \times 0.01 + 0.02 \times 0.99}$ = 0.31 \)
+
+4. **Combinatorics**:
+   - **Permutations**: \( n! \) for unique items.
+   - **Combinations**: \( $\binom{n}{k} = \frac{n!}{k!(n-k)!}$ \)
+   - **Multiplication Rule**: \( n_1 $\times n_2 \times \dots \times n_r$ \)
+
+5. **Key Calculations**:
+   - **Password Example**: \( 7^2 $\times$ P(10,4) = 49 $\times$ 5040 = 246960 \)
+   - **Dice Probability**: \( P($\text{at least one 6 in 4 throws}$) = 1 - (5/6)^4 = 0.518 \)
+
+![[lecture_08_neu.pdf]]
+
+
+1. **Random Variables**:
+   - **Definition**: A function that assigns numerical values to outcomes in a sample space.
+   - **Discrete vs. Continuous**: 
+     - Discrete: Countable outcomes (e.g., dice roll).
+     - Continuous: Uncountable outcomes (e.g., measurement of height).
+
+2. **Probability Distributions**:
+   - **Discrete**:
+     - **Bernoulli**: \( $f(x) = p^x(1-p)^{1-x}$\) for \( $x \in \{0,1\}$ \).
+     - **Binomial**: \( $f(x) = \binom{n}{x}p^x(1-p)^{n-x}$ \) for \( $x \in \{0,1,...,n\}$ \).
+     - **Uniform**: \( $f(x) = \frac{1}{m}$ \) for \($x \in \{1,2,...,m\}$ \).
+   - **Continuous**:
+     - **Uniform**: \( $f(x) = \frac{1}{b-a}$ \) for \( $x \in [a,b]$ \).
+     - **Normal (Gaussian)**: \( $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-(x-\mu)^2/(2\sigma^2)}$ \).
+
+3. **Expected Value and Variance**:
+   - **Discrete**:
+     - **Expected Value**: \( $E(X) = \sum x \cdot f(x)$ \).
+     - **Variance**: \( $Var(X) = E((X-E(X))^2)$ \).
+   - **Continuous**:
+     - **Expected Value**: \( $E(X) = \int_{-\infty}^{\infty} x \cdot f(x)dx$ \).
+     - **Variance**: \( $Var(X) = E((X-E(X))^2)$ \).
+
+4. **Key Calculations**:
+   - **Bernoulli Distribution**:
+     - \( $E(X) = p$ \), \( $Var(X) = p(1-p)$ \).
+   - **Binomial Distribution**:
+     - \( $E(X) = np$ \), \( $Var(X) = np(1-p)$ \).
+   - **Uniform Distribution (Discrete)**:
+     - \( $E(X) = \frac{m+1}{2}$ \), \( $Var(X) = \frac{m^2-1}{12}$ \).
+   - **Uniform Distribution (Continuous)**:
+     - \( $E(X) = \frac{a+b}{2}$ \), \( $Var(X) = \frac{(b-a)^2}{12}$ \).
+   - **Normal Distribution**:
+     - \( $E(X) = \mu$ \), \( $Var(X) = \sigma^2$ \).
+
+5. **Normal Distribution**:
+   - **Standard Normal (Z-Score)**: \( $Z = \frac{X-\mu}{\sigma} \sim N(0,1)$ \).
+   - **68-95-99.7 Rule**: 68% data within \( $\mu \pm \sigma$ \), 95% within \( $\mu \pm 2\sigma$ \), 99.7% within \( $\mu \pm 3\sigma$ \).
+
+![[lecture_09.pdf]]
+
+
+
+Here are the key points and calculations from the provided data:
+
+1. **Simple Linear Regression**:
+   - **Model**: \( $Y = \beta_0 + \beta_1X + \epsilon$ \), where \( $\epsilon \sim N(0, \sigma^2$) \).
+   - **Estimation**: Parameters \( $\beta_0$ \) and \( $\beta_1$ \) are estimated using least squares method.
+
+2. **Least Squares Estimators**:
+   - **Slope (β̂₁)**: 
+     \[$$
+     \hat{\beta}_1 = \frac{\sum_{i=1}^{n}(Y_i - \bar{Y})(X_i - \bar{X})}{\sum_{i=1}^{n}(X_i - \bar{X})^2}
+     $$\]
+   - **Intercept (β̂₀)**:
+     \[$$
+     \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}
+     $$\]
+
+3. **Residual Analysis**:
+   - **Residuals**: \( $e_i = Y_i - \hat{Y}_i$ \).
+   - **Residual Plot**: Used to check model assumptions (linearity, constant variance, normality).
+
+4. **Coefficient of Determination (R²)**:
+   - **Formula**:
+     \[$$
+     R^2 = \frac{\text{RSS}}{\text{TSS}} = 1 - \frac{\text{ESS}}{\text{TSS}}
+     $$\]
+   - **Interpretation**: Measures goodness of fit (0 ≤ R² ≤ 1).
+
+5. **Variance Estimation**:
+   - **Residual Sum of Squares (RSS)**:
+     \[$$
+     RSS = \sum_{i=1}^{n}(Y_i - \hat{Y}_i)^2
+     $$\]
+   - **Variance Estimator**:
+     \[$$
+     \hat{\sigma}^2 = \frac{RSS}{n-2}
+     $$\]
+
+6. **Key Calculations**:
+   - **Example Calculations**:
+     - For given data, compute \( $\hat{\beta}_0$ \), \( $\hat{\beta}_1$ \), and \( $\hat{\sigma}^2$ \).
+     - Calculate R² to assess model fit.
+
+
+
+![[lecture_10.pdf]]
+
+
+1. **Confidence Intervals**:
+   - **Point Estimator**: Provides a single estimate of a population parameter.
+   - **Interval Estimator**: Provides a range of values within which the parameter is expected to lie.
+   - **Formula for Mean (σ known)**:
+     \[$$
+     \left[ \bar{X} - z_{1-\alpha/2} \frac{\sigma}{\sqrt{n}}, \bar{X} + z_{1-\alpha/2} \frac{\sigma}{\sqrt{n}} \right]
+     $$\]
+   - **Formula for Mean (σ unknown)**:
+     \[$$
+     \left[ \bar{X} - t_{n-1,1-\alpha/2} \frac{S}{\sqrt{n}}, \bar{X} + t_{n-1,1-\alpha/2} \frac{S}{\sqrt{n}} \right]
+     $$\]
+
+2. **Statistical Tests**:
+   - **Hypothesis Testing**: Involves setting up a null hypothesis (H₀) and an alternative hypothesis (H₁), then determining whether to reject H₀ based on sample data.
+   - **Z-Test**: Used when the population variance is known.
+     - **Test Statistic**: \( $Z = \frac{\bar{X} - \mu_0}{\sigma / \sqrt{n}}$ \)
+   - **T-Test**: Used when the population variance is unknown.
+     - **Test Statistic**: \( $T = \frac{\bar{X} - \mu_0}{S / \sqrt{n}}$ \)
+   - **Two-Sample T-Test**: Compares the means of two independent groups.
+     - **Test Statistic**: \( $T = \frac{\bar{X} - \bar{Y}}{\sqrt{\frac{S_X^2}{n} + \frac{S_Y^2}{m}}}$ \)
+
+3. **Key Calculations**:
+   - **Example 1**: 95% confidence interval for bonbon package weights.
+     - **Result**: [63.84, 65.08]
+   - **Example 2**: Testing machine adjustment with t-test.
+     - **Result**: Null hypothesis not rejected, machine does not need adjustment.
+   - **Example 3**: Two-sample t-test for bonbon weights.
+     - **Result**: Reject H₀, second company’s bonbons are heavier.
+
+![[lecture_11.pdf]]
+
+
+1. **Point Estimation**:
+   - A point estimator (e.g., sample mean) estimates the parameter θ from a random sample.
+   - Example: For a normal distribution, the arithmetic mean estimates the expected value.
+
+2. **Confidence Intervals**:
+   - A range [gl, gu] where θ is likely to lie with probability 1−α.
+   - Types: One-sided (e.g., [gl, ∞)) and two-sided (finite interval).
+
+3. **Statistical Hypothesis Testing**:
+   - **Null Hypothesis (H0)**: Statement to be tested (e.g., μ = 15).
+   - **Alternative Hypothesis (H1)**: Opposing statement (e.g., μ ≠ 15).
+   - **Test Statistic**: A function of the sample data to assess H0 vs. H1.
+   - **Rejection Region**: Values of the test statistic leading to rejection of H0.
+   - **Errors**:
+     - **Type I Error**: Rejecting a true H0 (controlled by significance level α).
+     - **Type II Error**: Failing to reject a false H0 (related to test power).
+
+4. **Z-Test and T-Test**:
+   - **Z-Test**: Used when population variance is known or sample size is large.
+   - **T-Test**: Used when population variance is unknown; relies on sample standard deviation and t-distribution.
+
+5. **Two-Sample T-Test**:
+   - Compares means of two independent groups.
+   - **Assumptions**: Normality, independence, homogeneity (if variances are equal).
+   - **Test Statistic**: Accounts for sample means, standard deviations, and sizes.
+   - **Degrees of Freedom**: Calculated using Welch-Satterthwaite equation for unequal variances.
+
+6. **Examples**:
+   - **One-Sample Test**: Chocolate box weights using z-test (known variance).
+   - **Two-Sample Test**: Comparing bonbon weights using t-test with Welch-Satterthwaite adjustment.
+
+7. **Key Concepts**:
+   - **p-value**: Probability of observing test statistic under H0; compared to α.
+   - **Confidence Interval and Hypothesis Test Relationship**: Rejecting H0 if the parameter lies outside the confidence interval.
+
+8. **Future Topics**: Data preparation and decision trees.
+
+![[lecture_12.pdf]]
+
+
+1. **Machine Learning Overview**:
+   - Involves data preparation, model building, and evaluation.
+   - Follows the CRISP-DM process: Business understanding, data understanding, modeling, evaluation, and deployment.
+
+2. **Data Preparation**:
+   - **Handling Missing Data**: Strategies include filtering, marking, or imputing missing values (e.g., mean, median, or model-based imputation).
+   - **Handling False Data**: Identify and correct errors through analysis or expert consultation.
+   - **Feature Engineering**: Create new features from existing data (e.g., deriving age from birthdate) or combine attributes for better model performance.
+
+3. **Decision Trees**:
+   - **Structure**: A tree with nodes representing tests on attributes, leading to leaf nodes with class predictions.
+   - **Construction**: Built by recursively splitting data to minimize entropy (disorder). Information gain determines the best split.
+   - **Example**: Medicine recommendation based on blood pressure and age.
+   - **Pros and Cons**: Easy to interpret but may overfit or require discretization for numerical attributes.
+
+4. **Model Evaluation**:
+   - **Train-Test Split**: Evaluate models on independent test data to avoid overfitting.
+   - **Metrics for Regression**: Mean Squared Error (MSE), Mean Absolute Error (MAE).
+   - **Metrics for Classification**: Confusion matrix, accuracy, precision, recall, F1 score.
+     - **Accuracy**: Overall correctness.
+     - **Precision**: Correct predictions among positive predictions.
+     - **Recall**: Correct predictions among actual positive instances.
+     - **F1 Score**: Harmonic mean of precision and recall.
+
+5. **Advanced Topics**:
+   - **Ensemble Methods**: Random Forests and Gradient Boosting improve decision trees by reducing variance and bias.
+   - **Model Comparison**: Use validation sets to compare models and avoid overfitting.
+
+6. **Summary**:
+   - **Skills Acquired**: Data preparation, decision tree classification, and model evaluation.
+   - **Future Topics**: Dashboards and summaries for data visualization and reporting.
+
+
+![[lecture_13.pdf]]
+
+1. **Model Evaluation Basics**:
+   - **Overfitting**: When a model fits the training data too well, leading to poor performance on new data.
+   - **Underfitting**: When a model is too simple to capture the data's structure, resulting in poor performance on both training and test data.
+
+2. **Data Splitting**:
+   - **Train-Test Split**: Commonly used to evaluate model performance. Typical splits are 80% for training and 20% for testing.
+   - **Train-Validation-Test Split**: Used to compare different models by having separate training, validation, and test sets.
+
+3. **Evaluation Metrics**:
+   - **Regression Metrics**:
+     - **Mean Squared Error (MSE)**: Average of squared differences between predicted and actual values.
+     - **Mean Absolute Error (MAE)**: Average of absolute differences.
+     - **Mean Absolute Percentage Error (MAPE)**: Average of percentage differences.
+   - **Classification Metrics**:
+     - **Confusion Matrix**: Tabulates true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN).
+     - **Accuracy**: (TP + TN) / (TP + TN + FP + FN).
+     - **Precision**: TP / (TP + FP).
+     - **Recall**: TP / (TP + FN).
+     - **F1 Score**: Harmonic mean of precision and recall, given by \( $F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$ \).
+
+4. **Example Calculations**:
+   - **Confusion Matrix Example**:
+     - Accuracy: \( $\frac{4 + 1}{10} = 0.5$ \)
+     - Precision: \( $\frac{4}{6} \approx 0.66$ \)
+     - Recall: \( $\frac{4}{7} \approx 0.57$ \)
+     - F1 Score: \( $\frac{2 \times 0.66 \times 0.57}{0.66 + 0.57} \approx 0.31$ \)
+
+5. **Interpreting Results**:
+   - **Baseline Comparison**: Always compare model performance against a naive reference (e.g., predicting the majority class).
+   - **Contextual Relevance**: Accuracy alone may not indicate practical usefulness; consider domain context and baseline performance.
+
+6. **Summary**:
+   - **Skills Acquired**: Understanding of evaluation metrics and their application in assessing machine learning models.
+   - **Future Topics**: Advanced evaluation techniques and model optimization strategies.
+