--- name: data-analysis-plus description: "Enhanced data analysis with Python/R code templates, visualization gallery, statistical tests, and automated report generation. Covers hypothesis testing, regression, clustering, time series, and more." metadata: author: opencode version: 2.0 tags: data-analysis, statistics, visualization, python, r compatibility: opencode license: MIT --- # Data Analysis Plus Enhanced data analysis with code templates, visualization gallery, and statistical methods. ## Features - **Code Templates**: Python/R ready-to-use templates - **Visualization Gallery**: Charts for every analysis type - **Statistical Methods**: Hypothesis testing, regression, clustering - **Automated Reports**: Decision-ready output formats - **Data Validation**: Quality checks before analysis ## Quick Reference | Analysis Type | Python Template | R Template | |---------------|-----------------|------------| | Descriptive | `df.describe()` | `summary(df)` | | Hypothesis | `scipy.stats.ttest_ind()` | `t.test()` | | Regression | `sklearn.linear_model` | `lm()` | | Clustering | `sklearn.cluster.KMeans` | `kmeans()` | | Time Series | `statsmodels.tsa` | `forecast::auto.arima()` | ## Python Templates ### Data Loading ```python import pandas as pd import numpy as np # CSV df = pd.read_csv("data.csv") # Excel df = pd.read_excel("data.xlsx") # JSON df = pd.read_json("data.json") # Database import sqlalchemy engine = sqlalchemy.create_engine("sqlite:///data.db") df = pd.read_sql("SELECT * FROM table", engine) ``` ### Descriptive Statistics ```python # Basic stats df.describe() # By group df.groupby("category").agg({ "value": ["mean", "median", "std", "count"] }) # Correlation df.corr() ``` ### Data Cleaning ```python # Missing values df.isnull().sum() df.fillna(df.mean()) df.dropna() # Duplicates df.duplicated().sum() df.drop_duplicates() # Outliers Q1 = df["value"].quantile(0.25) Q3 = df["value"].quantile(0.75) IQR = Q3 - Q1 df = df[(df["value"] >= Q1 - 1.5*IQR) & (df["value"] <= Q3 + 1.5*IQR)] ``` ### Hypothesis Testing ```python from scipy import stats # T-test group1 = df[df["group"] == "A"]["value"] group2 = df[df["group"] == "B"]["value"] stat, p_value = stats.ttest_ind(group1, group2) # Chi-square contingency = pd.crosstab(df["cat1"], df["cat2"]) chi2, p_value, dof, expected = stats.chi2_contingency(contingency) # ANOVA f_stat, p_value = stats.f_oneway(group1, group2, group3) ``` ### Regression ```python from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split X = df[["feature1", "feature2"]] y = df["target"] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) model = LinearRegression() model.fit(X_train, y_train) print(f"R²: {model.score(X_test, y_test)}") print(f"Coefficients: {model.coef_}") ``` ### Clustering ```python from sklearn.cluster import KMeans from sklearn.preprocessing import StandardScaler scaler = StandardScaler() X_scaled = scaler.fit_transform(df[["feature1", "feature2"]]) kmeans = KMeans(n_clusters=3, random_state=42) df["cluster"] = kmeans.fit_predict(X_scaled) ``` ### Time Series ```python import pandas as pd from statsmodels.tsa.seasonal import seasonal_decompose # Parse dates df["date"] = pd.to_datetime(df["date"]) df = df.set_index("date") # Decompose decomposition = seasonal_decompose(df["value"], model="additive", period=12) decomposition.plot() ``` ## R Templates ### Data Loading ```r library(readr) library(readxl) # CSV df <- read_csv("data.csv") # Excel df <- read_excel("data.xlsx") # JSON library(jsonlite) df <- fromJSON("data.json") ``` ### Descriptive Statistics ```r # Basic stats summary(df) # By group library(dplyr) df %>% group_by(category) %>% summarise( mean = mean(value, na.rm = TRUE), median = median(value, na.rm = TRUE), sd = sd(value, na.rm = TRUE), n = n() ) # Correlation cor(df[, sapply(df, is.numeric)], use = "complete.obs") ``` ### Hypothesis Testing ```r # T-test t.test(value ~ group, data = df) # Chi-square chisq.test(table(df$cat1, df$cat2)) # ANOVA aov_result <- aov(value ~ group, data = df) summary(aov_result) ``` ### Regression ```r # Linear regression model <- lm(target ~ feature1 + feature2, data = df) summary(model) # Logistic regression model <- glm(binary_target ~ feature1 + feature2, data = df, family = "binomial") summary(model) ``` ### Clustering ```r # K-means library(cluster) df_scaled <- scale(df[, c("feature1", "feature2")]) kmeans_result <- kmeans(df_scaled, centers = 3) df$cluster <- kmeans_result$cluster ``` ## Visualization Gallery ### Chart Selection Guide | Question Type | Chart | Python | R | |---------------|-------|--------|---| | Trend over time | Line | `matplotlib` | `ggplot2` | | Comparison | Bar | `seaborn.barplot` | `ggplot2::geom_bar` | | Distribution | Histogram | `seaborn.histplot` | `ggplot2::geom_histogram` | | Relationship | Scatter | `seaborn.scatterplot` | `ggplot2::geom_point` | | Composition | Pie/Stacked Bar | `matplotlib.pyplot.pie` | `ggplot2::geom_bar(position="fill")` | | Correlation | Heatmap | `seaborn.heatmap` | `pheatmap` | ### Python Visualization ```python import matplotlib.pyplot as plt import seaborn as sns # Line plot plt.figure(figsize=(10, 6)) sns.lineplot(data=df, x="date", y="value", hue="category") plt.title("Trend Over Time") plt.savefig("trend.png", dpi=300, bbox_inches="tight") # Scatter plot plt.figure(figsize=(10, 6)) sns.scatterplot(data=df, x="feature1", y="feature2", hue="target") plt.title("Feature Relationship") plt.savefig("scatter.png", dpi=300, bbox_inches="tight") # Heatmap plt.figure(figsize=(10, 8)) sns.heatmap(df.corr(), annot=True, cmap="coolwarm", center=0) plt.title("Correlation Matrix") plt.savefig("heatmap.png", dpi=300, bbox_inches="tight") ``` ### R Visualization ```r library(ggplot2) # Line plot ggplot(df, aes(x = date, y = value, color = category)) + geom_line() + labs(title = "Trend Over Time") + theme_minimal() # Scatter plot ggplot(df, aes(x = feature1, y = feature2, color = target)) + geom_point() + labs(title = "Feature Relationship") + theme_minimal() # Bar plot ggplot(df, aes(x = category, fill = category)) + geom_bar() + labs(title = "Category Distribution") + theme_minimal() ``` ## Statistical Methods ### Hypothesis Testing Decision Tree ``` Is the data categorical? ├── Yes → Chi-square test └── No → Is the data normally distributed? ├── Yes → T-test (2 groups) / ANOVA (3+ groups) └── No → Mann-Whitney U (2 groups) / Kruskal-Wallis (3+ groups) ``` ### Sample Size Calculator ```python def sample_size计算器(effect_size, alpha=0.05, power=0.8): from statsmodels.stats.power import TTestIndPower analysis = TTestIndPower() n = analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power) return int(np.ceil(n * 2)) # Total sample size ``` ## Report Templates ### Executive Summary ```markdown # [Analysis Title] ## Key Findings 1. [Finding 1 with evidence] 2. [Finding 2 with evidence] 3. [Finding 3 with evidence] ## Methodology - Data: [source, timeframe, sample size] - Methods: [statistical tests used] - Limitations: [caveats] ## Recommendations 1. [Action 1] 2. [Action 2] 3. [Action 3] ## Appendix - Charts: [list of visualizations] - Statistical tables: [p-values, confidence intervals] ``` ### Technical Report ```markdown # [Analysis Title] - Technical Report ## Data - Source: [database/file] - Records: [count] - Variables: [list with types] - Missing values: [summary] ## Methods - [Method 1]: [justification] - [Method 2]: [justification] ## Results ### [Test 1] - Statistic: [value] - p-value: [value] - Effect size: [value] - Confidence interval: [range] ## Code [Reproducible code] ``` ## Best Practices 1. **Start with EDA** - Understand data before analysis 2. **Validate data** - Check quality, missing values, outliers 3. **Document assumptions** - State methodology choices 4. **Report uncertainty** - Confidence intervals, p-values 5. **Visualize results** - Charts communicate better 6. **Make reproducible** - Save code, set seeds 7. **Peer review** - Have someone check your work ## Common Pitfalls | Pitfall | Solution | |---------|----------| | P-hacking | Pre-register hypotheses | | Correlation ≠ causation | Use experiments when possible | | Selection bias | Random sampling | | Survivorship bias | Include failures | | Simpson's paradox | Stratify analysis |

Data Analysis Plus

Description

Latest Changes

Install Data Analysis Plus with One Click