Driving Smart Decisions: Predictive BI Models in Python#

Exploratory Data Analysis (EDA)#

EDA is the process of summarizing and visualizing data to discover initial patterns, spot anomalies, and test assumptions. Typical EDA steps include:

• Range and Distribution: Check the range (min/max) and distribution (histograms, boxplots).
• Missing Values: Identify incomplete or null entries.
• Relationships: Look for relationships between variables (scatter plots, correlation matrices).

Example code snippet for EDA using pandas and matplotlib:

1
import pandas as pd
2
import matplotlib.pyplot as plt
3
import seaborn as sns
4

5
# Load your CSV data
6
df = pd.read_csv("data.csv")
7

8
# Summary statistics
9
print(df.describe())
10

11
# Check for missing values
12
print(df.isnull().sum())
13

14
# Histogram of a feature
15
df['sales'].hist(bins=20)
16
plt.title("Sales Distribution")
17
plt.xlabel("Sales")
18
plt.ylabel("Frequency")
19
plt.show()
20

21
# Correlation heatmap
22
plt.figure(figsize=(8, 6))
23
sns.heatmap(df.corr(), annot=True, cmap="coolwarm")
24
plt.title("Correlation Matrix")
25
plt.show()

Through these initial EDA steps, you can start forming hypotheses about which variables are essential for your predictive tasks.

Data Cleaning and Transformation#

Data cleaning involves handling missing values, detecting outliers, and ensuring consistent data types. Data transformation manipulates the variables to improve model performance. Common transformations include:

• Handling Missing Data: Drop rows/columns or impute values (mean, median, or more sophisticated methods).
• Encoding Categorical Variables: Convert categorical variables into numeric labels through one-hot encoding or label encoding.
• Scaling: Normalize or standardize numerical features to improve model convergence (especially critical for distance-based algorithms and neural networks).

Below is a sample workflow for handling missing values and encoding categories:

1
import numpy as np
2

3
# Drop rows with missing target values
4
df = df.dropna(subset=['target'])
5

6
# Impute numeric missing values with mean
7
df['age'].fillna(df['age'].mean(), inplace=True)
8

9
# One-hot encode categorical columns
10
df = pd.get_dummies(df, columns=['category_column'], drop_first=True)
11

12
# Standardize a column
13
from sklearn.preprocessing import StandardScaler
14
scaler = StandardScaler()
15
df['income_scaled'] = scaler.fit_transform(df[['income']])

Feature Engineering and Selection#

After your data is cleaned, consider creating new features (feature engineering) that can better capture the underlying relationships. Common techniques:

• Interaction Features: Combine two or more features (e.g., ratios, products).
• Domain-Specific Transformations: E.g., extracting day of week from a timestamp.
• Binning: Convert continuous features into discrete bins to capture non-linear relationships.

At the same time, not all features add value; some may even introduce noise. Feature selection techniques (like univariate tests, recursive feature elimination, or tree-based feature importances) help you keep the most informative features, which can improve model performance and reduce overfitting.

Linear Regression Example#

Assume you have a dataset containing advertising spend (features) and corresponding sales (target). The goal is to predict future sales based on how much you spend on advertising channels.

1
import pandas as pd
2
from sklearn.model_selection import train_test_split
3
from sklearn.linear_model import LinearRegression
4
from sklearn.metrics import mean_squared_error
5

6
# Sample DataFrame
7
data = {
8
    'TV': [230.1, 44.5, 17.2, 151.5, 180.8],
9
    'Radio': [37.8, 39.3, 45.9, 41.3, 10.8],
10
    'Newspaper': [69.2, 45.1, 69.3, 58.5, 58.4],
11
    'Sales': [22.1, 10.4, 9.3, 18.5, 12.9]
12
}
13
df = pd.DataFrame(data)
14

15
X = df[['TV','Radio','Newspaper']]
16
y = df['Sales']
17

18
# Split the data
19
X_train, X_test, y_train, y_test = train_test_split(X, y,
20
                                                    test_size=0.2,
21
                                                    random_state=42)
22

23
# Initialize and train
24
lr_model = LinearRegression()
25
lr_model.fit(X_train, y_train)
26

27
# Predictions
28
y_pred = lr_model.predict(X_test)
29

30
# Evaluate
31
mse = mean_squared_error(y_test, y_pred)
32
rmse = mse**0.5
33
print("Root Mean Squared Error (RMSE):", rmse)
34

35
# Check coefficients
36
print("Coefficients:", lr_model.coef_)
37
print("Intercept:", lr_model.intercept_)

Classification Example with Logistic Regression#

For classification, let’s say your goal is to predict whether a customer will buy a product (Yes/No) based on variables like age, income, and marketing channel.

1
import pandas as pd
2
from sklearn.model_selection import train_test_split
3
from sklearn.linear_model import LogisticRegression
4
from sklearn.metrics import accuracy_score
5

6
# Sample DataFrame
7
data = {
8
    'age': [25, 32, 47, 51, 62],
9
    'income': [50000, 60000, 80000, 120000, 70000],
10
    'channel_web': [1, 1, 0, 0, 1],
11
    'purchased': [0, 1, 1, 0, 1]
12
}
13
df = pd.DataFrame(data)
14

15
X = df[['age','income','channel_web']]
16
y = df['purchased']
17

18
X_train, X_test, y_train, y_test = train_test_split(X, y,
19
                                                    test_size=0.4,
20
                                                    random_state=42)
21

22
# Initialize and train
23
log_model = LogisticRegression()
24
log_model.fit(X_train, y_train)
25

26
# Predictions
27
y_pred_class = log_model.predict(X_test)
28

29
# Evaluate
30
accuracy = accuracy_score(y_test, y_pred_class)
31
print("Classification Accuracy:", accuracy)

Regression Metrics#

• Mean Squared Error (MSE): Average of squared errors between predicted and actual values.
• Root Mean Squared Error (RMSE): Square root of MSE, often more interpretable because it’s in the same units as the target variable.
• Mean Absolute Error (MAE): Average of absolute errors. It’s less sensitive to outliers than MSE.
• R² (Coefficient of Determination): Measures how much variance in the target is explained by the features.

Example code:

1
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
2

3
mse = mean_squared_error(y_test, y_pred)
4
rmse = mse**0.5
5
mae = mean_absolute_error(y_test, y_pred)
6
r2 = r2_score(y_test, y_pred)
7

8
print("MSE:", mse)
9
print("RMSE:", rmse)
10
print("MAE:", mae)
11
print("R-squared:", r2)

Classification Metrics#

• Accuracy: Fraction of correct predictions.
• Precision: Among predicted positives, how many are truly positive.
• Recall: Among actual positives, how many are predicted correctly.
• F1 Score: Harmonic mean of precision and recall.

Example code:

1
from sklearn.metrics import precision_score, recall_score, f1_score
2

3
precision = precision_score(y_test, y_pred_class)
4
recall = recall_score(y_test, y_pred_class)
5
f1 = f1_score(y_test, y_pred_class)
6

7
print("Precision:", precision)
8
print("Recall:", recall)
9
print("F1 Score:", f1)

Confusion Matrix and ROC Curves#

A confusion matrix shows counts of true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN). The ROC curve plots the true positive rate against the false positive rate at various threshold settings.

1
from sklearn.metrics import confusion_matrix, roc_curve, auc
2
import matplotlib.pyplot as plt
3

4
cm = confusion_matrix(y_test, y_pred_class)
5
print("Confusion Matrix:")
6
print(cm)
7

8
# ROC Curve
9
y_pred_prob = log_model.predict_proba(X_test)[:,1]
10
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)
11
roc_auc = auc(fpr, tpr)
12

13
plt.plot(fpr, tpr, label='ROC Curve (area = %0.2f)' % roc_auc)
14
plt.plot([0, 1], [0, 1], 'r--')
15
plt.title("Receiver Operating Characteristic")
16
plt.xlabel("False Positive Rate")
17
plt.ylabel("True Positive Rate")
18
plt.legend(loc="lower right")
19
plt.show()

Ensemble Methods#

Ensemble methods combine multiple models to produce a more robust prediction. Examples include:

Example with Random Forest:

1
from sklearn.ensemble import RandomForestClassifier
2

3
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
4
rf_model.fit(X_train, y_train)
5
y_pred_rf = rf_model.predict(X_test)

Neural Networks and Deep Learning#

Neural networks excel at capturing complex relationships, particularly for unstructured data like images or text. Python libraries like TensorFlow and PyTorch enable building deep learning architectures. For BI tasks like tabular data forecasting or classification, you can experiment with multi-layer perceptrons (MLPs).

Simple example of a Keras-based neural network for regression:

1
import tensorflow as tf
2
from tensorflow.keras import Sequential
3
from tensorflow.keras.layers import Dense
4

5
model = Sequential()
6
model.add(Dense(64, activation='relu', input_shape=(X_train.shape[1],)))
7
model.add(Dense(32, activation='relu'))
8
model.add(Dense(1))  # single output neuron for regression
9

10
model.compile(optimizer='adam', loss='mean_squared_error')
11
model.fit(X_train, y_train, epochs=50, batch_size=32)
12

13
# Evaluate
14
loss = model.evaluate(X_test, y_test)
15
print("Test Loss:", loss)

Time Series Forecasting#

When dealing with sequential data (e.g., monthly sales, website traffic), time series forecasting models can be used. Popular methods include:

• ARIMA (AutoRegressive Integrated Moving Average)
• Prophet (by Facebook)
• LSTM Neural Networks for more complex patterns

Example with the statsmodels library for ARIMA:

1
import pandas as pd
2
import matplotlib.pyplot as plt
3
from statsmodels.tsa.arima.model import ARIMA
4

5
# Suppose df has a 'date' column and a 'sales' column
6
df['date'] = pd.to_datetime(df['date'])
7
df.set_index('date', inplace=True)
8

9
# Split into train/test
10
train = df.iloc[:-12]  # all but last 12 months
11
test = df.iloc[-12:]
12

13
model = ARIMA(train['sales'], order=(1,1,1))
14
model_fit = model.fit()
15
forecast = model_fit.forecast(steps=12)
16
plt.plot(test.index, test['sales'], label='Actual')
17
plt.plot(test.index, forecast, label='Forecast')
18
plt.legend()
19
plt.show()

Model Deployment#

Common approaches to deploying models include:

• REST APIs: Wrap your model in a Flask or FastAPI service that other applications can request predictions from.
• Batch Predictions: Run scheduled jobs that load data, score it, and store results.
• Cloud Services: Use platforms like AWS Sagemaker, Google Cloud AI Platform, or Azure ML to host and scale your models.

Example of a simple Flask API:

1
from flask import Flask, request, jsonify
2
import pickle
3
import numpy as np
4

5
app = Flask(__name__)
6

7
# Load trained model
8
with open('lr_model.pkl', 'rb') as f:
9
    model = pickle.load(f)
10

11
@app.route('/predict', methods=['POST'])
12
def predict():
13
    data = request.get_json()
14
    features = np.array(data['features']).reshape(1, -1)
15
    prediction = model.predict(features)
16
    return jsonify({'prediction': float(prediction[0])})
17

18
if __name__ == '__main__':
19
    app.run(debug=True)

Monitoring and Maintenance#

Once deployed, models can “drift” as the underlying data changes. Regularly monitor performance metrics, retrain or update models when performance drops, and maintain transparency regarding how the model makes decisions (consider interpretable AI techniques if needed).

Model monitoring best practices:

• Track metrics over time (accuracy, precision/recall, etc.).
• Create alert thresholds for significant performance drops.
• Maintain version control for your model and data schema.