Advanced Data Manipulation Tactics in Python#

Lists#

A list is a mutable data structure that can hold items of varying data types. You can add, remove, and modify elements at any time.

Example of creating and manipulating a list:

1
# Creating a list
2
my_list = [1, 2, 3, 4, 5]
3

4
# Appending an item
5
my_list.append(6)
6

7
# Removing an item
8
my_list.remove(2)
9

10
# Slicing a list (getting elements from index 2 to 4)
11
sub_list = my_list[2:5]
12

13
print("Original list:", my_list)
14
print("Sliced list:", sub_list)

Key Operations with Lists
• Appending, extending, or inserting items
• Removing items (by value or index)
• Slicing to extract sub-lists
• Combining lists using the + operator or extend()

Tuples#

Tuples are similar to lists, but they are immutable. Once created, you cannot change the contents of a tuple. This is useful for storing data that should remain constant.

1
my_tuple = (1, 2, 3)
2
# Trying to modify a tuple results in an error
3
# my_tuple[0] = 10  # This will raise a TypeError

Dictionaries#

Dictionaries store data as key-value pairs, making them invaluable for quick lookups or when you want to label your data.

1
grades = {
2
    "Alice": 85,
3
    "Bob": 92,
4
    "Charlie": 88
5
}
6

7
# Accessing a value
8
print("Alice's grade:", grades["Alice"])
9

10
# Adding a new key-value pair
11
grades["David"] = 90
12

13
# Iterating over a dictionary
14
for student, grade in grades.items():
15
    print(student, grade)

Key Operations with Dictionaries
• Adding and deleting key-value pairs
• Accessing values by their keys
• Iterating over keys, values, or both
• Using dictionary comprehensions for quick transformations

Sets#

A set is an unordered collection of unique elements. It’s particularly helpful for membership tests and for computing intersections, unions, and differences between collections.

1
set_a = {1, 2, 3, 4}
2
set_b = {3, 4, 5, 6}
3

4
# Union
5
set_union = set_a.union(set_b)
6
# Intersection
7
set_intersection = set_a.intersection(set_b)
8
# Difference
9
set_diff = set_a.difference(set_b)
10

11
print("Union:", set_union)
12
print("Intersection:", set_intersection)
13
print("Difference:", set_diff)

Numpy Arrays vs. Python Lists#

Numpy arrays are contiguous in memory and support a rich set of vectorized operations, allowing operations on entire arrays without explicit loops in Python. This can result in massive performance gains compared to plain lists.

Example performance difference:

1
import numpy as np
2
import time
3

4
# Large data size
5
size = 10_000_000
6

7
# Using Python lists
8
l1 = range(size)
9
l2 = range(size)
10
start_time = time.time()
11
result_list = [x + y for x, y in zip(l1, l2)]
12
end_time = time.time()
13
print("Python list addition took:", end_time - start_time, "seconds")
14

15
# Using Numpy arrays
16
a1 = np.arange(size)
17
a2 = np.arange(size)
18
start_time = time.time()
19
result_array = a1 + a2
20
end_time = time.time()
21
print("Numpy array addition took:", end_time - start_time, "seconds")

Creating and Reshaping Arrays#

Numpy arrays can be created from Python lists or generated using built-in functions (np.zeros, np.ones, np.arange, etc.). Reshaping allows you to alter the dimensionality of your data without creating a copy (when feasible).

1
import numpy as np
2

3
# Creating an array from a list
4
arr = np.array([1, 2, 3, 4, 5])
5

6
# Creating arrays with specific shapes
7
zeros_arr = np.zeros((2, 3))  # 2x3 array of zeros
8
ones_arr  = np.ones((3, 2))   # 3x2 array of ones
9

10
# Reshaping an array
11
initial_arr = np.arange(12)   # [0, 1, 2, ..., 11]
12
reshaped_arr = initial_arr.reshape((3, 4))
13
print(reshaped_arr)

Boolean Masking and Advanced Indexing#

Boolean masking allows you to filter arrays based on conditions, returning only the elements that meet specific criteria. You can also use advanced indexing to manipulate data in non-linear ways.

1
import numpy as np
2

3
data = np.array([10, 20, 30, 40, 50])
4
mask = data > 25
5
filtered_data = data[mask]  # [30, 40, 50]
6

7
# Advanced Indexing
8
indices = [0, 2, 4]
9
selected_data = data[indices]  # [10, 30, 50]

Series and DataFrames: A Quick Overview#

A Series is a one-dimensional labeled array, while a DataFrame is a two-dimensional labeled data structure with columns possibly containing different data types. Most data manipulation in Pandas centers around performing transformations on DataFrames.

1
import pandas as pd
2

3
# Creating a Series
4
data_series = pd.Series([1, 3, 5, 7], index=["a", "b", "c", "d"])
5
print("Series:\n", data_series)
6

7
# Creating a DataFrame from a dictionary
8
data_dict = {
9
    "Name": ["Alice", "Bob", "Charlie"],
10
    "Age": [25, 30, 35],
11
    "City": ["New York", "Los Angeles", "Chicago"]
12
}
13
df = pd.DataFrame(data_dict)
14
print("\nDataFrame:\n", df)

Indexing, Filtering, and Slicing#

Just like Numpy, Pandas supports advanced indexing, boolean filtering, and slicing operations. You can use .loc[] and .iloc[] for label- and integer-based indexing, respectively.

1
# Label-based indexing using .loc
2
result_loc = df.loc[0, "Name"]  # "Alice"
3

4
# Integer-based indexing using .iloc
5
result_iloc = df.iloc[1, 2]  # "Los Angeles"
6

7
# Boolean filtering
8
adults = df[df["Age"] > 26]
9
print("\nAdults:\n", adults)

Data Cleaning Techniques#

Real-world datasets often have missing values, duplicates, or incorrect data types. Pandas provides multiple tools to tackle these issues efficiently.

1. Handling Missing Values

   • df.isnull() to detect null values
   • df.dropna() to remove rows or columns with missing values
   • df.fillna(value) to fill missing values with a specified value or strategy

2. Removing Duplicates

   • df.drop_duplicates() removes duplicate rows

3. Changing Data Types

   • df.astype(dtype) can convert columns to specific data types

Example:

1
# Handling missing values
2
import numpy as np
3

4
df_with_nans = pd.DataFrame({
5
    "A": [1, 2, np.nan, 4],
6
    "B": [np.nan, 2, 3, 4],
7
})
8
df_filled = df_with_nans.fillna(0)  # Replace NaN with 0
9

10
# Removing duplicates
11
df_with_dups = pd.DataFrame({
12
    "A": [1, 1, 2, 2],
13
    "B": [3, 3, 4, 4]
14
})
15
df_no_dups = df_with_dups.drop_duplicates()

Data Transformation and Aggregation#

Pandas excels at transforming and aggregating tabular data. Common techniques include:

1. Applying Functions to Columns

   1
   def compute_ratio(row):
   2
       return row["A"] / (row["B"] if row["B"] != 0 else 1)
   3
   
   4
   df["ratio"] = df.apply(compute_ratio, axis=1)
   

2. Vectorized Operations

   1
   df["C"] = df["A"] + df["B"]
   2
   df["D"] = df["A"] * df["B"]
   

3. GroupBy and Aggregate
   Grouping your data to compute aggregate statistics, such as sum, mean, or count per group.

   1
   grouped = df.groupby("City").agg({
   2
       "Age": "mean",
   3
       "Name": "count"
   4
   })
   5
   print(grouped)
   

Merging, Joining, and Concatenating DataFrames#

When working with multiple datasets, you’ll often need to combine them. Pandas provides a suite of functions:

1. pd.concat() to stack DataFrames vertically or horizontally.
2. pd.merge() for SQL-style merges (like JOIN).
3. df.join() for merging on the index.

1
df1 = pd.DataFrame({
2
    "Name": ["Alice", "Bob"],
3
    "Age": [25, 30]
4
})
5
df2 = pd.DataFrame({
6
    "Name": ["Charlie", "David"],
7
    "Age": [35, 40]
8
})
9

10
vertical_concat = pd.concat([df1, df2], ignore_index=True)

1
employee_df = pd.DataFrame({
2
    "EmployeeID": [1, 2, 3],
3
    "Name": ["Alice", "Bob", "Charlie"]
4
})
5
salary_df = pd.DataFrame({
6
    "EmployeeID": [1, 2, 3],
7
    "Salary": [70000, 80000, 90000]
8
})
9

10
merged_df = pd.merge(employee_df, salary_df, on="EmployeeID")
11
print("\nMerged DataFrame:\n", merged_df)

DateTime Indexing and Resampling#

Convert your time-related column to a DateTimeIndex and make it the DataFrame’s index for easier slicing and resampling.

1
date_range = pd.date_range(start="2021-01-01", periods=7, freq="D")
2
ts_df = pd.DataFrame({
3
    "date": date_range,
4
    "sales": [100, 120, 130, 115, 140, 150, 160]
5
})
6
ts_df["date"] = pd.to_datetime(ts_df["date"])
7
ts_df.set_index("date", inplace=True)
8

9
# Resample monthly and compute sum
10
monthly_sales = ts_df.resample("M").sum()
11
print(monthly_sales)

Shifting, Lagging, and Rolling Computations#

Time-based analyses often require lagged features or rolling averages.

1
ts_df["sales_lag1"] = ts_df["sales"].shift(1)
2
ts_df["rolling_mean"] = ts_df["sales"].rolling(window=3).mean()
3
print(ts_df)

Vectorization for Performance#

Vectorization leverages built-in operations that apply across entire arrays or Pandas Series/DataFrames without explicit Python loops. It’s usually faster due to optimized C-level implementations.

1
import numpy as np
2
import pandas as pd
3

4
large_df = pd.DataFrame({
5
    "A": np.random.rand(10_000_000),
6
    "B": np.random.rand(10_000_000)
7
})
8

9
# Vectorized operation
10
large_df["C"] = large_df["A"] + large_df["B"]  # Highly efficient

Apply, Map, and Vectorized Functions#

When you need custom logic that can’t be easily vectorized, Pandas supports apply() and map(). However, keep in mind these can be slower than fully vectorized operations.

1
def custom_function(x):
2
    return x * x - 2*x
3

4
# Using apply on a Series
5
large_df["A_custom"] = large_df["A"].apply(custom_function)
6

7
# Using map (only for Series)
8
large_df["A_map"] = large_df["A"].map(custom_function)

Memory Optimization#

For very large datasets, memory can quickly become a bottleneck. Several strategies can alleviate memory pressure:

1. Downcasting Numeric Types

   • Convert float64 to float32 or int64 to int32 if the precision range allows.

   1
   large_df["A"] = pd.to_numeric(large_df["A"], downcast="float")
   

2. Efficient Loading

   • Use dtype specifications when reading data from files (e.g., pd.read_csv(filename, dtype={"col": "float32"})).

3. Chunk Loading

   • If a file is too large, read in chunks and process each chunk, possibly appending results to an HDF5 store or a database.

Parallel Processing and Multiprocessing#

Python code can be parallelized with the multiprocessing module or external libraries like Dask or Ray for distributing operations across multiple cores or even multiple machines.

1
import multiprocessing
2

3
def process_data(chunk):
4
    # Perform data cleaning or computations here
5
    return chunk.mean()
6

7
if __name__ == "__main__":
8
    with multiprocessing.Pool(processes=4) as pool:
9
        results = pool.map(process_data, [chunk1, chunk2, chunk3, chunk4])
10
    overall_mean = sum(results) / len(results)
11
    print("Overall mean:", overall_mean)

1
import dask.dataframe as dd
2

3
ddf = dd.read_csv("very_large_file.csv")
4
# Perform transformations just like in Pandas
5
ddf = ddf[ddf["value"] > 0]
6
result = ddf.groupby("category")["value"].mean().compute()

Modularizing Your Code for Scalability#

Rather than writing one huge script, break your pipeline into modular, testable functions or classes. Each module should handle a specific stage (e.g., reading input, cleaning data, transforming data, storing output).

Directory structure example:

1
data_pipeline/
2
  |-- __init__.py
3
  |-- read_data.py
4
  |-- clean_data.py
5
  |-- transform_data.py
6
  |-- main.py

Each module can have a straightforward interface. For instance, read_data.py might define a function read_csv_file(path) that returns a Pandas DataFrame.

Maintaining Clean Code with Logging and Error Handling#

Use Python’s built-in logging library instead of overusing print statements. Logging can be configured with different levels (DEBUG, INFO, WARNING, ERROR, CRITICAL).

1
import logging
2

3
logging.basicConfig(level=logging.INFO)
4

5
def clean_data(df):
6
    logging.info("Starting data cleaning process.")
7
    # Perform cleaning steps
8
    return df

Clear error handling ensures that your pipeline fails gracefully and can be debugged easily:

1
def safe_division(a, b):
2
    try:
3
        return a / b
4
    except ZeroDivisionError as e:
5
        logging.error("Division by zero. Returning None.")
6
        return None

Version Control for Data and Notebooks#

In collaborative settings, it’s crucial to track changes in your code and data transformations:

• Use Git or another VCS for your Python scripts and notebooks.
• Store large or unique data files in a data version control system like DVC or Git LFS if needed.
• Maintain a changelog for important data transformations.