Generating Mock Data

April 12, 2020
data pandas numpy
Estimated Reading Time: 2 minute(s)


You may find yourself in a situation where you’d like to generate mock data, like when writing a blog post series on taking a pipeline and model to production. Luckily numpy and pandas makes this task incredibly easy.

For my use case, I wanted to generate a Pandas DataFrame with one independent column temperature_celsius and one dependent column ice_cream_sales_euro. The goal was to make a data set where temperature_celsius would affect ice_cream_sales_euro.

On first attempt, working with the data showed to easy of a relation between the two, so we should generate some noise and add that in. This adds variation and unexplainable variance.

First we’ll import our required packages;

import datetime
import pandas as pd
import numpy as np

Its a good idea to set a random seed, so our code is reproducible.

np.random.seed = 42

Now we can create our index column, which will be the year, month and day and list our column names.

how_many_days = 365
today_last_year = datetime.datetime.now().date()
index = pd.date_range(today_last_year-datetime.timedelta(how_many_days), periods=365, freq='D')

columns = ["temperature_celsius", "ice_cream_sales_euro"]

Now we can create the data for our temperature_celsius column and the noise we mentioned above

temperature_celsius_data_x = np.arange(how_many_days)
temperature_celsius_data_delta = np.random.uniform(-1, 3, size=(how_many_days,))
temperature_celsius_data = (.1 * temperature_celsius_data_x) + temperature_celsius_data_delta

noise_data = np.random.normal(loc=25000, scale=10000, size=(how_many_days,))

And now we can take all the data and create our ice_cream_sales_euro_data column

ice_cream_sales_euro_data = (temperature_celsius_data * 1200) + noise_data

At this point we’re ready to create a single numpy array and output the data as a Pandas DataFrame and save that to a csv file.

data = np.array([temperature_celsius_data, ice_cream_sales_euro_data]).T

df = pd.DataFrame(data, index=index, columns=columns)
df.to_csv("ice_cream_shop.csv")

And the full script

import datetime
import pandas as pd
import numpy as np

np.random.seed = 42

how_many_days = 365
today_last_year = datetime.datetime.now().date()
index = pd.date_range(today_last_year-datetime.timedelta(how_many_days), periods=365, freq='D')

columns = ["temperature_celsius", "ice_cream_sales_euro"]

temperature_celsius_data_x = np.arange(how_many_days)
temperature_celsius_data_delta = np.random.uniform(-1, 3, size=(how_many_days,))
temperature_celsius_data = (.1 * temperature_celsius_data_x) + temperature_celsius_data_delta

noise_data = np.random.normal(loc=25000, scale=10000, size=(how_many_days,))

ice_cream_sales_euro_data = (temperature_celsius_data * 1200) + noise_data

data = np.array([temperature_celsius_data, ice_cream_sales_euro_data]).T

df = pd.DataFrame(data, index=index, columns=columns)
df.to_csv("ice_cream_shop.csv")

Python, Pandas and SQL (Oh my!)

July 11, 2019
pandas python sql databases
comments powered by Disqus