大众号：尤而小屋

作者：Peter
修改：Peter

大家好，我是Peter~

本文是根据机器学习的相关规矩办法对IC电子产品的数据发掘，首要内容包括：

  1. 数据预处理：针对数据去重、缺失值处理、时刻字段处理、用户年纪分段等
  2. 词云图制造：不同用户对不同品牌brand和品种category_code的偏好
  3. 相关规矩发掘：针对不同性别、不同品牌的相关信息发掘

本文关键词：电商、相关规矩、机器学习、词云图

数据基本信息

导入数据

In [1]:

importpandasaspd
importnumpyasnp
#显现所有列
#pd.set_option('display.max_columns',None)
#显现所有行
#pd.set_option('display.max_rows',None)
#设置value的显现长度为100，默认为50
#pd.set_option('max_colwidth',100)
importtime
importos
fromdatetimeimportdatetime
importmatplotlib.pyplotasplt
importseabornassns
%matplotlibinline
#设置中文编码和负号的正常显现
plt.rcParams['font.sans-serif']=['SimHei']
plt.rcParams['axes.unicode_minus']=False
importmissingnoasms
frompyecharts.globalsimportCurrentConfig,OnlineHostType
frompyechartsimportoptionsasopts#装备项
frompyecharts.chartsimportBar,Scatter,Pie,Line,Map,WordCloud,Grid,Page#各个图形的类
frompyecharts.commons.utilsimportJsCode
frompyecharts.globalsimportThemeType,SymbolType
importplotly.expressaspx
importplotly.graph_objectsasgo
fromplotly.subplotsimportmake_subplots#画子图
importjieba
fromsnownlpimportSnowNLP
fromsklearn.clusterimportKMeans
fromsklearn.preprocessingimportLabelEncoder
fromsklearn.preprocessingimportMinMaxScaler
importwarnings
warnings.filterwarnings("ignore")

In [2]:

#数据中存在中文，指定读取的编码格局
df=pd.read_csv("ic_sale.csv",
encoding="gb18030",#windows体系需求指定类型；mac不需求
converters={"order_id":str,"product_id":str,"category_id":str,"user_id":str}
)
df.head()

Out[2]:

基本信息

In [3]:

#1、数据shape
df.shape

Out[3]:

(564169,11)

In [4]:

#2、数据字段类型
df.dtypes

Out[4]:

event_timeobject
order_idobject
product_idobject
category_idobject
category_codeobject
brandobject
pricefloat64
user_idobject
ageint64
sexobject
localobject
dtype:object

In [5]:

#3、数据描述计算信息
df.describe()

Out[5]:

price age
count 564169.000000 564169.000000
mean 208.269324 33.184388
std 304.559875 10.122088
min 0.000000 16.000000
25% 23.130000 24.000000
50% 87.940000 33.000000
75% 277.750000 42.000000
max 18328.680000 50.000000

In [6]:

#4、总共多少个不同客户
df["user_id"].nunique()

Out[6]:

6908

数据预处理

数据去重处理

In [7]:

df.shape#去重前

Out[7]:

(564169,11)

In [8]:

df.drop_duplicates(ignore_index=True,inplace=True)

In [9]:

df.shape#去重后

Out[9]:

(561214,11)

特征信息

In [10]:

stats=[]
forcolindf.columns:
stats.append((col,
df[col].nunique(),
round(df[col].isnull().sum()*100/df.shape[0],3),
round(df[col].value_counts(normalize=True,dropna=False).values[0]*100,3),
df[col].dtype)
)

stats_df=pd.DataFrame(stats,
columns=['特征名','属性个数','缺失值占比','最大属性占比','特征类型'])
stats_df.sort_values('缺失值占比',ascending=False,ignore_index=True)

缺失值处理

In [11]:

df=df[df["price"]>0]

In [12]:

df.isnull().sum()

Out[12]:

event_time0
order_id0
product_id0
category_id0
category_code128662
brand27132
price0
user_id0
age0
sex0
local0
dtype:int64

In [13]:

ms.bar(df,color="red")#缺失值可视化
plt.show()

最后直接填充缺失值:missing

In [14]:

df.fillna("missing",inplace=True)#填充missing

时刻字段处理

In [15]:

df["event_time"].value_counts()

Out[15]:

1970-01-0100:33:40UTC1302
2020-04-0916:30:01UTC51
2020-04-0816:30:01UTC49
2020-04-0616:30:01UTC46
2020-04-0516:30:01UTC44
...
2020-07-2813:10:35UTC1
2020-07-2813:10:21UTC1
2020-07-2813:09:37UTC1
2020-07-2813:08:23UTC1
2020-08-1317:16:24UTC1
Name:event_time,Length:389813,dtype:int64

从上面的成果中看到：1970-01-01 00:33:40最多，其实便是时刻字段的缺失值

In [16]:

#去掉最后的UTC
df["event_time"]=df["event_time"].apply(lambdax:x[:19])
#时刻数据类型转化：字符类型---->指定时刻格局
df['event_time']=pd.to_datetime(df['event_time'],format="%Y-%m-%d%H:%M:%S")
#提取多个时刻相关字段
#df['month']=df['event_time'].dt.month
#df['day']=df['event_time'].dt.day
#df['dayofweek']=df['event_time'].dt.dayofweek
#df['hour']=df['event_time'].dt.hour

用户年纪分段

In [17]:

#不同性别下的年纪分布
fig=px.box(df,y=["age"],color="sex")
fig.show()

#不同年纪段人数计算
fig=plt.figure(figsize=(12,6))
sns.countplot(df["age"])
plt.title("CountsofDifferentAge")
plt.show()

针对年纪字段的分箱操作：

In [19]:

df["age"]=pd.cut(df["age"],bins=4,precision=0)
df["age"]#分段之后的age字段显现

Out[19]:

0(16.0,24.0]
1(33.0,42.0]
2(24.0,33.0]
3(16.0,24.0]
4(16.0,24.0]
...
561209(16.0,24.0]
561210(16.0,24.0]
561211(16.0,24.0]
561212(16.0,24.0]
561213(16.0,24.0]
Name:age,Length:561175,dtype:category
Categories(4,interval[float64,right]):[(16.0,24.0]<(24.0,33.0]<(33.0,42.0]<(42.0,50.0]]

不同地区用户的消费水平对比

In [22]:

fig=px.scatter(df[df["brand"]!="missing"],#除掉missing数据
#x="local",
y="price",
facet_col="age",
color="local",
size="price"
)
fig.show()

不同年纪段和性别的品牌偏好

In [23]:

age_brand=df.groupby(["age","sex","brand"]).size().reset_index().rename(columns={0:"number"})
age_brand.head()

Out[23]:

age sex brand number
0 (16.0, 24.0] 女 a-case 32
1 (16.0, 24.0] 女 acana 0
2 (16.0, 24.0] 女 accesstyle 3
3 (16.0, 24.0] 女 action 0
4 (16.0, 24.0] 女 activision 3

In [24]:

#完成排序功用-降序
age_brand=age_brand.sort_values(["age","number"],ascending=[True,False],ignore_index=True)
age_brand.head()

Out[24]:

age sex brand number
0 (16.0, 24.0] 男 samsung 11884
1 (16.0, 24.0] 女 samsung 11882
2 (16.0, 24.0] 男 apple 4561
3 (16.0, 24.0] 女 apple 4283
4 (16.0, 24.0] 男 missing 3354

In [25]:

#条件挑选
age_brand=age_brand.query("number>0&brand!='missing'")

In [26]:

fig=px.treemap(
age_brand,#传入数据
path=[px.Constant("all"),"age","sex","brand"],#传递数据路径
values="number"#数值显现
)
fig.update_traces(root_color="lightskyblue")
fig.update_layout(margin=dict(t=30,l=30,r=25,b=30))
fig.show()

品牌数量词云图

In [27]:

age_brand.head()

Out[27]:

age sex brand number
0 (16.0, 24.0] 男 samsung 11884
1 (16.0, 24.0] 女 samsung 11882
2 (16.0, 24.0] 男 apple 4561
3 (16.0, 24.0] 女 apple 4283
6 (16.0, 24.0] 男 ava 3317

In [28]:

brand_list=age_brand["brand"].value_counts().reset_index()
brand_list.columns=["word","number"]
brand_list.head(10)

Out[28]:

word number
0 samsung 8
1 darina 8
2 huion 8
3 aquapick 8
4 amigami 8
5 sjcam 8
6 rockstar 8
7 franke 8
8 bridgestone 8
9 tailg 8

In [29]:

information_zip=[tuple(z)forzinzip(brand_list["word"].tolist(),brand_list["number"].tolist())]
#绘图
c=(
WordCloud()
.add("",information_zip,word_size_range=[20,80],shape=SymbolType.DIAMOND)
.set_global_opts(title_opts=opts.TitleOpts(title="品牌词云图"))
)
c.render_notebook()

不同品牌的不同品种category_code

category_code处理

检查有多少种不同的category_code和对应的数量，运用value_counts()办法：

In [30]:

df["category_code"].value_counts()

Out[30]:

missing128662
electronics.smartphone101502
computers.notebook25917
appliances.kitchen.refrigerators20296
electronics.audio.headphone20049
...
kids.swing8
country_yard.watering5
sport.snowboard3
apparel.costume2
apparel.shoes2
Name:category_code,Length:124,dtype:int64

定论：除掉missing部分，最多的是electronics.smartphone，即：电子智能手机，其次便是电脑笔记本

In [31]:

fig=px.bar(df["category_code"].value_counts()[1:30])#前30个category_code
fig.show()

只选取需求的字段：

In [32]:

df=df[df["category_code"]!="missing"]#去除missing部分
df=df[["category_code","brand","age","sex","local"]]

将category_code字段进行切割处理：

In [33]:

df["category_code"]=df["category_code"].apply(lambdax:x.split(".")if"."inxelse[x])
df.head()

Out[33]:

category_code brand age sex local
0 [electronics, tablet] samsung (16.0, 24.0] 女 海南
1 [electronics, audio, headphone] huawei (33.0, 42.0] 女 北京
3 [furniture, kitchen, table] maestro (16.0, 24.0] 男 重庆
4 [electronics, smartphone] apple (16.0, 24.0] 男 北京
5 [appliances, kitchen, refrigerators] lg (16.0, 24.0] 男 北京

category_code词云图

In [34]:

data=df["category_code"].tolist()
data[:3]

Out[34]:

[['electronics','tablet'],
['electronics','audio','headphone'],
['furniture','kitchen','table']]

In [35]:

importitertools
#经过chain办法从可迭代目标中生成；展开成列表
sum_data=list(itertools.chain.from_iterable(data))
sum_data[:10]

Out[35]:

['electronics','tablet','electronics','audio','headphone','furniture','kitchen','table','electronics','smartphone']

In [36]:

category_code_number=pd.value_counts(sum_data).to_frame().reset_index()
category_code_number.columns=["category_code","number"]
category_code_number.head()

Out[36]:

category_code number
0 electronics 156709
1 appliances 150331
2 kitchen 107852
3 smartphone 101502
4 computers 76877

In [37]:

information_zip=[tuple(z)forzinzip(category_code_number["category_code"].tolist(),category_code_number["number"].tolist())]
#绘图
c=(
WordCloud()
.add("",information_zip,word_size_range=[20,80],shape=SymbolType.DIAMOND)
.set_global_opts(title_opts=opts.TitleOpts(title="商品品种词云图"))
)
c.render_notebook()

根据相关规矩建模

根据性别sex

查找频频项集-male

In [38]:

male=df[df["sex"]=="男"]
male.head()

Out[38]:

category_code brand age sex local
3 [furniture, kitchen, table] maestro (16.0, 24.0] 男 重庆
4 [electronics, smartphone] apple (16.0, 24.0] 男 北京
5 [appliances, kitchen, refrigerators] lg (16.0, 24.0] 男 北京
6 [appliances, personal, scales] polaris (24.0, 33.0] 男 广东
17 [appliances, kitchen, kettle] tefal (33.0, 42.0] 男 广东

In [39]:

importefficient_aprioriasea
male_list=male["category_code"].tolist()
#itemsets：频频项rules：相关规矩
itemsets,rules=ea.apriori(male_list,
min_support=0.005,
min_confidence=1
)
一个频频项

In [40]:

len(itemsets[1])

Out[40]:

60

In [41]:

itemsets[1]#一个频频项集

#字典的值value的降序排列
dict(sorted(itemsets[1].items(),key=lambdax:x[1],reverse=True))
二个频频项

In [43]:

len(itemsets[2])#总个数

Out[43]:

84

In [44]:

#两个频频项集
dict(sorted(itemsets[2].items(),key=lambdax:x[1],reverse=True))
三个频频项

In [45]:

len(itemsets[3])#总个数

Out[45]:

32

In [46]:

#三个频频项集
dict(sorted(itemsets[3].items(),key=lambdax:x[1],reverse=True))

Out[46]:

{('appliances','kitchen','refrigerators'):10209,
('audio','electronics','headphone'):10154,
('electronics','tv','video'):8876,
('appliances','environment','vacuum'):8069,
('appliances','kitchen','washer'):7235,
('appliances','kettle','kitchen'):6389,
('computers','mouse','peripherals'):6359,
('furniture','kitchen','table'):5626,
('appliances','hood','kitchen'):4487,
('appliances','blender','kitchen'):4439,
('appliances','kitchen','microwave'):3830,
('air_conditioner','appliances','environment'):3806,
('appliances','personal','scales'):3423,
('computers','network','router'):3318,
('components','computers','hdd'):2598,
('appliances','kitchen','meat_grinder'):2361,
('components','computers','cpu'):2055,
('appliances','kitchen','oven'):1958,
('appliances','environment','fan'):1952,
('computers','keyboard','peripherals'):1940,
('computers','peripherals','printer'):1802,
('appliances','environment','water_heater'):1753,
('computers','monitor','peripherals'):1733,
('components','computers','cooler'):1717,
('cabinet','furniture','living_room'):1550,
('chair','furniture','kitchen'):1513,
('appliances','hair_cutter','personal'):1388,
('air_heater','appliances','environment'):1341,
('appliances','dishwasher','kitchen'):1329,
('furniture','living_room','shelving'):1314,
('appliances','kitchen','mixer'):1288,
('construction','screw','tools'):1194}

查找频频项集-female

In [47]:

female=df[df["sex"]=="女"]
female.head()

Out[47]:

category_code brand age sex local
0 [electronics, tablet] samsung (16.0, 24.0] 女 海南
1 [electronics, audio, headphone] huawei (33.0, 42.0] 女 北京
7 [electronics, video, tv] samsung (16.0, 24.0] 女 北京
8 [computers, components, cpu] intel (42.0, 50.0] 女 浙江
10 [computers, notebook] asus (42.0, 50.0] 女 广东

In [48]:

importefficient_aprioriasea
female_list=male["category_code"].tolist()
#itemsets：频频项rules：相关规矩
itemsets,rules=ea.apriori(female_list,
min_support=0.005,
min_confidence=1
)
一个频频项

In [49]:

len(itemsets[1])#总个数

Out[49]:

60

In [50]:

#一个频频项集
dict(sorted(itemsets[1].items(),key=lambdax:x[1],reverse=True))
二个频频项

In [51]:

#两个频频项集
dict(sorted(itemsets[2].items(),key=lambdax:x[1],reverse=True))
三个频频项

In [52]:

#三个频频项集
dict(sorted(itemsets[3].items(),key=lambdax:x[1],reverse=True))

根据品牌brand

In [53]:

brand_category=df.groupby(["brand"])["category_code"].sum().reset_index()
brand_category

#去重功用-set
brand_category["category_code"]=brand_category["category_code"].apply(lambdax:list(set(x)))
brand_category

importefficient_aprioriasea
brand_list=brand_category["category_code"].tolist()
#itemsets：频频项rules：相关规矩
itemsets,rules=ea.apriori(
brand_list,
min_support=0.05,
min_confidence=1
)
#三个频频项集
dict(sorted(itemsets[3].items(),key=lambdax:x[1],reverse=True))

#两个频频项集
dict(sorted(itemsets[2].items(),key=lambdax:x[1],reverse=True))

#一个频频项集
dict(sorted(itemsets[1].items(),key=lambdax:x[1],reverse=True))

定论

  1. 从消费用户的年纪来看，平均在33岁，属于主力消费且有必定经济实力的人群；

  2. 从用户的产品偏好来看，用户首要喜爱：三星、苹果、ava(主营儿童产品，比方儿童头盔、摩托车)、tefal(特福，首要家电产品，比方蒸锅、不粘锅等)

  3. 从用户搜索的产品品种来看，用户更重视的是smartphone、kitchen、electronics；也就说：智能手机、厨房用品和电子产品是用户的重视点

  4. 从相关规矩发掘到的信息来看：

    • 男性/女性的相关产品信息可能是electronics与smartphone，appliances与kitchen，或者computers与notebook
    • 在同一个品牌中，appliances和kitchen；以及audio--->electronics--->headphone是首要相关产品