pandas ist ein Modul, das Datencontainer anbietet, ähnlich den DataFrames in R Die wich gsten Datencontainer sind: pandas.

Transkript

1 Lecture 9. Modules - pandas Matthias Bieg

2 pandas: Intro

3 Was ist pandas pandas ist ein Modul, das Datencontainer anbietet, ähnlich den DataFrames in R Die wich gsten Datencontainer sind: pandas.series, und pandas.dataframe Zusätzlich zu den Containern gibt es eine Reihe von Analysetools, sowie die Möglichkeit zum Plo en

4 pandas.series Series sind Listen Die Element können mit Iden fiern versehen werden, um einen intui ven Zugriff auf die Liste zu gewährleisten Die einzelnen Elemente können von unterschiedlichem Typ sein (integer, float, string, etc...) idx1 idx2... idxm-1 idxm d1 d2... dm-1 dm

5 pandas.dataframe DataFrames sind 2 dimensionale Matrizen Die Zeilen uns Spalten können mit Iden fiern versehen werden, um einen intui ven Zugriff auf einzelne Elemente zu gewährleisten Die einzelnen Elemente können von unterschiedlichem Typ sein (integer, float, string, etc...) Col1 Col2... Coln-1 Coln idx1 d1,1 d1,2... d1,n-1 d1,n idx2 d2,1 d2,2... d2,n-1 d2,n idxm-1 dm-1,1 dm-1,2... dm-1,n-1 dm-1,n idxm dm,1 dm,2... dm,n-1 dm,n

6 pandas: Object Creation

7 Creating a Series In [12]: import numpy as np import pandas as pnd mu = 181 sigma = 10. # List of random body heights male_heights = [ sigma * i + mu for i in np.random.randn(10) ] # List of Index Names index = [ "Individual"+str(i) for i in range(1, 11) ] # Create pandas.series object male_heights_series = pnd.series(male_heights, index=index) print male_heights_series Individual Individual Individual Individual Individual Individual Individual Individual Individual Individual dtype: float64

8 Create a DataFrame By passing a numpy array In [20]: import numpy as np import pandas as pnd # Create a list of lists wit random elements mu = 181 l = [ [ j * sigma + mu+offset for j in np.random.randn(5) ] for offset in range(8) ] # Defin column and row identifiers columns = [ "Individual"+str(i) for i in range(1, 6) ] index = [ "group"+str(j) for j in range(8) ] # Create DataFrame male_heights_dataframe = pnd.dataframe(np.array(l), columns=columns, index=index) print male_heights_dataframe Individual1 Individual2 Individual3 Individual4 Individual5 group group group group group group group group

9 By reading a tab-separated le Das pandas Modul bietet eine Methode zum Einlesen von tab-separierten Datein in pandas.dataframe Objekte: Die Methode pandas.read_csv In [49]: import pandas as pnd male_heights_filename = "data/male_heights.csv" # Read CSV file into pandas.dataframe male_heights_dataframe = pnd.read_csv(male_heights_filename, sep="\t", index_col=0) print male_heights_dataframe Individual1 Individual2 Individual3 Individual4 Individual5 group group group group group group group group

10 pandas: View Data

11 top and bottom of DataFrame In [32]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # View top of Dataframe print df.head(3) # View bottom of DataFrame print df.tail(2) Individual1 Individual2 Individual3 Individual4 Individual5 group group group Individual1 Individual2 Individual3 Individual4 Individual5 group group

12 Index, column, values In [34]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Inspect row identifiers print df.index # Inspect column identifiers print df.columns # Inspect values print df.values Index([u'group0', u'group1', u'group2', u'group3', u'group4', u'group5', u'group6', u'group7'], dtype='object') Index([u'Individual1', u'individual2', u'individual3', u'individual4', u'individual5'], dtype='object') [[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]]

13 Summary statistics of DataFrame rows In [41]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Get summary statistics of DataFrame columns print df.describe() Individual1 Individual2 Individual3 Individual4 Individual5 count mean std min % % % max

14 columns In [43]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Get summary statistics of DataFrame rows print df.t.describe() group0 group1 group2 group3 group4 group5 \ count mean std min % % % max group6 group7 count mean std min % % % max

15 Sort by column values In [48]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) print df.sort(columns="individual1") Individual1 Individual2 Individual3 Individual4 Individual5 group group group group group group group group /usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:5: FutureWarning: s ort(columns=...) is deprecated, use sort_values(by=...) """

16 Summary Methode DataFrame.head(x) DataFrame.tail(x) DataFrame.index DataFrame.columns DataFrame.values DataFrame.describe() DataFrame.sort(columns=[c1, c2,...]) Bedeutung Gibt die ersten x Zeilen zurück Gibt die letzten x Zeilen zurück Gibt die Zeilen Iden fier zurück Gibt die Spalten Iden fier zurück Gibt die Werte des DataFrames zurück Gibt Summarysta s ken für die einzelnen Spalten aus Gibt sor erten DataFrame zurück

17 Data Accession

18 Getting In [58]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Select a single column print df["individual2"] group group group group group group group group Name: Individual2, dtype: float64 In [56]: # Selecting via [], which slices the rows print df[0:3] Individual1 Individual2 Individual3 Individual4 Individual5 group group group

19 Selection by Label In [61]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Getting a row print df.loc["group1"] Individual Individual Individual Individual Individual Name: group1, dtype: float64 In [63]: # Getting selected columns of all rows print df.loc[:, ["Individual2", "Individual4"]] Individual2 Individual4 group group group group group group group group

20 In [65]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Slice rows by row ids print df.loc["group3": "group6", :] Individual1 Individual2 Individual3 Individual4 Individual5 group group group group In [66]: # Accessing a single cell print df.loc["group3", "Individual2"]

21 Selection by Position In [67]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Select row via position of the passed integers print df.iloc[3] Individual Individual Individual Individual Individual Name: group3, dtype: float64 In [70]: # Slice rows and columns print df.iloc[2:4, 1:3] Individual2 Individual3 group group

22 In [72]: # Select by integer positions of rows and columns print df.iloc[[0, 3, 4], [1, 2]] Individual2 Individual3 group group group In [74]: # Slice rows explicitly print df.iloc[0:3, :] Individual1 Individual2 Individual3 Individual4 Individual5 group group group In [75]: # Slice columns explicitly print df.iloc[:, 0:3] Individual1 Individual2 Individual3 group group group group group group group group

23 Boolean Indexing In [81]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) # Using a single columns values to select data print df[df["individual1"] > 180] Individual1 Individual2 Individual3 Individual4 Individual5 group group group group group In [82]: # Get all values fullfilling a criterion print df[df > 190] Individual1 Individual2 Individual3 Individual4 Individual5 group0 NaN NaN NaN NaN NaN group1 NaN NaN NaN NaN NaN group2 NaN NaN NaN NaN NaN group3 NaN NaN NaN NaN NaN group4 NaN NaN NaN group NaN NaN group NaN NaN group7 NaN NaN NaN

24 In [88]: # Using the isin() method for filtering df2 = df.copy().loc[:, "Individual1":"Individual4"] # Put new column with subgroup information to the end of DataFrame df2["subgroup"] = ["one", "two", "two", "three", "two", "one", "three", "one"] print df2 Individual1 Individual2 Individual3 Individual4 Subgroup group one group two group two group three group two group one group three group one In [89]: print df2[df2["subgroup"].isin(["one", "three"])] Individual1 Individual2 Individual3 Individual4 Subgroup group one group three group one group three group one

25 Setting Values In [102]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0).loc[:, "Individua l1":"individual3"] # Set a new column using a oandas.series heights_individual4 = [183., 182., 170., 172., 165., 185., 188., 190.] individual4_series = pnd.series(heights_individual4, index=["group"+str(i) for i in range(8)]) df["individual4"] = individual4_series print df Individual1 Individual2 Individual3 Individual4 group group group group group group group group

26 In [103]: # Setting values by label df.loc["group0", "Individual1"] = 5 # Setting values by position df.iloc[1, 1] = 6 print df Individual1 Individual2 Individual3 Individual4 group group group group group group group group In [104]: # Setting values by boolean indexing df[df > 180] = 10. print df Individual1 Individual2 Individual3 Individual4 group group group group group group group group

27 Summary Operator DataFrame[string] DataFrame[integer] DataFrame[integer1:integer2] DataFrame.loc[x, y] DataFrame.iloc[x, y] DataFrame[Bedingung] Bedeutung Gibt die Spalte mit dem Label string zurück Gibt die Spalte an Posi on integer zurück Gibt einen Slice der Spalten von Posi on integer1 bis integer2 zurück Gibt die Spalten mit Labels in y aus den Zeilen mit Labels aus x zurück. Hierbei sind x und y entweder skalare Strings, Listen von Strings, oder Slices von Strings Gibt die Spalten an den Posi onen in y aus den Zeilen an den Posi onen aus x zurück. Hierbei sind x und y entweder skalare Integers, Listen von Integers, oder Slices von Integers Gibt komple en DataFrame zurück, setzt jedoch Werte im DataFrame, die die Bedingung nicht erfüllen auf NaN

28 pandas.datafram.apply

29 Method Specs pandas.dataframe.apply(func, axis=0, args=()) Argument Default Bedeutung func - Funk on, die auf Spalten, bzw. Zeilen angewandt wird axis 0 Auf welche Achse soll di Funk on angewandt werden (0:=Zeilen, 1:=Spalten) args () Posi onale Argumente der Funk on

30 Apply to rows In [15]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) def mean(x): return (sum(x)/len(x)) # Apply a function to rows row_means = df.apply(mean, axis=0) print row_means Individual Individual Individual Individual Individual dtype: float64

31 Apply to columns In [18]: # Apply a function to columns col_means = df.apply(mean, axis=1) print col_means group group group group group group group group dtype: float64

32 Kombinieren von DataFrames

33 pandas.concat In [27]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) print df df_pieces = [df[:1], df[1:3], df[3:]] print pnd.concat(df_pieces) Individual1 Individual2 Individual3 Individual4 Individual5 group group group group group group group group Individual1 Individual2 Individual3 Individual4 Individual5 group group group group group group group group

34 pandas.merge In [42]: import pandas as pnd left = pnd.dataframe(np.array([["foo", 1], ["foo", 2]]), columns=["key", "lval"]) print left key lval 0 foo 1 1 foo 2 In [43]: right = pnd.dataframe(np.array([["foo", 4], ["foo", 5]]), columns=["key", "rval"]) print right key rval 0 foo 4 1 foo 5 In [44]: left_right_merged = pnd.merge(left, right, on="key") print left_right_merged key lval rval 0 foo foo foo foo 2 5

35 pandas.dataframe.append() In [45]: import pandas as pnd df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) print df Individual1 Individual2 Individual3 Individual4 Individual5 group group group group group group group group In [46]: s = df.iloc[7] print df.append(s, ignore_index=true) Individual1 Individual2 Individual3 Individual4 Individual

36 pandas.dataframe.plot()

37 Method Specs pandas.dataframe.plot(kind=plot_type) Die Plot Methode eines pandas DataFrames kann dazu benutzt werden um eine bes mmte Plotart für alle Spalten zu erzeugen. Das Schlüsselwortargument kind gibt hierbei an welche Art von Plot erzeugt werden soll.

38 Beispiel: Boxplots In [31]: import pandas as pnd %matplotlib inline df = pnd.read_csv("data/male_heights.csv", sep="\t", index_col=0) In [32]: Out[32]: # Plotte Boxplot für alle Gruppen (d.h. Zeilen) df.t.plot(kind="box") <matplotlib.axes._subplots.axessubplot at 0xad9c82cc>

39 Beispiel: Histogramme In [27]: import numpy as np import matplotlib.pyplot as plt df = pnd.dataframe(np.array([np.random.randn(1000), [ i + 10 for i in np.random.ran dn(1000) ]])) In [30]: fig = plt.figure(figsize=(10, 3)) # Plotte Histogram für die erste Zeile plt.subplot(1,2,1) df.iloc[0].t.plot(kind="hist") # Plotte Histogram für die zweite Zeile plt.subplot(1,2,2) df.iloc[1].t.plot(kind="hist") Out[30]: <matplotlib.axes._subplots.axessubplot at 0xadcfad2c>