SurajDataScientist/Exploratory_Data_Analysis_using_python_Libraries

1import streamlit as st2import numpy as np3 4st.title("Numpy")5 6#tab1, tab2, tab3, tab4, tab5, tab6, tab7 = st.tabs(["Overview", "Creating an array", "Accessing the elements", "array methods", "Arithmetic and statistical operations", "Sorting","Loading array and files."])7 8 9 10 11import streamlit as st12 13 14page = st.sidebar.radio("Choose topic", [15    "Introduction", 16    "Creating an array", 17    "Accessing the elements", 18    "Array methods", 19    "Arithmetic and statistical operations", 20    "Sorting",21    "Loading array and files"22])23 24 25if page == "Introduction":26    st.title("Introduction")27    28 29 30#with tab1:31    32    st.write("Numpy stands for Numerical Python. Numpy is one of the most widely used python library. It is a library which helps us create a array(data type which stores same kind of values) upto 64 dimensions. It is very helpful in data analysis as it is efficient, accurate and has wide range of application to EDA.")33    34    st.image("https://decodingdatascience.com/wp-content/uploads/2023/10/numpy1.png", width = 500)35    36    st.write("The main function of numpy is n-dimensional array, which means it helps us in mathematical calculations upto n(64) dimensions of array.")37    st.write("We have come across a word called array - let's explore it!") 38    39    st.markdown("""40    array is a datatype and it's properties are:-41    1. Can hold only same type of data42    2. It stores data sequentially43    3. It is mutable44    """)45    st.markdown("""46    Attributes of array:-47    1. ndim - It tells us the rank of array48    2. shape - It tells shape of the array and is in tuple49    3. dtype - It tells us the datatype of array50    4. size - It tells us the no. of elements present in a array.51    """)52    st.write("Numpy is not only limited to data analysis but it even used in Machine Learning and Deep Learning")53 54#with tab2:55 56 57 58 59elif page == "Creating an array":60    st.title("Creating an Array")61    st.code("np.array([1, 2, 3])", language="python")62    st.write("Use `np.array` to create basic arrays.")63    64 65    st.title("Creating an array using numpy module")66    st.write("We can create numpy array in two methods - A) Default method, B) using array - tools method")67    st.write("Here in an array we can identify the dimension of an array by counting the number of square bracket [] at starting and ending of an array.")68    st.write("If there is 1 [ ], it denotes 1 dim, if there are 2 [[]], it denotes 2 dim and so on till 64 dimension.")69    70    st.header("A. Default method")71    st.write("default tool - creating an array using default function called array. while creating array we can have 3 parameters they are np.array(object,ndmin,dtype)")72    73    st.write("(A)Default method - for example create a array and assign it to var1:-")74 75 76    st.write("This syntax creates 1-d array")77    78    with st.echo():79        var1 = np.array([1,2,3,4])80    st.code(str(var1), language="python")81    82    st.write("This syntax creates 2-d array")83        84    with st.echo():85        var2 = np.array([[11,22,33,44,55],[54,56,59,58,50],[99,98,97,95,90],[33,38,31,30,32],[65,62,97,68,60]])86    st.code(str(var2), language="python")    87    88    st.write("How to create an array with desired dimension")89    st.write("This will return you a 10 dimension array.")90    with st.echo():91        var3 = np.array([1,2,3,4,5,5],ndmin = 10)92    st.code(str(var3), language="python")93    94 95    96 97    st.header("(B) Array - tools method")98    st.write("numpy has provided other method of creating array other than default method where thes functions are not time consuming and are easy to use. like zeros, ones, full, arange etc.")99    100    st.write("1. **zeros(shape,dtype)** - it creates a n dimensional array where all the elements are 0")101    with st.echo():102        a1 = np.zeros((10,),dtype=int)103    st.code(str(a1), language="python")104    105    st.write("2. ones(shape,dtype) n-dim array where all the elements are 1")  106    with st.echo():107        a2 = np.ones((3,4,5))108    st.code(str(a2), language="python")109    st.write("3. full(shape,value,dtype) - n dim array where we can specify the elements")  110    with st.echo():111        a3 = np.full((3,4,5),99)112    st.code(str(a3), language="python")113    st.write("4. eye - It creates a array with all diagnal elements 1 and rest all elements 0 ")114    with st.echo():115        a4 = np.eye(10)116    st.code(str(a4), language="python")117    st.write("5. arange() - always returns 1dim array for for a values in range")    118    with st.echo():119        a5 = np.arange(1,10)120    st.code(str(a5), language="python")121    st.write("6. linspace() - returns only 1 d array, returns float. only equal distance b/w element but no control on how many elements")    122    with st.echo():123        a6 = np.linspace(start=1,stop=10)124    st.code(str(a6), language="python")125    st.write("With this below syntax we can control how many desired elements we want in output. ")126    with st.echo():127        a7 = np.linspace(start=1,stop=20,num=10)128    st.code(str(a7), language="python")129    st.write("7. diagnol - returns 2-d array. Customize the elements of diagnal in an array.")130    with st.echo():131        a8 = np.diag([1,2,3])132    st.code(str(a8), language="python")133    st.markdown("""134    135     8. methods in this random module-136    137    - random() - to generate random value b/w 0 & 1 - can generate infinite val which are float.138    random(size= )) in random by using size we can create values b/w 0 &1 in any dimension.139    140    - np.random.randint(start,end,size of dim)- returns only int values - here ending value is not included(exclusive).141    142    - random.choice([]) - choose random values from our choice of defined set143      random.choice([],size()) - in this we can even mention size where we get random choice values in specified dim.144      np.random.choice([],size=(),p[]) - Here p gives the weightage to the elements or % of weightage of randomness to value in our choice of values.145      """)    146    147    with st.echo():148        r1 = np.random.random()149    st.code(str(r1), language="python")150    with st.echo():151        r2 = np.random.randint(1,15)152    st.code(str(r2), language="python")    153    st.write("picks random values b/w 1 & 9 but in 2-dim array")    154    with st.echo():155        r3 = np.random.randint(1,10,size=(5,6),dtype=np.int8)156    st.code(str(r3), language="python")157    with st.echo():158        r4 = np.random.choice(["p1","p2","p3"],size=(3,4))159    st.code(str(r4), language="python")160    with st.echo():161        r5 = np.random.choice([1,2,3,4,5,6],size=(3,4),p=[0.1,0,0,0.1,0.1,0.7])   162    st.code(str(r5), language="python")163 164#with tab3:165 166 167 168elif page == "Accessing the elements":169    st.title("Accessing Elements")170    st.code("a[0]  # First element", language="python")171    st.write("Use indexing to access elements in the array.")172 173  174 175    st.header("How to access the elements from an array")176    st.write("We have 3 methods 1. index based slicing, 2. integer based indexing, 3. boolean based indexing")177    st.write("1. index based slicing - accessing elements through index positions and slicing technique with respect to the array size.")178    with st.echo():179        var1 = np.arange(1,21).reshape((4,5))180        var2 = np.arange(50,100).reshape((2,5,5))181    182    with st.echo():183        b1 = var1[0:3]184    st.code(str(b1), language="python")185    st.write("syntax for indexing 2d-array -  var_name[rowindex,columnindex]") 186    with st.echo():187        b2 = var2[1,3]188    st.code(str(b2), language="python")189    190    191    st.markdown("""192    In index based technique we have slicing based indexing and it's uses are as follows:- 193    - when we want to access a sequence of multiple elements194    - sub array and main both will be in same element195    - sub array will be a view of the original array means both are having the same memory location.196    197    """)198    199    st.write("slicing based technique to access elements from an array") 200    with st.echo():201        b3 = var2[1:3 , 0:4]202    st.code(str(b3), language="python")203    st.markdown("""204    2. integer based indexing205    - when we want to access a sequence or arbitary multiple elements206    - sub array will be always be in 1-dim array207    - sub array will be a copy of the original array means both are having the different meory location.208    209    """)    210    st.write("syntax is var_name[row index no. of element],[column index no. of element]. we can access multiple elements by giving its rspective row and column index.")211    with st.echo():212        b4 = var2[[0,0,0,0],[1,0,2,1],[1,3,4,2]]213    st.code(str(b4), language="python")214    215    216    st.write("3. boolean based technique. Here intially for eg. our default array var1 contains elements and let's apply a condition and store the result related to the condition perfomed on var1 as mask.")  217    st.write("Now this mask will return us the boolean values True and False according to the condition for each element present in the array.")218    st.write("In next line let us apply mask to var1 - var1[mask] and it will return the values of the array where conditions are satisfied i.e, where True is presnt in mask.")219    with st.echo():220        mask = var2 % 5 == 0221        b5 = var2[mask]222    st.code(str(b5), language="python")    223 224 225#with tab4:226 227 228elif page == "Array methods":229    st.title("Array Methods")230    st.code("a.reshape(2, 3)\na.flatten()", language="python")231    st.write("NumPy provides many useful methods to manipulate arrays.")  232 233    234    st.title("Important array methods")235 236    st.write("Array methods help us modify the array and use it according to our requirements.")237    st.write("some of the important methods we are going to discuss are reshape(), view(), copy(), flatten(), ravel(), repeat(), tile(), astype(), unique(), concatenation(), nan(), inf()")238 239    st.write("1. reshpe() - reshape is a function which helps us to modify the shape of the existing array.")240    st.write("this reshape is temporary change. if we want to change it permanently then we should assign it to our exsiting var or new var.")241    st.write("The following code will return us the a 3-d array with 5 - 2d array and where each 2d array conatins 1 row and 5 columns")242    with st.echo():243        s1=np.arange(1,51).reshape((5,1,10))244 245    st.code(str(s1), language="python")246 247 248    st.write("2. view() - changes will reflect in both new and old array ")249    250 251    st.write("3. copy() - change in new array won't change original array ")252    253    st.write("4. flatten() - flatten converts any n-dim array to 1-d array. Here the new 1-d array which is created is a copy but not a view. ")254    with st.echo():255        f1 = np.arange(1,51).reshape((5,1,10))256        f1.flatten()257    st.code(str(f1), language="python")258    259        260    st.write("5. ravel() - ravel is similar to flatten but it is the view which means the both original and the new array are changed.")261    with st.echo():262        f2=np.arange(1,51).reshape((5,1,10))263        f2.ravel()264 265    st.code(str(f2), language="python")266 267    st.write("6. repeat() - repeat is a function which repeats the elements present inside an array 'n' no. of times. it will always return 1-d array. Here the elements will be repeated 10 times as we can customize / contol the number of repeats by mentioning inside the function after var. ")268    with st.echo():269        r1=np.arange(1,4)270        np.repeat(r1,10)271         272    st.code(str(np.repeat(r1,10)), language = 'python')    273 274    st.write("7. tile() - tile is function which repeats the entire array - just like how tiles are fixed for flooring. it will copy/repeat the array in specified shape(row,col) For eg. in the below code the entire array will repeat 2 times row wise and 3 times column wise.") 275    with st.echo():276        c2=np.arange(1,17).reshape((4,4))277        np.tile(c2,(2,3))278 279    st.code(str(np.tile(c2,(2,3))),language="python" )   280 281 282 283    284    st.write("8. astype() - numpy method to convert data from one data type to another. For eg. in the below code by default it will create an array of int data type but here we are converting the data type into float.") 285    with st.echo():286        as1=np.arange(1,21,dtype=np.float32).reshape(4,5)287 288    st.code(str(as1), language="python")289 290    st.write("9. infinite - np.inf - data to represent infinite. We don't have any default method in python represent infinite, but as it a crucial factor for analysis numpy has a method to repesent infinite.")291 292    with st.echo():293        i1 = np.arange(1,21,dtype=np.float32).reshape(4,5)294        i1[1,1]=np.inf295 296    st.code(i1, language="python")    297        298 299    st.write("10. nan - to represent a empty cell we use nan")300 301    with st.echo():302        na1 = np.arange(1,21,dtype=np.float32).reshape(4,5)303        na1[1,1] = np.nan304    st.code(na1, language="python")305 306 307    st.write("11. Unique()- it helps us find out unique values and it's count in an n-d array")308 309    with st.echo():310        u1 = np.random.choice([1,2,3,4,5,6],size=(4,5))311        u2 = np.unique(u1,return_counts=True)312 313    st.code(np.unique(u2,return_counts=True), language="python")314 315 316    st.write("12. Concatenation - Concatenation helps us join 2 or more arrray's. Here this concatenation is done in 2 ways which are a) h-stack and v-stack b) concatenation")317    st.write("a) h-stack and v-stack does stack/joining array horizontally and vertically respectively. Here they both are applicable on 2-dim array only.")318 319    with st.echo():320        hv1 = np.arange(1,13).reshape((3,4))321        hv2 = np.arange(14,26).reshape((3,4))322        np.hstack([hv1,hv2])323        np.vstack([hv1,hv2])324 325    st.write("h-stack does horizontal concatenation and only condition is row should be equal in both array")326 327    st.code(np.hstack([hv1,hv2]), language = 'python')328 329    330    st.write("v-stack does vertical concatenation and only condition is column should be equal in both array")331    st.code(np.vstack([hv1,hv2]), language = 'python')332 333 334    st.write("b) concatenation - concatenation as a function is helpful for conatenating any n-dim array in any way i.e, in any depth, row or column by axis parameter.")335    st.write("syntax is - np.concatenate(([var1,var2]),axis=0).")336    st.write("Similarly for any n-dim array we just need to modify the axis to concatenate in any given array in any dimension.")337    with st.echo():338        c1 = np.arange(1,25).reshape((2,3,4))339        c2 = np.arange(26,50).reshape((2,3,4))340        np.concatenate(([c1,c2]),axis=0)341        np.concatenate(([c1,c2]),axis=0)342        np.concatenate(([c1,c2]),axis=0)343 344    st.write(" For eg. If there is a  two 3-d array as follows")345 346    st.code(c1, language = 'python')347    st.code(c2, language = 'python')348    349 350    st.write("here if axis = 0 then it does concatenation depth wise")351    st.code(np.concatenate(([c1,c2]),axis=0), language = 'python')352 353    st.write("if axis = 1 concatenation is done column wise")354    st.code(np.concatenate(([c1,c2]),axis=1), language = 'python')  355 356    st.write("if axis=2 concatenation is done row wise.")357    st.code(np.concatenate(([c1,c2]),axis=2), language = 'python')  358 359 360#with tab5:361 362 363 364elif page == "Arithmetic and statistical operations":365    st.title("Arithmetic & Stats")366    st.code("np.mean(a)\nnp.add(a, b)", language="python")367    st.write("You can perform element-wise arithmetic and stats.")368 369    370    st.header("Arithemetic functions on array using numpy methods.")371    st.write("We can perform all the basic arithmentic functions on any array using numpy methods as follows:-")372 373    with st.echo():374        a = np.array([10, 20, 30])375        b = np.array([2, 5, 3])376    st.code(f"""377    378    379    np.add(a, b)             = {np.add(a, b)}380    np.subtract(a, b)        = {np.subtract(a, b)}381    np.multiply(a, b)        = {np.multiply(a, b)}382    np.divide(a, b)          = {np.divide(a, b)}383    np.mod(a, b)             = {np.mod(a, b)}384    np.power(a, b)           = {np.power(a, b)}385    np.floor_divide(a, b)    = {np.floor_divide(a, b)}386    """, language="python")387 388    st.header("Statistical functions on array using numpy methods. ")389    st.write("We can perform basic statistics function on array like 1) mean, 2) median, 3) standard deaviation and 4) variance")390 391    with st.echo():392        stat1 = np.random.randint(10,50,size=(6,7))393 394 395    st.code(f"""396    397    np.mean(stat1)   = {np.mean(stat1)}398    np.median(stat1)   = {np.median(stat1)}399    np.variance(stat1)   = {np.var(stat1)}400    np.standard deviation(stat1)  = {np.std(stat1)}401    402    403    """)    404 405    406 407 408elif page == "Sorting":409    st.title("Sorting")410    st.write("Sort() is a funtion provided by numpy by which we can sort the elements presnt in an array. we have axis parameter inside sort function which acts as a tool to sort row wise, column wise and depth wise.")411    with st.echo():412        s1 = np.random.choice(range(1,20),size=(3,4))413        s2 = np.sort(s1)414 415    st.code(s1, language="python")  416    st.code(s2, language="python")417 418 419    st.write("Sorting in 3-d array")420 421    with st.echo():422        s3=np.random.choice(range(1,20),size=(3,2,3))423 424    st.code(s3, language="python")425 426    st.write("Here if we apply axis = 0 it does depth wise sorting.")427    with st.echo():428        np.sort(s3,axis=0)429    430    st.code(np.sort(s3,axis=0), language="python")431 432    st.write("Here if we apply axis = 1 it does column wise sorting.")433    with st.echo():434        np.sort(s3,axis=1)435    st.code(np.sort(s3,axis=1), language="python")436 437    st.write("Here if we apply axis = 2 it does row wise sorting.")438    with st.echo():439        np.sort(s3,axis=2)440    st.code(np.sort(s3,axis=2), language="python")441 442 443 444    st.header("argsort - (Its imporatnce and how to use it)")445 446    st.write("In the sort function which we discussed above it sorts the data but it does not sort/change the adjacent row/coloumns which are also equally important while maipulating the data. So due to this standlone sorting it corrupts the data and is not appropriate for further analysis.")447    st.write("So to solve this issue of sort() function we have a another function called argsort which sorts even the adjacent row and columns.")448    st.write("argsort - it gives the sorted index value of the element.")449    st.write("The output of argsort tells you where each element should be in a sorted array, not the position of a value within the original array.")450    st.write("For example, if argsort returns [3, 1, 5, 2, 0, 4] when applied to array [16, 14, 15, 10, 18, 14], it means the smallest element is at index 3, the second smallest is at index 1, and so on.")451    st.write("So finally argsort gives us the numbers which should be arranged in fashion as explained above to get sorted array. ")  452 453 454    with st.echo():455        ag1=np.random.choice(range(10,20),size=(3,4))456 457    st.code(ag1, language="python")458 459    st.write("so here in the following eg. the sort is not only done on single column as it corrupts data so, here the entire row is changed based on column sorting.")460    with st.echo():461        ag2 = ag1[np.argsort(ag1[:,1])]462 463    st.code(ag2, language="python")    464        465    st.write("so here the sort is not only done on single row as it corrupts data so, here the entire columns are changed based on row sorting.")466    with st.echo():467        ag3 = ag1[np.argsort(ag1[:,1])]468 469    st.code(ag3, language="python")470 471 472    st.write("argsort helps us sort by not corrupting data, but it becomes a tedious job when we want to use it multiple times.") 473    st.write("**lexical sort**")474    st.write("So here we have another method know as lexical sort - lexsort(). So here we can sort more than 2 columns or 2 rows at once i.e, write code in single cell which eventually saves time and reduces mistakes")475 476    with st.echo():477        np.lexsort((ag1[:,0],ag1[:,2]))478        ag4 = ag1[np.lexsort((ag1[:,0],ag1[:,2]))]479 480    st.code(ag4, language="python")    481 482 483 484elif page == "Loading array and files":485    st.title("Loading array and files")486    487    st.write("Array are structured data types and can hold useful data which is finally used for analysis.")488    st.write("Here this data stored in form of array has many benefits like computation upto 64 dimension and numpy as a library is one of the best choice for analysis in this 21st century data driven world.")489    st.write("So here as the data i.e, array is crucial, so we have a method to store the array and load it to any working environment we want.")490 491    st.write(".npy for single array and .npz for multiple array.")492 493    st.write("Here in the below Eg. we are saving array z1 into array1.npy file")494 495    with st.echo():496        z1 = np.array([1,2,3,4])497        with open("array1.npy","wb") as a:498            np.save(a,z1)499 500 501    with st.echo():502        np.load("array1.npy")503 504 505 506 507    st.header("genfromtext")508 509    st.write("Many a times during analysis we are not directly given array to perform operations on it.")510    st.write("Here mostly we have to collect the data and save it in a csv file or the csv file is given to us directly.")511    st.write("So here the next big task is to upload this csv file into our evironment so that we generate a desired array.")512    st.write("So here we have a method provided by numpy named genfromtext to generate array from text files.")513 514    515    st.write("syntax is data123 = np.genfromtxt(\"C:/Users/HP/Desktop/Data.csv\", delimiter=',', skip_header=1)")516 517    518    519    st.write("By default it will assume as text file, so inside the method we have to give the path to the file and its extension, followed by delimiter which as a partition between each element of row & column and finally if we want to skip the first row as it generally contains the index of the data then we can mention skip_header = 1")520    521 522        523 524 525 526        527 528 529    530    531num = st.number_input("Enter number 0 for home page or 1 for EDA", min_value = 0, max_value = 2, step = 1, format = "%d")532if st.button("Go"):533    if num == 1:534        st.switch_page("pages/EDA.py")535    elif num == 0:536        st.switch_page("home.py")537