The two leading cryptocurrencies, Bitcoin and Ethereum, are using the same method to validate transactions and mint new crypto coins. Moreover, several multinational companies such as Amazon and Microsoft have welcomed [blockchain technology](https://www.almabetter.com/blogs/what-is-blockchain-technology) with open arms to make specific operations more efficient and profitable.

Typically, people know that blockchains consume massive amounts of energy. Still, the reasons why and how to reduce it will be critical to the back-and-forths of blockchain-based applications and Web3 in a time riddled with the climate crisis. Keep reading to find out how much energy is used by cryptocurrency mining. We will also cover the other environmental impacts of cryptocurrency.

## Environmental Impact of Blockchain Technology

As much as blockchain technology is a fascinating technology and is in line to be a key component of web 3.0, many detractors seem to keep their sight of a significant flaw: the energy-intensive nature of blockchain processes. Unfortunately, there is no direct way to calculate how much energy is used for Bitcoin mining. Still, the figure can be estimated from the network's hash rate and the consumption of commercially-available mining rigs. 

According to a few studies, Bitcoin mining is now consuming more energy than the entire nation of Argentina every year. For instance, one Bitcoin mining uses approximately 2000 kW of electricity. On the other hand, an average American household consumes roughly the same electricity in 70 days. In addition, according to a tabled theory, the emissions from Bitcoin mining could raise the earth’s temperature by two degrees. Considering the current climate situation of the world, this model will just worsen the conditions.

Here comes the most ironic aspect; there are several projects powered by blockchain technology aimed at controlling climate change. 

## Can mining use less energy?

Generally, large-scale cryptocurrency miners are located where energy is ample, cheap, and reliable. However, processing crypto transactions and minting new coins does not need to be energy-intensive. On the other hand, considering the millions of dollars already invested in the framework of the current mining process, it will be a challenge to move to a more energy-efficient system. 

Let’s look at some interesting ideas aimed at taming the energy-thirsty blockchain technology. 

- The proof-of-stake (PoS) method of validating cryptocurrency transactions and minting new coins is an alternative to traditional cryptocurrency mining. In addition, the technique does not demand extensive computing power. Instead, the authority to validate transactions and operate the crypto network is granted based on the amount of cryptocurrency that a validator has agreed not to trade or sell.

- In a meeting between Elon Musk and a few CEOs of crypto mining companies, an idea for Bitcoin Mining Council was introduced, tasked with driving energy transparency. 

- The prevalent Ethereum network aims to reduce its energy consumption by about 99.9% in 2022 by shifting from a Proof of Work to a Proof of Stake consensus algorithm, just like several other projects have done. Proof of Stake will demand miners to show how much of a currency they have locked up to boost their odds of being selected to be the validator. In theory, this will lower the amount of energy needed to validate each transaction.

- Another impactful yet challenging idea is to move the Bitcoin operations next to oil fields and wind farms, thereby using all their excess energy. 


**Conclusion**

As highlighted earlier, there are many projects powered by blockchain technology to control climate change. Admittedly, utilizing this technology is quite challenging; however, with the correct use, blockchain technology can help preserve our mother earth, which should be our priority in any case. With our [Full Stack Web Development with Web 3 program](https://www.almabetter.com/courses/full-stack-web-development-with-web3), you can contribute to making blockchain technology a positive influence on the environment. 


Blockchain is one of the most revolutionary and innovative technologies of recent years, and it is not anywhere close to its full potential just yet. Several folks worldwide are intrigued by this technology. However, few critics are concerned about a significant drawback - cryptocurrency mining is highly energy-intensive.

169 images (13)-min.png

Impact of Blockchain on the Environment

The logistics industry is responsible for the movement of goods and materials around the world. It is a vital part of the global economy, and plays a key role in ensuring the efficiency and cost-effectiveness of supply chains of businesses of all sizes. Logistics companies provide a range of services, including transportation, warehousing, and order fulfillment. 
 
The industry is growing rapidly, due to the increasing globalization of businesses and the rise of e-commerce. The growth of the logistics industry is creating new opportunities for logistics companies, as well as for the businesses that they serve.
 
Logistics companies use a variety of information technology systems to manage their ecosystem. These include transportation management systems, warehousing management systems, and order fulfillment systems. Logistics companies also use a variety of communication systems, such as email, fax, and phone, to coordinate the movement of goods and materials.

## Data science in logistics


There is a growing trend of using [data science in businesses](https://www.almabetter.com/blogs/importance-of-data-science-in-business) across several industries. This includes the logistics business, where data science is leveraged to improve efficiency and optimize routes. This can involve using data to track trends in customer behavior, map out efficient routes, and predict demand. By using data science, logistics firms can improve their bottom line and better serve their customers.

## What problems are logistics firms currently facing and how can they be solved with Data Science?


As in any other industry, there are several challenges that come along the way in the logistics industry. Some of these challenges include:

- **Inefficient route planning**: This is a commonly occurring problem when it comes to the logistics business. Data science can be used to solve this issue by creating algorithms which can plan the most efficient and cost-effective routes. 

- **Lack of visibility**: Another common challenge encountered by logistics companies is the lack of visibility into their operations. This can be avoided using data science by creating dashboards and visualizations which can help managers get a clear insight into what goes on in their operations. 

- **Poor forecasting**: Another problem which can be solved using data science is poor forecasting. This is often caused by inaccurate data which can be cleaned and processed using data science techniques. This will allow for more accurate forecasting. 

- **Inaccurate tracking**: Data science can be used to create more accurate tracking systems. This will allow managers to track where their shipments are at all times. 

- **Inefficient customer service**: This is often caused by a lack of data. Data science can be used to collect actionable information about customer service interactions. This will allow for the creation of better customer service processes.



## What do Data Scientists do in logistics companies?


Data Scientists in logistics companies work on a variety of tasks related to improving the efficiency of the company's operations. This can include working on algorithms to optimize routes, developing models to predict demand, and analyzing data to identify trends. In addition to their work on specific projects, data scientists also often collaborate with other departments within the company to ensure that data is being used effectively across the organization.


## Which Logistics companies are recruiting Data Scientists in India

There are many logistics companies recruiting data scientists in India. Some of these companies are: 
- Amazon 
- Flipkart 
- Snapdeal 
- Paytm 
- Ola 
- Uber 
- MakeMyTrip 
- GoIbibo 
- Yatra 
- Cleartrip




**Conclusion**

The importance of data science in logistics is evident from the numerous advantages that it provides to the industry. Data science helps optimize operations and reduce costs within this sector. In addition, data science can help logistics companies improve their customer service and better understand customers’ needs. 

As the world of logistics becomes increasingly complex, data science will play an important role in helping logistics companies meet the challenges of the future.

To become a skilled Data Science Professional, enroll for [AlmaBetter’s Full Stack Data Science program](https://www.almabetter.com/courses/full-stack-data-science) today!


Logistics can be defined as the flow management of resources between the point of origin and point of consumption in order to meet customers’ needs. It involves the planning, coordination, and execution of transportation and storage of goods. Logistics is a critical component of supply chain management and is responsible for ensuring that materials and products are delivered to the correct destination, in the right quantity, and at the right time.

Untitled design.jpg

How can Data Science help logistics?

We strive to provide you with the most detailed and updated Full Stack Web Development program and diversify your skillset. Along with Polygon, we have collaborated with multiple hiring partners to provide high-paying jobs to students. 

This blog will discuss each module to understand the curriculum better. So, Let’s get started. 

## Module 1- Programming Web with Javascripts

In the first module, you will learn the fundamentals of programming and Javascript, which is a must for web developers. Next, you will be trained with essential developer tools necessary for creating, testing, and debugging a web application or software. You will also learn functions and object-oriented programming, two different yet crucial methods for storing and manipulating data. Lastly, you will learn data structures and algorithms.

## Module 2- Frontend Development

In the second module, you will start with understanding the web and the use and importance of Javascript on the browser. Next, you will be introduced to the React ecosystem. React is a Javascript framework that abstracts the view layer of an application. You will also learn the advanced concepts of React and its ecosystem. The second module also covers UI and UX, which are very important when developing web applications. We will make sure you master these skills before moving on to the next step. Lastly, you will build responsive web applications by utilizing all the prior learnings. 

## Module 3- Backend Development

In the third module, we will start with understanding Databases, Node.js, and Express.js. Node.js is an open-source server environment that allows you to run Javascript on a server. Express is a Node.js web application framework that provides rich features for web and mobile applications. Next, you will explore Microservices and MongoDB, which helps develop scalable applications with evolving data schemas. Finally, you will learn to build an end-to-end application using all the prior learnings. 

## Module 4- System Design and Architecture

In the fourth module, we will start with System Design and OS and Networking fundamentals. Next, we will explore Cloud services. Cloud Computing is a virtual platform that allows you to store and access your data over the Internet without any limitations. You will also learn advanced Data Structures and Algorithm concepts and their implementation in different programming languages. Lastly, we will dive into Graph Theory and its uses in various applications. 

## Module 5- Web 3.0 Blockchain Technology

In the fifth and final module of the curriculum, we will explore the future of the Internet, Web 3.0, and [Blockchain Technology](https://www.almabetter.com/blogs/what-is-blockchain-technology). Thanks to our partner, Polygon, you will be able to grasp and understand the advanced concepts efficiently. We will start with the fundamentals of Blockchain Technology and Ethereum and Smart Contracts. Next, we will learn Solidity,  an object-oriented, high-level language for implementing smart contracts. You will also learn to build full-stack dApps using React.js that will run on Ethereum and several other Layer 2 platforms. We will also explore Blockchain Cryptography and  Ethereum Virtual Machine (EVM). To wrap up the curriculum, we will learn about Decentralized Finance (DeFi) and finally work on building an end-to-end dApp.

## Why is AlmaBetter's Full Stack Web development with Web3 in collaboration with Polygon the best program available?


Do you want to be a part of the Internet revolution? Do you aspire to become a Web2+ Web3 developer?
Then, check out AlmaBetter’s latest program - [Full Stack Web Development with Web3 in collaboration with Polygon](https://www.almabetter.com/courses/full-stack-web-development-with-web3). Join India’s most prominent tech community of working professionals and students and 500+ hiring and referral partners and become a job-ready web developer in 28 weeks. 

## Here are the unique features of the program -

- Guaranteed placement with salary ranging from Rs 6 - 25 LPA
- Pay-after-placement model
- One time payment, lifetime access 
- One-on-one mentorship sessions and career guidance


AlmaBetter, in collaboration with Polygon, has created one of the finest Full Stack Web Development with Web3 programs. The program is perfect for folks looking to become job-ready Web2 + Web3 developers in 28 weeks. The curriculum is designed to perfection with quality content, robust learning, and an industry-relevant course structure covering everything from programming basics to Web 3.0 blockchain technology. 

Untitled (1080 × 1080 px) (1080 × 1080 px) (1).jpg

Full Stack Web Development with Web 3.0 curriculum

Web 3.0 allows websites and applications to process data in an intelligent human-centric method. The digital ecosystem is ready to move to a better, faster, and more secure environment with Web 3.0. Web 2.0 was helmed by three core elements of innovation - mobile, social, and cloud. However, it is being said that Web 2.0 will continue to reap the rewards, even if Web3 eventually replaces it. Meanwhile, as per Tim Berners-Lee, the Semantic Web is set to construct an absolute disruption worldwide, including India.

Web 3.0 in India will change the existing work processes efficiently and effectively.  By utilizing the semantic Web’s capabilities, India can extensively enhance its revenue return. Almost everyone is excited about the next generation of the Internet. Let’s dig deeper into whether India is ready to step into the future of the Internet.

## What is Web 3.0?
Web 3.0 is the next evolutionary step for the Internet. It is a decentralized online backdrop built on the blockchain at its most basic level. It will allow users to own platforms and apps developed on Web3 rather than a central authority. Users earn their privilege by building and maintaining the services. To know more about Web 3.0, check out our [blog](https://www.almabetter.com/blogs/a-beginner-s-guide-to-web-3-0).

## Is India ready for Web 3.0?
As humanity advances and technological breakthroughs occur, these shifts must showcase in our laws. Web 3.0 is deemed to be a game-changer in India in almost every aspect and sector in the near future. In simple words, Web 2.0 is all about search engines, while the semantic web is about finding engines. Since there has been an ongoing issue of the digital divide in India for a long time, citizens are concerned about Web 3.0 in India.

**The Government of India**

The IT Act originated amid the historic dot-com era. Its objective was to facilitate India’s move online by granting recognition to digital documents. One of its most striking features was the legal recognition of digital signatures. It was digital India’s first incursion into law-making for the virtual space. While the IT Act has adequate provisions to tackle the problems posed by Web 1.0 and Web 2.0 (such as hacking, cyber fishing, etc.), it doesn’t have a thorough strategy to deal with cybercrimes or to protect a user’s privacy. Also, many of its provisions are redundant. In simple words, the IT Act currently functions as a manual for radio in the age of augmented reality. 

With the next phase of the Internet fast approaching, it will be crucial for the government to ensure that they leave sufficient capacity to stimulate growth and intended usage while legislating on such technologies. The interests of users, developers, and shareholders have to be considered. 

**Entrepreneurs of India and Web 3.0**

Nascent entrepreneurs are excited about leveraging Web 3.0 in India to grab the semantic web's potential prospects. Several venture capitalists are planning to invest millions of dollars in Indian companies ready to adapt, leverage, and invest in semantic web technologies to promote it in India. Venture capitalists are already convinced that India is capable of reaping the benefits of Web 3.0 to facilitate productivity and revenue with more security and transparency. With the right plan of action, policies, and regulations, the lives of Indian citizens can be enhanced with Web 3.0  while creating more job opportunities.

India has one of the most influential communities of start-ups and innovators worldwide. Despite being a developing nation, India delivers one of the fastest-growing Internet consumer markets globally. Web 3.0 in India holds the possibility to make the country a comprehensive digital society with the benefits of digital technologies.

**Conclusion**

Web 3.0 is ready to recognize and rectify multiple inefficiencies of Web 2.0.  The state government of Telangana has accredited Web 2.0 and Web 3.0 blockchain start-ups known as India Blockchain Accelerator Program. The primary goal is to promote early-stage Web 2.0 and Web 3.0 start-ups to assist them in solving multiple real-life challenges. Investors consider that India is set to provide many opportunities to lead the rest of the world in successfully developing a blockchain and cryptocurrency ecosystem with Web 3.0 in India.


The world is ready to embrace the next Internet revolution, Web 3.0. Tech terms such as cryptocurrency, metaverse, and non-fungible tokens (NFTs) are making headlines almost every day. Along with the world, India is ready to enjoy and leverage the new decentralized Internet. 

India web 3 16_9.png

Is India ready for Web 3.0?

Data science is important for businesses because it can help them make better decisions by providing insights that are not readily available from traditional data sources. For example, data science can help businesses understand customer behavior, optimize marketing campaigns, and predict future trends. 

In the past, business decisions were often based on intuition or experience. However, with the advent of data science, businesses now have the ability to make informed decisions based on data. This has led to a new era of data-driven decision making, where businesses are able to use data to their advantage. 

## Common applications of data science in business

- **Improving customer service**: Data science can help businesses understand their customers better and provide them with improved service. 

- **Enhancing marketing**: Data science can help businesses target their marketing efforts more effectively and reach more potential customers. 

- **Optimizing operations**: Data science can help businesses streamline their operations and make them more efficient. 

- **Improving financial decision-making**: Data science can help businesses make better financial decisions by analyzing data and identifying trends. 


There are many other potential applications of data science in businesses. These are just a few of the most common ways that businesses are using data science to improve their operations. 

Data science is still a relatively new field, and there are many businesses that are not using it to its full potential. Data science can be a powerful tool for businesses, but it is important to use it correctly. Businesses should partner with experienced data scientists to ensure that they are using data science in the most effective way possible. 


## What Does a Data Scientist Do?

A data scientist is a professional who is responsible for analyzing and interpreting data. They use their skills to identify trends and patterns, and to make predictions about future events. Data scientists typically have a strong background in mathematics and statistics.

Data scientists typically work in a variety of industries, including healthcare, finance, retail, and manufacturing. They may also work in the government or in academia. 

## What are the responsibilities of a data scientist?

The responsibilities of a data scientist vary depending on their employer and the specific project they are working on. However, there are some common duties that all data scientists typically perform. These duties include: 

- **Collecting and cleaning data**: Data scientists typically begin a project by collecting data from a variety of sources. They then clean the data to remove any errors or inconsistencies. 

- **Analyzing data**: Once the data has been collected and cleaned, data scientists analyze it to identify trends and patterns. They use their findings to develop models and algorithms that can be used to make predictions about future events. 

- **Interpreting data**: Data scientists also interpret the data they have analyzed. They use their findings to provide insights and recommendations to their employer. 


## What skills does a data scientist need?

Data scientists need a strong background in mathematics and statistics. They also need to be skilled in programming languages such as R and Python. In addition, data scientists need to be able to effectively communicate their findings to others.

## What Does a Data Scientist Do in Business?

A data scientist in business is responsible for analyzing data to help improve business operations. This may include developing new methods for data collection and analysis, designing and running experiments, and creating and using predictive models. Data scientists may also be responsible for communicating their findings to business leaders and decision-makers.

## 5 Ways a Data Scientist Can Add Value to Business

- Helping businesses better understand their customers;
- Developing new and innovative ways to use data to improve marketing efforts;
- Creating models and algorithms to improve operational efficiency;
- Building custom applications to support business decision making; 
- Conducting research to identify new business opportunities.


**Conclusion**

There is no doubt that data science is one of the most important and valuable tools that businesses have at their disposal today. By harnessing the power of data, businesses can gain insights that would otherwise be hidden, and make better decisions that can lead to increased profits and improved efficiency.

However, data science is not a magic bullet, and it is important to remember that it is only one tool in the business arsenal. To be truly successful, businesses need to combine data science with other tools and techniques, such as market research, to get a complete picture of their customers and their needs.




Data science is a branch of computer science that deals with the extraction of knowledge from data. It is a relatively new field that has emerged from the intersection of statistics, machine learning, and computer science. 

business-man-expanding-futuristic-virtual-screen-modern-tablet.jpg

Importance of Data Science in Business

Here are the answers to the "Find The Hidden Words" challenge!

solutions puzzle.png

Find The Hidden Words - Solution

Web 3.0 is a term that is making headlines worldwide. You’ve probably heard or read about it, especially if you are interested in blockchain, cryptocurrency, and the future of the internet.  What exactly is Web 3.0? Is there a Web 2.0 and Web 1.0? And how will the Web 3.0 Internet transform social media and provide a possibility for decentralized ownership. 

In this blog, we will talk about the history, the present, and the future of the Internet.

## Web Numbers

Imagine the Internet and the crucial landmarks of how people have interacted with it since the advent of the World Wide Web. This journey unofficially diverges into three generations known as Web 1.0, Web 2.0, and Web 3.0.

The Web 1.0 era was mainly one-way media consumption where users would browse the internet to consume data and then leave as there was no easy way to share, react or engage.

Web 2.0 and the evolution of social networks like Facebook, Twitter, and YouTube changed this dynamic completely, steering towards an era of two-way communication, with the added prospect of user-generated content, which could be quickly distributed. 

The significant difference between Web 1.0, 2.0, and 3.0 is the level of involvement between users. With Web 1.0, users were restricted to viewing static pages.0. On the other hand, Web 2.0 allowed users to create and broadcast their content. Web 3.0 allows users to engage instantly by enforcing smart contracts. 


## What is Web 3.0?

Web 3.0 is the next evolutionary step for the Internet. It is a decentralized online backdrop built on the blockchain at its most basic level. It will allow users to own platforms and apps developed on Web3 rather than a central authority. Users earn their privilege by building and maintaining the services. 

Data Privacy is the key issue with the current state of the Internet. Social media networks have been collecting and exploiting user data for advertising revenue. We are already aware of the net worth of social media companies which is proof of how much they have earned through selling personal information. Web 3.0 will change the Internet for the better by introducing decentralization. 

For example, content creators on decentralized social network platforms can retain greater control of their digital essence while being rewarded for their valuable contribution to the platform.

## Blockchain, Cryptocurrencies, and other Web3 trends

The perfect example of Web 3.0 is a cryptocurrency, a decentralized currency that is not governed or controlled by a single government. Cryptocurrencies are managed by the people using and minting them in a simultaneous process.

The next significant element of Web3 is blockchain, the backbone of cryptocurrency. Blockchain will also be used to verify and validate ownership of digital assets soon. Moreover, just like NFTs (non-fungible tokens), other blockchain-based technologies are expected to develop where only verified content owners will gain monetary advantages from the assets. 

Web 3.0 is also expected to transform how monetization and marketing work on a larger scale, with Artificial Intelligence (AI) powering tech like Natural Language Processing (NLP). This technology makes interacting with computers more like interacting with fellow humans.




## Advantages of Web 3.0

-New Beginnings- Many experts believe that Web 3.0 will begin from scratch. Rather than using HTML as the primary language, it will rely on a new unidentified language. The experts believe that starting from scratch might be easier than improving the current web. 

-Data Ownership- End users will regain total control of their data while profiting from encryption security. On a case-by-case and permission basis, information can then be shared. 

-No Single Point of Control- Because mediators are eliminated, blockchains such as Ethereum provide a trustworthy platform with unbreakable rules and completely encrypted data.


## AlmaBetter’s Full Stack Web Development with Web3 in collaboration with Polygon

Do you want to be a part of the Internet revolution? Do you aspire to become a Web2+ Web3 developer?

Then, check out AlmaBetter’s latest program- Full Stack Web Development with Web3 in collaboration with Polygon. Join India’s largest tech community of working professionals and students and 500+ hiring and referral partners and become a job-ready web developer in 28 weeks. 


**Here are the unique features of the program -**

- Guaranteed placement with salary ranging from Rs 6 - 25 LPA
- Pay-after-placement model
- One time payment, lifetime access 
- One-on-one mentorship sessions and career guidance



## Conclusion

There’s no telling when the shift to Web 3.0 will be observed. It could be swift or a gradual one that perhaps we’re already experiencing right now. Either way, Web 3.0 is anticipated to alter how the web works, how we interact with it, and what we use it for in the future.


Humans are social creatures, always looking for ways to connect. Since the early days of the dial-up Internet, people have embraced new technologies to connect and communicate. Then came the next generation called Web 2.0, where users could create and share content easily on multiple platforms. The world is now ready to move to a better, faster, and safer Internet with Web 3.0.

businesswoman-using-digital-network-connection-interface-3d-rendering - Copy.jpg

A Beginner's Guide to Web 3.0

This blog will walk you through the intricate details of the press release and the plans outlined therein. 

Founded in September 2020, AlmaBetter is an Ed-AI platform offering Data Science and Web Development courses to students through a risk-free educational approach. With over 50,000 learners already enrolled, AlmaBetter is set to widen its horizons through the Web Development course. AlmaBetter aims to produce 30,000 Web Developers by the end of March 2023. While this is a bold promise, it is also a crucial step toward ensuring the gap between the skilled workforce and available positions is adequately bridged. To give more context to our readers, a study conducted by Belong found that a nominal 0.25% of the Indian software workforce is blockchain-ready. This data speaks volumes about the growing need for efficient Web3 developers today. 

With the robust, top-in-class training offered by AlmaBetter, the firm sees high potential in upskilling young talent and giving them a competitive edge in the job market. AlmaBetter is also host to over 500 hiring and referral partners spread across several Indian cities and therefore, AlmaBetter graduates are hired by top names in the industry. Furthermore, over 70% of AlmaBetter students get placed even before the completion of AlmaBetter’s exclusive placement program. This can be attributed to the excellently engineered and intricately structured course content and industry-relevant curriculum. The course content has been carefully curated in association with Polygon, emphasizing key Web3 concepts that help make students job-ready and at the top of their game. 

With a Net Promoter Score (NPS) of 87, AlmaBetter plans to bring more such offerings to the fore, helping create a dynamic educational ecosystem for young talents to thrive in. In addition, the partnership between AlmaBetter and Polygon will enable a cohesive environment for students to learn in, bringing together the best of training and technology under one roof. 

Find the links to the news story here:

[![348209_custom_site_themes_id_B6OzrqiSLHO4mVpAyRQ2_yourstory.png](https://almablog-media.s3.ap-south-1.amazonaws.com/348209_custom_site_themes_id_B6_Ozrqi_SLHO_4m_Vp_Ay_RQ_2_yourstory_2994b10ddf.png)](https://yourstory.com/the-decrypting-story/ethereum-scaling-project-polygon-almabetter-blockchain-education-web3/amp)


[![financial-express-logo.jpg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/financial_express_logo_jpg_fc5bf14c41.png)](https://www.financialexpress.com/education-2/polygon-and-almabetter-join-hands-to-impart-blockchain-and-web3-education/2491758/)

[![Inc42](https://almablog-media.s3.ap-south-1.amazonaws.com/Inc42_cce72a80ca.png)](https://inc42.com/buzz/polygon-and-almabetter-join-hands-to-address-talent-crunch-in-the-indian-web3-0-space/)

[![tc_fb_share_image.jpg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/tc_fb_share_image_jpg_bdbe38e8d1.png)](https://www.techcircle.in/2022/04/14/polygon-and-almabetter-collaborate-for-web3-education)

[![download (3).png](https://almablog-media.s3.ap-south-1.amazonaws.com/download_3_0141c168b2.png)](https://indiaeducationdiary.in/polygon-and-almabetter-partner-to-impart-web3-education/)

[![download (4).png](https://almablog-media.s3.ap-south-1.amazonaws.com/download_4_3717572bef.png)](https://www.medianews4u.com/polygon-and-almabetter-partner-to-impart-web3-education/)

[![1547537374888.jpg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1547537374888_jpg_429f4d188f.png)](https://www.highereducationdigest.com/polygon-and-almabetter-partner-to-impart-web3-education/)

[![0IpWgrRG_400x400.jpg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Ip_Wgr_RG_400x400_jpg_6a152b177d.png)](https://newslanes.com/2022/04/14/polygon-and-almabetter-partner-to-impart-web3-education/)

[![Coincrunch_logo_640_416.png](https://almablog-media.s3.ap-south-1.amazonaws.com/Coincrunch_logo_640_416_f140d3f489.png)](https://coincrunch.in/2022/04/14/polygon-partners-with-almabetter-to-address-web3-talent-crunch-in-india/)

Recently, several major media houses in the country published about the new Web Development with Web3 course launched by AlmaBetter in association with Polygon. Polygon is a leading decentralized Ethereum scaling platform. This collaboration aims to produce more than 30,000 Web3 developers by the end of March 2023, taking the total count of AlmaBetter students to 1 lakh. The articles contour the dire need to address the acute shortage of skilled developers in the crypto domain. 

collaboration 2 (1).png

AlmaBetter New Course Launch: Web Development with Web3 in Collaboration with Polygon

Along with mentioning that there is a gap between the demand and supply of skilled and job-ready workforce with relevant AI skills, the article also mentions how AlmaBetter aims to tackle this issue with their innovative Pay After Placement Model.

Read the full article here:

[![348209_custom_site_themes_id_B6OzrqiSLHO4mVpAyRQ2_yourstory.png](https://almablog-media.s3.ap-south-1.amazonaws.com/348209_custom_site_themes_id_B6_Ozrqi_SLHO_4m_Vp_Ay_RQ_2_yourstory_2994b10ddf.png)](https://yourstory.com/2021/06/edtech-startup-almabetter-cohort-courses-data-science-jobs/amp)

An article was published in YourStory regarding how AlmaBetter trains Data Science aspirants and with placements in high-growth startups and Fortune 500 companies.

Q2 release.PNG

How AlmaBetter helps fill data science jobs at Paytm, Zomato, Flipkart, and more

In this blog, we bring to you the essence of the press release and elaborate on the plans mentioned therein.

AlmaBetter, founded in Sep, 2020, is an Ed-AI platform that imparts Data Science skills to students in a risk-free manner. The company targets to train 25,000 data scientist aspirants and make them industry-ready by the end of March, 2022. This is a bold promise but is an important step towards addressing the widening gap between available jobs and the rate of upskilling observed. To give more context to the readers, a recent AIMResearch revealed that there were nearly 1,40,000 job vacancies in the Data Science job market in the 2nd week of June. This is the highest number of analytics jobs advertised in India since Analytics India Magazine started its research in 2015.

With robust training that AlmaBetter provides, the company sees a big opportunity to upskill young talents and give them an edge in securing these jobs. AlmaBetter-trained graduates get hired by top brands using their hiring portal- AlmaHire. The site functions as a gateway between their job-ready candidates and 200+ hiring partners. Nearly 70% of AlmaBetter students get placed even before the completion of their program. This can be attributed to an excellently-engineered and industry-relevant curriculum that focuses heavily on the practical usage of Data Science concepts. The instructors hail from premiere institutes of the country and adopt innovative and easy-to-grasp learning methods to make learning fun. 

Additionally, making education risk-free remains a non-negotiable proposition for the founders who themselves hail from very modest backgrounds. In a bid to help more young talents from economically weaker sections make their careers in the Data Science and Analytics field, the company has decided to allocate INR 2 Cr fund to provide scholarships, financial aid, and other implements required for online learning such as laptops, wi-fi connections, etc. 10% of all cohort seats will be reserved for extremely underprivileged students.

Find the links to the news story here - 

[![348209_custom_site_themes_id_B6OzrqiSLHO4mVpAyRQ2_yourstory.png](https://almablog-media.s3.ap-south-1.amazonaws.com/348209_custom_site_themes_id_B6_Ozrqi_SLHO_4m_Vp_Ay_RQ_2_yourstory_2994b10ddf.png)](https://yourstory.com/2021/06/almabetter-train-25000-data-science-aspirants-allocates-2cr-fund/amp)

[![The_Economic_Times_logo.png](https://almablog-media.s3.ap-south-1.amazonaws.com/The_Economic_Times_logo_f8f2cfebc7.png)](https://economictimes.indiatimes.com/industry/services/education/almabetter-to-train-25000-data-science-aspirants-allocates-rs-2-cr-fund/articleshow/83602970.cms?)

[![Outlook-Logo.png](https://almablog-media.s3.ap-south-1.amazonaws.com/Outlook_Logo_7fd013479e.png)](https://www.outlookindia.com/newsscroll/almabetter-to-train-25000-data-science-aspirants-allocates-rs-2-cr-fund/2103900)

[![hindu 1.png](https://almablog-media.s3.ap-south-1.amazonaws.com/hindu_1_008a6a64be.png)](https://www.thehindu.com/business/almabetter-to-train-25000-data-science-aspirants/article34843354.ece)

[![live mint.png](https://almablog-media.s3.ap-south-1.amazonaws.com/live_mint_8c5b209833.png)](https://www.livemint.com/companies/start-ups/edtech-almabetter-allocates-2-cr-to-train-data-science-aspirants-11624098598168.html)

[![download (3).png](https://almablog-media.s3.ap-south-1.amazonaws.com/download_3_0141c168b2.png)](https://indiaeducationdiary.in/almabetter-will-train-25000-data-science-aspirants-to-fill-talent-gap-in-the-data-science-sector/)

[![k12digest-new.jpg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/k12digest_new_jpg_83bb931d51.png)](https://www.k12digest.com/almabetter-will-train-25000-data-science-aspirants-to-fill-talent-gap-in-the-data-science-sector/)

[![download (5).png](https://almablog-media.s3.ap-south-1.amazonaws.com/download_5_7409de0158.png)](https://thegreaterindia.in/news/business/almabetter-to-train-25000-data-science-aspirants-allocates-rs-2-cr-fund/)

[![business-world-logo.png](https://almablog-media.s3.ap-south-1.amazonaws.com/business_world_logo_238a9467fe.png)](http://bweducation.businessworld.in/article/AlmaBetter-Allocates-2-Cr-Fund-To-Train-Aspirants/18-06-2021-393574/)

Recently, several prominent national media houses have published AlmaBetter's goal of training 25,000 Data Science aspirants by 2022. This is targeted towards narrowing the widening gap between Data Scientist job openings and rate of upskilling. The articles also outline how AlmaBetter has allocated large funds to facilitate students from economically underprivileged sections achieve 	excellence and build a career in this field. 

AlmaBetter Q2 2021 Press Release

Healthcare organizations worldwide are turning to advanced techniques such as Artificial Intelligence, Machine Learning, and Big Data to help make healthcare practices more efficient. 

Data and information have always been essential for decision-making and healthcare provision. With improved digitization in healthcare, a massive amount of data is also generated from other healthcare industry segments than hospitals and healthcare providers, for example, medical insurance, medical equipment, life sciences, and medical research.

It is safe to say that AI promises a better future in healthcare. According to a research conducted by Acumen, the global healthcare market will hit $8 billion by 2026. 

This blog will look at how AI and Big Data are used in healthcare. 

## AI Battles Serious Illness with Better Predictions

AI and big data deliver great value by boosting the speed with which scientists and healthcare experts can process and use data. With the use of analytical techniques, healthcare stakeholders can harness the power of data not only for the analysis of historical data but also for predicting future outcomes and selecting the most suitable action for the current situation. By integrating life sciences with big data, a company can battle dangerous conditions like cancer and cardiovascular disease. Here are some examples:

- Developing machine learning algorithms and devices to predict the risk of cardiovascular disease before it transpires.
- Increasing the accuracy of osteoporosis risk predictions in women, thereby lowering the risk window from ten years to two.


## New Drug Discovery

According to Daphne Koller, ML researcher and founder of Insitro, Big pharma is finding it challenging to develop new drugs.  Over the past few decades, developing drugs has become increasingly more problematic and costly, leaving patients with substantial unmet needs. According to a report, it can take up to 15 years to bring a new and potentially life-saving drug to market. However, AI technology can help researchers find the right participants for clinical trials and experiments which can boost the process. The technology can also help the experimenters monitor their medical responses more accurately and efficiently. 
 

## Utilizing Big Data To Fight Cancer

Big Data technology can also be beneficial in the fight against cancer. According to a report published by National Geographic, big data technologies can identify and reveal hidden patterns resulting in early detection of cancer. Early diagnosis of the condition can increase the chances of treating it. Big data technologies are proficient at interpreting genome sequencing to identify biomarkers for cancer and reveal groups at higher risk of cancer and find otherwise undiscovered treatments. The most advanced companies use big data techniques to speed their analyses and develop treatments faster and with more tangible results. 
 
## Detecting Healthcare Fraud

According to research from a Splunk report titled AI and ML in regulated industries, detecting and discovering patterns of strange and concerning behavior among many healthcare providers is complex and very time-consuming. Uncovering the source of illegal prescriptions and monitoring it more closely is an additional challenge. With machine learning programs and algorithms, healthcare organizations can see patterns across that data and when they deviate, allowing healthcare organizations to shift to preventing and identifying fraud versus a “pay and chase” approach.
 
## Quality of Electronic Health Records (EHR)

Conventionally, doctors would manually write down or type remarks and patient data after the patient’s visit, resulting in human error. However, with AI and deep learning-backed speech recognition technology, interactions with patients, clinical diagnoses, and potential treatments can be augmented and registered more accurately and in near real-time.
 
## Enhancing Population Health

Big data is playing a crucial role in improving the population’s health. An article in BuiltIn highlighted various businesses leveraging big data to help healthcare organizations and researchers follow the trends to improve health conditions. For example, few companies help organize and integrate billions of records so that pharmaceutical companies can conduct better research for clinical trials, raise the level of safety, and get products to market faster. 
 
## Conclusion

Artificial Intelligence and Big data show promises of development in the healthcare industry, and several VC funding continues to flow into the sector every year. Healthcare organizations have learned to trust algorithms to use them, and there are several startups and firms dedicated and working relentlessly to help and improve the healthcare industry. To help contribute to this revolutionary division in world healthcare, AlmaBetter’s [Full Stack Data Science program](https://www.almabetter.com/courses/full-stack-data-science) prepares you for any job role in Data Science, including positions in the healthcare sector. 


Healthcare is one of the fastest-growing growing parts of the economy. The world is going through a pandemic, and all eyes are on the industry and the new inventions that could save millions of lives.

16_9.jpg

Role of Big Data Analytics and AI in the future of healthcare 

Giving a modern spin to only a decade-old phenomenon, Web Developers seek to use Web3 to paint a new future in the technology world. Web 3.0 can be described as the advanced version of Web 2.0, the advent of 2005, and is based on three foundational blocks - Internet of Things (IoT), Blockchain technology, and Artificial Intelligence. 


Web3 was introduced with the idea of intricately transforming the digital era; and this is the tool AlmaBetter, in association with Polygon, will be offering its students in the newly launched Full Stack Web Development with Web3 course. The course is aimed at helping the Web Developers of the future become more adept in the skill-sets and technical talents the next generation job demands.

## About the Course


**Lifetime Access**: With this course, students will acquire lifetime access to all the course materials, including lecture recordings. This means students will be able to access and revise any topic under the course format at any time.

**Fully-refundable course fee**: Once students complete the course certification, they will receive a full refund of the program fees. This feature makes AlmaBetter’s course one of the best programs in the country for freshers and working professionals.

**One-on-one mentorship sessions**: Students will have the opportunity to attend 1:1 mentorship sessions with industry experts. These sessions will help boost students’ confidence and communication and also offer them a window into the workings of the web development industry. 

**Industry-relevant Capstone Projects**: To help students understand the concepts of Web3 and Web Development better, students will be roped in for real-life industry projects. These projects will be an excellent premise for students to implement their learnings in a real-time industry environment. 

**Access to exclusive Web Development Community**: Shared knowledge brings forward unique innovations. With our course, students will become a part of a dynamic web development community to share ideas and get inspired. The AlmaBetter community aspires to be the one-stop destination for students to broaden their knowledge and share tips and tricks with their peers. The impeccable user experience allows students to find topics of interest and participate in informative discussions swiftly.

**Cohort-based learning**: Our course structure follows a cohort-based learning approach in which students can learn through live classes, in addition to peer-based learnings, projects, and collaborative tasks. Students will also have access to live doubt resolution during lectures, live interaction with the moderator(s), and group discussion forums.

## Polygon - Our Learning and Hiring Partner


Polygon is a scalability platform that enables communication between Ethereum-supported blockchains. It helps developers build scalable user-friendly dApps with robust security measures. The aim of the partnership is to make Web3 technology available to web developers across the country, helping them build decentralised applications (dApps). Currently, Polygon has over 7000 companies building on top of their technology stack. 

-The demand for blockchain developers has seen a substantial increase in the past decade, and Almabetter in collaboration with Polygon, seeks to bridge the gap between the demand and supply of Web3 developers. 

-The Full Stack Web Development with Web3 course will be offered in partnership with Polygon. Under this partnership, we aim to establish a cohesive learning environment for our students.

-The curriculum has been designed by Polygon to equip students with strong foundational knowledge to build solutions on top of the Polygon stack.
 
-Through this partnership, AlmaBetter aims to produce 30,000 Web3 developers by March 2023.

-Polygon, as a part of this partnership, will be rewarding top performers with $MATIC coins on successful completion of the course.

## Why opt for AlmaBetter as your learning partner?


The Full Stack Web Development with Web 3.0 course has been designed to offer robust training to young professionals and help them acquire a competitive edge over their peers. One of the key USPs is the industry-relevant training and projects provided to students as part of the course, allowing them to prepare for real-world scenarios in a matter of 28 weeks. With mentorship sessions, Capstone projects, and a window into the sphere of Web Development, AlmaBetter can be the right choice for your career.

**Guaranteed Placement**: Once students have earned the course certification, they can choose to opt for the AlmaX program, which offers guaranteed placement with a minimum salary of 6 LPA.

**Student-Centred Curriculum**: Being an entirely online education platform, we understand the need to keep students engaged without the strenuous ritual of sitting through long lectures. To keep learning fun, we have gamified our live classes through the highly-interactive game-based learning platform, Kahoots.

**Learning Kit Components**: AlmaBetter offers the most comprehensive and fun learning experience with mentorship sessions, hands-on guidance, 500+ hours of learning, etc. Check out this link to know more about AlmaBetter Full Stack Web Development with Web3 Course Learning Kit Components. 

**500+ Hiring Partners**: AlmaBetter has partnered with more than 500 growing start-ups, medium-level corporate, and multinational conglomerates across several Indian cities. In the near future, AlmaBetter plans to expand its hiring network exponentially.

The ultimate goal of the Full Stack Web Development with Web3 course, launched in association with Polygon, is to offer the Web Developers of tomorrow a fitting platform. 




In the past few years, we have seen a paradigm shift in the world of technology. Whether it is the rise of the Metaverse (catapulted by Facebook’s rebranding as Meta) or the conscious transition into remote learning environments, the changes are irrefutable. Blockchain, too, has played a vital role in transforming the landscape of technology. 

16_9 (1).png

AlmaBetter, in association with Polygon, launches India's first Full Stack Web Development with Web3 course

Though its origins go back to the early 1980s, it was the digital boom that brought forward the phenomenon that blockchain is. Today, the technology is being leveraged by a string of industries, all working to make operations safer and more efficient. In this blog, we walk you through the fundamentals of blockchain technology and how it has transformed several sectors in the business space. 

## What is Blockchain?

In simple terms, a blockchain is an ever-growing digital ledger shared among the nodes of a computer network. It streamlines recording transactions and tracking assets in a business network. Anything of value can be tracked and traded on a blockchain network, tangible or intangible. Blockchain is famous for its essential role in cryptocurrency systems such as Bitcoin. 

The significant difference between a blockchain and a typical database is how the data is structured. A blockchain manages information together in blocks that carry sets of data. Blocks have limited capacities, and once loaded, it is locked and linked to the previous block, constructing a chain of data known as the blockchain.

## Importance of Blockchain

Every business runs on information. If the data is accurate and received swiftly, it can be very beneficial for the company. Blockchain is perfect for supplying information as it provides immediate, shared, and thoroughly transparent data stored on an unchangeable ledger accessible only to exclusive network members. A blockchain network can track payments, accounts, production, orders, etc. 

## Origin and goal of Blockchain

The primary goal of blockchain is to allow digital information to be recorded and distributed but not reworked. A blockchain is a foundation for a record of transactions that cannot be modified, deleted, or destroyed, and this is why they are also known as distributed ledger technology (DLT). 

The first-ever prototype was proposed in 1982 by cryptographer David Chaum, further worked on by computer scientist Stuart Haber and physicist W Scott Stornetta in 1991, where they presented a cryptographically secured chain of blocks. In 2008, Satoshi Nakamoto introduced the first decentralized blockchain, which eventually led to the creation of Bitcoin, the first cryptocurrency. The use of this technology exploded with the creation of several cryptocurrencies, non-fungible tokens (NFTs), and smart contracts. 


## Blockchain in different industries

Blockchain is also a reliable method of storing data about other types of transactions and not just monetary transactions. Several companies have already incorporated this technology, including Walmart, AIG, Pfizer, Unilever, etc. Let's have a look at different sectors where blockchain is being used:

**Healthcare**

Healthcare providers can leverage this technology to securely store their patients’ medical records. When a medical record is developed and signed, it can be easily written into the blockchain, providing patients with an assurance that the information cannot be altered. The personal medical data of several patients can be encoded and stored on the blockchain with a private key, thereby ensuring privacy. 

**Banking and Finance**

Possibly no industry benefits from integrating blockchain into its business operations more than banking. By incorporating this technology into banks, customers can see their transactions processed in less than 10 minutes. Moreover, banks can also exchange funds between institutions quickly and securely with blockchain. 

**Supply Chains**

According to Forbes, the food industry is progressively embracing blockchain to track the path and safety of edibles throughout the journey. Suppliers can leverage this technology to record the origins of materials they purchase, allowing companies to affirm the authenticity of their products and standard labels. 

## Future of blockchain

With many functional applications for the technology already being enforced and explored, blockchain is finally making its way into different sectors, in no small part, because of bitcoin and cryptocurrency. Blockchain technology makes business and government operations more efficient, secure, and accurate with fewer intermediaries.

The world is ready to step into the third decade of this game-changing technology; legacy companies will likely catch on to blockchain technology. Today, we see an accumulation of NFTs and the tokenization of assets. The following decades will prove to be a crucial period of growth for blockchain.

## Learn more about Blockchain Technology.

AlmaBetter's newest course launch: Full Stack Web Development with Web 3.0 course, covers all the essentials required to master blockchain technology. The curriculum includes the basics and the advanced concepts necessary to understand blockchain entirely. Enroll now and kickstart your journey as a software developer with AlmaBetter.


The blockchain industry can easily be referred to as one of the most innovative, ground-breaking discoveries of the 21st century. 

blockchain-business-symbol-concept (1) (1).jpg

What is Blockchain Technology?

Data Science is making headlines all over the world. This challenging domain is the hot new field that promises to revolutionize almost every industry. From government to healthcare, several industries are already capitalizing on the progress made in Data Science. Moreover, the demand for skilled Data Science professionals is rising every day, and at AlmaBetter, we aim to train and provide industry-ready folks to the world. 

Data Science offers a variety of job roles under its umbrella. AlmaBetter’s robust curriculum prepares students to excel and land any data science role efficiently. Let’s have a look at the various job roles in data science.


**Data Scientist**

Data Scientist is the most popular and general role in data science. A Data Scientist’s role includes all the aspects of a project, meaning, dealing with the business side to data collecting and analyzing, and eventually visualizing and presting. In short, a Data Scientist is responsible for understanding the challenges of business and offering solutions using Data processing and data analysis.

Data Scientists are usually also in charge of researching and developing new algorithms and approaches. Often, in MNCs, Data Scientists are team leaders of people with specialized skills. The skillset of a Data Scientist allows them to overlook a project and guide them from start to finish.


**Data Analyst**

Data Analyst is undoubtedly the second most known role in data science. Data Analysts are responsible for various tasks such as visualization, munging, and manipulating a massive amount of data. Occasionally, they have to perform queries on the database, and they are in charge of web analytics tracking and A/B testing analysis.

One of the most effective skills of a data analyst is optimization. They are responsible for creating and modifying algorithms to cull information from massive databases without corrupting the data.


**Data Engineers**

Data Engineers are accountable for building, designing, and strengthening data pipelines. Data Engineers also build and test scalable Big Data ecosystems for companies so that the data scientists can run their algorithms on data systems that are stable and highly optimized.


Moreover, Data Engineers are also responsible for working on batch processing collected data and matching its format to the stored data. In other words, Data Engineers make sure that the data is ready to be processed and analyzed. 


**Machine learning Engineers**

Currently, machine learning engineers are in demand and trending in the Data Science World. However, it is also one of the most challenging roles in Data Science. An Ideal Machine Learning Engineer should be familiar with many machine learning algorithms such as categorizing, clustering and classification. Also, they should be up-to-date with the latest research and advancements in the domain.

Moreover, Machine Learning Engineers are expected to have strong statistical and programming skills and basic knowledge of software engineering.


**Data Architect**

Data Architects share some shared responsibilities with Data Engineers. Both roles are required to ensure that the data is well-formatted and accessible for data scientists and analysts and improve the performance of the data pipelines.

A Data Architect also constructs the blueprints for data management so that the databases can be smoothly integrated, centralized, and protected with the most suitable security measures.


**How AlmaBetter prepares you to land any role in Data Science?**

AlmaBetter’s reverse-engineered curriculum covers full-stack data science concepts starting from Python, Analytics Frameworks such as SQL, Excel, tableau, math concepts such as Calculus, Linear Algebra, Probability & statistics to Machine Learning, NLP, Deep Learning, and ML Engineering.  The breadth of knowledge and industry-level approach ensured through our rigorous training is enough for you to land a high-paying job in the data science domain.


In this blog, we will explore what Data Science is and what are the related career prospects in India.

cropped_view_hands_typing_laptop_dd798108b3.jpeg

Most promising Job Roles In The World Of Data Science

Data Science is a vast field, and it takes a lot to start from scratch and build your future in this fantastic realm. A tremendous learning experience for students requires a well-designed curriculum; that's where AlmaBetter's Learning 3.0 comes in. AlmaBetter believes in democratizing quality education and is continuously working towards it. 


## What is Learning 3.0?


Before we discuss Learning 3.0, Let's understand 1.0 & 2.0. Learning 1.0 is the traditional learning method in offline school/college where students attend classes with their group for a fixed period of 5-6 days per week. Students connect with their friends, socialize with their peers, and directly access teachers/ professors for their queries. Students feel associated with the organization and have access to the alumni network for career and learning guidance. Students participate in discussions, extracurricular activities, prepare for the goal and have fun growing together in age and skill. Everything looks so positive, but here is the catch, students have to engage in a fixed period and be present physically to learn. It might sound okay, now let us consider working professionals as students looking to upskill or students who want to learn some skill that School does not provide. It will be tough for this segment to commit fixed timings and be physically present at multiple places.


Learning 2.0 is the program/course currently available on the Internet. Current programs offer self-paced learning and live lectures through an online setup that solve the problem of fixed timing and physical presence but miss the significant aspects such as peer learning, discussions, feeling of association with the organization, and direct access to doubt resolution. In the current online setup, students feel alone and quickly lose motivation towards learning.


Moreover, there are some other significant issues with the current education system -

1. Lack of outcome-based learning
2. Industry-relevant curriculum


AlmaBetter aims to tackle all the issues with Learning 3.0. Here are the key elements of Learning 3.0-

- **Outcome-Based Learning**- Your fees get refunded once you complete the course and enroll in our job guarantee program. We charge our hiring partners by providing skilled employees and onboard you for payment after the placement agreement. 
- **Student-Centred Curriculum**-  We focus on the curriculum required for the industry. We optimize our content based on the current requirements by industries.
- **Peer Learning for Students**- Our curriculum focuses on 80% group discussions. All students are encouraged to interact with the cohort, form a team of 5 members, study, and discuss topics every day. This is to ensure peer learning for students and remain engaged throughout the program.
- **Live interaction sessions with industry experts**-  AlmaBetter conducts live interaction sessions with industry experts every day from Monday to Saturday between 8 PM -10 PM. This helps students to resolve their doubts and develop a solid understanding of concepts. 
- **Monthly Mega- Event with amazing prizes**- We encourage students to participate in personal branding. AlmaBetter organizes the monthly mega-event "Engagement Superstars" and prizes up to INR 1.5 lakhs.


AlmaBetter cares about its students and focuses on resolving all the issues throughout the journey. Learning 3.0 assists in providing the best learning experience and eliminates the problems students face during their journey. 

Join AlmaBetter, the most student-centric institute, and start your journey with the revolutionized learning experience with Learning 3.0. 

The learning experience at any institute is an essential component for students. We at AlmaBetter believe in providing risk-free education and fun and a student-centered curriculum. We have come a long way from old learning systems, and AlmaBetter aims to revolutionize the learning experience for every Data Science enthusiast with Learning 3.0. 

learning.jpg

What is Learning 3.0?

The world has changed in many ways over the years. Technologies have advanced, politics have shifted, and populations have grown. But one thing that has remained constant is the importance of data. Data has helped humans navigate the world, discover new things, and improve the quality of life for billions of people.

Moreover, the data science market was valued at USD 31.05 billion in 2020, expected to reach USD 230.80 billion by 2026, registering a CAGR of 39.7 % during the forecast period, 2021-2026.


## Why Is Data Important?


Data. It’s the lifeblood of our modern world. Without data, our phones would be nothing more than dumb hunks of plastic. We wouldn’t find the fastest route to our destination or identify the best place to take our family on vacation.

With the world becoming increasingly digital, the importance of data has never been greater. Data is the foundation for the most advanced technologies, the basis for the most sophisticated algorithms, and the backbone of the most exciting innovations. 

Data is the starting point for the most critical decisions and the key to the most impressive accomplishments. No organization can succeed without data, and no strategy can succeed without understanding how to use data in the right ways.


## Emerging Trends of Data

Data has become an integral part of our lives. From the moment we wake up, we are bombarded with data in the form of news updates, emails, text messages, and a variety of other pieces of information. 
Our days are filled with data in the form of conversations, interactions, and experiences. When we sleep at night, we often intend to be better prepared for the day ahead, which usually involves more data.

So, let’s dive right into what are the emerging trends in the world of Data and how you can catch the bandwagon of technology:


**The growth of the Internet of Things (IoT):**

It will significantly impact the way data is used. The IoT refers to the collection of devices connected to the Internet. According to Mordor Intelligence, the IoT market reached $761.4 billion in 2020 and is expected to reach $1.39 trillion by 2026.

The IoT is expected to grow significantly over the next decade and will offer numerous benefits, including the ability to improve the way data is used. One of the most significant ways the IoT will impact the way data is used is by collecting data from a variety of devices.


**Big Data & Cloud Computing: **

Over the next decade, the biggest trend in data will be the continued growth of “Big Data” on a cloud-based infrastructure. Today's biggest challenge in the data industry is the storage of “Big Data.” 
As the amount of “Big Data” increases, the number of storage increases along with it. This creates a challenge for using “Big Data” to make better decisions. The Big Data market size is projected to grow from USD 162.6 billion in 2021 to 273.4 USD billion in 2026, at a Compound Annual Growth Rate (CAGR) of 11.0%
Cloud computing has become an essential part of data infrastructure. Cloud computing refers to using a network of computers that are accessible via the Internet. The term “cloud” is often used to describe the services provided by data stored on a cloud. Big data refers to the large volume of data stored in a cloud.


**Emphasis on Actionable Data: **

The ability to provide actionable data will be the differentiating factor for data providers. Actionable data is data that can be used to make decisions. For example, actionable data can be used to determine whether or not to greenlight a project or fire an employee.
Actionable data refers to data that is analyzed in a way that allows for the effective use of the data. For example, actionable data can improve the way products are sold. Actionable data can also be used to improve the way marketing is carried out.
Data must be used to make decisions, but it must also be used to take action. This shift in focus toward actionable data will impact the design of data-driven systems. 
For example, a data-driven transportation system may use data to decide the most efficient way to route a bus. Still, a person may determine the final decision on the bus rather than a computer.


**Hyper-Automation: **

Hyper-automation refers to using complex algorithms and automation to help streamline processes and analyze data. 
Hyper-automation will help reduce the amount of human intervention required to process and analyze large volumes of data. This will allow data to be used to make better decisions, which will increase the efficiency of data-driven systems.


**Use of A.I and M.L:  **

The use of artificial intelligence (AI) and Machine Learning will continue to increase over the next decade. The use of AI and machine learning has already become common in many industries, with the legal sector expected to be one of the first to utilize these technologies extensively. 
The use of AI and machine learning will continue to increase over the next decade, with a shift toward hyper-automation. AI and machine learning characterize this shift to perform tasks that humans once performed.


**Focus on Edge Intelligence: **

The future of data will be defined by the ability to provide edge intelligence. Edge intelligence refers to the processing and analyzing data as close to the source as possible. 
This will allow data to be used to make decisions closer to real-time, which will increase the effectiveness of data-driven systems. The focus over the next decade will be on edge intelligence, which will allow data to be used to make decisions that have a significant impact on the outcome of a system.
It refers to the locations where a network is used most heavily. This will allow for the increased use of data at the locations where it is needed most.



**Increased Used of Natural Language Processing: **

The future of data is moving toward the use of natural language processing. Natural language processing is an area of artificial intelligence that allows computers to process and analyze human language. 
Much of the growth of natural language processing over the next decade will be due to the increasing use of artificial intelligence to analyze human language.
NLP refers to the analysis of how people communicate using language. Over the next decade, NLP will become more prevalent as companies use it to process vast amounts of text for more effective communication.
The NLP market is predicted to be almost 14 times larger in 2025 than in 2017, increasing from around three billion U.S. dollars in 2017 to over 43 billion in 2025.


**Blockchain In Data Science: **

Over the next decade, the biggest data trend will be the increasing use of blockchain in data science. Blockchain can be called a digital ledger that can be used to store data securely. 
Over the next decade, the use of blockchain in data science will increase as companies utilize it to store large volumes of data. This will allow data to be used to make decisions that are further along in the decision-making process, which will increase the effectiveness of data-driven systems.
Blockchain is an innovative technology initially developed to support the digital currency bitcoin. However, blockchain has since been adapted for use in various industries, with data science being one of the fastest-growing industries utilizing blockchain. The use of blockchain in data science will allow for the secure storage of data while providing a decentralized platform for data scientists to share their work.


**Python for Programming: **

Python is an interpreted programming language that is well suited for data science. It is also easy to learn and popular for beginner data scientists. The continued focus on developing the python programming language will allow data scientists to learn a popular language well suited for data science without learning a framework specific to that language.


## Connecting the Data Points

The use of data in the future will continue to increase as more and more data becomes available to organizations. The focus over the next decade will be on the continued development of data science, which will allow data to be used to make decisions that have a significant impact on the outcome of a system. 
This will result in an increased ability for companies to become data-driven, which will lead to an increase in the effectiveness of data-driven systems. The use of data in the future will continue to be driven by the continued innovation of data science.
But are you equipped to join the trend of the future? Prepare yourself with the latest gadgets; technologies and learn the right way with a student-centric platform developed for future data scientists like YOU!
Join the revolution with AlmaBetter. 

Data is a vital resource in today's world. It is used to power the websites and apps we use daily, provides the information we require to make decisions, and informs us of our world's current state and where it is heading. Data can predict future trends and behaviors, and without it, our world would be a much different place. 

artificial-intelligence-technology-futuristic-science.jpg

Top 9 Data Science Trends to Keep you On Trend

As the other algorithms take the independent features and try to predict the dependent one, same for Xgboost, but it doesn’t take the actual dependent feature, instead it creates its own feature, isn’t that amazing?

Let's check how it works with the help of this table.

![1_9qG6qFcpx9Xd7zSNJzCAJA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_9q_G6q_Fcpx9_Xd7z_SN_Jz_CAJA_0083904149.png)

So, this is a simple example of predicting the IQ level with the help of age, but the column “**New feature (predicted)**” is the one which Xg-boost created by taking the average of all the IQ present in the table. As mentioned, it will take the new column and train itself, but this is not the main column/feature. Let’s have a look at this

![1_bDrXFj7w2AWshk0F_7qPRA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_b_Dr_X_Fj7w2_A_Wshk0_F_7q_PRA_858b150281.png)

This is the new and final feature from which the training begins in Xg-boost. Now it will try to predict the residuals with the help of decision trees.

![1_U7uHR2l3rxJESzkKXzbEIQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_U7u_HR_2l3rx_JE_Szk_K_Xzb_EIQ_a0b49e4aed.png)

As we know the splitting criteria in decision trees, with the help of information gain. So the root node will be split if it shows the maximum information gain, and this tree will be the base learner or the weak learner and will predict the new values for residuals and which when added to the previous** “New feature (predicted)” **will give the new values of IQ, now again the same procedures go and **Xg-boost learn from the mistakes of this base learner and will predict new values and so on until the loss or residuals are minimum, and we have our strong learner** at the end. The advantage of this approach is that the new learners are being added to the model by learning and correcting the mistakes of previous learners.

**_Let’s have a look to practical example in python_**

- !pip install xgboost
- Import xgboost as xgb

**Splitting and fitting the data**

- from sklearn.model_selection import train_test_split
- X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size=0.2)

**We need to change the format that Xg-boost can handle**

- D_train = xgb.DMatrix(X_train, label=Y_train)
- D_test = xgb.DMatrix(X_test, label=Y_test)

**we can define the parameters of our gradient boosting ensemble**

- param = {
- ‘eta’: 0.3,
- ‘max_depth’: 3,
- ‘objective’: ‘multi:softprob’,
- ‘num_class’: 3}

steps = 10 # The number of training iterations

**Training and testing**

model = xgb.train(param, D_train, steps)

preds = model.predict(D_test)

**Fighting with Overfitting**

This is a very usual case in this algorithm because the algorithm is going through the data so many times, so it may learn the underlying logic very well, but we have a remedy for that, by using the hyper-parameter “Gamma”.

_Minimum loss reduction required to make a further partition on a leaf node of the tree. The larger gamma is, the more conservative the algorithm will be._

_Reference - https://xgboost.readthedocs.io/_

Other parameters we can look into:- max_depth, and eta (the learning rate)


Special Notes –

1. We can use grid search for choosing the optimal hyper-parameters

2. Although it’s a boosting technique but it uses all the cores of our computer’s processor in a parallel manner, so we can also say this as a parallel technique but not directly, as the work is sequential.

References -

https://xgboost.readthedocs.io/

https://www.geeksforgeeks.org/xgboost-for-regression/

Let’s start with the full form, XG means the extended version of gradient boosting. As the name suggests it is a boosting technique, i.e models or trees are sequentially connected to each other in this Algorithm, that’s why it comes under ensembling. Here algorithm plays with the residuals only and tries to minimize the residuals which in turn reduce the loss and increase the accuracy, but as residuals are getting decreased so the model may overfit, but we have techniques to handle that, let’s start with the in-depth intuition of this algorithm along with the practical approach.

1_lgWmtrkfdukCBOXzAvap1A.jpeg

XGBoost In-depth Intuition

## Classification

Let us first look into the evaluation metrics used in classification models. Here we will not only get to know the metrics we will also see which ones are used in a specific business problem.

For a classification-based model, all the parameters are included inside a metric called **Confusion Matrix**.

AConfusion matrix is an M x M matrix used for assessing the performance of a classification model, where M is the number of predicted classes. The matrix compares the actual predicted values with those predicted by our classification model. This gives us a comprehensive view of how well our classification model is performing and what kind of metric is reliable.

Since we will be looking at binary classification, let us take a 2 x 2 matrix as shown below with 4 values:

![1_YLuQ4GZogyh8dUh8oK5Jqw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Y_Lu_Q4_G_Zogyh8d_Uh8o_K5_Jqw_a7c393b218.png)

Here ‘Actual Class’ is what actually happened and ‘Predicted Class’ is what my model predicted. Now let me explain the diagram with a realistic example. Let us imagine a classification model that predicts whether it will rain or not rain. ‘Rain’ is a positive class and ‘Not rain’ is a negative class. True positive (TP) is when my model predicts Rain and it actually rains. True Negative (TN) is when my model predicts Not Rain and it doesn't rain in real. Whereas False Positive (FP) is when my model predicts a positive class(Rain) but it turns out to be false and it doesn't rain. Similarly, False Negative (FN) is when my model predicts a negative class ( Not Rain) but it rains on that day.

Based on these four parameters we will define evaluation metrics like Accuracy, Recall, Precision, and F1_Score.

![1_ZrCk9fJHra1AR6k1pGwjFw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Zr_Ck9f_J_Hra1_AR_6k1p_Gwj_Fw_e6fb7fca73.png)

## ACCURACY

In very simple terms accuracy is an evaluation metric that measures the total number of predictions that my model got right out of the total number of predictions it made. In terms of confusion matrix, the formula for accuracy is;

![1_BVUXZfcoo9hlqFNS-09nwg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_BVUX_Zfcoo9hlq_FNS_09nwg_9997f63b17.png)

Accuracy takes into account both True positives and True negatives, many a time we might need any one of them, for which there are other metrics defined. Every term in the formula of Accuracy is present in the confusion matrix and we can calculate it accordingly. Many a time the model built is fairly accurate but it still can’t solve a business problem. Such cases arise when we have an imbalanced dataset i.e. a dataset that has a majority of observations in the positive class and very few observations in the negative class and vice versa.

For example, you build a model to predict if the email you got is spam or not. In this case, 1 maybe 2 emails out of 100 that you get, turn out to be spam. So if you build a model which shows all mails are legitimate you will still achieve an accuracy of 98% but it will neglect all the spam emails and the model does not work in any way. When accuracy is not a good metric to evaluate your model you need to look at other metrics.

## PRECISION

In very simple terms precision measures how precisely a model predicts the positive labels. In other words, out of all the positive predictions how many were correctly predicted. The formula for precision is given below:

![1_0AEAX2BxJluajn90O--RxA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_0_AEAX_2_Bx_Jluajn90_O_Rx_A_141ed23cb3.png)

The numerator of this formulae essentially tells us the positive observations that my model predicts correctly. The denominator essentially shows all the points that my models predicted as positive.

## RECALL

Also known as true positive rate(TPR) or Sensitivity, recall can be defined as out of all the actual positive values, how many of them were predicted correctly. The formula for the recall is given below :

![1_Q6SPtx9H0y0OwKf6mFdV1g.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Q6_S_Ptx9_H0y0_Ow_Kf6m_Fd_V1g_ae3370acbd.png)

The numerator is the same as precision. but the denominator has all the actual positive values irrespective of which class the model predicted it to.

Now let me give explain to you both precision and recall in realistic terms to get a better intuition. Imagine your friend gives you a present at every birthday for the past 10 years. In the 11th year she makes a bet, she says that I will give you your favorite PlayStation if you can correctly guess all the 10 presents that I gave you over the course of 10 years. Now you really want PlayStation so you try hard to recall each present one by one. Suppose you were able to recall 7 presents out of 10, therefore you have a recall of 7/10 i.e. 70 %. Now to further the bet she says okay you try to guess the other presents you can take infinite guesses. you try your best and were able to name all 10 presents in 20 tries. Though this makes your recall as 100% you were not precise enough, your Precision gets as low as 10/20 i.e. 50%. Hope this gives more clarity on both the terms.

**How do you choose between precision and recall for your business problem?**

The best explanation for this is to choose Precision when your False-positives cost you more than your False-negatives. For example imagine you sell gym products, your invest in a brand of protein supplements as suggested by your ML model. But it turns out that you got a false positive signal and your customers hate the supplement and you incur a huge loss. Any business problem which involves investing in your output prediction such as giving loans, customer churn analysis, other banking systems you use Precision.

Now imagine you have to deploy a model in the health sector, detecting cancer in patients. Here if you get any false negatives i.e. you get a prediction that the patient doesn't have cancer whereas, in reality, he has cancer, in this case, his life is at risk because of wrong predictions. So any business problem where a False-negative can cost you more than a False-positive you use Recall. Other places that you can use recall are credit card fraud prediction, all healthcare domains, etc.

What happens if your data has class imbalance but you cant choose between precision and recall as both hold equal importance. In such cases, you go for an amalgamation of both, there is a metric that combines both precision and recall and is named f1_score.

F1_score also known as F Measure is the harmonic mean of both precision and recall. F1 score can be calculated using the following formula.

![1_l6kChXQSMC_ahLDoPio20Q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_l6k_Ch_XQSMC_ah_L_Do_Pio20_Q_6c0961c875.png)

## REGRESSION

Evaluation metrics play a very significant role in building any machine learning model. The Mean squared error, root mean squared error, mean absolute error, R-Squared, and Adjusted R-squared are used to evaluate the performance of the model in regression algorithms.

Mean Squared Error(MSE) measures the arithmetic mean of the squared difference between the actual and predicted values in the dataset. It is a measure of the variance of residuals.

![1_QrjZBfBOWLKkhtt_6KYHig.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Qrj_Z_Bf_BOWL_Kkhtt_6_KY_Hig_5907a531cd.png)

Mean absolute error(MAE) measures the arithmetic mean of the absolute difference between the actual and the predicted values in the dataset. It measures the average of residuals.

![1_nFGg2hyuRgIafo66c0D0VQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_n_F_Gg2hyu_Rg_Iafo66c0_D0_VQ_2df7150a59.png)

Root Mean Squared Error(RMSE) is nothing but the square root of the mean squared error that we calculated earlier. It measures the root of variance i.e. standard deviation of residuals.

![1_iEc3eda4s_K5aLDKVIaGCw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_i_Ec3eda4s_K5a_LDKV_Ia_G_Cw_6bc46b0ba9.png)

R-squared measures the degree to which variance in the dependent variable can be explained by the independent variables(features). It will always range from 0 to 1, the higher the R-squared better is my model. It is a relative metric that can be used to compare when we apply more than one algorithm.

![0_Q2kREYwDCulazh8O.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Q2k_RE_Yw_D_Culazh8_O_a05fc1d302.png)

Adjusted R squared is a slight modification to the R-square value, this measures the variance in the target variable, explained by only the features that are helpful in making predictions, it will always be less than or equal to R² because it will penalize you for adding features which are not useful in making predictions. In the formula below n is the number of rows in the data and k is the number of columns in the data.

![0_eMBdeYzdnDpRovF-.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_e_M_Bde_Yzdn_Dp_Rov_F_af40415628.png)

## What to choose between MSE, RMSE, MAE, and MAPE?

Whenever you have to penalize the outliers present in the dataset you use either MSE or RMSE. I personally prefer RMSE over MSE as the latter does not have the same unit as my predicted value.

Use MAE when your predictions do not get affected by outliers as MAE is robust to outliers.

RMSE and MSE both are differentiable and are generally chosen over MAE, as MAE is not differentiable when actual value = predicted value.

Whenever your mean predicted values are large(in thousands or millions), choose MAPE(mean absolute percentage error) over the others as very large error values might look misleading.

R-Squared is a relative metric and should be preferred to compare models when we apply more than one algorithm.

Both R-Squared and the Error metrics hold important information, as the former tells us the relationship between independent and dependent variables and the latter tells us about how close are actual and predicted values.

## CONCLUSION

We often make the mistake of randomly applying these evaluation metrics to access our model’s performance, but data science is all about asking the question “why?”.That is exactly what we should be doing while choosing certain evaluation metrics over others. With this blog, I hope I am able to clear the intuitions between different evaluation metrics used in classification as well as regression analysis.

Happy Learning!

When it comes to evaluating your model’s performance there are different metrics that you can choose from. To choose the best evaluation metric to assess your model, it is necessary that you understand the advantages and disadvantages of each and every metric. As a Data Scientist, you might have come across statements like your model is extremely accurate but depending on the business problem your model is not performing up to the mark, in such cases maybe be you should go with a different metric.

1_Ae6Y7GunyNaOHpXz3l3EsA.png

Which evaluation metric should you choose for evaluating your Classification and Regression models?

WhatsApp is a fast, simple, and convenient way for family and friends to chat, create group texts, share photos and videos, send and receive documents, and engage in private, secure conversations anytime, day or night.

Now WhatsApp is on top among all other messaging channels in terms of their active users. Due to the pandemic, WhatsApp sees a 40% increase in usage.

![0_g3mji6EGTd2E-V7L.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_g3mji6_EG_Td2_E_V7_L_7a49259e6b.jpg)

Today, WhatsApp delivers roughly 100 billion messages a day

Let’s see the features required for WhatsApp system design one by one.

## Direct Messaging

We can chat with a friend or any number which we have in our contacts. If we have just two devices on the same network which need to transfer messages between each other then we need to know the address of both the devices and can exchange the message directly. But when the number of devices and the number of networks increases then it becomes increasingly difficult and we need to have a server in between that is responsible for receiving/sending the messages to appropriate clients.

![0_97q4mVjQs88nPwHM.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_97q4m_Vj_Qs88n_Pw_HM_2f6b726434.png)

As you can see above there are two clients in the diagram Client A and Client B. Here Client A needs to send a message to Client B. It initiates a request to the server. A TCP connection is established between the user and the gateway for communication.

As soon as a user becomes online a new TCP connection is established. When the user sends the message it travels through the gateway. Gateway is responsible for an abstraction of security, reverse proxy, rate limiting, etc. Then the request goes to the load balancer which is responsible for distributing the load across multiple servers. The messaging server then checks in the database to which server-client B is connected and forwards the request to client B.

There could be different scenarios when sending the message.

- Client A is offline: In this scenario, the message which needs to be sent is stored on the device’s database(e.g. SQLite) until the device becomes online.
- Client B is offline: In this scenario, the message gets saved in the database. As soon as the client becomes online the messages are sent to the client.

## Group Messaging

WhatsApp provides one important feature of group messaging. Around 256 people can enter one group and can interact together. It is helpful in establishing a collective conversation with others. It works in the same way as to direct messaging.

![1_27VYqe5f6d3sXdEjwpMscA.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_27_V_Yqe5f6d3s_Xd_Ejwp_Msc_A_b2b6bb5d8b.jpeg)

## Whatsapp Pay

WhatsApp Pay is an in-chat payment feature that allows users to make transactions via WhatsApp to their contact list. It is UPI-based payments service that allows you to both send and receive money. It was developed by the National Payments Corporation of India (NPCI).

![1_Hph0-VQ01R7w9QEL53k8cw.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Hph0_VQ_01_R7w9_QEL_53k8cw_da6d27ca05.jpeg)

## Online/Last Seen

This feature is used to see if the person is online or the person’s last online status.

![0_slhW62n1PzdRdaFW.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_slh_W62n1_Pzd_Rda_FW_bb8f7de3c9.jpg)

Whenever a client is performing any activity such as sending a message, opening a new message, uploading documents, or making a phone call this recent activity along with the time stamp is recorded for the user. With each new activity, the timestamp for the user is updated. Whenever user B wants to check the last seen of user A then it requests the database to get the timestamp of user A. If the timestamp is within the range current time — threshold value(e.g. 1 minute) then we can display the status as online else display the last seen as the timestamp of user A’s last activity timestamp.

## Message Acknowledgments

Let’s talk about message acknowledgments i.e. the concept of the single, double, and blue tick.

WhatsApp supports 3 types of message acknowledgments:

- Message sent
- Message delivered
- Message read.

![0_dammNM_LECWQC-dV.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_damm_NM_LECWQC_d_V_e85d76e1b7.jpg)

As soon as the server thread processes the message from the received message queue it sends the client that the server received the message. When the receiver receives the message or reads the message a new request is sent to the server which then updates the sender about the status of the message.

## Image Sharing

In a chat messaging system it’s common to share images, videos, and file attachments. This could be a long-running process and should be separated into another microservice that could handle such requests. As shown in the diagram below there’s another multimedia server that handles these types of file sharing requests. The client initiates an HTTP request to the Multimedia server. The multimedia server checks if the receiving client B is online. If yes then it sends the hash of the file to client B. Client B uses this hash to download the file.

![0_FTNEW06VsqT4TFU-.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_FTNEW_06_Vsq_T4_TFU_86ce510001.png)

If client B is offline then the file is stored inside secure cloud storage. As soon as client B comes online it receives the hash and fetches the files from the cloud. To make this file sharing faster we could use a CDN. A CDN stands for Content Delivery Network which caches the information geographically near the client’s location to reduce the latency and increase the download speed.

## How WhatsApp is using Data Science

This massive increase in WhatsApp usage over the last couple of years has opened many opportunities for businesses. WhatsApp chatbot is an important feature for WhatsApp businesses.


## WhatsApp Chatbots

WhatsApp Chatbot is an automated software powered by AI which runs on the WhatsApp platform. People communicate with WhatsApp chatbot via the chat interface like talking to a real person. It’s a set of automated replies that simulates a human conversation on WhatsApp.

![0_2NGKFxY4Ef2GGvSx.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_2_NGK_Fx_Y4_Ef2_G_Gv_Sx_0be81da4e9.jpg)

WhatsApp chatbots are actually helping companies in automating the things and services they provide.

There are official WhatsApp chatbot companies. They are working continuously to make this process smooth.

WhatsApp chatbot building company is a chatbot development company that creates chatbots for WhatsApp. Many of these companies partner with the official WhatsApp Business solution providers and help us get API access and guide us through the application process and after that, we have access to WhatsApp chatbots

## Checking Fake messages

WhatsApp has revolutionized the way information is spread. It connects more than 1.5 billion people and helps exchange an unbelievable number of 65 billion messages daily. Though WhatsApp is meant to connect with family, friends and colleagues, it is being misused by some to spread rumours and fake messages. This has led to unduly violence and social disorder. After a series of mob-lynching incidents, triggered by rumors circulating on WhatsApp, Indian Government is drafting an amendment to the Indian IT act. The draft as of now requires “intermediaries” like WhatsApp and Google to deploy technology-based automation tools with appropriate controls, for proactively identifying & disabling access to unlawful information or content. Most of the fake WhatsApp messages have features and motives such as

- A request to forward the message to many
- The threat of consequences if the request is ignored,
- Fake information source to add credibility,
- Details on the author and origin of the information is not cited
- A clear point in time is not mentioned, only generic time information is mentioned such as ‘last year’ or ‘yesterday’.

Data Science can be used to classify a WhatsApp text message into one of these five categories — fully true, mostly true, mostly false, fully false and undermined. With use of Data Science in application an individual receiving a forwarded text message will be able to get it verified.

![1_3PS1jkVdn3MMiI_foRglrg.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_3_PS_1jk_Vdn3_M_Mi_I_fo_Rglrg_ec41c92afa.jpeg)

In 2019, WhatsApp announced that it banned more than 2 million accounts for sending bulk automated messages and fake news per month. This initiative was taken up with the use of machine learning. Using Machine learning we can put a restriction on how many times a specific message can be forwarded including a number of texts.

![1_12SpC1TE3FLPTugehKuJhw.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_12_Sp_C1_TE_3_FLP_Tugeh_Ku_Jhw_d48845b205.jpeg)

## How WhatsApp can use Data Science in near future

**Virtual Assistants**

A Virtual Assistant or Digital Assistant is an application program that understands natural language voice commands and completes tasks for the user. They use Natural Language Processing(NLP) to match user text or voice input to run commands. Virtual assistants are basically cloud-based programs that require internet-connected devices or applications to work with their capabilities. Every time a command is given to the assistant, they tend to provide a better experience to the user based on past experiences using Machine Learning algorithms and learning. Some of the popular virtual assistants include Amazon Alexa, Cortana (Microsoft), Siri (Apple), and Google Assistant.

![1_9UrGpo8RvyxYVsX6Un9Guw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_9_Ur_Gpo8_Rvyx_Y_Vs_X6_Un9_Guw_5ab90c7aa8.png)

Here in our case WhatsApp can use virtual assistants to send and read out the messages by voice commands.

For this WhatsApp may not necessarily build virtual assistant but it can also integrate it with the virtual assistant which is already present in the device operating system ( Ex : Siri on Apple devices and Google assistants in Android devices)

**Search Keyword and show related advertisements**

Although WhatsApp is end-to-end encrypted but it can use some ML models which will not break the security and can come up with some keyword list which can be used for personalized advertisements. They can send the raw decrypted messages back to the WhatsApp servers, do some machine learning on-device, create a local advertising profile tailored to our preferences, and send limited data based on this data so that the security will not breach. As we all know, WhatsApp is a part of Facebook. This means Facebook could know if we are interested in cats without actually knowing the exact content of any of our messages. As per some Journals, WhatsApp may use the facebook data to monetize the platform by showing relevant advertisements with the help of facebook data only, but WhatsApp may monetize this way also.

**They may use multiple filters for video transformation**

As WhatsApp platform is used to interact with people and business will grow if the interaction is higher, so WhatsApp can do something which can help people in interacting more with the help of Deep Fake.

![1_zsk8mIv5JNQQUCieQTZtgA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_zsk8m_Iv5_JNQQU_Cie_QT_Ztg_A_132e7e5d5b.png)

Deepfakes are synthetic media in which a person in an existing image or video is replaced with someone else’s likeness. While the act of faking content is not new, deepfakes leverage powerful techniques from machine learning and artificial intelligence to manipulate or generate visual and audio content with a high potential to deceive. The main machine learning methods used to create deep fakes are based on deep learning and involve training generative neural network architectures, such as auto encoders or generative adversarial networks (GANs).

## Conclusion


- WhatsApp is on top among all other messaging channels with 2.5 Billion users around the globe.
- It is using so many features that help its customers to interact virtually anytime anywhere.
- It uses Data Science for checking fake content, chatbots, and predicting fraud in WhatsApp payments.
- There is a huge scope where data science can be used in integration with WhatsApp without even breaching the security of its users, in near future.


That’s it for this article and we hope you have great learning through this article!!

It feels good to operate with a team that is so self-reliant and motivated. I would like to thanks Bhanu Shahi, Jatin and Rishab Kesarwani for contributing in this article.

Happy learning!1





In this blog, we are going to discuss the system design of WhatsApp. We are going to focus on implementing some key features and will extend our idea further.

0_b0N1Sod_SITKwWKx.gif

WhatsApp System Design

**What is a Random Variable ?**

As the name suggests, it is a term that associates itself to the concept of randomness and uncertainty. There are numerous such incidents that occur in our day-to-day life whose outcomes are difficult to predict with utmost certainty. For example, while playing a game of basket ball, the certainty with which you put the ball in the basket from the half line depends on your expertise in the game. Similarly, you can’t accurately guess the number of likes you might get in your next Instagram post. It has to depend on a wide range of factors such as visual appeal, message in the caption, and your brand/product outreach.

Here the phrase 'number of likes on next Instagram post' can be coined as a random variable. The number of likes on a given post is a variable that can take any random integer value starting from 0 till infinity. Although the probability of you getting likes anything greater than 10000 might be bleak if you are a small brand. The probability of you having 5 billion likes is essentially 0 even if you are the largest brand in the world since the current population of the world is 7.9 billion and more than half of the world population is not on the internet and especially not on Instagram. Similarly, the total number of likes on a decent post not crossing 10 likes might be bleak as well, if your account has at least 100 followers.

![1_uwu-MQyYmnaWxrbJX4wS4Q.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_uwu_M_Qy_Ymna_Wxrb_JX_4w_S4_Q_02ce842230.jpeg)

_Probability Distributions_

We can clearly infer from the above experiment that, the probabilities of number of likes taking certain value widely varies over the entire range of random variables. And this distribution of probabilities over the entire range of values for the random variable is known as Probability Distribution of the random variable.

1. _Discrete Distributions_

These kind of distributions describe the variation of probabilities with respect to discrete number of countable values for the random variable. For example, when we roll a die, we have only six outcomes : 1, 2, 3, 4, 5 and 6. If our random variable is the number appearing on a rolling die, we have only six discrete values for our random variable. And probability of each of those value is 1/6, given its an unbiased die. This type of discrete distribution where all probabilities are same is also known as uniform discrete distribution.

_(i) Uniform discrete distribution_

A uniform discrete distribution comprises of probabilities that are equal for all the values that a random variable can take. Probabilities for the outcomes of rolling an unbiased die is a classic example of uniform discrete distribution, where each of the outcomes 1, 2, 3, 4, 5 and 6 has a probability 1/6 to occur.

If we have k outcomes for a random variable X, probability of getting an outcome x is given by :

![0_TrIuvw4IfCQUN60t.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Tr_Iuvw4_If_CQUN_60t_d0ef95269f.png)

import numpy as np

import matplotlib.pyplot as plt

y = np.full(6, 1/6)

plt.bar(range(1,len(y)+1), y)

plt.ylim(0,1.)

plt.ylabel(‘Probability’)

plt.xlabel(‘Outcomes of rolling an unbiased die’)

![1_1LnAIN-e2dvgJVHkMPZb5A.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_1_Ln_AIN_e2dvg_JV_Hk_MP_Zb5_A_71c7d57950.png)

_(ii) Binomial Distribution_

This distribution is based on the concept of n independent and identical bernouli trials. A bernouli trial consists of two outcomes : ‘success’ and ‘failure’. In a given trial, let the probability of getting a a success in be p and probability of getting a failure is 1-p. We conduct n such trials that are independent of each other. We now want the probability of getting x number of successes in those n trials. The probability is given by :

![0_0Eq67qiupHxPfWvp.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_0_Eq67qiup_Hx_Pf_Wvp_75fc0e4737.png)

def factorial(n):

 if n > 0:
 
 return n*factorial(n-1)
 
 return 1
 
n,p = 100, 0.3

x = np.arange(0,n+1,1,dtype=int)

y = []

for x_ in x:

x_ = int(x_)

y.append( (factorial(n)/factorial(x_)/factorial(n-x_)) * ((1-p)**(n-x_)) * p**x_ )
 
plt.bar(x,y)

plt.xlim(10,50)

plt.show()

A typical binomial distribution with n= 100 and p =0.3, looks like :

![1_iDgWajp4lXo-m_J5sraUjg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_i_Dg_Wajp4l_Xo_m_J5sra_Ujg_d595a394fd.png)

_(iii) Geometric Distribution_

This distribution models the number of trials on which the first success occurs. If probability of success in a given trial is p, the probability distribution of x number of such trials is given by :

![0_Bu7omhllETscYOIM.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Bu7omhll_E_Tsc_YOIM_0dc70aa839.png)

n,p = 25, 0.2

x = np.arange(0,n+1,1,dtype=int)

y = (1-p)**x * p

plt.bar(x, y)

plt.show()

A geometric distribution with p = 0.2 looks like :

![1_qHF3hud8tT9F1A88VxmfKg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_q_HF_3hud8t_T9_F1_A88_Vxmf_Kg_c3b59f6b88.png)

_(iv) Poisson Distribution_

A Poisson distribution is defined as :

![0_jbhKW8tn4jB8c4I1.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_jbh_KW_8tn4j_B8c4_I1_c326b92d0b.png)

where X is the random variable that determines the number of occurrences of certain event in a time interval, and λ is the average number of occurrences of certain invent in a time interval.

lamda = 20

x = np.arange(0,40,1,dtype=int)

y = []

for x_ in x:

x_ = int(x_)

y.append(lamda**x_*np.exp(-lamda)/factorial(x_))

plt.bar(x,y)

plt.show()

A poisson distribution with λ = 20 looks like :

![1_aernZ79oz413QgfVTdF62Q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_aern_Z79oz413_Qgf_V_Td_F62_Q_ffe19ef3b2.png)

2. Continuous Distributions

A continuous distribution is a probability distribution function for a continuous random variable i.e. a random variable with infinitely many possible values between any two random variable values. For example the height or weight of a person can be considered as a continuous random variable, whereas the probability of a person having certain height/weight gives rise to the concept of continuous probability distribution.

_(i) Normal Distribution_

This distribution has lot of applications in statistics. It has two important parameter μ and σ, μ represents mean of the distribution, while σ represents the standard deviation of the distribution. Now the normal distribution is defined as :

![0_Xbe98TNZYsFuDjCG.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Xbe98_TNZ_Ys_Fu_Dj_CG_6c7370a4d8.png)

mu, sig = 10, 5

x = np.arange(0,20,.1)

y = 1/(sig*np.sqrt(2*np.pi)) * np.exp(-(x-mu)**2/(2*sig**2))

plt.plot(x,y,linewidth=3)

plt.show()

A normal distribution with μ = 10 and σ = 5, looks like :

![1_HXuUA-KEtZifQC1ApUFJNw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_H_Xu_UA_K_Et_Zif_QC_1_Ap_UFJ_Nw_13a9399690.png)

(ii) Exponential distribution

Exponential distribution is nothing but the continuous version of geometric distribution. It’s mathematical formulation is given by :

![0_b32cWUtB1awnD96N.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_b32c_W_Ut_B1awn_D96_N_b0ccb1fe69.png)

lamda = 2.

x = np.arange(0,3.,.05)

y = lamda*np.exp(-lamda*x)

plt.plot(x,y,linewidth=3)

plt.show()

Exponential distribution with λ = 2, looks like :

![1_MDTSx5mXfNwmOWiD0JUkKg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_MDT_Sx5m_Xf_Nwm_O_Wi_D0_J_Uk_Kg_a9e64042ff.png)

Now that we have learnt about different kind of distributions, lets try to create a possible probability distribution for the total number of likes for next Instagram post based on our earlier discussion. Suppose you are an average Instagram user with a decent 500 follower base. And you have prepared a quite engaging picture for your next Instagram post. The probability distribution for the number of likes is most likely going to be a right skewed distribution. A right skewed distribution is a distribution with most number of observations in the lower end of values for the random variable compared to those on the higher end. Similarly, a left skewed distribution is a distribution with most number of observations in the higher end of values for the random variable compared to those on the lower end.

from scipy.stats import skewnorm

a = 3

mean, std = 70, 140

fig, ax = plt.subplots(1, 1)

x = np.linspace(0,600, 200)

ax.plot(x, skewnorm.pdf(x, a, mean, std),'r-', lw=5)

r = skewnorm.rvs(a, mean, std, size=5000)

ax.hist(r, density=True, histtype='stepfilled', alpha=0.2, bins=50)

ax.legend(loc='best', frameon=False)

plt.ylabel('Probability')

plt.xlabel('Number of likes in post')

plt.show()

![1_uKe3jMOfbioxTjJZ076UKw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_u_Ke3j_M_Ofbiox_Tj_JZ_076_U_Kw_783cdd3b81.png)

_Cumulative Distribution function_

Cumulative distribution function of a random variable X is defined as the sum of all the probabilities of values of X that are less than a certain value x. In mathematical terms, this can be written as:

For discrete distributions, CDF is defined as:

![0_07HElgo1GEzMWHlp.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_07_H_Elgo1_G_Ez_MW_Hlp_90251e58b6.png)

If f(x) is the continuous probability density function, CDF is defined as:

![0_V64Q2ePpVpQAhfNR.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_V64_Q2e_Pp_Vp_Q_Ahf_NR_a1193b81ff.png)

As x → ∞ , the cumulative distribution function P( X ≤ ∞) →1, which shows the sum of all the probabilities in a probability density function is 1.

_Joint Distributions_

Sometimes the probability density function is not only a function of a single variable but multiple different random variables. For example, P(X=x,Y=y) = f(x,y) is a function of two different random variables. Here f(x,y) is nothing but the Joint distribution function of both variables X and Y.

Similar to what we observed in single random variable, sum of all the probability of a joint distribution also equals 1 i.e.

![0_EydPtmYNk2g9-QVB.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Eyd_Ptm_Y_Nk2g9_QVB_78ec53329d.png)

**Marginal Distribution**

Marginal probability distribution of a variable X is defined as the probability density function with respect to X while summing the probabilities over all other variables (Y).

In case of a discrete joint distribution, marginal probability distribution is defined as:

![0_3cvg9kWezELx5tyQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_3cvg9k_Wez_E_Lx5ty_Q_70cd2828f7.png)

While in case of a continuous joint distribution, marginal probability distribution is given by :

![0_hc0PPvUHP0gkPF0s.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_hc0_P_Pv_UHP_0gk_PF_0s_aed802294e.png)

If two random variables are independent, Joint probability distribution of X and Y can be written as :

![0_AUgC20mJRuApJ4Zp.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_A_Ug_C20m_J_Ru_Ap_J4_Zp_53618da84d.png)

**Inferential Statistics**

Inferential statistics if that domain of statistics which is very useful to determine the population statistic based on a sample statistic. This technique is often used since in most real world cases, the sample size is quite smaller than the actual population and we need to determine the statistics of the population. Lets take an example to better understand the theory behind inferential statistics.

Let X be a random variable. We randomly select a sample of size n from the population. A statistic is something that is a function of n-sized sample. For example the mean and variance of the sample can be called as statistics. Again, a statistic is nothing but a random variable as well. We can randomly select a large number of such n-sized samples from the population, take mean and variances of those samples and try to see their distribution.

![0_j_wIgAijJSMHrnXR.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_j_w_Ig_Aij_JSM_Hrn_XR_cff0121e73.png)

![0_76usR36Wi2chAnO0.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_76us_R36_Wi2ch_An_O0_ddcb1f15e0.png)

_Sample Mean and Central Limit Theorem_

Lets suppose we take large number of samples with n observations from a population having some arbitrary underlying distribution with mean μ and variance σ².

For large enough n, the sampling distribution of the sample mean resembles the normal distribution with mean μ and variance σ²/n, irrespective of underlying distribution.

![0_-fc8QziRk2VwS_03.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_fc8_Qzi_Rk2_Vw_S_03_b90479ada4.png)

![0_urW2I0K2mrkLSXCx.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_ur_W2_I0_K2mrk_LSX_Cx_d22ea7bbcd.png)

This is known as **Central Limit Theorem**

In most cases, σ² in unknown. In those cases we can use t-distribution to approximate the distribution of sample mean. The t-distribution makes use of sample variance S² instead of population variance σ², i.e.

![0_thOsupcEEdwkqh2j.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_th_Osupc_E_Edwkqh2j_0345ed8522.png)

**Sample Variance Distribution**

When we try to look at the sampling distribution of sample variances, we can say that the mean of the sample variance distribution will be equal to μ. But the variance of the sampling distribution of sample variance depends on the underlying distribution of the population. However, if the underlying distribution is a normal distribution, the sampling distribution of the sample variance resembles a χ² distribution as:

![0_W5D3zD_hBv9NzQUr.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_W5_D3z_D_h_Bv9_Nz_Q_Ur_f956693112.png)

**Confidence Intervals**

From the above discussed concepts of sampling distribution, we now will be able to estimate the parameters of the underlying distribution with certain confidence using confidence intervals. A 100(1-α) % confidence interval of the population mean is given by :

![0_odi7sCH2NO5aRbqV.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_odi7s_CH_2_NO_5a_Rbq_V_5125412180.png)

![0_MSJoXH3aCjglfmfR.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_MS_Jo_XH_3a_Cjglfmf_R_2963d58be5.png)

However if the standard deviation σ of underlying distribution is unknown, we can use t-statistic to come up with a confidence interval for the population mean μ, which is given by :

![0_fIoHYekiBfyKOTYP.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_f_Io_H_Yeki_Bfy_KOTYP_6e4c208296.png)

As the name suggests, it is a term that associates itself to the concept of randomness and uncertainty. There are numerous such incidents that occur in our day-to-day life whose outcomes are difficult to predict with utmost certainty.

1_LJmVD3zaPJvlEo__0j_PCQ.jpeg

The Game of Random Variables

Hi Guys, my name is Shirsh and I am a aspiring Data Scientist ,currently I am building my skill sets to excel in this domain , in this blog I will be summarizing all important Statistics Concept which I have learnt till date.

Einstien.png

Statistics Note

 Every time you browse online for either shopping or watching content, a recommendation system is guiding you towards the potential item you might purchase or watch. They are so widely used now that many of us benefit from them without even knowing. We can’t possibly look through all the products or content on a website, this is where the recommendation system comes into play by giving us a better search experience, while also showing us many more products we might not discover otherwise.
 
 In2006 Netflix famously offered a prize to solve a simple problem that had been around for years all online. Commerce giants are fighting to get better at solving this same problem given observations of a user’s past behavior predict which other things that the same user will like, for example, based on a person’s viewing history will she like the matrix or star wars.

![1_vlcXuQffW2E_7XA77Umfpw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_vlc_Xu_Qff_W2_E_7_XA_77_Umfpw_0a8f0765a0.png)

We can represent user preferences graphically as connections between people on one side and things on the other such as movies here, the strength of the connection is visually represented by the thickness of the lines so the problem is to fill out all of these missing connections we don’t yet know shown in red, but because of the messy unpredictable nature of people’s tastes, we can never perfectly predict what they will like because for one it involves a guess about the future based on something you’ve never seen and two the answer is always in flux because people’s tastes change over time. What we can do is try to estimate those values as best we can be using whatever data we have access to. So regardless of the context, we can describe the problem in the following mathematical form.

![1_D-plw760R2YdVWEMyWOeHg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_D_plw760_R2_Yd_VWE_My_W_Oe_Hg_24e3bb697e.png)

We use a matrix which is a collection of numbers organized into rows and columns where the rows are users and the columns are items. In this case movies and each cell in this matrix contains a number that describes how much each individual likes each movie for example we could use a scale from 0 to 5 where a 0 would represent I hate it 2 is neutral and 5 as I love it. At any given moment there will be some set of preference data for users and movies but it will be incomplete because users have only seen and rated a few movies each and so the problem boils down to trying to predict these missing values how should we make these predictions.

**There are two general approaches to this problem the first is called :**

Content filtering is to use the information we know about people in things as connective tissue for recommendations. It’s like asking a person questions about what kinds of movies she likes to figure out if she will like the matrix. It begins by labeling each person and movies with some known attributes or what we call features,

![1_9Yaxe8ZFWqvpEmuizmj9EQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_9_Yaxe8_ZF_Wqvp_Emuizmj9_EQ_64d9540ffc.png)

for example, action and comedy that means if a person likes comedy and hates action we can represent her as 3 0 each movie is also mapped to each feature in the same way, for example, that matrix has no comedy and lots of action so we could represent it as 0 4 and to determine whether someone will like a movie we need to multiply these factors together so we could represent the strength of the connection between the user and the matrix as 3 times 0 plus 0 times 4 equals 0 so our estimation is that she will hate the movie so to make our predictions we first need to gather this feature data for every user and movie to simplify let’s return to a matrix the representation we can store this information in two matrices. One defines the mapping between people and features the other defines the mapping between movies and futures and by multiplying these two matrices together using matrix multiplication we get an estimated strength of the connection between every person and movie and if we’d like we can normalize our data to make sure the scale stays between 0 and 4 by dividing all values by 8 and rounding to the nearest point 4 that’s one way to solve this problem is known as content filtering.

![1_9qDxRdqINlgqXIgBBaWtUA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_9q_Dx_Rdq_I_Nlgq_X_Ig_B_Ba_Wt_UA_97a99f96fb.png)

**The problem is :**

- it’s overly simplistic
- not very accurate

that’s because there are obviously more relevant features of a movie than just comedy or action. The obvious way to improve this is just to include more features if your call when Netflix started it did just this would ask new users to fill out a laundry list of questions about their preferences before presenting them with suggested movies and this leads to the problem of having to collect all of this preference data on users not only is it a burden for users, it’s also prone to failure as we aren’t always great at describing our own preferences. Sometimes we simply can’t explain why we like things we just do which brings us to another approach called

Collaborative filtering: The motivation for collaborative filtering comes from the idea that you will probably like things people with similar viewing habits. This idea was popularized in this 2009 paper by Cohen Bell and Wolinski. So to begin we can throw away the idea of dreaming up features used to connect people and movies. Instead, we flip things around and use the user preference data we do have to generate the features. For example, we might have this incomplete set of preference data and we will instead learn or discover the relevant features based on patterns in this data. This is done by simply reversing the problem.

![1_kZMIJKLgY4P7Mq4ecN_L2Q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_k_ZMIJK_Lg_Y4_P7_Mq4ec_N_L2_Q_8df783ed1f.png)

We first perform an approximate factorization into two matrices and we can do this using a machine-learning approach. The job of the machine-learning algorithm is to guess values for those matrices which will match the existing data and the preference matrix as closely as possible. The simplest approach is to simply guess numbers over and over until you arrive at a set of numbers that predict the data with the lowest error overall. Once this estimation is finished we can multiply the matrices as before to fill in all of the missing values.

It’s important to note that we won’t know exactly what to label these discovered features. So we call them latent features because they arise out of the underlying patterns and the data. You can think of them as an average or weighted sum of the patterns in the data are not based on a human-defined feature such as comedy. That’s the key insight behind this method. With content filtering the features come from the human mind whereas with collaborative filtering the features are extracted directly from the patterns in the data. This will predict the data in the same way but more accurately.

It’s also important to notice that this kind of prediction is actually a form of compression because we can represent a huge matrix of preference data by two smaller matrices of feature data. Once our feature data is determined or learned we can throw away the original data we started with because we can simply multiply the features together to recreate that data. The reason this is possible is that the preference data we started with is not random but follows patterns that exist across people. If instead the preference data was filled with random numbers we would not be able to compress it accurately because there would be no patterns to extract.

This collaborative filtering approach has also been applied to a wide range of contexts such as evaluating policy outcomes before they are implemented in the real world known as synthetic control. For example, if we wanted to understand the effect of gun control or minimum wage increases in some city we could look at the effect in similar cities first which have implemented those policies to approximate the outcome using the same mechanism. So it is the patterns we share that allow this method to work. Recommendation systems based on the ideas are in widespread use today whether it’s movies music news articles or anything else you search for online the things that are recommended to you are based on patterns the machine has observed in other people who are similar to yourself.

But Collaborative Filtering comes with its own shortcomings. One such problem in such kinds of systems is the problem of Cold-Start.

Cold- Start represents in a literal sense a situation when the engine is cold and we were are unable to start it. The user or visitor cold start simply means that when a recommendation system meets a new user for the first time and thus there is a lack of history about her preferences. Getting to know your user is critical in creating a great user experience for them.

**It is faced when we have**

- A new user.
- A new product
- A new recommender system is in place.

Remedies for the cold start problem is :

**When we have a new user :**

1.Start by recommending whatever is newer, cheaper, or popular. Popular items are highly likely to be liked by the user because it is popular and it is easier to build a profile on popular items.

2.Recommend demographically relevant products. For example, if it is a movie recommendation system you can recommend Punjabi songs to a user situated in Punjab.

3.Recommend product according to age, sex, or geography.

4. Use other sources of data like social network data.

**When we have a new recommender system: **

1.Design a Non-Recommender system that captures data to feed it later to the recommender system.

2. Get similar data from other sources.

## Six Examples of Recommender Systems

Recommender systems work behind the scenes on many of the world’s most popular websites. E-commerce websites, for example, often use recommender systems to increase user engagement and drive purchases, but suggestions are highly dependent on the quality and quantity of data that freemium (free service to use/the user is the product) companies already have in place.

**1. AMAZON**

When we buy something or browse anything on Amazon, we see recommended products based on our likes or search results on the page.

![0_eGT8Pezpho9ow6Si.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_e_GT_8_Pezpho9ow6_Si_426ec0b5e1.png)

As we can see above, I may have searched for products related to Sudha Murthy, but Amazon clusters it into segments based on other previous search results. A recommender system often biases the user’s opinion.


**2. IMDB**

When we rate a TV show or movie on IMDb it recommends other shows or movies based on important details like cast, genre, sub-genre, plot, and summary.

![0_crCUJQWlpRd1Onro.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_cr_CUJQ_Wlp_Rd1_Onro_8ad0d74e57.png)

As we can see above, I was recommended to rate Frozen Planet because I’ve watched David Attenborough’s wildlife documentary series. In this case, IMDb suggested this to me based on the cast of the series.


**3. FACEBOOK & INSTAGRAM**

Facebook and Instagram use recommender systems on a wider scale for suggesting friends and stories in the newsfeed.

![0_9cZOY7TI9sZU4aF8.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_9c_ZOY_7_TI_9s_ZU_4a_F8_4fd2e4842c.jpeg)

Often in the newsfeed section, the user is bombarded with articles similar to one another based on the likes and pages they follow. Some argue it creates an unconscious bias among the user, which may be bad or useful depending on how the user interprets something when they have only one side of the argument.


**4. YOUTUBE**

YouTube recommends videos based on previous clicks.

![0_zoQTGYZgfbM9yZYC.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_zo_QTGY_Zgfb_M9y_ZYC_79f1145e08.png)

As we can see above, the first video is about Google. While I’ve seen the video, YouTube recommends it to me again.


**5. GOOGLE**

Whenever we are logged in and search on Google, it is stored in our history and recommends products based on our previous searches. Not surprisingly, more than 95% of the company’s revenue is generated through advertising.

![0_NQjwfpk96WS2Ykh5.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_N_Qjwfpk96_WS_2_Ykh5_dd0a2332e7.png)

![0_0wUSuCnMG-rCrxxp.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_0w_U_Su_Cn_MG_r_Crxxp_367a9593b3.png)

From the above, we can see we were getting targeted with these advertisements based on recent search results.

**6. ANOTHER GOOGLE SUBSIDIARY- GMAIL**

![0_s9FwX7V6vsXNAd11.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_s9_Fw_X7_V6vs_XN_Ad11_8707ad13a5.png)

Gmail’s Sponsored Ad

Gmail recommends advertisements based on my personal inbox and tags they may have generated.

Other examples of recommender systems at work include movies on Netflix, songs on Spotify, and profiles on Bumble.


## To summarise:

In this article we understood how the Recommendation System works and the difference between Collaborative Filtering vs Content-Based Filtering and their working. Both methods have their own set of advantages and disadvantages. We defined the problem of a cold start and saw some ways to tackle it. In the end, we saw some use cases of recommendation systems in big tech giants. Recommendation systems are very widely used in the tech industry, there are many kinds of research going on to develop more features in the recommender. I'll be writing a second part for the codes(building recommender systems)and other technological advancements in this area of machine learning.

Happy learning!







Recommender systems are machine learning algorithms that help customers discover new products and services.

1_RbUFMnB4D2d_aeNigp4fTA.png

Recommender Systems — The most valued application of machine learning.

This is the continuation of my first part where our main focus here is on understanding the important probability distributions and some related underlying concepts behind modelling distributions of data.

Without further ado, let us cut to the chase.

**Important Discrete Distributions**

**Uniform Distribution:**

Let’s consider of a sample space: S = {1,2,3…k}. **Uniform distribution** holds equal probabilities which x can take. The outcomes are equally likely to occur(ex- tossing a coin) and we can equal assignment(1/k) to all outcomes.

Distribution: P(X=x) = 1/k, which is the pdf of X. From this we can infer that more the no of elements , the more is the variance in the sample.

**Bernoulli Distribution:**

A Bernoulli trial is an experiment which has (or can be regarded as having) only two possible outcomes — s (“success”) and f (“failure”).

![1_6h4W8-Jclwfk8JiF39u5bw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_6h4_W8_Jclwfk8_Ji_F39u5bw_0ebe7d83b6.png)

In short, we can say there’s 1 experiment with n trials called the sequence of n independent B trials. For example, tossing 3 coins simultaneously will be a sequence of 3 B trials together and trials are identical that is for each trial P({S})=θ. The trials are independent and identically distributed.

**Binomial Distribution:**

In Binomial distribution, we use two parameters (n,θ) and let S be a sample space which is the joint set of outcomes of n trials. n will be 3 for a sample space of tossing 3 coins simultaneously. The trials are independent of one another, the outcome of any trial does not depend on the outcomes of any other trials. Let’s have a look at the distribution function:

![1_C3gWHs0byNRKKG31NWqiiw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_C3g_W_Hs0by_NRKKG_31_N_Wqiiw_2fb66324f3.png)

**Geometric Distribution:**

In geometric Binomial distribution we usually end the experiment whenever there’s a first success. Here the random variable X is the no of trials on which first success occurs. Imagine of a experiment of tossing a coin whenever a head occurs. The values can be countably infinite since we don’t know when the success will occur and it can go till infinite times.

For X =x, there must be a run of (x-1) failures followed by a success. So, the distribution looks like:

![1_3rNUJF-NKYOoidseUOAVkg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_3r_NUJF_NKY_Ooidse_UOA_Vkg_48a9675c56.png)

**Negative Binomial Distribution:**

It is the special case of geometric Binomial distribution. In negative Binomial distribution the number of trials(n) is not fixed as in normal Binomial. In normal Binomial distribution we see the no of failures that counts before a success. The experiment continues (trials are performed) until a total of r successes have been observed. The PDF of a normal Binomial distribution is given by:

![1_8-B2iKw-9T9JQoO1t_DneQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_8_B2i_Kw_9_T9_J_Qo_O1t_Dne_Q_371b02d3c1.png)

**Poisson Distribution:**

Let’s go with an example, If we know on an average(across years) at a particular cross section, 30 vehicles pass by in a minute. So, what is the probability that 50 or 100 vehicles will pass by in 1 minute (based on past records)? well, this can be modelled by a Poisson Distribution. The distribution of P(X=x) is given by:

![1_IJBH1Rn4sM9aXtyaQUFBsg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_IJBH_1_Rn4s_M9a_Xtya_QUF_Bsg_f13b87d110.png)

Here x can take values till infinite because we don’t know the exact no of occurrence given the past records. λ is the parameter/rate of occurrence of an event. Also note that Poisson distribution is a positively skewed distribution.

**Important Continuous Distributions**

So, for continuous distributions point probabilities are always zero. That is P(X=x) = 0. We’ll use P(X ≤ x) and P(X<x) interchangeably. Please note that: P(X>x) = 1-P(X<x), also P(X<x) = 1-P(X>x). Keep in mind that area under a continuous

Uniform distribution:

The Uniform Distribution (also called the Rectangular Distribution) is the simplest distribution where we have equal probabilities or the probability distribution is uniform in a range.

![1_RCt8U3RrJ3AaETVCmG1QrA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_R_Ct8_U3_Rr_J3_Aa_ETV_Cm_G1_Qr_A_49930c0877.png)

The area of the rectangle is 1 or p × (b−a) = 1. PDF of X following a uniform distribution:

![1_C3h3RCx8NQX2mJ2nSaxF2w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_C3h3_R_Cx8_NQX_2m_J2n_Sax_F2w_4c1a3a0d58.png)

We can also generate random numbers between 0 & 1 using a uniform distribution. Well, this is how the CDF of a uniform distribution will be!

![1_1U9DIugZ5V_h5EsrYYz8WQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_1_U9_D_Iug_Z5_V_h5_Esr_Y_Yz8_WQ_2976e87693.png)

Normal Distribution:

The normal distribution is probably the most important distribution in all of probability and statistics. Many populations have distributions that can be fit very closely by an appropriate normal (or Gaussian) bell-shaped curve. The tails of this distribution doesn’t touch x-axis, instead touches at -∞ to ∞ hovering above the bell shaped curve. Examples can be height, weight, marks scored follows a normal distribution.

A continuous random variable X is said to have a normal distribution with parameters µ and σ > 0. The distribution is symmetrical about µ and the pdf of X is given by:

![1_nKLb4SBqVP1o3-uFu4RQuw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_n_K_Lb4_S_Bq_VP_1o3_u_Fu4_R_Quw_10ebf13689.png)

![1_MBbjCHVjJg7As7KwdulYlg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_M_Bbj_CH_Vj_Jg7_As7_Kwdul_Ylg_bbcac9389c.png)

**Standard Normal Distribution:**

This is a special case of Normal distribution. The normal distribution with parameter values µ = 0 and σ = 1 is called the standard normal distribution. We cannot find the explicit value of X by integrating so the distribution is provided by transforming to a Z score[(X-µ)/σ], then scaled to find the values of X.

P(X ≤ x) or P(X < x)= P(Z< (X-µ)/σ) → CDF of a Normal distribution

![0_T4bpSxJeu2sjr85A.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_T4bp_Sx_Jeu2sjr85_A_3856478b37.png)

The total spread is 6σ which constitutes 3σ on the left and 3σ on right of the mean(µ).

**Exponential Distribution:**

X is said to have an exponential distribution with the rate parameter λ (λ > 0) if the pdf of X is:

![1_vRDuesPDtJQd70KTdun1uw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_v_R_Dues_P_Dt_J_Qd70_K_Tdun1uw_2dc8b65fa2.png)

Here λ denotes the rate of occurrence/waiting time till the next event occurs. This waiting time is modelled by a exponential distribution. In simple words, waiting time between two events. Life expectancy of a light bulb is modelled by an exponential distribution in Poisson process with parameter λ.

![1_TBSAblVOJHlrlFo5sebyUw.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_TBS_Abl_VOJ_Hlrl_Fo5seby_Uw_3612ab4b06.jpeg)

There are many other distributions like Chi-Square, t-distribution, F-distribution and many more. There’s no way bounding ourselves from google search and exploring these vast range of topics. We could have included here but it will be an overflow of this story and already it has been extended much and there’s many new things coming up next. So please don’t hesitate to scroll down below ;)

**JOINTLY DISTRIBUTED RANDOM VARIABLES**

It is basically the generalization of two or more random variables. We consider the typical case of two random variables that are either both discrete or both continuous. In cases where one variable is discrete and the other continuous, appropriate modifications are easily made. Suppose in a survey, apart from collecting the weights of students in a class we are interested to go with the heights of the students as well since both of them possess a relationship while calculating the heights of the students in a class. For a 2D random variable, possible value of [X1,X2] are either finite or countably infinite in number.

_Joint Probability mass function for discrete Bi-variate probability distribution:_

![1_Za34w_AOcV791fkyVBx0wQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Za34w_A_Oc_V791fky_V_Bx0w_Q_09817e3a3b.png)

![1_lYZkqvOqsv8L7ns9wdp5bQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_l_Y_Zkqv_Oqsv8_L7ns9wdp5b_Q_fb48dfe46f.png)

The case where both variables are continuous is obtained easily by analogy with the discrete case on replacing sums by integrals. Thus the joint probability function for the random variables X and Y (or, as it is more commonly called, the joint density function of X and Y ) is defined by:

![1_CnyCFoSjqJaDtwawCTzaYA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Cny_C_Fo_Sjq_Ja_Dtwaw_C_Tza_YA_73acb7c9ee.png)

**Marginal Distribution**

It is defined for each random variable. This is the distribution of X alone without considering the values that Y can take and vice versa. The marginal distribution of a discrete random variable X and Y alone defined by a function g(x) and h(y) is respectively:

![1_qN2K0i4Ahol54jwnni2pig.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_q_N2_K0i4_Ahol54jwnni2pig_19cfbad195.png)

The marginal distributions(Marginal PDF) of X alone and Y alone for a continuous case are respectively:

![1_1cBaqH0SE4OYnhKN-sDgHw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_1c_Baq_H0_SE_4_O_Ynh_KN_s_Dg_Hw_16ece18df6.png)

![1_m9Tj4pLX5jIB8PA6V7FUxg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_m9_Tj4p_LX_5j_IB_8_PA_6_V7_F_Uxg_4a0409ff63.png)

**Independent Random Variables**

The random variables X and Y are said to be independent if for any two sets of real numbers A and B is: P{X ∈ A, Y ∈ B} = P{X ∈ A}*P{Y ∈ B}

When X and Y are discrete random variables, the condition of independence is given by: p(x, y) = p(x)*p(y) for all x, y.

In the jointly continuous case, the condition of independence is equivalent to: f(x, y) = f(x)*f(y)

**Conditional Probability Density Function (Conditional PDF)**

The relationship between two random variables can often be clarified by consideration of the conditional distribution of one given the value of the other. The conditional distribution of Y|X = x is:

![1_kTKavsQ5KXSwWccNScOruQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_k_T_Kavs_Q5_KX_Sw_Wcc_N_Sc_Oru_Q_d16d35ac1f.png)

Similarly, the conditional distribution of X|Y = y is:

![1_R9aePp2G9nP4mFGOEkqSIw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_R9ae_Pp2_G9n_P4m_FGO_Ekq_S_Iw_9875836dd5.png)

Now, if X and Y are jointly continuous, then, for any set A:

![1_HLCImpfjAH2OHQ0toOwPUw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_HLC_Impfj_AH_2_OHQ_0to_Ow_P_Uw_eccefb67b1.png)

**Conditional Expectation for discrete case:**

Most often we are interested in finding out the weight of a person given the height of the person. Generally we want to find the expected value of X given Y. If [X,Y] is a discrete random vector, the conditional expectations are:

![1_M1rJaK0NSdnHZnZjCg0EPQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_M1r_Ja_K0_N_Sdn_H_Zn_Zj_Cg0_EPQ_f9f385e964.png)

**For continuous case:**

![1_HJDbAR8xt7CRG1uDwjj8jA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_HJ_Db_AR_8xt7_CRG_1u_Dwjj8j_A_1c1754edf9.png)

Alternatively this function E(X | y) is also known as Regression of Mean. The graph of this function is called the regression of X on Y or how the expectation of X varies with y.

**Covariance & Correlations**

The degree of association between X & Y determines how associated these variables or the variations are. Covariance determines the direction of linear relationship between 2 variables, whereas a related quantity to covariance is Correlation which determines the strength of the relationship between X and Y. Covariance & Correlation is calculated respectively by:

Cov(X,Y) = E[XY]-E[X]*E[Y] and,

Corr(X,Y) = Cov(X,Y)/sqrt(Var(X)*Var(Y))

Covariance is positive if X ∝ Y, negative if X ∝ 1/Y or vice-versa interchangeably. If the correlation(ρ) comes out to be 1, it signifies that X & Y are 100% correlated. The correlation coefficient takes a value in the range -1≤ρ≤1.

**End Notes**

We are almost on the finishing line. Let me tell you, statistics as a whole is a vast area and there’s no end to this massive sea. We might end up doing a PHD in statistics if we were to master each of its counterparts. In this article, I have tried my best to frame the important ones starting from random variables(discussed in part1), went through some major probability distributions and concepts which one can have to build your toolkit as a statistician and derive meaningful insights!

Thankyou for reading. Please share your feedbacks incase you really enjoyed spending few time over these two parts of my detailed story!


This is the continuation of my first part where our main focus here is on understanding the important probability distributions and some related underlying concepts behind modelling distributions of data.

1_RCt8U3RrJ3AaETVCmG1QrA.png

Probability Distributions in Statistics

MakeMyTrip is an Indian online travel company founded in 2000. Headquartered in Gurugram, Haryana, the company provides online travel services including flight tickets, domestic and international holiday packages, hotel reservations, rail, and bus tickets. With over 47% market share, MakeMyTrip is India’s first and biggest travel company. In fact, one in every four passengers at an airport is their customer. As of 31 March 2018, they have 7 million monthly active users with14 company-owned travel stores in 14 cities, over 30 franchisee-owned travel stores in 28 cities, and counters in four major airports in India. MakeMyTrip has offices in New York, Singapore, Kuala Lumpur, Phuket, Bangkok, and Dubai.

MakeMyTrip is planning to expand its user base to 100 million by making use of the latest tools and techniques in the field of Data Science and Analytics. Driven by the desire to provide an exceptional travel experience for its customers, they have continuously stayed ahead of the curve by developing technology and products to meet the ever-changing demands of the rapidly evolving travel ecosystem. Using data in building one of the best dynamic pricing systems to cater for the needs of the customers by providing a truly personalized experience to each of its users is the key to the MMTs success story.

## Dynamic pricing for a dynamic market

A well-developed ML algorithm can even learn and make pricing suggestions in real-time, dynamic pricing is something that even small retailers and e-commerce players can now use to compete in the retail market. This allows retailers to set product prices based on supply and demand, also known as dynamic pricing. To put it plainly, ML is valuable because it automates a task that is almost impossible for humans to do manually.


![0_DYAhqenBzgrG81Bu.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_DY_Ahqen_Bzgr_G81_Bu_d987685d1f.png)

ML-powered software gets information from data to throw up dynamic pricing solutions. But before that, the retailer needs to know his customers and what data is incoming.

_ML works on a simple philosophy — the larger the data sets, the better the learning process, and the better the outcome. Over time, ML-based software only improves in performance._

Analysts can take other factors into account for dynamic pricing. For example, an analyst could choose weather, demand, operating costs of the company, competition, the minimum price, and the best price, etc.

In the end, for a competitive pricing strategy, ML solutions can repeatedly scrape the web to collect important information about prices set up by competitors for similar products, what consumers’ opinions were about products, as well as the pricing history over the last number of days/weeks.

## How to develop a general dynamic pricing model

The most important aspect that is to consider is the level of granularity you are aiming for. For example, are you look at a single customer level or an entire segments?

![0_hCHpydk__gx7qsVW.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_h_C_Hpydk_gx7qs_VW_827bb66c96.jpg)

Another crucial factor is defining strategic goals that align with business goals. Profit maximizing is obvious, right? But you could also choose goals for getting new customers or satisfying old customer satisfaction.

The ML-based dynamic pricing model can then be developed once the answers to the above points come in. The model will predict whether someone will make a purchase at a price best optimized at that moment in time. The models can be used either using the Generalized Linear Models (GLMs), or the Deep Learning methods.

- **Dynamic pricing based on groups**

Customer segmentation is the practice of dividing the customers base into groups of individuals that are similar in specific ways relevant to marketing. This can be as simple as a split A/B test or more sophisticated by predicting a higher willingness to pay based on machine type, location, demographic information, etc.

![0_za0e415a7vAv59qv.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_za0e415a7v_Av59qv_3ab5519b25.jpg)

Customer segmentation relies on identifying key differentiators that divide customers into groups that can be targeted. Information such as a customers’ demographics (age, race, religion, gender, family size, ethnicity, income, education level), geography (where they live and work), psychographic (social class, lifestyle and personality characteristics) and behavioural (spending, consumption, usage and desired benefits) tendencies are taken into account when determining customer segmentation practices.

MakeMyTrip’s customer segmentation operates under the fact that every customer is different and that their price would be better served if they target specific, smaller groups such that those consumers would find relevant and lead them to buy something. MMT also hope to gain a deeper understanding of their customer's preferences and needs with the idea of discovering what price segment each group finds most valuable to and tailored to more accurately discounts and coupons toward that segment.

MakeMyTrip segregated users into different buckets to understand what they can do differently for different user cohorts and how they can build a more relevant product experience for the users. Each of the users is dissected into the following categories:

- When a user has not historically searched with us
- When a user has searched with us before
- Searched for a flight
- Searched for a hotel
- Searched for a holiday destination
- Searched for more than one i.e. flights and hotels both, hotels and holidays etc.
- When a user has searched and booked with us before and is coming back again
- When a user has an upcoming booking
- If the booking was on the day of travel
- When a user has completed a trip with us


In each of the above scenarios, MakeMyTrip needs to serve a very different kind of information to the user. For example, if a user has not searched with MMT earlier, they present the user with the most relevant content which inspires the user to travel. If a user has already searched, they identify the most relevant information which will aid a user in making a booking. When a user has made a booking, the most important post-booking scenarios which a user needs to be made aware of needs to be highlighted.

**2. Dynamic pricing based on time**

My friend used to always discuss this form of dynamic pricing — having a price go up or down based on time. In its most basic form, you’ll see this purely in a lot of industries— at the end of the month prices are lower as salespeople push for quotas. We see people wait for the festive season to book their holidays as the price usually goes down. Time-based pricing is a pricing strategy in which businesses set flexible prices for products or services based on current market demands.

![0_LcHNPB1Lnhto0edt.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Lc_HNPB_1_Lnhto0edt_66ba0c122f.png)

At MakeMyTrip, one of the core components to make the user experience seamless and immensely relevant is to show the right information to the user at the right time and with the right offers. Most of you would agree that travel booking involves a lot of planning, be it leisure trip or business travel. Depending on where, with whom and under which context one is travelling, the user preferences and the choice varies. MakeMyTrip strives to solve this by considering user preferences, search history, and leveraging their years of travel booking experience.


**3. Price Elasticity of Demand**

price elasticity of demand (PED), describes how the quantity demanded by consumers will respond to a change in price. Dynamic pricing models utilize vast amounts of historical data on demand and prices to determine how the nature of the relationship between these two variables. Generally speaking, price elasticity is determined by factors such as availability of substitutes; the proportion of the consumer’s budget that must be allocated to purchasing the item; degree of necessity; and brand loyalty.

While historical data can, in theory, give an accurate estimate of consumers’ price elasticity, the predictive ability of such estimate is often weak, and it requires more data points than most firms have available to obtain a statistically valid result. However, to increase the level of sophistication, it can be relevant to include an analysis of patterns in price elasticity determinants, to which heuristics and qualitative interpretation can be applied.

For MakeMyTrip, it is crucial to know how high the elasticity of the demand is, in which periods and how the price willingness of the customers behaves depending on it. In simple terms, this means: “How strongly do my potential guests react to price changes?”

Based on that, a well-founded decision can be made about the room price for a specific day/period to exploit the sales potential as much as possible. If the price elasticity is ignored, the hotel may be empty if the price is too high and full if the price is too low, but the rooms will be sold at far too low a price.

Ideally, the hotelier would have to recalculate the price elasticity every day or approx. once a week. Only then would he have the optimal basis to price his rooms correctly daily.

## Benefits of Dynamic Pricing Model

**1. It can be used as a way to boost sales**

Dynamic pricing is frequently thought of as a means for firms to raise their prices.

![1_tVH1_0TOeIDpHdtOBT3qQA.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_t_VH_1_0_T_Oe_I_Dp_Hdt_OBT_3q_QA_e4259f6ce7.jpeg)

Although this is true to a degree, the method may also be utilised to reduce pricing. A reduced price can sometimes spark sluggish sales, allowing a company to reach its sales revenue goals for a day, a month, or even longer. A simple flash sale is one approach to encourage the usage of dynamic pricing at the location.

**2. It can be used to maximize profits**

![0_fPK22zzKbXuyWJZU.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_f_PK_22zz_Kb_Xuy_WJZU_2302eb2604.png)

Dynamic pricing is a method that may be utilised to optimise earnings if rivals are selling goods or services at a much higher price. If you know what potential consumers desire ahead of time, you may change the pricing of products depending on their purchasing habits.

**3. It can create higher levels of demand**

Because empty seats equal zero revenue, dynamic pricing is commonly utilised at events.

![0_PBsFSyFrGPXL83Gl.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_P_Bs_F_Sy_Fr_GPXL_83_Gl_343eb683e5.png)

If seats are still available on the day of the event, certain customers may be able to get them for a reduced price. This helps you to optimise earnings while also allowing you to take advantage of whatever revenues are available at the moment. This procedure can be found at MakeMyTrip.

**4. It provides more insights into customer behaviours**

The demand curve for each client becomes easy to compute with dynamic pricing. This curve depicts the lowest and highest amount a customer is prepared to pay for a certain transaction. This curve is created using a variety of data points, including the device being used for shopping. More insights into customer behaviour may be gained with this additional data, which makes it more likely that a sale can eventually happen.

![0_BQdb9Xmtoy91epCe.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_B_Qdb9_Xmtoy91ep_Ce_6699424547.jpg)

Businesses that use machine learning-based models for dynamic pricing see an improvement in profits of 10% or more, as well as a 10% rise in sales. Dynamic pricing, which adjusts rates in real-time, does this for your company.

**Conclusion:**

Most proponents of dynamic pricing argue that we should simply state that we use dynamic pricing on our site. This is fantastic, but it still doesn’t address the cognitive dissonance of wondering if you’re receiving the best value possible.

It’s worth noting that all of the above solutions address both transparency and the sensation of being left out. That said, while dynamic pricing is unlikely to get us in legal problems, we must be mindful of our brand, image, and any potential PR reaction that might have a long-term impact. We’ve discovered that pricing transparency is critical, and we’ll continue to push for it.

It feels good to operate with a team that is so self-reliant and motivated.  I would like to thanks Md Asif and Prashant Bharadwaj for their invaluable contribution to this article 


**References:**

- https://www.makemytrip.com/blog/tamil/node/21852
- Mobile Application Analytics — MakeMyTrip | by Srijan Rana | MakeMyTrip-Engineering | Medium
- Customer Story: MakeMyTrip — Databricks
- Wisdom of the better few | Proceedings of the fifth ACM conference on Recommender systems
- Dynamic Pricing Guide: How to Implement a Dynamic Pricing Strategy (priceintelligently.com)
- https://www.rateboard.io/en/blog/details/price-elasticity-in-the-hotel-industry
- https://blog.pricebeam.com/using-price-elasticity-in-dynamic-pricing-models

With E-commerce and online travel companies’ activity generating too much data for a team of humans to handle, Machine learning technology is now being used to help these companies make critical business decisions. ML solves this issue because it can process data faster without stopping and delivering the best-in-class end-to-end experience to its users.

1_Gcxwd8XFH3q9pHCT8743Dg.jpeg

MakeMyTrip Dynamic Pricing

**Machine learning** is one of the technology that has become more and more popular with time and machine learning is the subset of the Artificial Intelligence which comes to your knowledge when you are connected to IT industry. Most of the companies like Netflix, Google and smaller companies uses Machine learning algoithms to predict the insights from the data.

Although terms like artificial intelligence, machine learning and deep learning are used interchangeably but, they are not the same thing. Machine learning is the subset of artificial intelligence and deep learning is a subset of machine learning. Here, the key takeaway is:

- Artificial Intelligence deals with imitating human intelligence.
- Machine Learning deals with imitating how humans learn.
- Deep learning deals with imitating how human brain processes data with the help of neurons.

## How Great Personalities Think about Machine Learning?
**Alan Turing’s** vision towards machine learning is being explained in one of his seminal paper such as “ **Machine learning is an application of artificial intelligence where a computer/machine learns from the past experiences (input data) and make future predictions. The performance of such a system should be at least human level**.”

A Computer Scientist **Arthur Samuel** who had experience for very long years in artificial intelligence, has described machine learning as “**the study that gives computer the ability to learn without being explicitly programmed.**”

**Tom M. Mitchell’s** technical definition is explianed as “**A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.**”

Machine learning in todays world is being treated as the emerging technology where all the problem in any of the field can be treated with solution of this technology. In my view machine learning is the concept where we are making the machines to learn from the previous data it may be labelled or not labelled with the foundation of mathematics and statistics.

Types of Learning used by Machine

- **Supervised learning**: This type of learning includes input data with labels which makes machines to identify them with the features which have been already predefined. In this type of learning by machines are supervised.

- **Unsupervised learning**: This type of learning not includes input data with any labels which makes machines hard to identify them. It is hard to tell what is good learning and what is not. Here machines are not supervised

- **Reinforcement learning**: This type of learning are directed of using task oriented algorithms, where learning will be achieved through series of steps.

## Supervised Learning

Machines often learn from sample data that has both an example input and an example output. For example, one data-sample pair may be input data about an list of animals which are labelled and the associated output is the corresponding labelled identified animals. Given enough of these input-output samples, the machine learns how to construct a model that is consistent with the samples it trained on. From there, the model can be applied to new data that it has never seen before — in this case, which animal is given as input. After learning from sample data, the model applies what it has learned to the real world.

This class of machine learning is called “supervised learning,” since the desired predicted outcome is given, and the model is “supervised” to learn the associated model parameters. Humans know the right answer, and they supervise the model as it learns how to find it. Since humans must label all of the data, supervised learning is a time-intensive process.

![a1.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/a1_1dec695a44.jpg)

Supervised Learning can be further classified into two types:

- Regression
- Classification

**Regression**

Regression learning allows us to predict continuous outcome variables based on the value of one or more predictor variables. In simple words, it estimates the relationship between the input and output variables. The goal of regression model is to build a mathematical equation that defines y as a function of the x variables.

**Classification**

Classification methods have a similar goal to regression models. Thus classification is a predictive modeling technique where a label is predicted for a given input. The difference between regression and classification is that the dependent attributes are numerical for regression while they are categorical for classification. In classification the output variable is categorical, that means it can be values like yes-no, true-false, spam-not spam, etc.

**Unsupervised Learning**

In unsupervised learning, the machine learns from data for which the outcomes are not known. It’s given input samples, but no output samples. Unsupervised learning is less common in practical business settings, but it is attractive: you don’t need labeled data and can avoid the human effort and cost of doing so. Unsupervised learning is potentially applicable in many more areas, since it’s not narrowly restricted to applications with labeled data.

![a2.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/a2_fe61372345.jpg)

**Reinforcement Learning**

As we know that supervised learning requires input and output examples. Reinforcement learning is like unsupervised learning in the sense that outputs are usually not given. The central concept of reinforcement learning is based around an “agent” (a computer or robot) that is interacting with an “environment” (here defined as everything that is not the agent). The agent performs actions on the environment (for instance, a robot takes a step forward). Then, the environments will then provide some sort of feedback to the agent, usually in a form called the “reward.” By “reward”, we don’t mean we give the machine a jolt of electrons. We literally just add to the program’s reward counter. The agent’s goal is to maximize the number in that counter. Critically, however, no one is telling the agent how to maximize the reward or explaining why it gets a reward. That’s what the agent figures out for itself by taking actions and observing its environment.

In many forms of reinforcement learning, the agent does not know what the objective is because it does not have examples of success. All it knows is whether it receives the reward or not. Reinforcement learning has echoes of human psychology — the brain experiences something good and a dopamine rush makes a person want more of it. A bad experience, like touching a hot stove, causes pain that discourages the person from repeating the behavior. However, despite the parallels to human psychology, humanizing it too much is a mistake.

**1. Virtual Personal Assistants**

Siri, Alexa, Google Now are some of the popular examples of virtual personal assistants. As the name suggests, they assist in finding information, when asked over voice. All you need to do is activate them and ask “What is my schedule for today?”, “What are the flights from Germany to London”, or similar questions. For answering, your personal assistant looks out for the information, recalls your related queries, or send a command to other resources (like phone apps) to collect info. You can even instruct assistants for certain tasks like “Set an alarm for 6 AM next morning”, “Remind me to visit Visa Office the day after tomorrow”.

Machine learning is an important part of these personal assistants as they collect and refine the information on the basis of your previous involvement with them. Later, this set of data is utilized to render results that are tailored to your preferences.

Virtual Assistants are integrated to a variety of platforms. For example:

- Smart Speakers: Amazon Echo and Google Home
- Smartphones: Samsung Bixby on Samsung S8
- Mobile Apps: Google Allo

**2. Predictions while Commuting**

Traffic Predictions: We all have been using GPS navigation services. While we do that, our current locations and velocities are being saved at a central server for managing traffic. This data is then used to build a map of the current traffic. While this helps in preventing the traffic and does congestion analysis, the underlying problem is that there are fewer cars that are equipped with GPS. The machine learning models in such scenarios helps to estimate the regions where congestion can be found on the basis of daily experiences.

Online Transportation Networks: When booking a cab, the app estimates the price of the ride. When sharing these services, how do they minimize the detours? The answer is machine learning. Jeff Schneider, the engineering lead at Uber ATC reveals in an interview that they use ML to define price surge hours by predicting the rider demand. In the entire cycle of the services, ML is playing a major role.

**3. Videos Surveillance**

Imagine a single person monitoring multiple video cameras! Certainly, a difficult job to do and boring as well. This is why the idea of training computers to do this job makes sense.

The video surveillance system nowadays is powered by AI that makes it possible to detect crime before they happen. They track unusual behaviour of people like standing motionless for a long time, stumbling, or napping on benches etc. The system can thus give an alert to human attendants, which can ultimately help to avoid mishaps. And when such activities are reported and counted to be true, they help to improve the surveillance services. This happens with machine learning models doing their job at the backend.

**4. Social Media Services**

From personalizing your news feed to better ads targeting, social media platforms are utilizing machine learning for their own and user benefits. Here are a few examples that you must be noticing, using, and loving in your social media accounts, without realizing that these wonderful features are nothing but the applications of ML.

- People You May Know: Machine learning works on a simple concept: understanding with experiences. Facebook continuously notices the friends that you connect with, the profiles that you visit very often, your interests, workplace, or a group that you share with someone etc. On the basis of continuous learning, a list of Facebook users are suggested that you can become friends with.

- Face Recognition: You upload a picture of yourself with a friend and Facebook instantly recognizes that friend. Facebook checks the poses and projections in the picture, notice the unique features, and then match them with the people in your friend list. The entire process at the backend is complicated and takes care of the precision factor but seems to be a simple application of ML at the front end..

- Similar Pins: Machine learning is the core element of Computer Vision, which is a technique to extract useful information from images and videos. Pinterest uses computer vision to identify the objects (or pins) in the images and recommend similar pins accordingly.

Machine learning is one of the technology that has become more and more popular with time and machine learning is the subset of the Artificial Intelligence which comes to your knowledge when you are connected to IT industry. Most of the companies like Netflix, Google and smaller companies uses Machine learning algoithms to predict the insights from the data.


0_urFGgT5PpHqbQcDK.jpeg

Machine Learning: Makes Human to Train Them

Contributors : F M Nurul, Rohan Roney, Smruti Ranjan, Roopesh Valluru


### What is E-learning ?

The Internet has become one of the vital ways to make available resources for research and learning for both teachers and students to share and acquire information. Technology-based e-learning encompasses the use of the internet and other important technologies to produce materials for learning, teach learners, and also regulate courses in an organization. Typically, e-learning is conducted on the Internet, where students can access their learning materials online at any place and time.

### Why is E-learning important?

- It is flexible when issues of time and place are taken into consideration. Every student has the luxury of choosing the place and time that suits him/her. The adoption of e-learning provides the institutions as well as their students or learners the much flexibility of time and place of delivery or receipt of according to learning information.

- E-learning enhances the efficacy of knowledge and qualifications via ease of access to a huge amount of information.

- It is able to provide opportunities for relations between learners by the use of discussion forums. Through this, e-learning helps eliminate barriers that have the potential of hindering participation including the fear of talking to other learners. E-learning motivates students to interact with other, as well as exchange and respect different point of views. E-learning eases communication and also improves the relationships that sustain learning. E-Learning makes available extra prospects for interactivity between students and teachers during content delivery.

- E-learning is cost effective in the sense that there is no need for the students or learners to travel. It is also cost effective in the sense that it offers opportunities for learning for maximum number of learners with no need for many buildings.

- E-learning always takes into consideration the individual learners differences. Some learners, for instance prefer to concentrate on certain parts of the course, while others are prepared to review the entire course.

- E-learning helps compensate for scarcities of academic staff, including instructors or teachers as well as facilitators, lab technicians etc.

- The use of e-Learning allows self-pacing. For instance the asynchronous way permits each student to study at his or her own pace and speed whether slow or quick. It therefore increases satisfaction and decreases stress

### Shortcomings of E-learning?

Let us look a some of the shortcomings of E-learning :

- E-learning as a method of education makes the learners undergo contemplation, remoteness, as well as lack of interaction or relation. It therefore requires a very strong motivation and time management skills in order to reduce such effects.

- With respect to clarifications, explanations, and interpretations, the e-learning method may be less effective that traditional methods of learning. The learning process is much easier face-to-face with instructors or teachers.

- When it comes to improvement of learner’s communication skills, e-learning may have a negative effect. Though learners might have an excellent academic knowledge, they may not possess the needed skills to deliver their acquired knowledge to others.

- Since tests and assessments in e-learning are frequently supervised by proxy, it may be difficult, if not impossible, to control or regulate activities such as cheating.

- E-learning may also be subject to piracy, plagiarism, cheating, inadequate selection skills, and inappropriate use of of copy and paste.


Despite the shortcomings, it is essential to use the data analytics and machine learning tools to improve the state of e-learning by personalizing content and resources to students’ needs and progress. ML has to intervene so as to make content of the e-learning platforms more fun and interactive, using the insights obtained by analyzing past user-content engagement data. With the onset of pandemic worldwide and increasing commuting restrictions, there has been an increasing demand to improve the standard of imparting online education on a mass scale. Ed-tech companies now need to put extra efforts in building their trust among students, as more and more students are going online for their studies.

### Impact of ML in E-learning

![0_Dcn4Xi2KMdHxf7ln.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Dcn4_Xi2_K_Md_Hxf7ln_91f0b5e404.png)

Machine learning for education. So, what does it bring? It is already changing the global landscape of online education. As per the research report released by Technavio, the global market of online courses based on machine learning technologies will grow with the CAGR higher than 16% by 2022! This rapid growth is bound to happen due to affordable large volume data storage and its efficient processing. From the perspective of making the e-learning process more engaging and result-driven, machine learning and artificial intelligence are already turning it into a more convenient way of getting the knowledge as compared to usual courses or corporate training. What are the features that they can offer to both sides of the process — students and universities?

### Learning process customization and personalization

Machine learning and education processes are closely interconnected. ML algorithms analyze how the students perceive and explore the information they are given. It helps the system to either draw the user back and go through some learning points again or let them step further. ML also helps the teachers to monitor and trace the learning process individually. As compared to traditional methods in classrooms, where the goal is to deliver the course, but not ensure that everyone got it, ML gives an advantage of more profound information perception. This type of learning is offered by Edtech and MagicBox learning systems.

### All-round content analysis

![0_eCMGv0QFjzDAS4Ov.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_e_CM_Gv0_Q_Fjz_DAS_4_Ov_a6f058a6e9.jpg)

Using machine learning in education helps to move the industry of online learning to the new level by making the content more up-to-date and relevant to an exact request. How? ML technologies analyze the content of online courses and help to figure out whether the quality of the offered information meets the applicable standards on one hand, and on the other, it shows how the users perceive the data and do they understand what is taught. Hence users receive the information under their personal needs and abilities, and the overall learning process improves greatly.

Have a quick look at Udemy, one of the world’s largest educational marketplaces, which connects students and instructors and offers over 150K different courses. It boasts a great feature of personalized recommendations. By answering several simple questions, its users get an optimized set of different courses and relevant content upon their request. The feature is a good contribution to platform usability.

![0_5OqICCv_7vpOVjjM.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_5_Oq_IC_Cv_7vp_O_Vjj_M_d3e058f8ec.jpg)


There is another application of machine learning in education that deals with scores and grading. As each online course reflects the learning abilities of a large number of students, they have a lot of experience in grading them. ML technologies turn the grading process into a couple of seconds issue. We’re speaking more of the exact sciences in this perspective. There are spheres, where machines cannot replace teachers, but even in those cases, they can help to improve the existing grading and evaluating approaches.

In addition to that, applying machine learning in education today is the easiest way to detect cases of plagiarism. The application of ML technologies in this sphere turned out to be highly revenue-generating. Would you believe that a company that uses ML technology to reveal plagiarism costs $1.735 billion? A year ago a project called Turnitin was purchased by a privately held media company exactly at this price.

### Personal e-learning courses

Here both machine learning and artificial intelligence help to give each student a personal course. It is compiled based on the exact user’s request — rather a time-saving solution! The program gives precise answers to the users whenever they need support.

As still machines cannot replace humans, this kind of studying is applicable only for those disciplines, where there are exact answers and it will not help to explain some philosophical questions, for example. But the amount of data it can offer to precise requests is astronomical! Have a glance at the Wizcabin project, in which the e-learning automation tool converts its users’ storyboards automatically into interactive courses.


### Time-consuming tasks automatization

Administrative work is extremely important and extremely boring. Let the machines do this kind of task! ML helps to automate scheduling and content delivery processes and free your time and resources for other, more engaging parts of e-learning process development.

### What are the Challenges Faced by Artificial Intelligence and Machine Learning in Education?

The use of machine learning for education enhancement and optimization leads to outstanding results! But there are not only benefits but also serious risks of machine learning and artificial intelligence application in e-learning. If these technologies could solve all the problems and replace the traditional system of education, it would already have happened. What are the problems of ML and AI in the education sector today?

**1. Prediction accuracy**

![0_RTHzwjfdcufitCi6.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_RT_Hzwjfdcufit_Ci6_b2ea62d9d9.png)

Machine learning for education algorithms analyze the interests of online students, make predictions about their further needs, and give recommendations on courses. The statistical tools here cannot always reflect real needs. Let’s imagine, we see a student, who sits in a noisy place like a cafe or some workspace, and he chooses a text format of information delivery, as right at the moment it suits him better. We cannot say that he would make the same choice at home.

Thus, the student’s preferences are not that vivid for machine learning and prediction scenario is not 100% accurate. The reason for that is that ML doesn’t have enough data for use. To make predictions, people use information that they see every day, remember from the past, dealing with it during some period. To make this information digitally accessible, we need to collect and describe almost every moment of a student’s life, which seems to be impossible in today’s world.

Furthermore, it is not that easy to recommend a course and be sure that you offer the content that is really needed. Quite often learning management systems rely on the student’s job title and role, the department he\she works in and some skills they list. Their existing experience and the true level of knowledge are not always reflected correctly by a machine and here such things as instructor observations and feedback from peers and managers cannot be done by a machine with the same accuracy a human can render.

**2. Human factor and old-fashioned approach to e-learning**

The methodologies used by teachers and professors are highly adaptive when used in a classroom. When it comes to the remote educational process, we face an issue of behavior patterns and individual approach. The real-life process allows us to see how two different people perceive the same data and amend the approach if needed.

IT systems can measure the progress of each student and correct the learning plan, but they cannot invent the methodology on their own, and in some cases the approach that ML and AI are taught is not the best and rather old-fashioned.

In future this issue can become a great driver of methodologies further development as many non-vivid problems of the present system will become obvious and IT systems will offer even greater education methodologies. Until it didn’t happen, the human factor here is an issue.

**3. Adaptive learning development costs**

![0_EV-KFQrH__rAg56i.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_EV_KF_Qr_H_r_Ag56i_55ea9a46cb.jpg)

This is the model when machine learning is used to run the interaction between student and e-learning platform, analyze the unique needs of each user and offer customized resources and learning activities. No doubt this is a great technology and in the nearest years it will be more widely introduced by the key e-learning web platforms.

But as for today, the development of this approach is quite costly and time-consuming. The most expensive part here is the granularity of content. This system requires constant updating and monitoring, as the developers need to understand how deep the information adaptation of each learner should be — curriculum, course, or model level? Rules and predictions also need continuous attention of developers, and these things altogether form quite a big budget.

**4. Data cleansing**

Speaking about technological challenges, it’s worth mentioning the problem of data cleansing. The process of figuring out which data is incorrect or unnecessary and its removal is crucial for making the educational content relevant and up-to-date.

But which data and in which volume is needed to improve the learning process? What are the criteria for its relevance? The methodologies are still not perfect and today they cannot replace humans in these processes.

### Examples of ML Applications in Education

Moving forward from descriptions to examples, let’s check it out, how are machine learning and education interconnected? Who is creating these applications and how do they work?

- Content Technologies, Inc. — one of the world’s leading AI and ML technologies developers that render solutions for business processes automation and intelligent instruction design, is offering two platforms that are interesting to look at. The first one is called Cram101. The web platform uses ML to segment the content of textbooks and outline the most useful parts of it, create study guides, make chapter summaries, and multiple choice practice tests.

- Quite a similar service is offered by JustTheFacts101 service. It also highlights the most significant parts of the text and makes the summaries of chapters. Then it archives the data and makes it available on Amazon.

- The e-learning platform called Netex Learning makes it possible for the educators to create digital curriculum, and offer different content across platforms and integrate different forms of information delivery — video\audio\texts. The company has also invested in the development of personalized learning cloud platform, that is applicable in the workplaces, where employers can design customizable learning systems with apps; gamification and simulations; virtual courses; self-assessments and others.

- A company called OTTER is offering a great service based on AI. They’ve developed an application that allows users to set the keywords in advance of recording a class or a course. It’s great to support for getting exactly what you need in a nice summary.

### Companies That Use Machine Learning for Education Projects

Machine learning is widely used in many spheres. Let’s have a quick look at a couple of examples of how it is applied in education projects by businesses.

- A Swedish EdTech company called Hubert.ai has developed an assessment system that they call a cognitive assistant. Its goal is to turn the surveys into speaking tests by asking students follow-up questions. It uses machine learning algorithms to make a deep text analysis that automatically categorizes feedback. It helps to make an assessment in education much more engaging and effective.

- Getting ready for school examinations becomes much easier with ML and AI technologies. A project called MobyMax uses these technologies to compile a highly adaptive curriculum for K-8 subjects.

- It provides personalized lessons for each student based on their individual needs and helps to fix learning gaps. The users have their personal lesson plans and practice sheets. The application also features different contests, certificates, games, and daily smiles to motivate students not to stop their studies.

- A popular education tools platform called Quizlet Learn has developed a service for students based on machine learning algorithms, that help not only to study but also to prepare for different tests. It does this by creating tailored study plans based on current knowledge levels and upcoming test dates. The core of this algorithm is books and paper summarization and the creation of smaller study guides for textbooks and revision materials.

### Conclusion

The most competitive feature of any educational project today is the ability to offer customized content in an engaging format with personal guidance and support. Some 10 or 15 years ago, which is nothing for the conservative sphere of education, the online world wouldn’t be capable of offering this kind of education.

Massive implementation of machine learning in education has changed this situation and it has completely replaced humans in many spheres of the learning process. It’s not only better data processing that they can do but a real enhancement of the e-learning industry. The content becomes more relevant, place, and time do not matter anymore, students have virtual assistants that guide them all way long.

The capabilities of these technologies and their potential make them the most significant tools of transforming the whole e-learning industry and bringing the tomorrow of e-learning closer!

Our digital product development team knows how to make these technologies work for you and turn your e-learning project into the most annoying competitor to the websites that you benchmark today!

The Internet has become one of the vital ways to make available resources for research and learning for both teachers and students to share and acquire information.

0_ydj-GxepU8baci_v.jpg

Machine Learning in Ed-Tech

Before beginning, I would like to thank my co-authors of this blog Rajat Pal, Shubham Sharma, Shubham Kumar for their contribution.

One day my mother asked me to accompany her to the hospital where she was receiving treatment for her knee issues. I attempted to ignore her, assuming that it would take a long time because we would have to wait in line to meet Dr. once we were at the hospital. I had no choice but to accompany her. I went to reception to register, and as I was going to ask for a registration slip after sharing the information, I received a text from the hospital confirming my registration with the token number.

While waiting for our turn, I noticed my mother looking through her bag for something, and upon inquiry, I found she was looking for a test report. We heard our token number called while looking for reports, and I thought the whole day was a waste because we had to come back tomorrow with reports, but we went to see the doctor because our turn had come. He began his normal query with the registration number, and while looking at the laptop, he stated that our reports appeared to be in order. He wrote a prescription and instructed us to pick it up from the pharmacist/reception. The utilization of computers and networking in the hospital made me both thrilled and amazed.

As data science enthusiasts, we were intrigued by the amount of data generated by the hospital and wondered how we could make use of it. When I got home, I started looking into how machine learning enhances the efficiency of the health sector, and I was amazed by the work done.

The massive amounts of data generated from Electronic medical records, billing, clinical systems, data from wearables, and a variety of research projects continue to generate massive amounts of data. This creates a significant opportunity for healthcare practitioners to improve patient care by leveraging actionable insights from past patient data. Of course, data science is the driving force behind it. Data scientists all over the world are steadily transforming the healthcare industry with superior machine learning and analytics. They’re trying to optimize every area of healthcare operations by tapping the potential of data, from increasing care delivery to achieving operational experience.

Here is the broad classification of health care where ML is found very useful and is emerging day by day.

1. Use of ML for doctors and hospitals.

2.Drug discovery and clinical trials.

3.Diagnosis for predicting disease on basis of symptoms.

4.Diagnosis for improvement.

5.Recovery analysis

![1_x_qt2Pee0yD-l4aJIOLuKA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_x_qt2_Pee0y_D_l4a_JIO_Lu_KA_e1698ad498.png)

**In a country like India where it became difficult to get doctor appointments. In this case, how does Data science is used?**

According to the Indian government, there is one doctor for every 1457 persons in the country. So doctors are unable to maintain a healthy work-life balance as a result of these factors, which make life difficult for them. We’ve also come across many articles in which doctors have committed suicide as a result of work stress. On the other side, we hear about people who are suffering because there aren’t enough doctors to care for them. Getting an appointment at AIIMS usually takes 20–25 days. Many people mainly from rural areas travel a long distance only to obtain a better prescription.

We can use the internet to increase the reach of doctors now that it has reached every corner of the country. Many platforms are now available via which you can contact any specialized doctor and obtain a prescription while sitting at home.

- Practo

- Doctor24X7

- DocsApp

- Curofy

![0_DI68oCRzfavVeau_.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_DI_68o_C_Rzfav_Veau_8f6d6fa4f8.jpg)

Using these platforms, also solved the problem of information loss. Patients’ prescriptions and diagnoses are documented, which is useful for future treatment. Now doctors can better comprehend their patients if they have a thorough record of their disease therapy.

Patients have the option of receiving therapy from a professional on these platforms. On the other side, it aids doctors in better managing their patients. This may also assist us in lowering the costs associated with treatment.

With the rise in the number of fitness bands and smartwatches, we can now collect a huge amount of data such as heart rate, spo2 (which measures the oxygen level in the human body), temperature, sleeping pattern, and so on. Combining these data with a person’s location, height, weight, diet, and habits can help us to prevent disease.

PAI(Personal Activity Intelligence) is functionality that comes standard in smartwatches. The PAI score is computed based on our heartbeat, and if we can maintain a high PAI number, it can help us reduce our risk of cardiovascular disease.

Rapid economic development is causing a lot of environmental changes and unhealthy lifestyles, increasing the number of people suffering from critical sickness conditions like Cardiovascular disease, Lung ailment, Hepatitis is a disease of the liver, Cancer, Kidney Illness.

We can predict these diseases at an early stage using machine learning, which saves a person’s life and money.

**How does drug discovery happen and how does ML is helping in drug discovery and labs.**

![0_qa7yGIVlYZ05ysT7.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_qa7y_GI_Vl_YZ_05ys_T7_c7e5b29ef6.jpeg)

We all know that the volume of laboratory testing is growing at the same rate as the demand for healthcare. Many businesses like BenevolentAI, Airamatrix are now getting involved in different aspects of the treatment process, from diagnosis through therapy and drug development with the help of ML.

Here, I will talk about Drug research and development using ML.

Have you ever wondered how a drug is coming to market? Can you even guess the time it could take generally for a drug to reach us? It takes roughly 10 years for the drug to come into the market after starting the process of discovery. It’s almost 1/4th of career time for an individual. It’s a long process.

Not only its a time taking process but also corporations invest billions of rupees In drug research to clinical trials. In the field of drug development, computational solutions can dramatically cut the cost of bringing new medications to market. By 2028, according to Bekryl, AI has the potential to save $70 billion in the drug research process. As biomedical data is so complicated, utilizing algorithms to develop novel medications is now more feasible than ever.

There are many steps of the drug discovery process that can be aided by machine learning: Here are two important steps

1. ML is used in designing the chemical structure of medicine.

2. Examining a drug’s effect — both in fundamental preclinical research and clinical trials, which generate a large amount of biological data. Machine learning can help in the discovery of new patterns in your data.

![0_71WRU-XP_f5mMIof.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_71_WRU_XP_f5m_M_Iof_3ee053792d.png)

_Now let’s deep dive into how Machine learning plays a role in the drug discovery process._

Researchers want to find compounds that can interact with a promising therapeutic target to create desirable biological effects once the target has been discovered. For hit compound(molecule that exhibits the desired type of activity) discovery and the subsequent hit-to-lead selection process, it is critical to construct pharmacologically relevant screening assays. By refining the screening criteria, lead compounds are picked from a pool of hits, allowing for the selection of the most promising molecules for further development.

QSAR(quantitative structure-activity relationship) is labor, time, and cost-effective strategy for obtaining compounds with desired biological properties because no substance needs to be manufactured or evaluated before computational evaluation. As a result, QSAR is widely used in companies, universities, and research institutions all over the world.

QSAR modeling has become increasingly important in choosing compounds for synthesis and/or biological evaluation. The QSAR models can be used to identify hits as well as optimize hit-to-lead conversion. Several optimization cycles could be used to create a desirable balance between which are essential to developing a new, safe, and effective medicine.

Previously modeling of the QSAR was restricted to a small number of congeneric chemicals and simple regression approaches. Nowadays, QSAR modeling has expanded, varied, and evolved to include the modeling and virtual screening (VS) of very large data sets containing thousands of different chemical structures and the use of a wide range of machine learning approaches.

A good quality data collection is required to develop a superb QSAR model utilizing artificial intelligence.

These are some of the factors that could influence medication development:

· Structure variables: shear rate, particle sizes, molecular weight, Melting point, Quantum based features, etc.


· Particle variables: adhesive strength, resistance, valency, hybridization, etc.

· ADME properties: Absorption, distribution, metabolism, and excretion Other qualities, such as toxicity, environmental conditions can be utilized in specific instances.

You’ll need to examine the data and construct a model to design or optimize rational drugs after you’ve built the data collection.

![0_Lp5dd6z8dkyzke7q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Lp5dd6z8dkyzke7q_80d25f071a.png)

That is why QSAR predictive models are built using AI approaches. It has been shown to outperform existing methods in terms of predicting a molecule’s activity from its physicochemical characteristics while also lowering costs.

You can also execute a feature or input selection to extract the features that have the greatest impact on your model.

To be more specific, neural networks have been employed to develop predictive QSAR models in recent years.

Because ANNs do not require a prior understanding of how input and output are connected, they may swiftly adjust to non-linear and linear relationships.

It can also teach you how to use statistical methods to investigate the properties of molecules, reduce the dimensionality of the data set, create a prediction model, and test it against real-world data to determine its accuracy.

![0_TEP-XUyB0-liY0MO.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_TEP_X_Uy_B0_li_Y0_MO_ca0437aaaa.png)

Machine learning has been attempted by scientists and engineers from research institutions and pharmaceutical businesses such as Roche and Pfizer to extract useful information from clinical data acquired in clinical trials. The interpretation of this data in the context of medication safety is a hot topic in academia right now.

Clinical trials are also the most costly part of the medication development process. It is critical to apply the experience gathered during past clinical trials in the early stages of medication development to lower expenses. This is accomplished in two steps:

Machine learning could be used to examine and understand biological data from research experiments to forecast a drug’s effects and adverse effects.

The interpretation of biological data should be aided by data from clinical trials evaluated with machine learning.

It is conceivable to design superior preclinical tests to come up with the most effective medicines with the fewest adverse effects if those two approaches are developed concurrently.

**How Machine Learning is helping in Diagnosing Diseases?**

What will be your reaction if an alien invaded our planet and blinded 45% of people that can be saved in India alone and killed many more? panic? Plan of action? Well, something of that sort is happening right under our nose and we completely ignore it. In our case, it’s not aliens but diseases that are preventable causing havoc in the lives of underprivileged and needy people but we have a superhero disguised in form of Machine Learning/Deep learning that’s there for our rescue.

In India alone, there is a shortage of 127,000 eye doctors, and of all the patients that go blind 45% of them suffer from vision loss even before diagnosis.

So here comes our Avenger to the rescue named as ‘Inception Neural Network’ that was built in collaboration with Google and Indian Doctors Which produced an overall staggering 0.95 F-1 Score, it performed even better than highly-trained doctors when it came to diagnosing a disease. Here are some reviews that this artificial neural network received in its first attempt —

_“The study by Gulshan and colleagues truly represent the brave world in medicine” -Dr. Andrew Beam( Harvard Medical school)._

_“Google just published this paper in JAMA and it actually lives up to the hype” — Dr. Luke Oakden-Rayner ( University of Adelaide)._

Apart from diagnosis, another question that might come up is — “Can you show exactly where the disease is?” and our hero never disappoints and uses the power of HeatMap to show exactly where the issue is.

![1_estiM5Bc6-yI4CqqAZa0Ew.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_esti_M5_Bc6_y_I4_Cqq_A_Za0_Ew_783a31f4a7.jpeg)

We were successful in diagnosing a preventable disease that can cause blindness but is it limited to only that? Not at all — there are several others like Breast cancer, tumors, and other types of cancers that can be prevented if diagnosed on time.

So these were discussions of diseases that a Doctor can detect as well ( although with less accuracy) but the problem here is a shortage of those doctors. Machine Learning being our savior cannot just possess normal powers it has to be Supernatural and guess what? it does have those. How? Well, it might not be a shocker but it can even predict only through images what our most highly skilled Doctors cannot! and in certain cases, it was able to predict diseases before they happen!! and not to mention the accuracy and availability of them is tremendous.

What’s the best thing with technology you can do? it certainly is Saving a life and the best use of that technology in recent times should go to our healthcare industries to ensure we indeed have a brighter future.

**How does data science is improving diagnosis to decrease failure rates?**

![0_U6rjkgVpjFgx5NAm.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_U6rjkg_Vpj_Fgx5_N_Am_c6d1f29ac5.jpg)

Diagnostic failure rates remain high despite having access to large amounts of health data. According to the National Academies of Sciences, Engineering, and Medicine, around 5% of adult patients in the United States are misdiagnosed each year. More than 12 million persons are included in this figure. Furthermore, postmortem investigation results suggest that diagnostic errors are responsible for about 10% of patient deaths.

Machine learning is being used in hospitals to speed up the diagnosis and analysis process. Doctors can now seek assistance with the diagnosis so that they can move on to the treatment phase as quickly as possible.

Using machine learning to analyze diagnosis reports or CT images, finding anomalies or even cancerous cells can be done with superhuman precision.

Machine learning is beneficial in oncology, pathology, dermatology, and other fields since it improves the precision and accuracy of the results. The amount of time it takes to complete analysis has also dropped, allowing lab personnel to work more efficiently.

Reviewing machine learning data allows clinicians to better assess the severity of a condition, which helps to speed up therapy. This also assists doctors in prioritizing patients and increasing the success rate of disease prevention.

Machine learning also aids in the review of the prescriptions issued to patients; it will compare the prescriptions to those provided to other persons with similar symptoms and will issue a warning if the doctor has recommended another medication or if a valuable medication has been ignored.

How hospitals are using data science to improve patient care and experience?
Hospitals gather data from patients who have visited their facility and ask them to rate the medical staff, services, and overall experience. They can come up with more information after analyzing the data provided by the patients.

**Obstacles :**

1. The datasets are insufficiently large.

2. Inconsistent labels

3. The characteristics are subjective and do not have universal acceptance.

4. Laboratory medicine is undergoing a digitization and automation phase.

**Conclusion:**

Medicine is entering a new era thanks to data science and machine learning. It’s fascinating to consider where it could go. Soon, having incorporated machine learning knowledge that assesses patient status in real-time, what’s going on with similar patients in other healthcare systems, what suitable clinical trials are underway, and the efficacy and cost of new treatment alternatives will be extremely frequent.

Opportunities and therapies that were once just a concept are now becoming a reality. We live in the age of machine learning, in which algorithms can help us prevent and treat diseases by evaluating data and assisting clinicians in making better judgments.

Machine learning is providing a lot of new capabilities to doctors, nurses, and healthcare staff, allowing them to better care for their patients. “The secret of patient care is in caring for the patient,” declared Dr. Francis Peabody many years ago. While machines take on additional jobs, humans can focus on what they do best: caring for others and assisting them.

_That's the end of this blog…Thanks for reading our blog. Hope it is found to be informative. If you have enjoyed do give a clap…Happy learning !!! Also do let us know your ideas on ML in health care……_

_Reference links :_

1. https://deepsense.ai/machine-learning-in-drug-discovery/

2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6262347/

3. https://www.neuraldesigner.com/solutions/drug-design

4. https://www.sciencedirect.com/science/article/pii/S2001037019303988

In this blog, We have taken case studies where Machine learning is playing a vital role for people and also the business.

0_LDOv5uJtrqvKglpt.jpeg

Machine Learning Case Study in Health Care Industry

## What are some of the applications of NLP?

1. Grammarly, Microsoft Word, Google Docs

2.Search engines like DuckDuckGo, Google

3.Voice assistants — Alexa, Siri

4.News feeds- Facebook, Google News

5.Translation systems — Google translate


## Why text preprocessing ?

Computers are great at working with structured data like spreadsheets and database tables, but we humans usually communicate in words, not in tables. Computers couldn’t understand those. To solve this problem, we have to come up with some advanced techniques. In NLP, we use some very smart techniques that convert languages to useful information like numbers or some mathematically interpretable objects so that we could use them in ML algorithms based upon our requirements.

Machine Learning needs data in numeric form. We first need to clean the textual data and this process to prepare(or clean) text data before encoding is called text preprocessing, this is the very first step to solve the NLP problems. SpaCy, NLTK are some libraries used to make our tasks of preprocessing easier.

## Steps involved in preprocessing:

**Cleaning**

1.Removing URL-

![1_8iZpGdPTldV7Ki_E5mVYdQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_8i_Zp_Gd_P_Tld_V7_Ki_E5m_V_Yd_Q_bcf682aa6d.png)

# Importing re library to remove url.
import re

def clean_url(text):
  return re.sub(r'http\S+, "", text)
dataset['Clean_text']=dataset['Review'].apply(clean_url)

2. Removing punctuations and numbers . Punctuation is basically the set of symbols [!”#$%&’()*+,-./:;<=>?@[\]^_`{|}~]:

def clean_punctuations(text):
  return re.sub('[^a-zA-Z]',' ', text)
  
  dataset['Clean_text']=dataset['Clean_text'].apply(clean_punctuations)
  
 ** 3. Converting all to lower case**
 
 def clean_lower(text):
  return str(text).lower()
dataset['Clean_text']=dataset['Clean_text'].apply(clean_lower)

**4. Removing stopwords**

import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords
stop=set(stopwords.words('english'))
def clean_stopwords(text):
 return ' '.join([item for item in text if item not in stop])
dataset['Clean_text']=dataset['Clean_text'].apply(clean_stopwords)


**5. Tokenization-**

It’s a method of splitting a string into smaller units called tokens. A token could be a punctuation, word, mathematical symbol, number etc.

![1_GaMrBWjMxvVsMo3S9iLzng.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Ga_Mr_B_Wj_Mxv_Vs_Mo3_S9i_Lzng_e16f7c274a.png)

from nltk.tokenize import word_tokenize

def clean_tokenization(text):
  return word_tokenize(text)
  
  dataset['Clean_text']=dataset['Clean_text'].apply(clean_tokenization)
  
  
 ** 6. Stemming and Lemmatization-**
 
-  Stemming algorithms work by cutting off the end or the beginning of the word, taking into account a list of common prefixes and suffixes that can be found in an inflected word. This indiscriminate cutting can be successful in some occasions, but not always, and that is why we affirm that this approach presents some limitations.

from nltk.stem import PorterStemmer
stemmer=PorterStemmer()
def Clean_stem(token):
  return [stemmer.stem(i) for i in token]
  
  dataset['Clean_text']=dataset['Clean_text'].apply(Clean_stem)
  
  
- Lemmatization, on the other hand, takes into consideration the morphological analysis of the words. To do so, it is necessary to have detailed dictionaries which the algorithm can look through to link the form back to its lemma. Look into the figure for clear picture.

from nltk.stem.snowball import SnowballStemmer

lemma = SnowballStemmer("english")

def clean_lemmatization(token):
  return [lemma.lemmatize(word=w,pos='v) for w in token]
  
  dataset['Clean_text']=dataset['Clean_text'].apply(clean_lemmatization)
  
  
![0_9BvN65J6sjvA3IzF.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_9_Bv_N65_J6sjv_A3_Iz_F_c26fe26d58.png)

![1_zfXXmhBs5Oe61KAmUzwl8A.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_zf_X_Xmh_Bs5_Oe61_K_Am_Uzwl8_A_aa19d495a8.png)

**7. Removing small words having length ≤2**

After performing all required process in text processing there is some kind of noise is present in our corpus, so like that i am removing the words which have very short length.

![0_u7_mWlExGor58thZ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_u7_m_Wl_Ex_Gor58th_Z_fdbda16b09.png)

def Clean_length(token):
  return [i for i in token if len(i)>2]
  
  dataset['Clean_text']=dataset['Clean_text'].apply(Clean_length)
  
  **8. Convert the list into string back**
  
  
![0_aIU4HDsck_q86aWF.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_a_IU_4_H_Dsck_q86a_WF_c4f1b377fd.png)

def convert_to_string(list1):
  return ' ' .join(list1)
dataset['Clean_text']=dataset['Clean_text'].apply(convert_to_string)

Now we are all set to vectorize our text.


## Vectorizing

1. CountVectorizer- It converts a collection of text documents to a matrix of token counts: the occurrences of tokens in each document. This implementation produces a sparse representation of the counts.

![0_73FCorAbb0sTI7u0.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_73_F_Cor_Abb0s_TI_7u0_ef48f6fe02.png)

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(max_df = 0.9,min_df = 10)

X = vectorizer.fit_transform(dataset['Clean_text'])

2. TF-IDF: In TF-IDF we transform a count matrix to a normalized tf: term-frequency or term-frequency times inverse document-frequency representation using TfidfTransformer. The formula that is used to compute the tf-idf for a term t of a document d in a document set is:

![0_9rDpN5R8aJMR7dqe.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_9r_Dp_N5_R8a_JMR_7dqe_bfa845a52e.png)

from sklearn.feature_extraction.text import TfidfVectorizer as tf

tf = tf(max_df = 0.9,min_df = 10)

X = tf.fit_transform(dataset['Clean_text'])

**Note-**

In CountVectorizer we only count the number of times a word appears in the document which results in biasing in favour of most frequent words. This ends up in ignoring rare words which could have helped is in processing our data more efficiently.

To overcome this , we use TfidfVectorizer .

In TfidfVectorizer we consider overall document weightage of a word. It helps us in dealing with most frequent words. Using it we can penalize them. TfidfVectorizer weights the word counts by a measure of how often they appear in the documents.

That’s all folks, Have a nice day :)

NLP is a subdomain of AI that enables machines to read, understand and analyze human languages like text, speech, etc. The ultimate goal of NLP is to help computers understand language human behavior. It is the driving force behind things like virtual assistants, speech recognition, sentiment analysis, automatic text summarization, machine translation and much more.

1_GaMrBWjMxvVsMo3S9iLzng.png

Introduction to NLP

The above picture shows us how far we have come in the field of information technology. Since the inception of the first computer designed by Charles Babbage in the early nineteenth century, we have come a long way. Today, Computers are not just smaller but smarter as well. In recent decades, we have reached a breakthrough in the field of computer science with astonishing advancements in the field of transmitting human intelligence to computers.

Today, we have machines that can mimic human behavior and make predictions by understanding the pattern of thousands of terabytes of information (popularly known as Big Data). You may come across various terminologies to describe the recent developments such as AI, Data Science, Machine Learning, and Deep Learning. But let’s not get ahead of ourselves. Let’s start from the basics and understand each term individually.

1. Artificial Intelligence (AI): Artificial Intelligence is an umbrella term used to define all the technological advancements and algorithms that enable machines to demonstrate human intelligence.

2. Machine Learning (ML): Machine Learning is a subset of artificial intelligence. ML involves learning the pattern of real-life scenarios from the input data based upon certain algorithms to provide accurate predictions.

3. Deep Learning: Deep learning is a subset of Machine Learning which is concerned with algorithms that mimic the workings of a human brain. These algorithms which have similar structure and function to a human brain are popularly known as Neural Networks.

![a3.png](https://almablog-media.s3.ap-south-1.amazonaws.com/a3_e7a2adfbfb.png)

You can understand the relationship between AI, ML, and Deep Learning from the diagram shown below:-

Since we have covered the basic terminologies, let’s deep dive into the workings of Machine Learning.

## What is Machine Learning?

In layman's terms, machine learning involves the use of algorithms to make the machine learn on its own from a given input dataset. In other words, it involves the study of those computer programs which automatically improve their performance through experience.

The Term Machine Learning was first coined by Arther Samuel in 1952. He described ML in the following words.

_“Machine learning is the practice of giving computers the ability to learn without being explicitly programmed to do so.”_

After Arther, a more formal and widely popular definition of Machine Learning was propounded by Tom Mitchell. Let us get a wider perspective about Machine Learning from his words mentioned below.

_“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.”_

Now let us understand the meaning of the ETP definition of ML by viewing the examples mentioned in the purple boxes below.

![a4.png](https://almablog-media.s3.ap-south-1.amazonaws.com/a4_eaddb7c48e.png)

## Tasks of Machine Learning

**Supervised Learning**

It is defined by its use of labeled datasets to train ML algorithms in order to classify data or predict outcomes accurately. As input data is entered into the model, it adjusts its parameters until we get a fitted model that appropriately translates the relationship between the independent and dependent variables. Some of the techniques used in supervised Machine Learning include linear regression, logistic regression, neural networks, and naïve Bayes. 

There can be two instances while undergoing supervised learning, namely:-

**1. Regression**: We define a supervised ML problem as a regression problem if the dependent variable or the target variable is continuous in nature. For instance forecasting, the weather report of a city-based upon technical factors such as wind speed, humidity levels, seasonal variations, etc can be an example of a regression problem.

**2. Classification**: We define a supervised ML problem as a Classification problem if the dependent variable or the target variable is categorical in nature. For example: if a bank wants to analyze whether to give a loan to a particular account holder based upon CIBIL score, account balance, past loans repayment, etc. This is an example of a classification problem since our dependent variable can only take two values, YES (the bank will accept the loan application) and NO (the bank will reject the loan application).

![a5.png](https://almablog-media.s3.ap-south-1.amazonaws.com/a5_66c71fbb43.png)

**Unsupervised Machine Learning**

When the target variable is absent from the initial dataset then we use Unsupervised Machine Learning. Since we don’t have a target set of values to supervise our model’s working, we use this ML technique to analyze and cluster unlabeled datasets. Hereby Cluster, we mean the formation of segments based upon the pattern of the input dataset. Cluster analysis finds the common points or similarities between the data objects and categorizes them as per the presence and absence of those common characteristics. Some of the most commonly used ML algorithms in Unsupervised scenarios include k-means clustering, neural networks, Principle Component Analysis, and probabilistic clustering methods.

For example, clustering is a very commonly used technique to build recommendation systems for OTT, finding customer segments for airline loyalty programs from a given dataset of air passengers by Delta Airlines, and much more.

![a6.png](https://almablog-media.s3.ap-south-1.amazonaws.com/a6_1d1f9833e0.png)

**Reinforcement Learning**

Reinforcement machine learning is a behavioral machine learning model that is similar to supervised learning, but the algorithm isn’t trained using sample data. This model learns by using a trial and error approach while working in an environment. The model’s output is assessed by the environment and the outcomes are labeled as desired and undesired. The desired outcomes are rewarded so the algorithm tries to maintain a streak of the desired outcomes consecutively. Therefore, a sequence of successful outcomes will be reinforced by the environment to develop the best recommendation or policy for a given problem statement.

For example, the IBM Watson system competed against the previous champions of Jeopardy! the challenge in 2011 and won by using this form of ML algorithm.

This was all for today. We’ll discuss more about the various techniques of Machine learning in future blogs. so stay tuned and like this article, if you gained some clarity on machine learning.

References

1. Machine-learning mastery articles.

2. wikimedia.com

3. zldoty article on what is machine learning.

4. IBM machine learning article.

5. Alma Better Course Materials.



The above picture shows us how far we have come in the field of information technology. Since the inception of the first computer designed by Charles Babbage in the early nineteenth century, we have come a long way.

0_xNQbHYTQ12yx3r9v.jpeg

Introduction to Machine Learning

Before we begin with A/B testing, it is of the utmost importance that we understand Hypothesis testing first.

In simple words hypothesis is an idea that can be tested, and in hypothesis testing we are trying to reject the STATUS QUO, thereby trying to find an alternative change or innovation

Let us suppose that we are primarily concerned with using the resulting sample to test some particular hypothesis concerning them. As an illustration, suppose that a construction firm has just purchased a large supply of cables that have been guaranteed to have an average of breaking strength of at least 7,000 psi. To verify this claim, the firm decided to take a random sample of 10 of these cables to determine their breaking strengths. They will then use the result of this experiment to ascertain whether or not they accept the cable manufacturer’s hypothesis that the population mean is at least 7,000 pounds per square inch.

**Mathematically, a statistical hypothesis is usually a statement about a set of parameters of a population distribution. It is called hypothesis because it is not known whether or not it is true**.

A primary problem is to develop a procedure for determining whether or not the values of a random sample from this a population are consistent with the hypothesis. For instance, consider a particular normally distributed population having an unknown mean value of θ and known variance of 1. The statement “θ is less than 1” is a statistical hypothesis that we could try to test by observing a random sample from this population. If the random sample is deemed to be consistent with the hypothesis under consideration, we say that the hypothesis level has been “accepted”; otherwise we say we cannot “accept it”.

In “accepting” a given hypothesis we are not actually claiming it is true but rather we are saying that the resulting data appears to be consistent with it.

**Significance levels**

Consider a population having distribution Fθ, where θ is unknown, and suppose we want to test a specific hypothesis about θ. We shall denote this hypothesis by Hθ and call it null hypothesis. For example, if Fθ is a normal distribution function with mean θ and variance equal to 1 then two possible null hypotheses about θ are:

a) Hθ : θ=1

b) Hθ : θ≤1

Thus the first of these hypotheses states that the population is normal with mean 1 and variance 1, whereas the second states that it is normal with variance 1 and a mean less than or equal to 1. Note that the null hypothesis in (a), when true, completely specifies the population distribution; whereas the null hypothesis in (b) does not. A hypothesis that, when true, completely specifies the population distribution is called a simple hypothesis and one that doesn’t is called a composite hypothesis.

Level of significance is the degree of significance in which we accept or reject the null-hypothesis. 100% accuracy is not possible for accepting or rejecting a hypothesis, so we therefore select a level of significance that is usually 5%.

This is normally denoted with α and generally it is 0.05 or 5% , which means your output should be 95% confident to give similar kind of result in each sample.

**Critical Region**

Suppose now that in order to set a specific null hypothesis H0, a population sample of size n — say X1,…Xn — is to be observed. Based on these n values, we must decide whether or not to accept H0. A test for H0 can be specified by defining a region C in n-dimensional space with the proviso that the hypothesis is to be rejected if the random sample X1,…,Xn turns out to lie in C and accepted otherwise. The region C is called the critical region. In other words, the statistical test determined by the critical region.

![1_NOOw2pTExfpnU-hHI7Bxcg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_NO_Ow2p_T_Exfpn_U_h_HI_7_Bxcg_e44aef2041.png)

In other words, the statistical test determined by the critical region C is the one that

accepts H0 if (X1,X2,…Xn) ∉ C

rejects H0 if (X1,…,Xn) ∈ C

It is important to note when developing a procedure for testing a given null hypothesis H0, that, in any test, two different types of errors can result. The first of these, called a type I error, is said to result if the test incorrectly calls for rejecting H0 when it is indeed correct. The second, called a type II error, results if the test calls for accepting H0 when it is false.

![alma1.png](https://almablog-media.s3.ap-south-1.amazonaws.com/alma1_d148312ed5.png)

![1_74IK0jG9PSMFLe7dtMdf5w.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_74_IK_0j_G9_PSMF_Le7dt_Mdf5w_1c9e083f00.jpeg)

Sounds a little too mathematical, doesn’t it? Let us understand the types of error with a very fun example

Suppose we have a crush on someone and are unsure if we should invite them out on a date. So what are the probabilities of each scenario?

![1_Jv4FtqDKuvOON12p7TucCQ.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Jv4_Ftq_D_Kuv_OON_12p7_Tuc_CQ_e0464b76f3.jpeg)

Let us assume the null hypothesis to be H0: They don’t like you (and you don’t ask them out)

The truth can be put in 4 scenarios:

1. They don’t like you and you don’t ask them out ←Correct decision
2. They like you and you don’t ask them out(missed chance)/False negative← Type II error
3. They don’t like you and you ask them out / False positive ← Type I error
4. They like you and you ask them out ← Correct decision

Only if Mathematics could solve the matters of love!

**P Value**

The P value, or calculated probability, is the probability of finding the observed, or more extreme, results when the null hypothesis (H 0) of a study question is true — the definition of ‘extreme’ depends on how the hypothesis is being tested.

If your P value is less than the chosen significance level then you reject the null hypothesis i.e. accept that your sample gives reasonable evidence to support the alternative hypothesis. It does NOT imply a “meaningful” or “important” difference; that is for you to decide when considering the real-world relevance of your result.

Example : you have a coin and you don’t know whether that is fair or tricky so let’s decide null and alternate hypothesis

H0 : a coin is a fair coin.

H1 : a coin is a tricky coin. and alpha = 5% or 0.05

Now let’s toss the coin and calculate p- value ( probability value).

Toss a coin 1st time and result is tail- P-value = 50% (as head and tail have equal probability)

Toss a coin 2nd time and result is tail, now p-value = 50/2 = 25%

and similarly we Toss 6 consecutive time and got result as P-value = 1.5% but we set our significance level as 95% means 5% error rate we allow and here we see we are beyond that level i.e. our null- hypothesis does not hold good so we need to reject and propose that this coin is a tricky coin which is actually.

**One tailed and Two tailed tests**

One Tailed Test: A test of a statistical hypothesis , where the region of rejection is on only one side of the sampling distribution , is called a one-tailed test.

For ex: Let us compare the mean of a sample to a given value x using a t-test. Our null hypothesis is that the mean is equal to x. A one-tailed test will test either if the mean is significantly greater than x or if the mean is significantly less than x, but not both. Then, depending on the chosen tail, the mean is significantly greater than or less than x if the test statistic is in the top 5% of its probability distribution or bottom 5% of its probability distribution, resulting in a p-value less than 0.05.

![1_Ee2ziJaU4tLhqjXibOTElA.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Ee2zi_Ja_U4t_Lhqj_Xib_OT_El_A_6ac5467cfe.gif)

![1_LUHDtXJQwpwTIDx8ddKOlQ.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/1_LUH_Dt_XJ_Qwpw_TI_Dx8dd_K_Ol_Q_752e7dff53.gif)

Two Tailed Test: A two-tailed test is a statistical test in which the critical area of a distribution is two-sided and tests whether a sample is greater than or less than a certain range of values. If the sample being tested falls into either of the critical areas, the alternative hypothesis is accepted instead of the null hypothesis.

For ex: We wish to compare the mean of a sample to a given value x using a t-test. Our null hypothesis is that the mean is equal to x. A two-tailed test will test both if the mean is significantly greater than x and if the mean significantly less than x. The mean is considered significantly different from x if the test statistic is in the top 2.5% or bottom 2.5% of its probability distribution, resulting in a p-value less than 0.05.

![1_xdycC_fw_N3oX-T_9JVH0Q.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/1_xdyc_C_fw_N3o_X_T_9_JVH_0_Q_0fa454fad4.gif)

**Popular Hypothesis Testing Methods:**

1. When Variance is known: Z-Test:

We use Z- test when the following conditions are true:

> Sample size is greater than 30

>Population data is normally distributed, this doesn’t matter if the first condition is true

>Exclusion/Inclusion of one data point is independent of inclusion/exclusion of another data point

>Sample is randomly selected

>All Sample sizes are equal

**One-sample Z-Test:**

![1_HdvcvRkltDRXD0Owov7Wpw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Hdvcv_Rklt_DRXD_0_Owov7_Wpw_6fe651c615.png)

**Two-sample Z-Test:**

![1_6nxSZBsW0eHeG4rwM99Y8g.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/1_6nx_SZ_Bs_W0e_He_G4rw_M99_Y8g_baab371c82.gif)

2. **When Variance is not known: T-Test:**

T-tests are conducted when the following conditions are true:

> Population variance is unknown

> Sample size is less than 30

**One-sample T-Test:**

![1_SnH0k6c_EJuomKpBd6cgBw.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Sn_H0k6c_E_Juom_Kp_Bd6cg_Bw_f321905607.jpeg)

**Two-sample T-test:**

![1_cWypWuTDHh9qxkt-PLA3bw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_c_Wyp_Wu_TD_Hh9qxkt_PLA_3bw_160d3cf577.png)

**A/B Testing:**

So What is A/B Testing?

A/B testing is a method of statistical hypothesis testing wherein two variants of a web page are compared against one another to understand which one performs better for a conversion goal.

![AB Testing.png](https://almablog-media.s3.ap-south-1.amazonaws.com/AB_Testing_1bb608c9c9.png)

**_Why do A/B testing?_**

We do A/B testing because cannot have a 100% accuracy of how the next 100,000 people who visit the web page will behave. We cannot wait until that happens for optimizing the experience because it will be too late by then.

Therefore we use the information we have now to make predictions of future behavior by statistical analysis. Or in simple words, keep a track on the 1,000 customer behavior to estimate the behavior of the next 99,000 users.

If we can optimize our statistical analysis will incredible efficiency, this will prove extremely beneficial for running the business.

Let us look at an example of sales before and after A/B testing

![1_0gDJj0K5MlJPW-RxnXdw8g.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_0g_D_Jj0_K5_Ml_JPW_Rxn_Xdw8g_4e3bac58a6.png)

**Where should we be careful?**

Since we are trying to make predictions about the population, our sample needs to accurately represent the population.

_Let us look at a split testing dashboard to understand Confidence level and margin of error:_

![1_3tsYC8hyMloZg2mRGPfISA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_3ts_YC_8hy_Mlo_Zg2m_RG_Pf_ISA_fcc37edf3d.png)

The original page has a conversion rate of 10.3% plus/minus 1%. The 10.3% is the true mean. This 1% is our margin of error. Although we want to have 100% confidence of true mean. In real life scenarios that would inaccurate.

Hence 10.3% ± 1.0 % at 95% confidence is our actual conversion rate for this page

So, what we are saying is that if 20 samples were taken, we can say with complete certainty that 19 0f those would contain true conversion rate within 95% confidence intervals.

The confidence interval is an observed range in which a given percentage of test outcomes fall. We manually select our desired confidence level at the beginning of our test, and the size of the sample we need is based on our desired confidence level.

The range of our confidence level is then calculated using the mean and the margin of error.

The upper bound of the confidence interval is found by adding the margin of error to the mean. The lower bound is found by subtracting the margin of error from the mean.

**Errors:**

As we already mentioned, A/B testing will not be done for the entire population but a small sample for example lets say 1,000 people and make a hypothesis that Variation B will perform better than Variation A.

If Variation A has better conversion then no change needs to be made since this is our original page.

If Variation B performs better then we have to determine if the change was statistically large enough for it to effect the population and if yes then we must make changes according to Variation B.

So why can’t we directly do this? Because of the variance of samples. There can be 4 scenarios that occur when we run our test.

1. Test says Variation B is better and Variation B is actually better.
2. Test says Variation B is better but Variation B is not better- ←Type I error
3. Test says Variation B is not better but Variation B is better ← Type II error
4. Test says Variation B is not better and Variation B is actually not better

**Type I Errors & Statistical Significance**

A type I error occurs when we incorrectly reject the null hypothesis.

To put this in AB testing terms, a type I error would occur if we concluded that Variation B was “better” than Variation A when it actually was not.

Remember that by “better”, we aren’t talking about the sample. The point of testing our samples is to predict how a new page variation will perform with the overall population. Variation B may have a higher conversion rate than Variation A within our sample, but we don’t truly care about the sample results. We care about whether or not those results allow us to predict overall population behavior with a reasonable level of accuracy.

Type I error is related to Statistical Significance.

**Type II Errors & Statistical Power**

A type II error occurs when the null hypothesis is false, but we incorrectly fail to reject it.

To put this in AB testing terms, a type II error would occur if we concluded that Variation B was not “better” than Variation A when it actually was better.

Just as type I errors are related to statistical significance, type II errors are related to statistical power, which is the probability that a test correctly rejects the null hypothesis.

AB tester are mostly interested in Statistical Significance, but over a course of test the values will go from significant to non significant. Hence it is important to have a large enough sample space over a consistent time period to account for the variability of population.

It is important to take the sample size and time pre-decided in advance. Do not use Statistical significance as a stopping factor because the test results will oscillate between significant and not significant. The only time you should use significance as the deciding factor is at the endpoint.

![1_wxb84XBTl-5rZoeuzCKt9w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_wxb84_XB_Tl_5r_Zoeuz_C_Kt9w_072dfd0011.png)

**A/B Testing Examples:**

![1_dwjapdbPOJqILb0FkILAgA.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_dwjapdb_PO_Jq_I_Lb0_Fk_IL_Ag_A_e5d6859686.jpeg)

![1_pWEBIIcioEveVnN-yJrjtg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_p_WEBI_Icio_Eve_Vn_N_y_Jrjtg_71fa82ab9a.png)


In simple words hypothesis is an idea that can be tested, and in hypothesis testing we are trying to reject the STATUS QUO, thereby trying to find an alternative change or innovation

AB Testing.png

Hypothesis Testing: Understanding A/B testing

Considering the current pandemic scenario, we might want to know whether a certain new drug developed by a pharmaceutical company will be able to reduce the number of active cases in the world. Of course before trying the following drug on a mass scale, the government might be interested in the effectiveness of the drug based on the results from a couple of thousands of test cases.
For example, When testing the new vaccine on a random sample of 2500 previously uninfected people, its found that these people are 70 % less likely to contract to COVID-19 compared to those who did not take the vaccine.
Now, the question is : Is this significant enough? Could this be just the result of chance, or is this vaccine really effective ? It is in these cases that the hypothesis testing comes in and has an very important role to play.
Statistical Inference
Statistical inference is the process of drawing conclusions about the population based on a sample from that population. This technique really comes in handy when it becomes physically impossible to acquire stats and information about the entire population. We can ask numerous questions about the population and might be able to draw some definitive conclusions based on single randomly drawn sample from that population.
Suppose, we survey a total of 50 different higher education institutions in India. We found that out of 950 professors in those institutions, 640 are male and 310 are female.
Based on this statistic can we come to a make a claim that there are less than 30% preference for female professors in Indian institutions ? Does the data provide enough evidence to infer a gender bias based on this statistic ?
Steps in Hypothesis testing :
Given a claim, identify the null hypothesis and the alternative
hypothesis, and express them both in symbolic form.
Given a claim and sample data, calculate the value of the test statistic.
Given a significance level, identify the critical value(s).
Given a value of the test statistic, identify the P-value.
State the conclusion of a hypothesis test in simple, non-technical
terms.
Null hypothesis : The null hypothesis is a statement that the value of a
population parameter (such as proportion, mean, or standard deviation) is equal to some claimed value. This is the first assumption that you make about any event and is often quite mundane and doesn’t signify any progress or change.
Alternative Hypothesis : The alternative hypothesis is the statement that the statistic has a value that somehow differs from the null hypothesis.This statement is often progressive in nature. The symbolic form of the alternative hypothesis must use one of these symbols: ≠, <, >.
Of the two symbolic expressions obtained so far, let the alternative
hypothesis be the one not containing equality so that it uses the symbol < or > or ≠. Let the null hypothesis be the symbolic expression that the statistic equals the fixed value being considered.
Two salient features :
Rare event rule : Under the assumption that null hypothesis is True, if probability of certain event is bleak, i.e. if the event is more likely to have occurred by chance, then our assumption is supposed to be incorrect and we reject null hypothesis in favor of alternate hypothesis.
In any case, we either reject or fail to reject the Null hypothesis. We never accept a null hypothesis.
After formulating both the hypothesis, we conduct the hypothesis test under the assumption that the null hypothesis is true. Our goal is to collect enough evidence in favor of alternate hypothesis so as to be able to reject null hypothesis. However if our results are statistically insignificant, we fail to reject the null hypothesis. To be able to say if the results are statistically significant or not, we need to understand the concepts of test statistic and p value. But before discussing these terms in detail, lets learn about the concept of and significance level, critical region and types of test we run based on our hypotheses.
Significance Level
The significance level, also denoted as alpha or α, is a measure of the strength of the evidence that must be present in our sample before we will reject the null hypothesis and conclude that the effect is statistically significant. The researcher determines the significance level before conducting the experiment.
The significance level is the probability of rejecting the null hypothesis when it is true. For example, a significance level of 0.05 indicates a 5% risk of concluding that a difference exists when there is no actual difference. Lower significance levels indicate that we require stronger evidence before we will reject the null hypothesis.
In other words, the significance level is a statistical way of demonstrating how confident you are in your conclusion. If you set a high alpha (0.25), then you’ll have a better shot at supporting your alternative hypothesis, since you don’t need to find as big a difference between your test groups. However, you’ll also have a bigger chance at being wrong about your conclusion.
Critical region is the region of values that corresponds to the rejection of the null hypothesis at some chosen significance level α. Critical Value is the permissible limit value beyond which results are statistically significant and Null hypothesis is rejected.
Types of Test

![0_Q5T2Zaqs3IhxpmOV.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Q5_T2_Zaqs3_Ihxpm_OV_743f11eed2.gif)
The types of test that we conduct is based on the alternate hypothesis (Ha)
If Ha : parameter < value, then we conduct left-tailed test, where critical region lies on the left tail of the distribution.

If Ha : parameter >value, then we conduct right-tailed test, where critical region lies on the right tail of the distribution.

If Ha : parameter ≠ value, then we conduct a two-tailed test, where critical regions lie on both the left and right tails of distribution.

Test Statistic
A Test statistic is a mathematical formula that allows researchers to determine the likelihood of obtaining sample outcomes if null hypothesis were true.
The value of test statistic is used to make a decision regarding the null hypothesis. The larger the test statistic, the further the distance, or the number of standard deviations, a sample mean is from the population mean stated in null hypothesis. Examples of test statistics that are used in hypothesis testing are :
Test statistic for proportions :

Test statistic for mean :

If standard deviation σ of population is not known, but however the sample variance is known, we use a t-statistic with (n-1) degrees of freedom, instead of z-statistic.

When we want to test the measure deviation of sample variance (S²) from the population variance (σ²), we use χ² statistic with (n-1) degrees of freedom.

P-value Approach
After we are done calculating the statistic of a parameter for a given sample, We need to calculate the P-value of the given statistic. P-value is nothing but the probability of observing a sample parameter that is as or more extreme than the observed and is given by the area under the tails with limits defined according to the type of test we conduct. Lets take the example of the z-statistic that is calculated assuming that our sample parameter follows a normal distribution, which also follows from the Central Limit Theorem.

Now, we use significance levels during hypothesis testing to help us determine which hypothesis the data support. We compare your p-value to our significance level. If the p-value is less than our significance level, we can reject the null hypothesis and conclude that the effect is statistically significant. In other words, the evidence in our sample is strong enough to be able to reject the null hypothesis at the population level.
Example 1:
A principal at a certain school claims that the students in his school are above average intelligence. A random sample of thirty students IQ scores have a mean score of 112.5. Is there sufficient evidence to support the principal’s claim? The mean population IQ is 100 with a standard deviation of 15.
Step 1: State the Null Hypothesis. The accepted fact is that the population mean is 100, so: H0: μ=100.
Step 2: State the Alternate Hypothesis. The claim is that the students have above average IQ scores, so: H1: μ > 100. The fact that we are looking for scores “greater than” a certain point means that this is a right-tailed test.
Step 3: State the α level. If you aren’t given an α level, use 5% (0.05).
Step 4: Find the test statistic using this formula:

For this set of data: z= (112.5–100) / (15/√30) = 4.56.
Step 5: Considering we are doing a right-tailed test, calculate the P-value for z statistic using the z table
P-value ∼ 0.00001.
Step 6: Compare the P-value to significance level α. If p < α, results are significant and null hypothesis is rejected. Else, we fail to reject the Null hypothesis.
Since p < α = 0.05, we reject the null hypothesis, in favor of alternate hypothesis and effectively conclude that principal’s claim is True based on the statistical test conducted.
Example 2
The average height of students in a batch is 100 cm and the standard deviation is 15. However, the teacher believes that this has changed, so he decides to test the height of 75 random students in the batch. The average height of the sample comes out to be 105. Is there enough evidence to suggest that the average height has changed?
Step 1: State the Null Hypothesis. In our case it will be the average height of students in the batch is 100.
H0 : μ = 100
Step 2: State the Alternate Hypothesis. In our case, the teacher claims that the actual value has changed. He doesn’t know whether the average has gone up or down, but he believes that it has changed and is not 100 anymore.
Ha : μ ≠100
So if the alternate hypothesis is written with a ≠ sign that means that we are going to perform a 2-tailed test because chances are it could be more than 100 or less than 100 which makes it 2-tailed.
Step 3: State the α level. If you aren’t given an α level, use 5% (0.05).
Step 4: Find the test statistic using this formula:

For this set of data: z= (105–100)/(15/√7.5) = 2.89
Step 5: Considering we are doing a two-tailed test, calculate the P-value for z statistic using the z table
P-value = 0.003852
Step 6: Compare the P-value to significance level α. If p < α, results are significant and null hypothesis is rejected. Else, we fail to reject the Null hypothesis.
Since p < α = 0.05, we reject the null hypothesis, in favor of alternate hypothesis and effectively conclude that teacher’s claim is True that the average height of students in the batch has changed.
Statistical errors
We often come across terms like statistical errors as a result of conducting hypothesis tests. If the result of the test corresponds with reality, then a correct decision has been made. However, if the result of the test does not correspond with reality, then an error has occurred. There are two situations in which the decision is wrong. The null hypothesis may be true, whereas we reject H0. On the other hand, the alternative hypothesis Ha may be true, whereas we do not reject H0. Two types of error are distinguished: Type I error and type II error.

Type I error
The first kind of error is the rejection of a true null hypothesis as the result of a test procedure. This kind of error is called a type I error (false positive) and is sometimes called an error of the first kind.
Type II error
The second kind of error is the failure to reject a false null hypothesis as the result of a test procedure. This sort of error is called a type II error (false negative) and is also referred to as an error of the second kind.
Power of Hypothesis Test
The power of a hypothesis test is the probability that the test rejects the null hypothesis (H0) when a specific alternate hypothesis (Ha) is true i.e., it indicates the probability of avoiding a Type-II error. The statistical power ranges from 0 to 1, and as statistical power increases, the probability of making a type II error (wrongly failing to reject the null hypothesis) decreases. If probability of making Type II error is β, power of hypothesis test is given by 1-β.


0_eKJufdSbd82WuxTh.jpeg

Hypothesis Testing

Contributors: F M Nurul Huda Pathan, Soumya Ranjan Mishra


Uber is an American technology company that provides ride-hailing, food delivery (Uber Eats), package delivery, couriers, freight transportation, and, through a partnership with Lime, electric bicycle and motorized scooter rental. Most of its revenues come from ride-hailing.

![0_bXrBIiHPWGpYUgct.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_b_Xr_B_Ii_HPW_Gp_Y_Ugct_6f92d092fb.png)

Let’s now discuss how the ride-hailing operations are done in Uber.

- Uber provides different service options. It includes a private ride in a car with a driver for up to four passengers. Depending on the location, It also offers other levels of service at different prices including black luxury vehicles, newer or premium level vehicles, cars with leather seats, sport utility vehicles, minivans, vans, hatchbacks, electric cars, hybrid vehicles, motorcycles, auto-rickshaws, actual taxicabs, lower-cost shared transport with other passengers going in the same general direction.
- It determines the fees and terms on which drivers transport riders. It uses a dynamic pricing model. Fares fluctuate depending on the local supply and demand at the time of service. Customers are quoted the fare in advance. Service is generally accessed via a mobile app.
- Users need to set up a personal profile with a name, phone number, other information, and payment preference, which could be a credit card, e-commerce payment system, or, in some cases, cash. After the service is complete, the customer may be given the option to provide a gratuity to the driver, which is also billed to the customer’s payment method.
- Drivers usually provide a vehicle, which could be owned, rented, or leased. Drivers must meet requirements for age, health, car age, and type, have a driver’s license and a smartphone or tablet and may be required to pass a background check. In many cities, vehicles must pass annual safety inspections and/or must have an emblem posted in the passenger window. Some cities also require drivers to have a business license. There may be accommodations for hearing-impaired drivers. Drivers may be notified before accepting a trip if it will be longer than 45 minutes.
- After each trip completion, drivers and customers may rate each other, and users with low ratings may be deactivated.

Now, let’s discuss how an Uber Trip system can be designed. The system must have the following standard features.

- Riders will be able to view the available drivers in nearby locations.
- Riders will be able to request rides from source to destination.
- Nearby drivers will get notified of the ride, and one of the drivers will confirm the ride.
- Pickup ETA will be shown to customers when a ride is dispatched to pick up the rider.
- Once the trip is completed, the rider must be shown the trip details such as the trip map, price, etc. in the ride receipt.

The Uber Trip System architecture will have the following components.

![0_UEKqmR8KI3lZ5RZy.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_UE_Kqm_R8_KI_3l_Z5_R_Zy_30628a90f6.png)

**Pre-Trip Component:**

It has the following sub-components as

- Driver Location Manager: This component will be responsible for maintaining the changing driver locations. It will ingest the messages emitted by the drivers’ uber app containing their current locations and update the car location index.
- Trip Dispatcher: It will be responsible for handling the trip requests from users and dispatch a driver in response to those requests.
- Arrival ETA Calculator: This component will be responsible for calculating the ETA for the driver to reach the rider once the driver has accepted the ride.

**On-Trip Component:**

It has the following sub-components which include

- Trip Recorder: GPS signals transmitted from the ride when the trip is in progress are recorded. These GPS signals will be then recorded in a data store which will be used by subsequent sub-component systems such as Map Matcher and Pricing Calculator.
- Map Matcher: It will be responsible for matching the trip on the actual map using algorithms specialized for this purpose.
- Price Calculator: It will then use the recorded trip information for computing the price which users have to pay for the trip.

**Data Storage:**

- Driver Locations: Here, the driver location information such as latitude and longitudes will be stored.

- Trip Records: This database will record trip details. These details can be used to draw insights by the company to draw patterns and business insights.

**How are the Pre-Trip Components designed?**

Pre-trip components perform the following actions.

- Update Driver Location -It involves updating the driver locations as drivers keep changing their locations.
- List nearby drivers-It comprises the steps involved in fetching the list of nearby drivers.
- Requesting for a ride-It includes a series of steps involved in requesting and dispatching a ride.

These components are designed to show the cars in the nearby locations and dispatch the ride on users’ requests. The location of the cars can be stored in an in-memory car location index, which can support high read and write requests. The driver app will keep on sending the updated car locations, which will be updated in this index. The index will be queried when the rider app requests for nearby car locations or places a ride request.

Car Location Index will maintain the driver locations in a distributed index, which is spread across multiple nodes. Nodes contain driver information containing their locations, the number of people in the car, and if they are on a trip. These pieces of information are aggregated as objects and are distributed across the nodes using some mechanisms (e.g., city, product, and so forth) along with replication.

![0_7WnNcprc7f24h8aJ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_7_Wn_Ncprc7f24h8a_J_4fb43218f2.png)

Some properties that a car index location must contain are-

- It must store the current locations of all drivers and must support fast queries by location and properties such as driver’s trip status.
- It must support a high volume of reads and writes.
- The data stored in these indexes must be ephemeral, and we don’t need long-term durable storage.

ETA for the car arrival is calculated by considering factors such as route, traffic, and weather account. There are two primary components of estimating an ETA given an origin and destination on a road network such as

- calculating the route from origin to destination with the least cost
- estimating the time taken to traverse the route.

Every intersection of a road on a map is represented by a node and each road segment by a directed edge.

![0_zFHYbq-Q7OJSfNU7.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_z_FH_Ybq_Q7_OJ_Sf_NU_7_2772a8a585.png)

We can use Dijkstra’s algorithm to find the shortest path between source and destination. This algorithm has a time complexity of O(N*logN) to find the shortest path for N nodes. But ride-hailing apps like Uber require a highly scaled approach to determine the routes. To solve this we can use partitioning the entire graph, pre-computing the best path within partitions, and interacting with just the boundaries of the partitions. It will give a time complexity of O(√N*log√N). Thus the complexity to find the shortest path can be reduced.

After the optimal route is found, we can find the estimated time to traverse the road segment by taking traffic into account, which is a function of time, weather, and real-life events. This traffic information can be used to populate the edge weights on the graph.

![0_txZI9c8zPDKhR5I7.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_tx_ZI_9c8z_PD_Kh_R5_I7_95ace90df7.png)

**How are On-Trip and Post-Trip Components designed?**

These components are responsible for handling the processes which involve from start to completion of the trip. It keeps track of the locations of the ride when the trip is in progress. It is also responsible for generating the route map and computing the trip cost when it gets completed.

![0_2WaRWfdtZFJL11Ts.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_2_Wa_R_Wfdt_ZFJL_11_Ts_8283dba997.png)

GPS locations sent by the rider/driver app while the trip is in progress are stored in a database. Some of the characteristics of this database are-

- It must support a high volume of writes.
- The storage must be durable.
- It must support time-series-based queries such as the location of the driver in a given time range.

Map Matcher module takes in raw GPS signals as input and outputs the position on the road network. The input GPS signals consist of latitude, longitude, speed, and course. The positions on a road network consist of latitude (on an actual road), longitude (on a real road), road segment id, road name, and direction/heading. The route map generated by Map Matcher is used for-

i) finding the actual position of the driver on the road network

ii) calculating the fare prices.

**Optimizations**

- One of the problems with the proposed design is that both the heavy reads and writes are directed to the data store. We can optimize the system by separating the read traffic from the writes by caching the raw driver locations in a distributed cache (Redis). Redis is an open-source, in-memory data structure store, used as a database, cache, and message broker. The raw driver locations from the Redis cache are leveraged by MapMatcher to create the matched map and persist it in the data store.

![0_E1vrZ-UbVlpiblfh.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_E1vr_Z_Ub_Vlpiblfh_f109fd56f5.png)

- We can also improve the predicted ETAs by applying machine learning on top of the ETA calculation mechanism mentioned above. The first challenge of using machine learning involves feature engineering to come up with features such as region, time, trip type, and driver behavior to predict realistic ETAs.

- Another exciting challenge would be to select the machine learning model, which can be used for predicting the ETAs. For this use-case, we would rely on using a non-linear(e.g., ETA doesn’t linearly relate with an hour of the day) and non-parametric(there is no prior information on the interaction between different features and ETA) model such as Random Forest, Neural networks and KNN learning. We can further improve the ETA prediction by monitoring customer issues when the ETA prediction is off.


**Extended requirement**

- In the design above, the drivers were being notified of the nearby riders, and they were accepting the rides they wanted. As an extension of the existing implementation, we may extend the system to enable automated matching of drivers to riders.

- The drivers are matched to riders based on factors such as the distance between drivers and riders, the direction in which the driver is moving, traffic, and so forth. We can apply the naïve approach of matching riders to the nearest driver.

- However, this approach doesn’t tackle the problem of something called a trip upgrade, which occurs when a rider is matched to a driver and, at the same time, an additional driver, and an additional rider enters the system. The trip upgrade scenario is illustrated in the images below.

![0_4dzMH0-px6HNSgFI.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_4dz_MH_0_px6_HN_Sg_FI_4c5c52e9e4.png)

![0_9rub24i069RmF0WB.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_9rub24i069_Rm_F0_WB_b9634c6c7e.png)

**How to handle datacenter failures?**

- It doesn’t happen very often, but there could be an unexpected cascading failure or an upstream network provider could fail.
- Uber maintains a backup data center and the switches are in place to route everything over to the backup data center.
- The problem is the data for in-process trips may not be in the backup data center. Rather than replicate data they use driver phones as a source of trip data.
- What happens is the Dispatch system periodically sends an encrypted State Digest down to driver phones. Now let’s say there’s a data center failover. The next time the driver phone sends a location update to the Dispatch system will detect that it doesn’t know about this trip and ask them for the State Digest. The Dispatch system then updates itself from the State Digest and the trip keeps on going like nothing happened.

## Conclusion

Uber is a company with a complicated history. Still, its founders had made something that was impossible. They survived sabotages, strikes, and discontent with the governments of different countries around the whole world. The necessity to provide people with good services at affordable prices is urgent nowadays. Not all people can use the quality they want. Uber opens new perspectives and possibilities.

As a final word, we would like to suggest some points that might be improved in system design are as follows:

1.Reduce Driver Downtime
2.Own Local Event Recommendations
3.Customize the experience
4.Create Loyalty Programs

_Finally, we had a word on Uber system design. I think our work will help you in certain ways. Clap for us. If you suggest adding something more, feel free to reach me. See you next time with another blog!!!_

**Reference**

https://www.techtakshila.com/system-design-interview/chapter-3/

https://www.geeksforgeeks.org/system-design-of-uber-app-uber-system-architecture/

https://www.educative.io/blog/uber-backend-system-design


This article aims to make you understand the System Design of the Uber application. We hope you will enjoy the blog. Let’s Start!

0_FOcH1AddJTt9fcYY.png

How should you approach to design an Uber Trip System?

**Disclaimer**: This article is written with humorous intent. The Reserve Bank of India is admired globally and is the epitome of excellence in monetary policy and economic decision-making. Indian Police officers, in fact all UPSC personnel, are crème de la crème meritocrats. We do not condone any heinous activities.

The professor and his team are planning the heist in the RBI bank because they came to know that India is developing at an incredible rate. The professor is explaining how to minimize the risk of getting caught, by the Indian police, using the Gradient Descent Algorithm.

**Tokyo**: What is this Gradient Descent Algorithm?

**Professor**: Gradient Descent Algorithm is used to minimize a function by optimizing its parameter.

**Raquel**: Which function does it minimize?

**Professor**: It minimizes the cost function.

**Denver**: What is the cost function, and what is the need to minimize it?

**Professor**: The cost function represents the error between the actual value and the predicted value. Put another way: by minimizing the cost function, we are minimizing the error between the actual value and the predicted value.

**Rio**: How will this FUNCTION help us not to get caught?

**Professor**: By minimizing the function of the risk of getting caught. The Gradient Descent Algorithm is used to minimize the given function. In our case, it is a risk function. It does so by first calculating the slope of a function by the first-order derivative of the function and, second, it moves in the opposite direction to the gradient by alpha times the gradient at that point. (Professor took the chalk and wrote this on the blackboard)

**New Value = Old Value - Step Size**

**Professor**: In the world of mathematics, the ‘Step Size’ will be equal to the product of Learning Rate and Slope. So the above equation can be written like this.

![1_xXGUZiMrAoKnhDb4FC9iDg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_x_XGU_Zi_Mr_Ao_Knh_Db4_FC_9i_Dg_3edb867d17.png)

**Tokyo**: Professor! In a simple language, please.

**Professor**: (Professor looked at Tokyo and slowly nodded and took the black color remote of the projector and presented this slide) In simple language, suppose this is a function that we want to minimize. And we want to find the value of X when Y is minimum. For this, we will be using Gradient Descent Algorithm.

![1_sBHEzOHEtZmCpPLU7JMd6A.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_s_BH_Ez_OH_Et_Zm_Cp_PLU_7_J_Md6_A_7126f9c4bf.png)

**Denver** ((hastily answered): zero, zero X value should be zero for Y to be minimum.

**Raquel**: Why use Gradient Descent when we can clearly see the minimum value?

**Professor**: In this case, we are using a single variable. So the minimum value of a function is apparent. But what if there are multiple variables in the function? Say, 10 or 50? Then, how would you inspect the minimum value of that function? And by the minimum, I mean “local minima”, not global minima; because this algorithm finds the local minima of the function. For this very reason, we are using the gradient descent algorithm. Here we will start with some approximate values, and then we will be moving towards local minima. Let’s see this diagram.

![1_LNbvbTC1yIU1Tm-VRO8IPg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_L_Nbvb_TC_1y_IU_1_Tm_VRO_8_I_Pg_102f578dfb.png)

**Tokyo**: Hold on, Professor. Do you want to risk our life by taking some approximate values?

**Rio**: Tokyo… calm down Tokyo.

**Raquel**: I am still not satisfied with this algorithm. Ah…Just show me what the graph of risk looks like after we enter the bank.

**Rio**: Yeah and also the risk function that we have to minimize. We’re supposed to minimize this function with respect to what? I am just asking what is X in our risk function?

**Professor**(calmly stares at Rio and then Raquel for a few seconds and then again to Rio and says): Time…We have to minimize the risk function with respect to time. Here’s how the graph of risk versus time looks like (professor took the remote and changed the slide) We need to escape from the bank at point A because after that it will become almost impossible to escape.

![1_R513AJ_NNQq9DtfKYMHZXw.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/1_R513_AJ_NN_Qq9_Dtf_KYMHZ_Xw_c59bf9018a.jpeg)

**Denver**: Okay. I want to ask why we are focusing so much on the Gradient Descent Algorithm. Can you please give some of the benefits of this algorithm?

**Professor**: Sure. There are around 4 major benefits of using this algorithm.

1. Fast enough to scale on big data.

2. The exact method will take more time. And we will never do it 
exactly, so we have to go with approximation.

3. The loss function gives the direction of the Optimal Solution.

4. Easy to understand.

See this graph of what this algorithm does. (Professor took the remote and showed this slide to everyone)



**Professor**: It is relatively very easy to understand.

**Tokyo**: Easy! What do you mean that I am a fool and everyone here is a genius? Come on, Professor. We will just keep on calculating these so-called minimum values of a function, and the Indian police will get us and maybe even shoot us. The lecture is over, and I am done with this ALGORITHM (Tokyo angrily left the class).

Meanwhile, the Indian Government came to know about this plan from their intelligence agency, and they have decided to contact AlmaBetter for a better understanding of this algorithm.



The professor and his team are planning the heist in the RBI bank because they came to know that India is developing at an incredible rate. The professor is explaining how to minimize the risk of getting caught, by the Indian police, using the Gradient Descent Algorithm.

bank.png

Gradient Descent

## INTRODUCTION

_Generative Adversarial Networks (GANs) are nothing but a framework for estimating generative models via an adversarial process._

As I told you before that the most common definition for GAN may be it connects the imaginary world to reality. Before going further let’s know when and how GAN was introduced as a reinforcement Deep Learning technique? A generative adversarial network (GAN) is a class of machine learning frameworks designed by Ian Goodfellow and his colleagues in 2014. Two neural networks contesting with each other in a game (in the form of a zero-sum game, where one agent’s gain is another agent’s loss).

This technique basically learns the pattern and statistics behind the training set. Using them the technique generates new data from given random noise with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that aren’t completely real but look the same as the given images. These are capable of generating wisdom from garbage data. They can be used for generating images, videos, alter images, and much more! The recent research in GANs is focused on generating Deep Fakes (of humans) and detecting them.

**STATISTICS BEHIND GAN**

_Probability_

Probability is a measure of how likely an event is to occur. For example- Today there is a 60% chance of rain. There is a 50% chance of getting head in one toss. The odds of winning the lottery are a million to one. If an event is certain to happen, the probability of the event is 1. If an event doesn’t happen, the probability will be 0 and the probability never be negative.

_Dependent and Independent Events_

Two events are independent if the result of the second event is not affected by the result of the first event. Two events are dependent if the result of the first event affects the outcome of the second event so that the probability is changed.

![1_cKYm34YHJ2Cxot3eNvt_dA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_c_K_Ym34_YHJ_2_Cxot3e_Nvt_d_A_4dbe019e97.png)

_Joint Probability_

A joint probability is a statistical measure the calculates the likelihood of two events occ during together and at the same point in time. A joint probability is the probability of event Y occurring at the same time event X occurs.

![1_kX1opO2awDKDb0klQ6nqJQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_k_X1op_O2aw_DK_Db0kl_Q6nq_JQ_7603190ecc.png)

_Conditional Probability_

Conditional probability is defined as the likelihood of an event or outcome occurring, based on the occurrence of a previous event or outcome. Conditional probability is calculated by multiplying the probability of the preceding event by the updated probability of the succeeding, or conditional, event.

![1_kX1opO2awDKDb0klQ6nqJQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_k_X1op_O2aw_DK_Db0kl_Q6nq_JQ_7603190ecc.png)

_Bayes Theorem_

Baye’s theorem, named after 18th-century British mathematician Thomas Bayes, is a mathematical formula for determining conditional probability. The theorem provides a way to revise existing predictions or theories given new or additional evidence.

Let A and B be two events with P(B) not equal to zero. The conditional probability of A given B is:

![1_nU_GNYbvuHmr_IOHdWh49w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_n_U_GN_Ybvu_Hmr_IO_Hd_Wh49w_2dbb5d5cbd.png)

## Generative v/s Discriminative Modeling

**Generative Models**

Generative models are models where the focus is the distribution of individual classes in a dataset and the learning algorithms tend to model the underlying patterns/distribution of the data points. These models use the intuition of joint probability in theory, creating instances where a given feature (x)/input and the desired output/label (y) exist at the same time.

Generative models use probability estimates and likelihood to model data points and distinguish between different class labels in a dataset. These models are capable of generating new data instances. However, they also have a major drawback. The presence of outliers affects these models to a significant extent.

Examples of generative models :

* Naive Bayes Classifier

* Hidden Markov Model

- Linear Discriminant Analysis

GANs too, are generative models. But unlike others, the novelty lies in its training evaluation measures, i.e. the adversaries.

**Discriminative Models**

Discriminative models, also called conditional models, tend to learn the boundary between classes/labels in a dataset. Unlike generative models, the goal here to find the decision boundary separating one class from another. So while a generative model will tend to model the joint probability of data points and is capable of creating new instances using probability estimates and maximum likelihood, discriminative models(just as in the literal meaning) separate classes by rather modeling the conditional probability and do not make any assumptions about the data point. They are also not capable of generating new data instances. Discriminative models have the advantage of being more robust to outliers, unlike generative models.

Examples of discriminative models :

* Logistic Regression

* Support Vector Machine

- Decision Trees

In GANs, discriminators introduce the ‘adversarial’ nature which we are about to discuss in the next section.

![1_ExDqA5QMDnrg2FVGZr-jmg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Ex_Dq_A5_QM_Dnrg2_FVG_Zr_jmg_7f0c21be9c.png)

**GAN, Architecture, and Working**

Here, we discuss the actual architecture of GANs and see how it works.

![1_T1CJvRBQnpNFKZw0bU-wVA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_T1_C_Jv_RB_Qnp_NFK_Zw0b_U_w_VA_d3478955ed.png)

Here is the basic architecture of how GAN works and how the training goes on. We will discuss the architecture block by block. Let’s take the example of generating pictures of dogs through GAN.

Latent Random variables may be termed as noise. The data is from a random distribution. The data taken here will lead to the final desired pictures. Real-world images are those images whose are the ream images and data from the real distribution. This is the actual distribution. The goal of a GAN is to generate data belonging to this distribution from the noise mentioned in the earlier sub-section. In this case, we will take various real dog images. The purpose of the discriminator is to classify the image as real or fake. So, the discriminator can accurately be classifying the real images as real or fake images as fake. Again there will occur a backpropagation to reduce the noise and those who are classified as fake will go for a further generator process to train again. In our example, the discriminator can be a basic CNN-based binary image classifier.

_The Discriminator is nothing but a trainable evaluation metric that tells how good/bad is the quality of the generated data._

The Generator is the main component of the GAN. In this part, the generative network attempts to learn the pattern, trends, and statistics of the distribution and generate data belonging to it.

Now I guess you will able to understand why it’s called two nets are competing. We will also discuss why it’s called where one agent’s gain is another agent’s loss in the mathematics part.

![1_A75a2pQgC4oXE_683O2A8Q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_A75a2p_Qg_C4o_XE_683_O2_A8_Q_f98e120242.png)

![1_5g6wTmog1i3lyDOfEQtdwg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_5g6w_Tmog1i3ly_D_Of_E_Qtdwg_77b5e8a5fd.png)

Here the training has started from taking noise from the random data to the generator and it trained on that data and now other real images are taken from real-world dog images and both images from real-world and generator will go to discriminator to get identified and classified as dogs images are real or fake.

In the first image now discriminator identified a dog and no dog as it’s shown above the probability values for being a dog. Thus, those samples will be backpropagated to the discriminator where the weights will get updated again till we get the probability value for being a dog only. Now in the second image, all the images will be calculated their probabilities according to being a dog but we need the probability value must be 1 for a dog. That’s why we backpropagated those samples that are having a probability of being a dog. In the second image, we don’t require samp-le real images because the discriminator has already aware of the trends and patterns to identify a dog.

![1_kEpdCE4EKeidEuqqkblWQQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_k_Epd_CE_4_E_Keid_Euqqkbl_WQQ_89cf651904.png)

![1__gOeQL5yzAcQcHqjfzjY6w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_g_Oe_QL_5yz_Ac_Qc_Hqjfzj_Y6w_68d4ace044.png)

Here the same happens what I have explained above and we can see the random noise is getting a dog-like shape. While running the training process for lots of epochs our predicted and generated images will be getting more meaningful and the epoch will run until the loss is minimized.

Before going to the loss function I would like to discuss the difference between CNN and DCNN.

![1_3-AZm4dU7UWZbkqu0Rg5Ow.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_3_A_Zm4d_U7_UW_Zbkqu0_Rg5_Ow_f99951ea87.png)

For detailed knowledge of CNN, you may refer to: https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53

![1_UPWZz1mVNuPp3lzmqOe9bg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_UPW_Zz1m_V_Nu_Pp3lzmq_Oe9bg_c1dbc27e4d.png)

DCNN is also known as Deconvolutional Neural Network. Using simple words, CNN we input the image and it goes through convolution, pooling, and then flattening into a vector form for the training process. DCNN is the opposite of it and it can create an image from the flattened vector. It’s the reverse one.

For detailed knowledge of DCNN, You may refer to: https://searchenterpriseai.techtarget.com/definition/deconvolutional-networks-deconvolutional-neural-networks

## Training Loss

The GANs paper defines the training loss as:

![0_pwYsn7G4fUioIQbC.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_pw_Ysn7_G4f_Uio_I_Qb_C_6d09bdfd77.png)

This is the Minmax loss, the discriminator tries to maximize it and the generator tries to minimize log(1 — D(G(z))) as it cannot touch log(D(x)).

However, the paper also suggests modifying this loss for the generator to maximize log(D(G(z))).

![1_7ut69PuFs3fCIxbpjWV-Cg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_7ut69_Pu_Fs3f_C_Ixbpj_WV_Cg_e94fa05115.png)

Here we get the intuition of why it’s called as to where one agent’s gain is another agent’s loss. Because at the same time discriminator tries to maximize the loss function and the generator tries to minimize the loss function.

## Application of GAN

![1_dr-Yh1IEyiEwX7SeOs4UNQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_dr_Yh1_I_Eyi_Ew_X7_Se_Os4_UNQ_018768021e.png)

![1__-EuYRVhSB1618YyvNeqTw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Eu_YR_Vh_SB_1618_Yyv_Neq_Tw_c7ce9e48d3.png)

![1_ldF15CEH5l_F9OH1Hmx7Og.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_ld_F15_CEH_5l_F9_OH_1_Hmx7_Og_778d429cd7.png)

![1_JEdTyAcwtBP6ghsKroeMSg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_J_Ed_Ty_Acwt_BP_6ghs_Kroe_M_Sg_de2dbd05ac.png)

![1_4M08bPYJBMqPnWk8He8EWQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_4_M08b_PYJB_Mq_Pn_Wk8_He8_EWQ_51bf9e4822.png)

In this method, the clearest and resolution enhanced photo of a moon can be processed using lots of the moon’s real images.


- DeepFake Generation and Detection is one of the latest research topics in GANs which essentially generates edited or tampered images that look realistic to the naked eye. This can be easily done manually with the help of photo/video editing software and hence, is vulnerable to misuse. DeepFake detection can be derived as a corollary to generation.

**Types of GAN:**

- _Vanilla GAN_

![1_iNiNHDE91c7DoadBdolVRA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_i_Ni_NHDE_91c7_Doad_Bdol_VRA_4d2ad1eeda.png)

- _Deep Convolutional GANs (DCGAN)_

![1_O0yjB72a95My4_LTsqWPUw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_O0yj_B72a95_My4_L_Tsq_WP_Uw_8fbfb701ab.png)

- _Conditional GANs (CGAN)_

![1_4u4BDOmmt2Hy7R19oYvAWA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_4u4_BD_Ommt2_Hy7_R19o_Yv_AWA_19bb4c84d8.png)

- _Super Resolution GAN_

![1_ar8sMeQn9J09ZI75lbOw-Q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_ar8s_Me_Qn9_J09_ZI_75lb_Ow_Q_edd490ecbd.png)

- _info GAN_

![1_JKM_Xb8xBpMKZb26xgNyJg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_JKM_Xb8x_Bp_MK_Zb26xg_Ny_Jg_809f261227.png)


## CONCLUSION

The main purpose of this article is to get the basic knowledge of the adversarial training evaluation of a generative model. We have discussed generative and discriminative modeling, and how they differ from each other. We discussed how GANs work and the idea behind it. We learned the training aspect of the GANs as well as the issues that arise while actually training the network.

_Finally, we had a word on some well-known flavors of GANs.I think my work will help you in certain ways. If you suggest adding something more, Feel free to reach me. Just drop me a mail. See you next time with another blog!!_!

## References:

- Original paper: https://arxiv.org/abs/1406.2661
- https://machinelearningmastery.com/what-are-generative-adversarial-networks-gans/
- https://en.wikipedia.org/wiki/Generative_adversarial_network
- https://towardsdatascience.com/a-comprehensive-guide-to-generative-adversarial-networks-gans-fcfe65d1cfe4
- https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/



This article aims to make you understand Generative Adversarial Networks (GANs), their types, working principle, applications, Math, and Statistics behind them. The most common definition for GAN may be it connects the imaginary world to reality. I hope you will enjoy the blog. Let’s Start!

1_JEdTyAcwtBP6ghsKroeMSg.png

Generative Adversarial Networks (GANs): Connects Imagination To Reality.

## What is Feature Engineering ?

Feature engineering is the process of using the domain knowledge to create relevant features in order to make Machine Learning algorithms more accurate. Feature engineering impacts noticeably on the performance of model, If feature engineering performed correctly, this helps the model to perform well.

## What is Time Series ?

A time series is a sequence of data points that occur in successive order over some period of time, time series tracks the movements of the chosen data points over specified period of time with data points recorded in regular intervals.

In a time series, the data is captured at equal intervals and each successive data point in the series depends on its past values.

**Which are the different types of features in Time- Series ?**


**I. Date-time Based Features :**

Time series has a Date-time feature from which we can extract various factors like Minute, Hour, Business hour, Weekend, Week days, Seasons, Quarter, Holiday, Leap year, Start of a month, Start of year, End of year and many more, which plays major role in finding the seasonality, trend of the data, and also plays major role in interpreting data.

![1_ElnszjeOo_vGcFnqeSYMTQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Elnszje_Oo_v_Gc_Fnqe_SYMTQ_f6f7170db9.png)

**II. Lag Features :**

Lag features are target values from previous period. For an example, if we would like to forecast the sales in period ‘t’, we can use sales of previous month i.e ‘t-1’ as a feature, this would be lag of 1. Similarly, to find sales in period ‘t+2', we use sales of month ‘t-2’ as a feature and so on. Increasing the lag let’s say up to 4, this will allow the model to make predictions 4 steps ahead.


![1_iUVkJyHXPTQpcTy7gcWUdg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_i_U_Vk_Jy_HXPT_Qpc_Ty7gc_W_Udg_750d9e6264.png)

Lag values chosen will depend on the correlation of individual values with its past values. ACF (Auto correlation Function) and PACF (Partial Function) plots helps to determine the lag at which correlation is significant.

- ACF: The ACF plot is a measure of the correlation between the time series and its lagged version.

![1_fjYag5p2cJuNlZ5p3Xc_zQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_fj_Yag5p2c_Ju_Nl_Z5p3_Xc_z_Q_493375b865.png)

- PACF: The PACF plot measures correlation of residuals with the next lag value.

![1_bVY20jdHzM2XLZNrsikK4A.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_b_VY_20jd_Hz_M2_XLZ_Nrsik_K4_A_a88bd2fcac.png)


**III. Rolling Window Feature :**

Rolling window features calculates some statistics based on past values. It is called as the rolling window method because the window would be different for every data point, and it looks like a window that is sliding with every next point.

![0_lEJnc1PaXBBgoVkZ.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/0_l_E_Jnc1_Pa_XB_Bgo_Vk_Z_2dde2e32af.gif)

The Forecast point : Point in time a prediction is being made.

Feature Derivation Window (FDW) : A rolling window, relative to the forecast point.

Forecast Window (FW) : Range of Future values which we wish to predict.

![1_ODrCFOTq2iz52lIHQd9bfw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_O_Dr_CFO_Tq2iz52l_IH_Qd9bfw_42a3f4893c.png)

**IV. Expanding Window feature :**

Its a type of window includes all previous data in series, help in keeping track of the bounds of observable data. Expanding function collects sets of all prior values for each time step, these lists of prior numbers can be summarized and included as new features.

![0_BjBsAyu1ik4pQuvC.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Bj_Bs_Ayu1ik4p_Quv_C_4be560ad73.gif)

**V. Domain Specific Features :**

With good domain knowledge, understanding the given problem statement, and final objective of problem, and sufficient knowledge about the given data is enough to engineer the domain specific features of the model.


Since, Feature Engineering plays a vital role in modelling process, feature engineering has to be done efficiently to enhance the predictive power of the model.

**References :**

- [Basic Feature Engineering With Time Series Data in Python (machinelearningmastery.com)

- Chapman & Hall/CRC Data Mining and Knowledge Discovery Series — Book Series — Routledge & CRC Press

Feature Engineering, isn’t it the first priority which we look for ?? Of course its yes..! So let’s look into the different features available for Time-Series.

1_B2knMGDeqb8PKKRpqJXmbg.png

Feature Engineering for Time Series problem

I’m just starting over into this field and everyday I keep learning and exploring bits and pieces with randomness in data, I don’t step back finding what significance it holds and what can be achieved out of it. So far, I have been enjoying my day mostly on Stack Overflow and various Data Science blogs.


My journey till now has been amazing and I’m yet to learn many things on the way. This is just a beginning with my exploratory Data Analysis and there are more new contents coming up. I hope this detailed analysis can help as well as set a stepping stone to many **Data driven enthusiasts** looking to break into this field.


So, there are no thumb rules for data exploration. If you are in a state of mind, that machine learning can only sail you away from every data storm, trust me, it won’t. After some point of time, you’ll realize that you are struggling at improving model’s accuracy. In such situation, data exploration techniques will come to your rescue. If you’re yet new to this battle of data exploration and want to have an overall idea of how this can be achieved step by step with ease, please don’t divert yourselves from this detailed exploration on **Airbnb** 2019 dataset comprising of the bookings made in NYC!

This San-Francisco based startup offers you someone’s home as a place to stay instead of a hotel. You might be thinking of another unicorn in town as to OYO Hotels which has kind of a relatable business model but **Airbnb** allows you to be host for anyone anywhere with rooms/beds available in your personal space. OYO Rooms and Airbnb are by no means similar to each other, in fact, they are almost as opposite as the sky and sea. So, having much said let’s just deep dive into our actuals on why are we basically here? Scroll below and have a feel!

This dataset has around ** 48,895** observations with 16 columns and it is a mix between categorical and numeric values. I have portrayed this detailed analysis as much simple as required to get a basic understanding even if someone is very new to this ;)

## <Data Exploration and variable Identification>

Since 2008, guests and hosts have used Airbnb to expand on traveling possibilities and present a more unique, personalized way of experiencing the world. Nowadays, Airbnb became one of a kind service that is used by the whole world. Data analysts become a crucial factor for the company that provided millions of listings through Airbnb. These listings generate a lot of data that can be analyzed and used for security, business decisions, understanding of customers’ and providers’ behavior on the platform, implementing innovative additional services, guiding marketing initiatives, and much more.

The very basic information about the dataset using df.info()

![1_bp3hM8qvwvwIIEsHgt_48w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_bp3h_M8qvwvw_II_Es_Hgt_48w_119ef103bf.png)

By basic inspection, a particular property name will have one particular host name hosted by that same individual but a particular host name can have multiple properties in an area. So, host_name is a categorical variable here. Also neighbourhood_group (comprising of Manhattan, Brooklyn, Queens, Bronx, Staten Island), neighbourhood and room_type (private,shared,Entire home/apt) fall into this category.

~id,latitude,longitude,price,minimum_nights,number_of_reviews,last_review, reviews_per_month, calculated_host_listings_count, availability_365 are **numerical variables.**

## Key Understandings so far?

A host on airbnb holds multiple properties in a neighbourhood group(boroughs of NYC) with different host-ids but a host with a particular property/listing in a particular neighbourhood of a neighbourhood group holds a same host-id(also not mandatory as there are exceptions where few hosts have different id’s for each listing/property in a neighbourhood)
Also the data tells, there might be cases where a particular host has co-hosted someone else’s property/listing in a neighbourhood on Airbnb.

Let’s not worry, I have provided my github code repo at the end. You’re free to explore and get the essence of the stories depicted here.

## Let’s hover over the price column and see its distribution and nature

I was curious to check the **distribution of price** over the entire dataset looking at the five-number summary of the data, later found out something like this.

I have used seaborn **distplot** to plot this distribution curve.

![1_H2-9rW62cFd1Icll3YUaeg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_H2_9r_W62c_Fd1_Icll3_Y_Uaeg_b574aa7f63.png)

The distribution has a positively skewed tail at the very extreme as we can see. Also getting the **skewness** as 19.118939 and **kurtosis** to be 585.672879, depicting the skewness value>1 and kurtosis is much high indicating presence of good amount of **outliers**, we will look later into this when we handle outliers!!

## Checking the relationship between numerical variables

The famous **price** vs the **minimum_nights**!

We’ll be finding the relationship between these two numerical variables using seaborn scatter plot as below:

![1_IasxTFQx8fyxKsIq2g5K7g.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Iasx_TF_Qx8fyx_Ks_Iq2g5_K7g_6fb662a7ca.png)

There’s an interesting lookout from the above scatter plot, what do you see?
many data points are clustured on 0 **price** range, few have min nights for stay but price is 0. looks like anomaly in price.
Let’s see the boxplot of this price column to have a feel of the presence of **outliers**. Don’t worry, I’ll be handling these outlier values!

![1_62L6rYwvs5L3vJHOFwpl1Q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_62_L6r_Ywvs5_L3v_JHO_Fwpl1_Q_d3d1ed2bb5.png)

Also let’s check the **correlation** matrix to understand how are the features interrelated with each other. I have plotted using seaborn heatmap to understand the strength between the variables used.

![1_7qy9f2OLY-SHzD1NiJ8NoQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_7qy9f2_OLY_S_Hz_D1_Ni_J8_No_Q_0e11da0af9.png)

## So, which are the most important features in this dataset from the above heatmap?

There’s correlation among host_id to reveiws_per_month & availability_365 (sequential color bar is used between value and color). Also there’s noticiable correlation between min_nights, no_of_listings_count & availability_365. Price also shows some correlation with availability_365 & host_listings_count.

no_of_reviews and reviews_per_month gives almost the same information. so we can carry out analysis with any of the two variable. Also, no_of_reviews is correlated to availability_365!

## Let’s understand outliers and figure out some ways to deal with such anomalies in data

I have done most of the common **data pre-processing steps** like missing values treatment, checking duplicate records but here comes the very important part in EDA which many overlook before fitting a ml model is removing the outliers values as many machine learning algorithms do not support **missing values** and also making ml models robust to **outliers**.

Well an **outlier** is a data point that lies outside the overall pattern in a distribution. Say we’re trying to understand the people’s income based on start and end of a project. We might measure income levels of our sample group at the start and the end. Imagine our results follow a linear distribution and looks like this:

![1_t91kV6cMh6PtZ8LzKFRROQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_t91k_V6c_Mh6_Pt_Z8_Lz_KFRROQ_9c11707b1d.png)

I think by now you’ve gotten an idea, what’s that point outside of that distribution doing here? well, you guessed correct that’s an outlier in this distribution of data. I’ll be more simple, suppose there’s a list of data points like: [5,7,8,7,8,10,11,11,15,10,10000]

So by now you must be able to get the intuition what can possibly a outlier value in this list of data points. I hope you guessed correct and going by that let’s also know where do this outlier values come from?

- An accidental response that was recorded correctly.
- A Data entry error.

Also, there can be any low range or high range outliers and you can pretty well find out using **IQR(Inter-quartile range)** approach. A commonly used rule says that a data point is an outlier if it is more than 1.5. IQR above the **third quartile** or below the **first quartile**. Said differently, low outliers are below **Q1–1.5*IQR** and high range outliers are above **Q3+1.5*IQR.**

So, let’s come to the main objective here(obviously you can do some google search to know about these). There’s also another way to handle outliers using the numpy percentile() function on the nth percentiles of the dataset: [25~Q1,75~Q3] and subsetting the dataframe to respective upperbound and lowerbound values for the variable containing the outliers.

In this Airbnb dataset, I have used the **IQR approach** to handle the outliers as it had performed the best in removing almost all the extreme values present.

Now, let’s check the box plot once again as by now I have removed the outliers.

![1_ZMqtR27548boRyBj4yydKA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Z_Mqt_R27548bo_Ry_Bj4yyd_KA_e8bf3fb379.png)

See? how good the box plot looks and the whiskers are also visible well enough which literally got squeezed above due to upperbound outliers.

Well, must be curious to see the distribution of price after removing of outliers?
Let’s check this/

![1_gMdTufLXF_0rlQwyaEwzeQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_g_Md_Tuf_LXF_0rl_Qwya_Ewze_Q_5ebec21703.png)

Now, lets do some **univariate analysis**(working with single variables)

## Univariate Analysis on Airbnb data
 
Next, I’ll be showing plots representing the count of Airbnb’s in different neighbourhood groups and neighbourhoods of NewYork City. From the plot, we can easily visualize that maximum number of houses or apartments listed on Airbnb.

![1_jR-PZz-WZSwrlileLnmj4w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_j_R_P_Zz_WZ_Swrlile_Lnmj4w_f5e37122bf.png)

Well, by now you have easily got to know the neighbourhood group having the highest no of listings in NYC.

![1_SHEx_TWyLQ71H9g7siW8KA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_SH_Ex_T_Wy_LQ_71_H9g7si_W8_KA_e053e49e6e.png)

Okay, let’s not forget to have an idea of the **count of room types** in NYC based on their listings in different neighbourhood groups.

![1_S2bkq9CRQ198ZrY0eOnczA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_S2bkq9_CRQ_198_Zr_Y0e_Oncz_A_18ec78d9d7.png)

## What to conclude from this?
**Manhattan** has more listed properties with **Entire home/apt** around 27% of total listed properties followed by **Brooklyn** with around 19.6%.

**Private rooms** are more in **Brooklyn** as in 20.7% of the total listed properties followed by Manhattan with 16.3% of them. While 6.9% of **private rooms** are from **Queens**.
Very few of the total listed have shared rooms listed on Airbnb where there’s negligible or almost very rare **shared rooms** in **Staten Island** and **Bronx**.

We can infer that **Brooklyn, Queens, Bronx** has more** private room** types while **Manhattan** which has the highest no of listings in entire NYC has more **Entire home/apt room** types.
let’s see some **multivariate/bivariate** analysis(working with more than one variables)
## Bi-variate analysis on Airbnb

Lets now check for distribution of price across: Manhattan, Brooklyn, Queens, Bronx & Staten Island.
Instead of checking distributions for each categories one by one we can simply do a violin plot for getting the overall statistics for each groups. But we’ll get to know the median of price/neighbourhood group.

![1_S2bkq9CRQ198ZrY0eOnczA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_S2bkq9_CRQ_198_Zr_Y0e_Oncz_A_18ec78d9d7.png)

As usual **Manhattan** being the most **costliest** place to live in, have price more than 140 USD followed by Brooklyn with around 80 USD on an average for the listings.

**Queens, Staten Island** are on the same page with price on listings.
1. Top neighbourhoods in NYC with respect to average price/day of Airbnb listings?

![1_JEAMQ6i6JfsO5-LQreYF0w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_JEAMQ_6i6_Jfs_O5_L_Qre_YF_0w_beba2622cb.png)

The bar plot above clearly depicts the neighbourhoods with listings having highest average price/day in each neighbourhood groups of NYC.

Among the top neighbourhoods in each neighbourhood groups, top 2 of them namely: **Fort Wadsworth & Sea Gate**, origins from **Staten Island & Brooklyn** respectively.

2. Room_types vs price on different neighbourhood groups?

![1_CuJerkGfmU4m7bWg8TdZ4w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Cu_Jerk_Gfm_U4m7b_Wg8_Td_Z4w_2bef8d0f66.png)

I have plotted the above seaborn facetgrid to have a perfect visualization plot for two categorical variables with a numerical variable(price) and compare their price ranges across various neighbourhood groups in NYC.

Looks like a property/listing with **Entire home/apt** as room_type wins the show at NYC followed by private rooms.

**Manhattan** has the highest price for room types with **Entire home/apt** ranging to nearly 240 USD/night, followed by **Private room** with 110 USD/night. And it’s obvious being the most expensive place to live in!
3. On an average for how many nights people stayed in each room_types?

![1_xO6TzDuMY9pscjLvro6wqw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_x_O6_Tz_Du_MY_9pscj_Lvro6wqw_828c4c2348.png)

It clearly indicates that people mostly prefer living in an **entire home/apt** on an average of more than **8 nights** followed by guests who stayed in shared room where average stay is **6–7 nights.**
4. How monthly reviews varies with room types in each neighbourhood groups?

![1_0X7hXYvrmfoStevsMQjt8Q.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_0_X7h_X_Yvrmfo_Stevs_M_Qjt8_Q_3c21a1e5e5.png)

Seaborn stripplot function always treats one of the variables as categorical and draws data at ordinal positions (0, 1, … n) on the relevant axis, even when the data has a numeric or date type. So what do we conclude by this another kind of scatter plot?

So, **Private rooms** received the most no of reviews/month where **Manhattan** had the highest reviews received for Private rooms with more than 50 reviews/month, followed by **Manhattan** in the chase.

Manhattan & Queens got the most no of reviews for Entire home/apt room type.

There were less reviews received from **shared rooms** as compared to other room types and it was from **Staten Island **followed by **Bronx**.

5. Top 10 reviewed hosts on the basis of reviews/month?

![1_tleefDZcILt_da38oI0kzA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_tleef_D_Zc_I_Lt_da38o_I0kz_A_57fcba4b9a.png)

Row NYC holds the title as the most reviewed host with more than 40 reviews/month on average. For other insights on hosts please do check my GitHub repo(link at the bottom)

6. Room_types and their relation with availability in different neighbourhood groups?

![1_f9jTPNZWQe1HflLwc20drg.png](https://almablog-media.s3.ap-south-
1.amazonaws.com/1_f9j_TPNZW_Qe1_Hfl_Lwc20drg_a6579d6b3b.png)

Looking at the above categorical box plot we can infer that the listings in **Staten Island** seems to be more available throughout the year to more than 300 days. On an average, these listings are available to around 210 days every year followed by **Bronx** where every listings are available for 150 on an average every year.

7. Let’s also see what can be done with latitude and longitude?

![1_BJIxHc7nUU4ZF3bxIG7NYQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_BJ_Ix_Hc7n_UU_4_ZF_3bx_IG_7_NYQ_bfc56bf3a9.png)

**Now, let’s check for the distribution of types of rooms across all neighbourhood groups of NYC!
**
![1_rGfHw7Ht53Km_Sb3dsNV5g.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_r_Gf_Hw7_Ht53_Km_Sb3ds_NV_5g_8bffe621de.png)

By the two scatterplots of latitude vs longitude we can infer there’s is very less **shared room** throughout NYC as compared to private and **Entire home/apt**.
95% of the listings on Airbnb are either **Private room or Entire/home apt**. Very few guests had opted for shared rooms on Airbnb.

Also, guests mostly prefer this room types when they are looking for a rent on Airbnb as we found out previously in our analysis.

**Let’s have an idea of the price variations as well, across these co-ordinates in a more clear way.
**
![1_NxETkeW8MvoISkWIsU3k-w.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Nx_E_Tke_W8_Mvo_I_Sk_W_Is_U3k_w_d98f3a0334.png)

The scatterplot showing the price variables across these co-ordinates in a more authentic way using the original NYC boroughs map by saving the original map image in my local directory and then reading the image using cv2 imread function.
We can infer that there are high range of prices across **Manhattan** followed by **Brooklyn and Queens** being the most costliest place to stay in NYC.

**Listings availability in a year throughout NYC??**

![1_HUjpqAzC2gURVNsV3NYzaA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_H_Ujpq_Az_C2g_URV_Ns_V3_N_Yza_A_7fee9ec271.png)

I’ve plotted the scatterplot depicting the availability of listings available throughout NYC in a year. I have used hues with different sizes based on the availability ranges.

**Bronx & Staten Island** has listings which are **mostly available** throughout the year, might be the case as they are not much costlier as compared to other boroughs as in Manhattan, Brooklyn & Queens.
I’ve reached almost the end of the analysis. There might be few analysis which can be done more. But there’s always an ending to a story!

## End notes

Through this exploratory data analysis and visualization, we gained several interesting insights into the Airbnb rental market. This Airbnb dataset for 2019 year appeared to be a very rich dataset with a variety of columns that allowed us to do deep data exploration on each significant column presented. After that, we proceeded with analyzing boroughs and neighborhood listing densities and what areas were more popular than another, their price variations, their availability as per room types. Also we emphasized on key findings like room types and their preferred stays by guests, the top reviewed hosts and their listings.
Next, we put good use of latitude and longitude columns to create a geographical heatmap color-coded by the price of listings

I have used Seaborn and Matplotlib for creating all the visualizations. This is just a glimpse of eda on the airbnb dataset and there’s no any predictions involved. Also, here’s my Github repo for the full code reference. Please do shower some claps on this if you like it _/\_

Thanks a lot for reading. Feel free to give any feedback!

Hi! this is Sudip. This is my first article and I’ve tried to frame the whole context of data exploration and visualization through one of my project. Currently I’m on the learning process and everyday gets interesting with newer concepts and ideas to formulate.

1_BbftTDf7KptN2Ft4YbIYHw.jpeg

Exploratory Data Analysis on NYC Airbnb 2019 dataset

Before starting I would like to thank Shweta singh,Keerthana for their support and the constant research they have been doing on this topic.


## What is DropBox?

_Dropbox is a cloud file storage device. It enables users to share the files across their devices and also with remote storage servers. It does that by allowing the users to create special folders on each of their computers or mobile devices, which the service then synchronizes so that it appears to be the same folder regardless of which computer is used to view it. Files placed in this folder also are typically accessible through a website and mobile apps and can be easily shared with other users for viewing or collaboration._

**These are the table of contents that will be followed throughout the blog:**

![1_31WrLaLAWR9chf4mVk5tuw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_31_Wr_La_LAWR_9chf4m_Vk5tuw_f5e61955dd.png)

The above flowchart describes the entire architecture of dropbox starting from the client-side, passing through the cloud server, and the several services that are provided by the DropBox services.

## Conclusions:

- The introduction of cloud-based services when it comes to storing large amounts of files has made our lives so much easier.
- We all have used storage-based cloud services at one point in time, but without knowing the entire architecture behind it.
- DropBox has recently seen an increase in its annual revenue in the first quarter in 2020, with paying users ending at 15.48 million, as compared to 14.31 million for the same period last year.

The reason for such a huge increase in the revenue of DropBox is because of the application of ML algorithms. They had a clear vision of their problem statement where they wanted to have a broader user base, and in turn, increase revenue. They used clustering algorithms to segment different user groups together, and provided benefits to users having less likelihood of using DropBox.

**Goals of a DropBox System:**

Given below are the several goals/objectives that are taken into consideration before designing the DropBox system UI:

- Users should have the flexibility to upload and download files whenever they wish to.
- The flexibility of sharing files/folders with anyone.
- Synchronization between devices.
- The system should allow the storage of large files.
- File operations should follow an ACID approach.
- Supporting offline editing. Flexibility to create/delete/edit files even when the users are offline. And they should get synced, as soon as the user becomes online.
- The system should also provide security services in the form of encryption to the files stored.

**Functional Requirements:**

The basic system behavior can be defined by the Functional Requirements. These are what the system does, or focuses on doing. In the simplest of terms, these requirements need to be met, if we want a proper functioning of our system.

- Users can be further divided into two parts: Free Users and Premium users. The free users will be provided with limited capabilities, limited network bandwidth, and storage. Premium service is a paid service, storage size is larger than that of free users.
- A user can also create a root folder if the person wants. And the benefit of the cloud service is that the changes made in the files if synchronized with the cloud will also get reflected in all the user's devices.
- The next question that arises is what is the maximum file size that is allowed? The maximum file size supported is 2Gb.
- A user should be able to share files or folders with other users. Once a file or folder is shared with other users, any updates to the file or the folder are synchronized automatically to all other users' devices. Sharing of a folder enables sharing of all files and subfolders under that folder.
- One of the major advantages of DropBox is that the user can also operate it in an offline mode. The changes get synchronized to their files whenever they switch to an online mode.
- File Versioning and easy rollback: Dropbox supports multiple versions of a file or folder that the user is operating on, such that if the user is facing some serious difficulty in their present version, the person can easily roll back to the previous version if he/she wants to.
- One of the major requirements can be its file search system. If a user has thousands of files stored in DropBox, the person should have the capability to search for the file he/she is looking for, instead of searching it manually which would be time-consuming.
- Providing a mechanism for resolving conflicts by the system if the same file is updated by a single or multiple users.

**Non-Functional Requirements:**

As we learned in the previous section that functional requirements are those requirements that are required for an optimized performance of the system, whereas non-functional requirements specify the approach of the system to deal with functional requirements. They don’t affect the functionality of the system, even if they are not met, the system will still perform its basic operations.

- DropBox should be highly available and fault-tolerant.
- Service needs to be highly scalable with increasing load.
- Minimizing network bandwidth consumption and file transfer latency through file synchronization. (This is one of the most important non-functional requirements as this is the reason the files get divided into smaller segments, also known as ‘chunks’. This will enable the user to modify/upload/download modified chunks of a file. Even if the action fails, the user just needs to retry that particular action with that chunk, instead of retrying for the entire file.)
- Guarantee on the ACID system.

**What is the ACID system in DropBox Design?**

Whenever we deal with system designs in a cloud storage platform, the concept of ACID is extremely important. ACID is made up of four independent functionalities including Atomicity, Consistency, Isolation, and Durability. Let’s get an idea of these four functionalities and what they represent.

- **Atomicity:** One of the properties of a cloud storage platform like dropbox is this if a file is changed from one version to a different one, that file is replicated/updated in all the other devices as well. Suppose the user is trying to access the updated file from a different device. The files get updated in the form of ‘chunks’, so the user trying to access the file from a different device may see the file getting updated in a transient way (that is in the form of chunks). The user should not see all the changes appearing in the form of chunks. To counter this, all the chunks in the user’s dropbox application are stored in a temporary file in a temporary location, where all the changes are first applied. After all the changes are applied to the file, we can link the actual file with the temporary file.

- **Consistency**: This functionality means users should not be able to read files that don’t make sense together. If a user has updated a file in one device, the second user operating on a different device should not see a partial change in that file. The system may crash if the second user tries to operate/modify that partially updated file.

- **Isolation:** In terms of DropBox service, this means that two different files having been dealt with differently by the user, should be totally independent of one another. Changing or modifying one file should not modify a different file. Secondly, if a file gets updated on multiple devices at the same time, one file will have to wait for the other write to complete first.

- **Durability**: Changes made to a file will not get lost. If a file has been created and then modified based on the choices of the user, and that file has been updated to the remote server, these changes should not be lost. This is one of the best advantages of cloud storage services, where changes are preserved.

In the next section we will be looking at some of the basic design APIs that DropBox uses:

![0_jYShK8d7BEgDhztm.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_j_Y_Sh_K8d7_B_Eg_Dhztm_670d75adc1.jpg)

## DropBox Design APIs:

API stands for Application Programming Interface. It acts as a medium through which two or more applications connect with each other. It acts as a messenger that delivers the request of a user to the application where the user is requesting information, and delivers the response back to the user. Different design APIs for Dropbox is as follows:

- File Upload (user token, File Metadata, File content)
- Update file metadata (user token, file id, file metadata)
- Delete file (user token, file id)
- Share file (user token, file id, second user-id, permissions)

**Overall Architecture of DropBox:**

Given below is the entire architecture of DropBox where the box on the left signifies the operations on the client-side, and that to the right are all the services that DropBox provides.

![1_Az--TfTDrOSfdeAUUxwnqg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Az_Tf_T_Dr_O_Sfde_AU_Uxwnqg_0c148dfc18.png)

We all have used DropBox once in a while, without actually knowing how the system works. This blog will cover all the insights related to the different functional requirements as well as the system design of DropBox. So let's start learning...

1_8_ZwhwsaQph-MO6nmpoOcw.png

DropBox System Design

We all are acquainted with e-commerce platforms and have been using them for various reasons, be it any online shopping platform like Flipkart, Amazon, and Netmeds. Due to the increase in the cases of the Covid-19 in the year 2020 and 2021, people have even started ordering grocery items from online retail stores like Grofers.

With the introduction of Artificial Intelligence, e-commerce platforms are making effective use of data such that the supply of essential items runs in parallel with the demand of items. 

Although there are multiple ways in which an MNC may consider using Machine Learning algorithms, we will be focussing on only those algorithms that are considered crucial to any e-commerce platform. These are the following topics that we will focus on:

_**Time Series Forecasting, Recommender Systems, Advanced NLP**_

### The use of Time Series in E-Commerce

![aa1.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/aa1_656859958e.jpeg)

Before discussing the details about the use of Time Series Analysis in E-Commerce platforms, let’s try to answer some questions:

- We all have heard about Big Billion Days in Flipkart right? According to Flipkart, the Big Billion Days is responsible for almost 40% of the overall annual sales. Out of the highest selling items, mobile phones are one of them. So what may happen, if the Flipkart distribution stores run out of mobile phones on the second day itself?

The above hypothetical scenario may lead to devastating consequences in terms of revenue and user experiences. So, it’s necessary for every E-Commerce platform to predict the number of items that they should have in stock before such a huge annual event.

In such cases, the concept of Time Series Forecasting comes into the picture. Time Series Forecasting consists of three extremely important components, and we will try to explain the use of each component in huge MNC’s such as Amazon and Flipkart.

The main components of a Time Series Analysis are as follows. We will not be discussing the residual component in this blog.

- **Trend**: In statistical terms, it is nothing but the change in mean over time. The trend gives a measure of the dependent feature with respect to time which can be at an yearly, monthly, weekly, daily, or even at an hourly level. With respect to our discussion on the Big Billion Days, suppose we are working on the dataset which gives us information on the number of mobile phones sold on a monthly basis and let’s say we are monitoring the sales of mobile phones. The rise/fall in the line chart depicts whether the sales are increasing/decreasing over time.

- **Seasonality**: In statistical terms, it is the change in standard deviation over time. Sticking to our example, let’s consider a hypothetical scenario. The number of mobile phones sold in Flipkart annually is much higher in the months of October-December, and the least sales are in the month of May. This is because of the ‘Big Billion Days’ and ‘Diwali Sale’ in the month of October and November. The rise in sales (in this case, mobile phones) at a particular time of the year is given by seasonality.

When it comes to E-Commerce platforms, customer satisfaction is always the primary objective. And this is the reason Time Series analysis is one of the most important modeling techniques that is used by every E-Commerce platform. Advanced Modeling architectures like **Facebook Prophet, ARIMA, and SARIMAX, **Machine Learning algorithms as well as advanced Deep Learning algorithms like **LSTM** are employed, to get the most accurate forecasting results.

### Recommender System in E-commerce

A recommended system works like a well-trained salesman who is good enough in cross-selling and upselling the products. The recommended system uses information like reviews and ratings of customers for different products to suggest good products with a higher price.
In an e-commerce context, recommender systems have an influence on financial performance as well as the intensity of consumer conversation.

Specifically, recommender systems can enhance e-commerce in three ways:

1. Conversion: Turning Browsers into Buyers Increasing the proportion of visitors to a website that makes a purchase. Retailers can use recommender systems to locate goods that best suit customer’s interests and inclinations, which may involve accidental purchases prompted by serendipity from the suggestions.

2. By increasing Cross-sell: Cross-selling improved by the recommended system by suggesting additional products and services to customers. If the recommendations are good, the typical order size increases. as an example, a site might recommend additional products within the checkout process, supporting those products already within the handcart.

3. By building loyalty: Building customer loyalty becomes an important aspect of business strategy. Recommender systems can improve loyalty by creating a value-added relationship between the location and therefore the customer. Whenever a customer visits an internet site, the system learns more about that customer’s preferences and interests and is increasingly ready to operationalize this information to personalize what’s offered. By providing each customer with an increasingly relevant experience, a corresponding improvement within the likelihood of that customer returning is achieved.

_**Content-based recommender systems**_

A content-based recommender makes recommendations to a user by matching the outline of an item and the profile of the user’s interest. There are three important aspects to a content-based recommender namely, matcher, the item descriptions, and the user profile.

“The user-profile” consists of different types of information:
- A description of the items-types that interest the user
- A user’s interaction history with the recommended system

**_Collaborative filtering systems_**

Collaborative Filtering systems make recommendations exclusively based on knowledge of users’ relationships to items
Within Collaborative Filtering CF we distinguish two important classes, user-based and item-based, supplemented with several optional variations

User-based CF algorithms start by finding a set of neighboring users who purchased or rated items overlapping this target user’s purchased or rated items. These likely-minded users are found using user similarity values.
An alternative to user-based CF techniques is item-based CF, a technique that compares each of the users’ purchased or rated items to related items and then combines the most similar items into a recommendation list.

### NLP in E-commerce

![aa2.jpeg](https://almablog-media.s3.ap-south-1.amazonaws.com/aa2_7fc5d1cc9f.jpeg)

Computers are generally not designed to understand what we communicate as humans. They communicate in code, which consists of long lines of ones and zeros. If we think that humans and computers can’t truly connect we are wrong, they are connected and can communicate. E.g. Computers respond daily to our search terms, even voice commands with smart products such as Siri, Alexa. These all things are today possible with the help of Data Science.

Retailers are nowadays selling online, and they want to increase customer satisfaction and NLP plays an important role not only in helping customers in the front end, by suggesting some keywords in the search bar, but also retailers at the back end.

It’s human nature if we don’t like the product we don’t buy during shopping offline, but it’s not the case here in online shopping. After getting the product the customer can return it, add comments to that particular product along with the ratings. We can check the ratings manually but let’s say if we have thousands of products. With the help of NLP, a company can understand the behavior of a customer to a particular product, and that will help the company to understand or to rate the seller. Additionally, with the help of search history and comments of a customer, we can predict what a customer actually wants, and show them the relevant ad, or recommend the product directly to save their time.

It can also be used in validating the addresses of customers for example if a customer is putting a wrong or incomplete address, With the help of NLP we can come up with a solution that shows the error if a customer will put this type of address. Also, the retailers have a lot of data of customers and their addresses with the help of which they can suggest the complete address to the customer while typing or adding a new address, which in turn is a good experience.

### Efficient Customer Support

Online e-commerce platforms are focusing more on automation, and at the same time trying to increase customer satisfaction. Companies invest a lot of their resources when it comes to Customer Support Executives so that they can attend to any customer whenever a ticket is raised. However, due to a large number of customers, it was often noticed that the ratio of the number of tickets raised and the number of executives are quite high, which made it impossible to attend to every customer, and this led to poor customer satisfaction.
To counter this issue, E-Commerce platforms are taking the aid of Machine Learning and Deep Learning algorithms, by building chatbots.

What are **Chatbots** and how do they work?

Chatbots are automated AI-driven programs that can help simulate a conversation. It is very similar to a customer getting his/her needs attended by an executive through a chatting platform. The major challenge in building a chatbot was the ineffectiveness of a machine to understand data, and at the same time storing the semantic information of a sentence being fed to it. The role of a chatbot is not only to understand what is being asked of it, but at the same time it should give replies to the customer, and the replies must have a proper grammatical structure to it as well. This is why companies use advanced Natural Language Processing tools like Transformers and Berts which have made life much easier for an E-Commerce platform. It’s like having an extra set of executives, working for the MNC.

**Conclusion**

- The need for NLP solutions is expanding as businesses increasingly communicate with consumers in their own language.

- According to previous industry research, the NLP market will expand at an annual rate of 18.4% and will be valued at $13.4 billion by 2020.

- It’s no surprise that it’s exploding. Retailers must get things right 

- the first time, in an inventive environment packed with time-pressed customers.

- Effective sales have always required two-way communication. That hasn’t changed, even though we’ve all gone digital. Thanks to NLP.

Written by
Jatin

https://medium.com/almabetter/data-science-in-e-commerce-1acaa5699f20

We all are acquainted with e-commerce platforms and have been using them for various reasons, be it any online shopping platform like Flipkart, Amazon, and Netmeds. Due to the increase in the cases of the Covid-19 in the year 2020 and 2021, people have even started ordering grocery items from online retail stores like Grofers.

1_lnvQuyXaIEAg_AVLpr57ng.png

Data Science in E-Commerce

This article aims to make you understand the basic data preprocessing of various data types before any machine learning process. The process of cleaning data prepares the available data for further machine learning process to reach the client’s requirement.

1_e7lYKpF5FJYjNMVPlQgaKg.png

DATA PREPROCESSING

The classic definition of machine learning says, how you learn from your experience and apply that learning for future prediction, with that being told it is a little confusing to understand unsupervised learning. So when you are dealing with unsupervised learning there is no Target variable. You can not calculate your model’s accuracy/performance, so how will you improve or how will you judge your learning.

So, when you start looking at the actual application of unsupervised learning you realize its true potential.

Imagine you are a part of a Data Science Team and your goal is to suggest a scheme or offer for a credit card holders for a bank, or suggest movies for an OTT platform subscriber in these type of problem where you are taking a new step for the first time, you don’t have any experience but just based on the Hidden Pattern in your Data you are about to take some action, this is what is the main power of Unsupervised learning is.

So Unsupervised learning is a Prerequisite in Supervised learning.

**Types of Unsupervised Learning:**

The most Unsupervised learning we see are-

· K means Clustering

· Hierarchical Clustering

· Recommendation Systems

Here we will briefly discuss Clustering techniques and the ideas behind them. More on the Recommendation system later.

**K means:**

The idea of K means is that, you cluster points based on the distance between them and then you can assume that points belonging to the same cluster will have similar characteristics. In easy words, you can say it is like grouping based on centroids of clusters. Now how those clusters are formed is a very interesting concept on its own.

K Means clustering works on the principle of Expectation-Maximization. It is a two-step process.

Imagine you have a huge distribution of Data Points. Our goal is to find k clusters in the distribution where each point of the cluster is closest to its cluster center than any other cluster’s center. This step can not be achieved at once(it is one of the reasons that Clustering takes time for a huge Dataset).

First, we will randomly choose k points in our distribution and then based on them assign each point to a particular cluster based on the distance between the points, this way we will form an initial cluster. Now when we have made an initial cluster, we will adjust the center of this cluster to the centroid position of the cluster, after this, we will again assign all the points based on this new cluster center.

![1_yHrIXPmeryYQGtxz4hK7mg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_y_Hr_IX_Pmery_YQ_Gtxz4h_K7mg_ef9740eb4e.png)

This process will be repeated again and again, till we converge to a point where not much change happens in the clustering process. So in our process, the step where we assign the points to the cluster is called the Expectation step, and the step where we adjust the cluster center is called the Maximization step.

As we have seen that, the lack of pre-requisite is the main problem in unsupervised learning. The biggest challenge in K means clustering is of finding the optimal value of K . It can also be the case that the business expects us to do the clustering for a pre-decided number K, but most of the time it is up to us to decide the optimal value of K.

To do that we have two metrics to calculate,

**WSS** — Within sum of squares = Sum of the distance of each point from the center

**Silhouette score**- The idea is similar to that of WSS, the formula will be a little different.

![1_i0jncWULWy6ZjI4ShITeVA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_i0jnc_WUL_Wy6_Zj_I4_Sh_I_Te_VA_d577cacd3e.png)

Once we have calculated these values for different values of K, we can plot the value of WSS/Silhouette score we are getting for each, and the point where the value starts to converge can be taken as the right value of K, which then has to be verified by the business anyway.

![1_IjZUxSAp0KBBJmoQpD7h9g.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_Ij_Z_Ux_S_Ap0_KBB_Jmo_Qp_D7h9g_bb6265d5b9.png)

For example, imagine a Credit Card company trying to divide its customers into segments, so if you find 3 or 4 as the optimal value of K, then they will be fine, but let’s say you found K=11 then that does not make much sense.

Another issue with K-means clustering is that people confuse it with the K nearest neighbors since both of them are classic machine learning techniques, the difference being that K means clustering in an Unsupervised learning Algorithm and K nearest Neighbor is a Supervised learning Algorithm.

**Hierarchical:**

This is the other most used way of clustering you have two options either you move from top to bottom, giving rise to Divise Hierarchical clustering or you move from bottom to top, giving rise to Agglomerative clustering.


Generally, we notice that Agglomerative Clustering is more used, as you can limit the grouping forming a given number of clusters.

So you start from a single data point and start grouping points that are closer to each other. So let’s say we are in the middle of the process, so you would have the chance to assign a particular point to any closer forming group and that distance between two groups or one group and one point is calculated by the idea of **Linkages**.

There can be many types of distances that we can have between two points in distribution, like Euclidean , Manhattan Distance, etc, but when we want to calculate the distance between two groups of data points it little tricky.

We may calculate the distance between the two closest points of two groups or two farthest points from both the groups, or we may take the distance from the centroid of one cluster to the centroid of the other, or you may take an average of the distance of all points in one group with all points in another group. All of these forms different kinds of Linkages.

But there is a more intuitive method, so for every group, we can calculate WSS, ie the sum of distances of all points from the center of that cluster.

So, now for G1, we calculate WSS1 and G2 we calculate WSS2 then we take G1 and G2 as a single group and calculate WSS12, now we find the value of

(WSS1+WSS2) — WSS12 = delta.

So now for 3 groups G1, G2, G3 we can use this delta value to find which groups are closer to each other. Closer groups will have a low value of delta.

This is called the Wards method of finding distances between two clusters of data points and is a backbone of what we know as Hierarchical Clustering. It may not tell the exact distance but will tell you which groups are closer.

What we get from Hierarchical clustering is a Dendrogram, and based on our understanding of how many clusters we want we can cut the dendrogram by a horizontal line, which gives us a clear view of the kind of clusters formed.

![1_G_snm6o3Z4IY5Zji000Ykw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_G_snm6o3_Z4_IY_5_Zji000_Ykw_2515e36246.png)

We saw , the two most classic clustering technique, more on similar topics later.

Hope , these clustering techniques are used with cautions and don't create a division based on some stereotype of the society.




The classic definition of machine learning says, how you learn from your experience and apply that learning for future prediction, with that being told it is a little confusing to understand unsupervised learning. 

1_HYq2ifXuQZJUXxbt62VBiw (1).png

Clustering — How we are “Closely Related”!

CatBoost is a new machine learning algorithm based on gradient boosting. This algorithm was developed by researchers and engineers at Yandex (Russian tech company) in the year 2017 to serve multi-functional purposes such as Recommendation systems, Personal assistants, Self-driving cars, Weather prediction, and many other tasks. This algorithm performs better than existing boosting algorithms like XGBoost, LightGBM etc in various aspects which includes training period, accuracy, computational power, tuning hyper parameters, implementation etc.

In CatBoost, the term ‘Cat’ refers to Category and ‘Boost’ refers to Boosting, this doesn’t mean that CatBoost is restricted only for categorical features it also supports Numerical and Text features but it has an effective handling technique for categorical data.

The main difference between CatBoost and other boosting algorithms is that the CatBoost uses symmetric trees. And builds them level by level.

![1_9gglhNoL2cgGKE_WC6YXIg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_9gglh_No_L2cg_GKE_WC_6_YX_Ig_81efed2332.png)

Symmetric tree is a tree where nodes of each level use the same split as shown in the above picture. This allows to encode path to leaf with an index which helps in decreasing prediction time, and its extremely important for low latency environments.

**How does CatBoost works ?**

Let’s look into the dataset which has 10 data points ordered in time.

![1_wNyplCXQUOWcGnPCtXUPrQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_w_Nypl_CXQUO_Wc_Gn_P_Ct_XU_Pr_Q_f95cad906d.png)


In order to Calculate residuals for each data point using a model, that has been trained on all the other data points at that time(suppose if we want to calculate residual of x5, we suppose to train model using x1,x2,x3,x4 data points), this procedure becomes computationally expensive when we have large set of data points. In that case instead of training different models for each data point, it trains only log(number_of_datapoints) models. If a model has been trained on n data points then that model is used to calculate residuals for the next n data points.

In the above dataset, we calculate residuals of x5,x6,x7, and x8, using the model that has been trained on x1, x2,x3, and x4 data points. This process is known as ordered boosting.

CatBoost divides a given dataset into random permutations and applies ordered boosting on those random permutations. By default, CatBoost creates four random permutations. With this randomness, we can further stop overfitting of our model, and the randomness can be further controlled by tuning parameters.

**What are the features of CatBoost ?**

- Great quality without parameter tuning : Since the default parameters of CatBoost itself gives better result, there is no need of tuning hyper parameters, which in turn reduces the time spent for tuning.

- Categorical features support : CatBoost supports working with non numeric factors because of which time spent for pre-processing the data will be eliminated, and improves the training result too.

- Fast and scalable GPU version : Training the model on GPU gives better speedup compared to training the model on CPU. CatBoost efficiently supports multi-card configuration for large datasets.

- Improved accuracy : CatBoost reduces over-fitting while constructing models with a novel gradient-boosting scheme.

- Fast prediction : CatBoost uses distributed GPUs, this feature enables CatBoost to learn faster and make predictions 13–16 times faster than other algorithms.

**When to use CatBoost ?**

- Where short training time on robust data is required.

- when you are working on dataset which has categorical features, and you want to get rid of converting these features into numerical format.

- when you need to choose model, which is incredibly faster than many other algorithms.

**When to not use CatBoost ?**

- Where tuning parameters of categorical features are necessary to optimize the model.


**How to implement the CatBoost algorithm ?**

i. Import the libraries/modules needed.

![1_TRAeEA0nNH8FuBiAem7lkg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_TR_Ae_EA_0n_NH_8_Fu_Bi_Aem7lkg_2357e7e8cc.png)

ii. Split your data into train and test.

![1_j1baCG0N3OIrhbVJn645uw.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_j1ba_CG_0_N3_O_Irhb_V_Jn645uw_71ce053df2.png)

iii. Train your model using CatBoost.

![1_jB0Gk4age6Gux8I1AdHmMQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_j_B0_Gk4age6_Gux8_I1_Ad_Hm_MQ_402add4db2.png)

iv. Finally, predict the model results and evaluate your model.

![1_mi3LBErqQwPBs-9abalXXQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_mi3_LB_Erq_Qw_P_Bs_9abal_XXQ_843f57656a.png)


**What are the applications ?**

i. Weather Forecasting : Accurate forecasting of meteorological elements (such as wind, temperature, humidity,rainfall etc) are critical and also can be widely used in many fields. The data is continuously observed and recorded by data mining technique and CatBoost algorithm is highly anticipated to discover future meteorological elements and to improve the accuracy of weather forecasting.

![1_0s4KLewvR5-Aei8gD8PaPg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_0s4_K_Lewv_R5_Aei8g_D8_Pa_Pg_2df8001e58.png)

ii. Fraud Detection : Set of process and analyses that allow business to identify and prevent unauthorized activities in various sectors which includes Finance, Banking, Insurance, Medicare etc.

![1_rdNzH3rDZ1F1yI6tTOxi5g.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_rd_Nz_H3r_DZ_1_F1y_I6t_T_Oxi5g_77f6108f52.png)

iii. Sales Forecasting : Nowadays sales forecasting has become a vital technology in retail industry, which helps business owners to accurately predict the sales of thousands of products and make optimum decisions based on them.

![1_keAyAxfcfIn9pFdlAQtJNg.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_ke_Ay_Axfcf_In9p_Fdl_A_Qt_J_Ng_d31da39f73.png)

iv. Loan Default Prediction : Loan default occurs when the loan borrower fails to make timely payment or stops making payment to bank or any financial company, therefore having a model that could predict loan defaulters would be beneficial to banks or financial institutes before approving the loan for particular customer .


![1_aRQ9OQozcGH6k7VCMAAMTQ.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_a_RQ_9_O_Qozc_GH_6k7_VCMAAMTQ_f8d52688d8.png)

and many more applications.

So this is the short overview of CatBoost algorithm, use this algorithm to train your model where you need your results to be two times faster than LightGBM, and twenty times faster than XGBoost.





‘CatBoost’ doesn’t it sound like we boosting some cat from the word ‘Cat’ + ‘Boost’, Let’s deep dive into it, to know what actually it is.

1_1qbSAsx6nAc0oyxtgd9Fpg.png

CatBoost: The Fastest Algorithm!

Human beings are the most advanced species on Earth. Our success as humans comes from the ability to communicate and share information. That is precisely where the concept of developing a language comes into the picture and when we talk about the human language it is one of the most diverse and complex part of us considering a total of close to 6500 languages.

Another very well known fact is that in the 21st century, according to industry insights only 21% of available data is present in a structured form. Data is being generated as speech, audio, tweets, and messages i.e. majority of the data is present in the textual format thus being highly unstructured in nature. Now in order to produce significant and actionable insights from this data it is important to get acquainted with the techniques of text analysis and natural language processing aka NLP.


### Introduction

**Text Mining** or **Text Analysis** is the process of deriving meaningful information from natural language text. It usually involves the process of structuring the input text, deriving patterns from the structured data and finally the interpreting and evaluating the output. On the other hand **NLP** refers to the artificial intelligence method of communicating with an intelligent system using the natural language. As text mining refers to process of deriving high quality information from the text the overall is to essentially turn the text into data analysis via the application of NLP. That is why text mining and NLP go hand in hand.

![aa1.png](https://almablog-media.s3.ap-south-1.amazonaws.com/aa1_d42806661f.png)

Natural language processing (NLP) is the intersection of computer science, linguistics and machine learning. The field focuses on communication between computers and humans in natural language and NLP is all about making computers understand and generate human language.



### Applications of NLP

1. One of the first and most important applications of NLP is **sentiment analysis** be it on twitter, facebook or any other social media forum.

2. **Chatbots** is another very important application. This is common in many of the company websites which automate certain tasks and Q&A through these bots.

3. Another application of NLP comes in the domain of **speech recognition** which also include the voice assistants like Siri, Cortana etc.

4. **Machine translation** is another use case one of the most famous one being the google translator translating from one language to another and that too in real time.

5. Other applications include **spell check, keyword searching and information extraction and advertisement matching**.

![aa2.png](https://almablog-media.s3.ap-south-1.amazonaws.com/aa2_363a9a6d18.png)


### Components of NLP

NLP can be divided into 2 major components:

1. **Natural Language Understanding**: This is the process of mapping the given input in natural language into useful representation and analysing those aspects of language.

2. **Natural Language Generation**: It is the process of generating meaningful phrases and sentences in the form of natural language from some internal representation.

![aa3.png](https://almablog-media.s3.ap-south-1.amazonaws.com/aa3_bbf8a9d7af.png)

The understanding part is much more difficult than the generation part as it takes a lot of time to understand a particular language, the intricacies of it. Going forward lets try and understand what are the steps involved in NLP.

![aa4.png](https://almablog-media.s3.ap-south-1.amazonaws.com/aa4_c43e9f0f5b.png)

### Various Steps for NLP

1. **Tokenisation**

Its the first step of NLP. It is the process of breaking strings into tokens which in turn are small structures used for tokenisation. Eg “This is NLP” can be broken down into 3 tokens namely This , is , NLP.

2. **Stemming**

It means normalising the words into its base or root form. Eg. Affects, affecting etc. comes from the base word “affect”. Stemming algorithm generally works by cutting the end or beginning of the word taking into account a list of common prefixes/suffixes that can be found in the infected word.

3. **Lemmatization**

It takes into account the morphological analysis of the word. It is necessary to have a detailed dictionary which the algorithm can look through to link the form back to root word called ‘lemma’. Similar to stemming, it maps several words into one common root. But unlike stemming the output is a proper word which is not always the answer in stemming.

4. **POS Tags**

Once we have the root forms of the token, next comes the POS tags . The grammatical type of the tokens is called the Parts of Speech or the POS tags. Eg. verb, noun, article, adjective etc. It indicates how a word functions in meaning as well as grammatically within the sentence.

5. **Named Entity Recognition**

It is the process of detecting named entities i.e. person name , company names, locations etc. It has 3 steps primarily: Noun phase identification, the phrase classification and entity disambiguation. Following is the example of NER

6. **Chunking**

This basically means picking up individual pieces of information and “grouping” them into bigger pieces. These are also called chunks. This helps in getting insights and meaningful information from the text.

![aa5.png](https://almablog-media.s3.ap-south-1.amazonaws.com/aa5_63eac5e516.png)

All of these steps and functions are performed using the Natural Language Toolkit Language (nltk) python library and is used for various other text analysis and NLP tasks.


### Major Challenges in NLP

There are still a number of Natural Language Processing limitations and problems:

- Contextual words and phrases and homonyms
- Synonyms
- Irony and sarcasm
- Ambiguity
- Errors in text or speech
- Colloquialisms and slang
- Domain-specific language
- Low-resource languages
- Lack of research and development

Although the future looks extremely tough and challenging and full of threats for NLP, the discipline is developing at a very fast pace like never before and we are likely to reach a level of advancement in the coming years that will make complex applications look possible.

NLP refers to the artificial intelligence method of communicating with an intelligent system using the natural language.

1_ZHEPb7DgpdqA5bpTyHqBDg.png

Basics of Natural Language Processing

## The world’s largest store in every pocket


With the accelerating adoption of mobile — US smartphone penetration exceeds 40 percent today and is projected to reach nearly 60 percent in three years — digital commerce is poised to explode, bringing shopping quite literally into the palms of many consumers’ hands.7 For some retailers, mobile is already a huge factor: at designer-fashion retailer Gilt, for instance, mobile accounts for about 50 percent of daily traffic and more than 30 percent of total sales. Mobile technologies will increasingly influence every stage of the customer’s shopping journey — from personalized promotions prompted by geotargeting to in-store research and price checks, as well as to payment capabilities that offer checkout options beyond waiting in line. A recent McKinsey survey of digital shoppers highlights how mobile technology can complement the in-store experience; for example, almost half of the consumers who conduct research on their mobile phones have done so while in stores, and half say they’re open to the idea of in-store mobile payments.8 Indeed, while just two years ago mobile accounted for only 3 percent of e-commerce sales, that figure will probably rise to 15 percent by the end of 2013.9


## Highly personalized marketing

Habits of consuming content have changed dramatically. US consumers doubled their spending on digital newspapers in the past seven years, for example, while halving their spending on print newspapers.10 As more consumers abandon print media for digital media, marketers follow: 44 percent of them now allocate at least half of their marketing budgets to digital media, up from only 31 percent in 2009.11

We’re already seeing that direct mail and newspaper circulars are playing a diminished role in retail marketing. Mass advertising will not disappear overnight, but its influence is certainly waning. Ads are shifting toward not just digitization but also personalization, powered by increasingly sophisticated algorithms and predictive models that analyze transaction data and digital-media trends (for example, what topics are hot on social networks). Already, 35 percent of what consumers purchase on Amazon and 75 percent of what they watch on Netflix come from product recommendations based on such algorithms.

![0_sO_GOq8Bj1_QkMK0.gif](https://almablog-media.s3.ap-south-1.amazonaws.com/0_s_O_G_Oq8_Bj1_Qk_MK_0_43101c8914.gif)


Company-directed marketing is also competing for attention with peer recommendations through social networks, user reviews, and the like. Our research shows that for the average consumer, peer recommendations carry ten times more weight than recommendations from salespeople. Indeed, social media could well make up 22 percent of marketing budgets in five years as retailers increase their spending to facilitate and influence peer connections about brands through paid ads and branded pages on social-media platforms such as Facebook, Ibotta, and Pinterest.


## A distribution revolution

Amazon already offers same-day delivery in ten cities and guarantees one- to two-day ground delivery in the continental United States. It is not unreasonable to think that consumers will expect comparable shipping speeds from all retailers — we expect same-day delivery to become available soon in at least the top 150 metropolitan statistical areas, which hold nearly 75 percent of the population.12 Furthermore, we believe retailers will offer shipping free of charge to their most loyal and profitable customers, as opposed to providing it only for those who make minimum purchases. We also expect to see third-party distribution services evolve and expand. Some companies may make big investments in distribution infrastructure and sell it as a service to other retailers, as Amazon and eBay do now. Others are beginning to invest in infrastructure to provide convenient and secure package-delivery locations: lockers and pickup boxes are appearing in groceries, convenience stores, and drugstores nationwide, and new services are sprouting up to let retailers ship packages for pickup at other retail locations or self-storage facilities.


Consumers have come to expect simple and seamless processes not only for receiving the products they’ve purchased but also for returning unwanted products. Free and easy returns — including the ability to return or exchange online purchases in stores — are becoming table stakes.


**New retail business models**


No doubt, retail competition just keeps getting tougher. Consider the ongoing blurring of lines between formats and sectors as retailers try to steal shopping trips and share from one another (for instance, fresh food is no longer the dominion of supermarkets alone but is also increasingly found in warehouse clubs, convenience stores, pharmacies, and even dollar stores). Furthermore, players across the value chain are encroaching on what used to be the exclusive turf of retailers. More manufacturers are selling directly to consumers; examples include Apple, Nike, and — via Vitacost.com — several consumer-product manufacturers. Tech players are also fighting for consumer retail dollars: Google offers more than one billion products for sale on Google Shopping and may soon open retail stores.13 Additionally, companies such as craigslist, eBay, and Etsy (home to almost a million small businesses) are creating marketplaces where individuals and entrepreneurs can sell their wares to the masses. Finally, rental and aftermarket-circulation models, such as Chegg for textbooks or Rent the Runway for designer fashion, are eating into traditional demand for retail goods.

Competition is coming from near and far as technology makes retailing much more global than it has ever been. UK online retailer ASOS.com, for example, offers free two-day shipping worldwide for a relatively small membership fee, and at times as a promotional offer to all customers. Until recently, retailers didn’t have to worry much about global competition until stores started sprouting down the street — nor did they have an opportunity to access global consumers from North America — but that is changing as technology helps break down barriers and generates new retail business models.

## Why cross-selling and upselling?

We need cross-selling and upselling because there is a very high probability of an existing customer buying a similar product at around 60–70% and around 5–15% for a new customer. If used effectively, they can increase your sales and, therefore, profits. And if backed with machine learning for marketing, cross-sell and upsell campaigns can significantly improve your business’ position on the market.


## Difference between cross-selling and upselling

Let us explain this with an example. If a customer has bought an iPhone 10, he may also be interested in buying AirPods. This would be an example of cross-selling. If he has an iPhone 10, he might be interested in buying an iPhone 12. This would be a classic example of upselling. Conversion to YouTube Premium would also be a very good example of upselling.

![0_Tqotj2_rekurFDPp.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_Tqotj2_rekur_FD_Pp_83586078b3.png)

![0_7_95KYmEP6OpE87I.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_7_95_K_Ym_EP_6_Op_E87_I_98458c3113.png)




## How Machine Learning… ?


**1. Machine Learning is a key to personalized recommendations and improves cross-selling and up-selling opportunities**

Once we have already collected more than enough data about our customers we have data about your customer’s age, location, gender, hobbies, buying history, marital status, etc. And even did customer segmentation for a better offering. That’s great, but machine learning algorithms can significantly improve our offer of personalization. Thanks to data-driven recommendations, customers will get the right offers at the right time, and, therefore, purchase more products. One example is — Amazon identifies which items are often purchased together. After that shows users potential complementary products. This trick ensures a better customer experience. As users receive exactly those recommendations which they may be waiting for.

![0_hONiWYxOAGDlcrwP.png](https://almablog-media.s3.ap-south-1.amazonaws.com/0_h_O_Ni_W_Yx_OAG_Dlcrw_P_3f77474b31.png)


Machine learning algorithms are usually divided into two categories: collaborative filtering and content-based filtering. However, combining both of these approaches is also a popular way of building a recommender system.

**2. Machine Learning sharpens sales prediction**

One of the best things about machine learning is that it never stops learning from new data. This ability allows forecasting our customers’ behavior and expectations in the future. On the basis of historical and new data, a machine learning model can increase the accuracy of the sales forecast. This is especially important and useful when you need to predict how our customers will perceive new products or services. Cross-selling and upselling strategies, and reduces the risk of inefficient marketing.

![1_95FKsaBs3DxJKcTtUDuRDA.png](https://almablog-media.s3.ap-south-1.amazonaws.com/1_95_F_Ksa_Bs3_Dx_J_Kc_Tt_U_Du_RDA_a283a7cb2d.png)

Apart from this, using machine learning and customer predictive analytics, it is possible to identify the most effective sources of contact with clients. Again, this can help us to polish your selling strategy and gain more profits.

**3. Machine Learning makes dynamic pricing possible**

Dynamic pricing is among the latest pricing trends — it implies continuous altering of product prices, in reaction to real-time demand and supply. This model allows better control of the pricing strategy, gives flexibility without reducing the brand value, and saves budget over the long run. And, dynamic pricing can be a great help when we are cross-selling or upselling our products.

![0_JZuuGqA_Op5v7l6x.jpg](https://almablog-media.s3.ap-south-1.amazonaws.com/0_J_Zuu_Gq_A_Op5v7l6x_ddd4557289.jpg)

However, it is virtually impossible to efficiently monitor other items and follow the real-time demand and supply manually, since there is too much data to check and analyze. But machine learning can solve the problem — a properly built model will take into account a lot of factors. It will take much more than we would be able to consider without such an algorithm, provide us with precise data, and ensure much faster responses to demand fluctuations.

## Conclusion

We have tried to give you a touch of the ML. We hope you have a great learning through this article.

It feels good to work with a awesome team. I would like to thank 
Sanchita Paul & Priyanshi Singh for her invaluable contribution to this article.

## References

- McKinsey & Company- https://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers#
- addepto- https://addepto.com/machine-learning-predict-reduce-customer-churn/

That’s all folks, Have a nice day :)




Over the past decade, US e-commerce has grown at an impressive clip of almost 18 percent a year. It now accounts for 8 percent of total retail sales.

0_cPhhsWSVOSV4BpcA.gif

Amazon Cross selling & Up Selling

Keeping that in mind, AlmaBetter aims to narrow the gap between Data Science job openings and the rate of upskilling in India with their Full Stack Data Science Program. AlmaBetter’s Program is India’s #1 Data Science program designed perfectly for young data science enthusiasts as working professionals to upskill and land a job in this demanding industry. AlmaBetter, with the help of their 200+ hiring partners, also offers a placement guarantee to students.

The Full Stack Data Science Program is devised to provide robust training to young talents and give them an edge in securing a job in this promising industry. Around 70% of Data Science Program students get placed before completing the program. The Data Science Program provides an excellently- engineered and industry-relevant curriculum that focuses heavily on the practical usage of Data Science concepts. Moreover, AlmaBetter believes that they will succeed when their student thrives. Hence, the Full Stack Data Science Program delivers a risk-free learning environment for all the students. 


## This Program Includes

Lifetime access to the program (recorded lectures, study materials, assignments, and projects)
Students can easily access all the learning elements and revise the challenging concepts multiple times with Lifetime access to the program. 

One-on-one mentorship sessions with industry professionals.
Mentorship sessions with industry experts are the perfect way to expand your industry knowledge while boosting your confidence and communication skills. Every mentor has a different story, and it helps you learn from them immensely. Students can also resolve their doubts through these mentorship sessions.

5+ Industry relevant Capstone Projects
Working on Capstone projects helps students boost their technical skills, team-building skills, and communication skills. Also, it is one of the best ways to showcase your skills and prove your credibility to recruiters.

Access to the exclusive Data Science community.
AlmaBetter Data Science Community is India’s #1 Data Science community, and it is the ideal place to build your network and connect with like-minded folks. The AlmaBetter Community is the perfect place to broaden the horizon of knowledge. Students can ask questions and share tips and tricks with people from diverse backgrounds and industries in the AlmaBetter Community. With the user-friendly browsing experience on the AlmaBetter Community app, students can swiftly find topics of interest and like-minded peers. Informative discussions are also a significant part of our data science community. One can easily follow or join a discussion and impart their knowledge on the discussion channels. 





Access to Data Science Lab 

Data Science requires mastery over technical concepts in mathematics. Thus          AlmaBetter’s Data Science lab hosts a wide variety of visual experiments which students can perform and see what’s going on underneath. 


Fully refundable program fee

AlmaBetter offers the Data Science Program with two different plans, Pro Flex and Pro Live. The subscription fees are fully refundable once students complete the program and earn the Full Stack Data Science Program certification. The minimal and refundable subscription fee makes it one of the most cost-effective Data Science programs.



Flexible Learning 

The Full Stack Data Science Progam allows students to attend live lectures or learn with pre-recorded lectures at their own pace. The flexible learning environment helps students and working professionals upskill according to their convenience.



## Why this is the #1 Data Science Program


Reverse Engineered Curriculum

AlmaBetter’s Full Stack Data Science Program curriculum covers full-stack data science concepts starting from Python, Analytics Frameworks such as SQL, Excel, tableau, math concepts such as Calculus, Linear Algebra, Probability & statistics to Machine Learning, NLP, Deep Learning and ML Engineering. These in-demand skills would help students get a wide range of jobs available in the Data Science industry today.


Visual Experiments

Data Science is a relatively technical field, and that’s why the this program includes conducting visual experiments that are intuitive, interactive, and make learning fun.


Student-centered curriculum

AlmaBetter’s Full Stack Data Science Program undoubtedly has the most student-centered data science curriculum. Online training programs can be distracting, and it can be strenuous for students to sit through long lectures. Hence, they have gamified their live classes through highly interactive Kahoots.


AlmaX Placement Drive

After completing the Data Science Program, students can opt for the placement drive. The AlmaX Placement Drive makes students job-ready with their robust training and gets them placed with their hiring partners with a minimum guaranteed salary in the range of 6-25 LPA.


200+ Hiring Partners

AlmaBetter has partnered with 200+ plus growing start-ups, medium-level corporate, and MNCs based out of different cities all over the country. Moreover, the AlmaBetter team is working towards expanding its hiring network. 


According to one of the data science pioneers, Andrew Ng, data science is the new electricity. India is in dire need of data scientists. Industries all over the world have realized the importance of using data. It is fair to say that we live in a world driven by big data. Hence, the demand for data scientists is increasing exponentially. Also, Data Science is an ever-growing industry, and the required skillset cannot be mastered in a short period. It requires a deliberate approach and consistent efforts.

young-indian-man-standing-outdoor-with-laptop-front.jpeg

AlmaBetter's Full Stack Data Science Program

Let's discuss each module one by one:

 


In the first module, you will learn the fundamentals of Python programming, understand various data structures in Python and get yourself familiar with object-oriented programming. Next, you will learn to do exploratory data analysis (which is almost 60% of what a Data Scientist does) using libraries such as NumPy and Pandas. Finally, you will learn to do Data Visualization using popular libraries such as Matplotlib and Seaborn. 

 

In the second module, you will be learning popular analytics tools such as SQL, Excel and Tableau. 

 

In the third module, you will learn all the essential mathematics required for Data Science. You will learn mathematical concepts such as Calculus, Linear Algebra, Probability & Statistics. These concepts will be covered through visual experiments and hence, even if you have not studied mathematics in the past, you will be able to understand all the concepts easily.

 

In the fourth module, you will learn about a special class of Machine Learning algorithms known as Supervised machine learning, where you will develop the notions of classification and regression. You will learn about various algorithms such as Linear Regression, Logistic Regression, Support Vector Machines, Neural Networks, Decision Trees and their ensembles. You will also become familiar with popular modeling techniques such as regularization, sampling, cross validation, handling class imbalance, hyper parameter tuning and much more.

 

In the fifth module, you will learn about another special class of machine learning algorithms known as unsupervised learning wherein you will develop understanding of algorithms such as principal component analysis, K-means clustering, hierarchical clustering, Gaussian Mixture Models etc. 

 

In the sixth module, you will learn about Deep Learning - a field within machine learning based on a special algorithm Neural Networks. Deep learning has become tremendously popular and useful in the last 5 years. Furthermore, you will learn about Natural Language Processing techniques which are used to deal with textual and audio data.

 

In the seventh module, you will learn to deploy a machine learning model in production along with understanding of database technologies such as SQL. This is the module where you will learn how to convert projects into products.



AlmaBetter's robust curriculum is ideal for every data science enthusiast. The quality of content is impeccable, and the curriculum is structured to perfection for a seamless learning experience.  We strive to provide you with the most detailed and updated Data Science educational program and diversify your skillset. We have also collaborated with multiple hiring partners to provide high paying jobs to the students. 

academy-certification-curriculum-knowldege-icon.jpeg

Full Stack Data Science Curriculum

Imagine a skill school that invests in its students and alleviates the problem of unaffordability of quality education by helping students gain in-depth knowledge at ZERO upfront cost. Sounds unrealistic, doesn’t it?




Well, we at AlmaBetter are committed to turning this surreal imagination into a reality. We impart Data Science training to select students on the “Pay After Placement” model where we charge them only after they start earning above a set minimum threshold salary. Our novel concept eliminates any financial risk that students may otherwise face in the current education system. 


WHY DATA SCIENCE, you ask?


We live in a data-driven world, and guess what’s next? A data-driven future! Data Science is progressively being utilized across a varied spectrum of business enterprises. As more and more businesses realize the capability and need of data-driven decision making, the demand for data science professionals has skyrocketed. On the flip side, the supply growth remains stunted. This can be imputed to three factors- 


Unavailability of Data Science as an established course in the mainstream college curriculum. 

Unaffordability of efficient and job-oriented online Data Science courses to a majority of potential learners. 

Disconnect between recruiters and candidates. 


The question that arises is that how do we augment the supply of skilled AI workforce to meet the ascending demand?



AlmaBetter is dedicated to providing ingenious solutions to the above question. We truly believe “Talent is evenly distributed across the society; however, opportunity is not”. With our innovative “Pay After Placement” model, we can make quality training accessible to talented individuals across India, thus breaking the shackles of a risk-inscribed education system. 


This model eradicates the concept of risk in education, so that students graduate only with skills, and not with debt!


Moreover, our objectives are aligned with student’s objectives (as we earn only when they earn), we bring our utmost honesty into making the student’s learning journey fun and enriching. Let us explain to you, HOW? 


At AlmaBetter, we are challenging the traditional education system by incorporating innovative measures at each step of the student’s learning journey -  


Industry Relevant Curriculum - Our curriculum is designed considering the relevance and usage of concepts and tools in the Data Science domain. 

Innovative Teaching Methodology - Our instructors (from premier colleges like IITs/ IIMs) focus on imparting learning through experiments and gamification to make the process gripping and enthralling. They truly add the ‘fun’ while explaining the ‘fundamentals’. 

Novel Evaluation Techniques - We challenge the adequacy of the evaluation process in the traditional evaluation system (student performance is assessed based on exams conducted twice/thrice in a semester). Instead, our evaluation mechanism leverages technology to intercept behavioral inhibitions to learning at a day-to-day level. This helps us to provide a customized learning experience at individual levels.

Mentorship - We methodize one-on-one mentorships for our students, where we connect them to accomplished and experienced Data Science professionals. Such interactions provide the students with essential problem-solving skills, instill confidence in them and help them understand the business applications of underlying theoretical concepts.


To conclude, we focus profoundly on imparting skills and values beyond the usual curriculum. Our company goal is not unilateral and we put our students first. Our aim is not just to create techies but to create the technology leaders of tomorrow.


Become a part of our family and rethink education with AlmaBetter! 

Imagine a skill school that invests in its students and alleviates the problem of expensive quality education by helping students gain in-depth knowledge at ZERO upfront cost. Sounds unrealistic, doesn’t it?

serious-indian-graduate-graduation-robe-with-crossed-arms-looking-forward.jpeg

Rethinking Education with AlmaBetter

In an interview with CNBC, the Co–founder of Microsoft – Bill Gates, owed his success to the assistance of his mentor – Warren Buffet, thus reflecting upon the importance of mentorship. Through mentorship, the more experienced person, aka the mentor, provides knowledge, guidance and counseling to the person at the receiving end, i.e the mentee, to help him realize and achieve his true potential. We have come a long way from old systems of mentorship like the guru–disciple tradition, to modern mentorship programs. And throughout the history of mentorship, we can see that it builds powerful connections that have a lifelong positive impact on both, the mentor and the mentee. 


Having someone who guides, supports and encourages you through every step of your learning journey is one of the best ways to develop new skills and grow as a person, isn’t it? According to a survey conducted by Sage, out of those who have a mentor, 97% admit that their contributions were beneficial. But more than 85% of the people who want a mentor, don’t have one!


At AlmaBetter, we realize and fulfil our student’s need for mentorship. As a part of our Learn. Earn. Return (LER) program, we organize mentorship sessions for our students. Here, we have listed out the key elements of mentorship sessions for you-  


1-on-1 sessions - Each student is connected to a mentor for personalized experience so that they can chart a career path and create strategies specific to the student’s goals. 


Weekly sessions with Industry Experts – These mentorship sessions are conducted weekly,  wherein experienced professionals from the Data Science Industry connect with our students for insightful sessions. 


Mentorship from diverse experts – We have a strong family of 50+ mentors who interact with our students on a rotation basis, thus ensuring that during a student’s journey at AlmaBetter, he can learn from 20+ mentors. 


This mentorship program was designed keeping in mind the pivotal role that mentors play in a student’s life. Let me throw some light on why we focus on mentorship sessions. 


Mentors help students develop strategies fitting their needs, talents and skills, thereby helping them in career progression. When students interact with the experts, they get a pulse of how the Data Science industry works and what the current market trends are. They learn about practical applications of theoretical concepts which they can readily apply when they enter the industry.



By addressing student’s questions regarding the industry and by sharing valuable insights with them in these interactive sessions, mentors help them develop effective communication skills. With a mentor's valuable feedback and constant encouragement, students become more confident. These skills come in handy when students sit for placements.


Along with helping students navigate in their career paths, these sessions also serve as a great outlet for data science professionals to give back to the industry as they help us in building a new capable AI workforce. 


For these reasons, AlmaBetter’s data science program comprises weekly mentorship sessions with experts. And if you're on a lookout for such a comprehensive program, you know where to find it!


Join us and receive support and guidance throughout your journey from industry's best talent!! 



In an interview with CNBC, the Co–founder of Microsoft – Bill Gates, owed his success to the assistance of his mentor – Warren Buffet, thus reflecting upon the importance of mentorship. Through mentorship, the more experienced person, aka the mentor, provides knowledge, guidance and counseling to the person at the receiving end, i.e the mentee, to help him realize and achieve his true potential. We have come a long way from old systems of mentorship like the guru–disciple tradition, to modern mentorship programs. And throughout the history of mentorship, we can see that it builds powerful connections that have a lifelong positive impact on both, the mentor and the mentee. 

smiling-elegant-woman-with-tablet-showing-ok-sign.jpeg

Mentorship sessions at AlmaBetter


 Let’s go through each component of the learning kit one by one:

 

1. Live Lectures

We provide you full access to join interactive live lectures where you can get your doubts resolved during the live class itself. You will also participate in gamified live quizzes. These lectures are delivered by industry experts and rich in industry-relevant content.

We start a live cohort every month or two. You can start attending classes with any cohort.

 

2. Lecture Recordings & Materials

If in case you miss any of the live classes, or if the class timings do not suit your schedule, you can resort to lecture recordings and materials and learn at your own pace. You can use the discussion forum to get instant doubt support. You will be provided with assignments and quizzes to practice.

 

3. Team Capstone Projects

At the end of each module, you will be doing a team Capstone project. At AlmaBetter, a capstone project is an excellent way to hone in your concepts, learn to collaborate in a team and develop excellent presentation and communication skills.

 

4. Mentorship Sessions & Industry Talk

We also provide you with various resources to supplement your learning experience. Mentorship is one of them. You can schedule one-on-one mentorship sessions with our mentors. All our mentors are experienced working professionals placed at reputed companies.

Every Saturday, we invite an industry Data Science expert for an AMA session. You can interact with the professional and gain industrial exposure.

 

5. Data Science Lab

We provide the most intuitive ways to learn Data Science through visual experiments. Our Data Science Lab will help you develop strong intuition of algorithms which is a must-have to crack nuanced interview questions.

 

6. Competency Certification Challenges

Once you complete 4-5 topics in a module you can take a competency challenge. Competency challenges are a great way to solidify your conceptual learning and benchmark yourself with other Pro students.

At AlmaBetter, we take responsibility for your learning outcomes, and we are committed to delivering them daily through our consistent efforts and innovative technologies. We at AlmaBetter work relentlessly to provide you with the best educational program; hence the Pro Program learning kit is designed to bring out the best in YOU. 

sideview-students-prepearing-exams-university.jpeg

Data Science Pro Program Learning Kit

Congratulations for being selected at AlmaBetter! 


We are delighted to have you on-board with us. You are all now the Pirates of your Cohort Ship, just about ready to begin one of the most thrilling, challenging and life-changing experiences of your lives. For a successful voyage and an exciting journey, you need to first prepare yourselves. In the next few days, you need to pack up your bags and keep things handy to launch a smooth learning experience.


Here are a few things you need to get ready and take care of, before beginning your voyage; the ‘To-Do list’ before the first day:


**1. Schedule your slots**

The foremost thing to manage in a drive is time – ‘Schedule your time’. Set up a time table for yourselves. List down the class hours, time of other day to-day works, office or college timings and sort it accordingly. 


On weekdays, AlmaBetter needs you to give 2 hours i.e. 8 – 10 p.m. IST. Arrange your dinner before or after this correspondingly, pertaining to the needs of your body. Ensure that you complete your other tasks before sitting for the classes to focus properly in lectures and avoid tension of pending work. You are required to give 5 hours (2+2+1) on Saturdays, following which you’ll be handed over assignments and projects to be submitted by Sunday evenings. Picture all of this in your head and decide how and when you need to devote time and organize appropriately.


**2. Basic amenities**

Ensure that you have all the right tools and equipment for your chosen course.


Specifications:

- Wifi/Network
- Windows/Mac


**3. Talk with the loved ones**

Having honest conversations with your friends, family and kin about your expectations regarding the course and future is crucial to avoid any disagreements or obstructions in the next 32 weeks. Let them know How you feel about the course, How do you want to go about it, What changes you need to make to adjust accordingly, What do you think of your future and so on. 


A peaceful environment and a silent davenport are necessary for effective learning. Prepare to set healthy expectations and confines with your loved ones ahead of time. 


**4. Arrange Substitutes**

When you’ll be busy attending classes, you need someone to look after your children or prepare dinner for the family. Gather in your family members and make arrangements as to who will take care of the family responsibilities and work while you’re busy in class.


A systematized setup at home and mutual understanding with your father, mother, siblings or spouse about the level of commitment required to be successful at AlmaBetter will allow you a smooth run in the upcoming 8 months.


**5. Resources to give you a start**

At AlmaBetter, you will be a part of an intensive, fast-paced program for a period of 32 weeks. Matching up the pace of curriculum and skills required is important, but at the same time, difficult. We’re immersing you to a new culture, especially taking into account the ones from a non-coding background. Herein, we desire creativity, ingenuity and commitment from you. You will have access to mentors and coaches to guide you, whilst you’ll have to be resourceful and enthusiastic to communicate and learn effectively. We recommend you to go through some coding or data science programs and videos on the loose in your free time to have an idea/overview and to give a bright kick start to your course.


**6. Self-designed Learning Environment**

One of the biggest perks of AlmaBetter course is the ability to attend classes from anywhere-whether that’s at your office desk or in the comfortable couch at home. You will have access to all your study material at home. All you need to do is create your own learning territory.


Wherever you choose to learn, make sure you have a quiet, sublime and dedicated workspace. Having your lovely provisions around like your favourite pen and note-pad, favourite flower vase makes you cheerful and jolly. Have a proper table and chair seating arrangement to make the sitting hours easy on your back. Ensure the setup at a secluded place to avoid distractions from children. Since all the sessions are to be conducted via Zoom Cloud Meetings, make certain that your work background is neat & tidy, and you have undisturbed access to your network or Wi-fi.


Keep up all of this ready and heads up for the class! We will see you on your First Day!



Here are a few things you need to get ready and take care of, before beginning your voyage; the ‘To-Do list’ before the first day:

smiling-freelancer-copywriter-using-laptop-working-from-home.jpeg

AlmaBetter Blog