Data Science Interview Questions

Categorical variables represent labels or groups, such as city, plan type, or device class, and usually require encoding for ML. Numerical variables represent measurable quantities such as age, revenue, or time-on-site and may require scaling.

CLT states that as the sample size grows, the distribution of the sample mean approaches a normal distribution, even if the original population isn’t normal. This is why we can use z-tests and confidence intervals in many practical settings.

• INNER JOIN returns only rows that match in both tables based on the join condition
• OUTER JOIN includes non-matching rows as well: LEFT JOIN keeps all rows from the left table, RIGHT JOIN keeps all rows from the right table, and FULL OUTER JOIN keeps all rows from both.

While running an algorithm on any data, to gather proper insights, it is very much necessary to have correct and clean data that contains only relevant information. Dirty data most often results in poor or incorrect insights and predictions which can have damaging effects. For example, while launching any big campaign to market a product, if our data analysis tells us to target a product that in reality has no demand and if the campaign is launched, it is bound to fail. This results in a loss of the company’s revenue. This is where the importance of having proper and clean data comes into the picture.

• Data Cleaning of the data coming from different sources helps in data transformation and results in the data where the data scientists can work on.
• Properly cleaned data increases the accuracy of the model and provides very good predictions.
• If the dataset is very large, then it becomes cumbersome to run data on it. The data cleanup step takes a lot of time (around 80% of the time) if the data is huge. It cannot be incorporated with running the model. Hence, cleaning data before running the model, results in increased speed and efficiency of the model.
• Data cleaning helps to identify and fix any structural issues in the data. It also helps in removing any duplicates and helps to maintain the consistency of the data.

The following diagram represents the advantages of data cleaning:

• A population is the full set of data points you care about, like all customers in a region
• A sample is a subset of the population used to estimate population properties

A probability distribution describes the likelihood of different outcomes. Common distributions include normal (continuous, symmetric), binomial (success/failure counts), Poisson (event counts over time), and exponential (time between events).

Feature Selection is a critical step in the machine learning pipeline. It aims to identify the most relevant features from a dataset, leading to improved model performance, reduced overfitting, and faster training times.

Feature Selection Techniques

Data analysis can not be done on a whole volume of data at a time especially when it involves larger datasets. It becomes crucial to take some data samples that can be used for representing the whole population and then perform analysis on it. While doing this, it is very much necessary to carefully take sample data out of the huge data that truly represents the entire dataset. There are majorly two categories of sampling techniques based on the usage of statistics, they are:

• Probability Sampling techniques: Clustered sampling, Simple random sampling, Stratified sampling.
• Non-Probability Sampling techniques: Quota sampling, Convenience sampling, snowball sampling, etc.

Overfitting: The model performs well only for the sample training data. If any new data is given as input to the model, it fails to provide any result. These conditions occur due to low bias and high variance in the model. Decision trees are more prone to overfitting. Underfitting: Here, the model is so simple that it is not able to identify the correct relationship in the data, and hence it does not perform well even on the test data. This can happen due to high bias and low variance. Linear regression is more prone to Underfitting.

RMSE: RMSE stands for Root Mean Square Error. In a linear regression model, RMSE is used to test the performance of the machine learning model. It is used to evaluate the data spread around the line of best fit. So, in simple words, it is used to measure the deviation of the residuals. RMSE is calculated using the formula:

• Yi is the actual value of the output variable.
• Y(Cap) is the predicted value and,
• N is the number of data points.

MSE: Mean Squared Error is used to find how close is the line to the actual data. So, we make the difference in the distance of the data points from the line and the difference is squared. This is done for all the data points and the submission of the squared difference divided by the total number of data points gives us the Mean Squared Error (MSE). So, if we are taking the squared difference of N data points and dividing the sum by N, what does it mean? Yes, it represents the average of the squared difference of a data point from the line i.e. the average of the squared difference between the actual and the predicted values. The formula for finding MSE is given below:

• Yi is the actual value of the output variable (the ith data point)
• Y(cap) is the predicted value and,
• N is the total number of data points.

So, RMSE is the square root of MSE.

Statistical analyses are classified based on the number of variables processed at a given time. Univariate analysis | Bivariate analysis | Multivariate analysis This analysis deals with solving only one variable at a time. | This analysis deals with the statistical study of two variables at a given time. | This analysis deals with statistical analysis of more than two variables and studies the responses. Example: Sales pie charts based on territory. | Example: Scatterplot of Sales and spend volume analysis study. | Example: Study of the relationship between human’s social media habits and their self-esteem which depends on multiple factors like age, number of hours spent, employment status, relationship status, etc.

Feature engineering transforms raw data into model-friendly signals that improve learning. It includes scaling numeric values, encoding categories, creating aggregates (e.g., users’ 30-day spend), extracting date parts, and handling missingness.

The test set is used to test or evaluate the performance of the trained model. It evaluates the predictive power of the model. The validation set is part of the training set that is used to select parameters for avoiding model overfitting.

A p-value is the measure of the probability of having results equal to or more than the results achieved under a specific hypothesis assuming that the null hypothesis is correct. This represents the probability that the observed difference occurred randomly by chance.

• Low p-value which means values ≤ 0.05 means that the null hypothesis can be rejected and the data is unlikely with true null.
• High p-value, i.e values ≥ 0.05 indicates the strength in favor of the null hypothesis. It means that the data is like with true null.
• p-value = 0.05 means that the hypothesis can go either way.

An interdisciplinary field that constitutes various scientific processes, algorithms, tools, and machine learning techniques working to help find common patterns and gather sensible insights from the given raw input data using statistical and mathematical analysis is called Data Science.

• It starts with gathering the business requirements and relevant data.
• Once the data is acquired, it is maintained by performing data cleaning, data warehousing, data staging, and data architecture.
• Data processing does the task of exploring the data, mining it, and analyzing it which can be finally used to generate the summary of the insights extracted from the data.
• Once the exploratory steps are completed, the cleansed data is subjected to various algorithms like predictive analysis, regression, text mining, recognition patterns, etc depending on the requirements.
• In the final stage, the results are communicated to the business in a visually appealing manner. This is where the skill of data visualization, reporting, and different business intelligence tools come into the picture.

• Correlation means two variables move together and can arise from confounders or coincidence
• Causation means one variable directly influences the other

Gradient: Gradient is the measure of a property that how much the output has changed with respect to a little change in the input. In other words, we can say that it is a measure of change in the weights with respect to the change in error. The gradient can be mathematically represented as the slope of a function. Gradient Descent: Gradient descent is a minimization algorithm that minimizes the Activation function. Well, it can minimize any function given to it but it is usually provided with the activation function only. Gradient descent, as the name suggests means descent or a decrease in something. The analogy of gradient descent is often taken as a person climbing down a hill/mountain. The following is the equation describing what gradient descent means: So, if a person is climbing down the hill, the next position that the climber has to come to is denoted by “b” in this equation. Then, there is a minus sign because it denotes the minimization (as gradient descent is a minimization algorithm). The Gamma is called a waiting factor and the remaining term which is the Gradient term itself shows the direction of the steepest descent. This situation can be represented in a graph as follows: Here, we are somewhere at the “Initial Weights” and we want to reach the Global minimum. So, this minimization algorithm will help us do that.

• Mean is the average and is sensitive to outliers
• Median is the middle value and is robust when the data is skewed
• Mode is the most frequent value and is useful for categorical or discrete data

• A Type I error is a false positive that rejects a true null hypothesis
• A Type II error is a false negative that fails to reject a false null hypothesis

Data is said to be highly imbalanced if it is distributed unequally across different categories. These datasets result in an error in model performance and result in inaccuracy.