2 Logistic Regression

1. Logistic function, odds ratio Because we need the linear function mx+c (right hand side) to range from negative infinity to infinity and probability on left hand side. Probability has a restrictive range (0 to 1) and odds ratio also has a restristrictive range 0 to infinity, we use log of odds that has the range of minus infinity to infinity. Interesting

1.1. Below is a link to an awesome theoretical explanation for logistic regression https://www.analyticsvidhya.com/blog/2021/08/conceptual-understanding-of-logistic-regression-for-data-science-beginners/ The odds of an event represent the ratio of the (probability that the event will occur) / (probability that the event will not occur): P/(1-P)

2. Cost function of logistic regression is calculated from maximum likelihood estimation of a Bernoulli distribution. The Bernoulli distribution is a discrete distribution having two possible outcomes labelled by n=0 and n=1 in which n=1 ("success") occurs with probability p and n=0 ("failure") occurs with probability q=1-p, where 0<p<1. It therefore has probability density function P(n)={1-p for n=0; p for n=1, P(n)=p^n(1-p)^(1-n).

2.1. Derivation can be found here: https://gaussian37.github.io/machine-learning-etc-mle/

3. Sigmoid function

4. Logistic regression is an extension of linear regression for binary classification problems. Logistic regression uses sigmoid function to convert linear dependence of independent variables to a binary problem. The cost function is defined so that a clear global minima can be found. For a more detailed explanation please use this link: https://towardsdatascience.com/introduction-to-logistic-regression-66248243c148

5. Likelihood function: In statistics, a likelihood function is a function that expresses the probability of a set of observations given a particular set of parameter values for a statistical model. It is defined as the product of the probability density or mass functions of the individual observations, with the assumption that the observations are independent of each other. The likelihood function is used in statistical inference to estimate the parameters of a statistical model. Given a set of observations and a statistical model, the maximum likelihood estimate (MLE) of the model parameters is the set of parameter values that maximize the likelihood function. This approach to estimation is called maximum likelihood estimation (MLE). For example, suppose we have a sample of data that is normally distributed with unknown mean and variance. We can specify a statistical model that assumes the data follows a normal distribution with some unknown parameters: the mean, μ, and variance, σ^2. The likelihood function for this model would be the product of the normal density functions of the individual observations, evaluated at the given values of μ and σ^2. To find the MLE of the mean and variance, we would find the values of μ and σ^2 that maximize the likelihood function. In general, the likelihood function can be thought of as a measure of how well a given set of parameter values fits the data. It is often used in statistical inference to compare different models or to test hypotheses about the parameters of a model. https://statlect.com/fundamentals-of-statistics/normal-distribution-maximum-likelihood

6. some links: https://towardsdatascience.com/logistic-regression-b0af09cdb8ad