Deep Learning Math


### Sub-field of ML: learning representations of data
Existing ML uses manually designed features
- often over-specified and incomplete
- take a long time to design and validate
DP
- Learned Features are easy to adapt, fast
- Deep learning provides a very flexible, (almost?) universal
- Effective end-to-end joint system learning

> speach recog not good
> visual perception good
> Question Answering good


> 4 + 2 = 6 neurons (not counting inputs) - biases has same number (for each resulting node)
> [3 x 4] + [4 x 2] = 20 weights
> 

> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F5986fe3d-198a-4608-959f-6bd5bc738065%2FUntitled.png?table=block&id=8f6601d1-344a-4488-b982-926c53650f7f&cache=v2)
> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F4110b2cb-bf0f-4221-b2c9-cc8c6f2c8786%2FUntitled.png?table=block&id=0e436d3f-24b4-4328-bc11-1e91e5aa5747&cache=v2)






![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Ff7503f3f-0f31-446e-b91b-32842c7f119d%2FUntitled.png?table=block&id=4e369bcf-c0fd-4d82-9523-8156a1c29aac&cache=v2)
Optimize (min. or max.)** objective/cost function 𝐽(𝜃)** Generate **error signal **that measures difference between predictions and target values
 



word representation is 74 72 65 65 in x0
but it is pool
so we need new presentation of word



## wordnet


WordNet: contains the list of synonyms/hypernyms ← using human resources
Word’s meaning is given by words that frequently appear close-by
### Context: set of words that appears nearby in the fixed-size window (ex: before/after 5 words)
Vector dimension = Number of words * Number of words

memroy O(n^2) poor
so we use dimensionality reduction (pca, svd) ← almost all value is 0 so l;we can
→ word embedding 이제 벡터곱으로 similar판단가능 (become dense vector) - from one hot vector


> We can make embedding for between other languagex
> vector similarity (cosine)

> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F30c1772d-b6df-4eed-ad3f-2777aebcef9a%2FUntitled.png?table=block&id=e97e701f-21dd-4ab7-8e3e-d039ad4d6d08&cache=v2)





# Language Modeling: Models ofP(text) - sentance


score sentance

> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F3c48b3c2-5971-4a4d-8c78-896f6b721be1%2FUntitled.png?table=block&id=70af901f-dd8c-4e16-a2b5-408adf172390&cache=v2)
> add all is exampel (linearfeatures)
> usually softmax the sum
> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F66fd7479-7a77-416a-9684-a30972d659e4%2FUntitled.png?table=block&id=4f96ddb9-9fa4-48df-b29f-2012e73945f1&cache=v2)

> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F7bcd216a-04a9-411b-9c95-c2c8728c44e7%2FUntitled.png?table=block&id=2b6a582b-5666-47cd-8fa5-0185c4b045bb&cache=v2)



soft max is for before output

![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F52fd4795-45f9-4e20-8f4a-b9a18df4d4d7%2FUntitled.png?table=block&id=5a5b7c77-f67f-486b-b148-0265844a4461&cache=v2)
# 1. P(text) - linear model
## CBOW
Predict word based on sum of surroundingembeddings

## Skip-gram
Predict each word in the context given theword [very good, good, neutral, bad, verybad]
Linear Models can’tLearn FeatureCombinations
# 2. Models of P(label |text)
## BOW
each word has its own 5 elements corresponding to
## DeepCBOW
CombinationFeatures - Each vector has “features” (e.g. is this an animate object? is this a positive word, etc.)

# ConvolutionalNetworks
### x/ CNN - pooling
weak for long distance feature extractor
don't have holistic view of
# x/ RNN
good for long disatnce fraeture extractor
 weakness - Indirect passing of information, credit assignment moredifficult
Can be slow, due to incrementalprocessing

## ModelingSentences w/n-grams

# P(text |text)

> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F547f55b2-840c-4e50-80a0-8b936f4a64c3%2FUntitled.png?table=block&id=7fa3ac7f-fd55-4575-b6b1-afbae72438fe&cache=v2)
> Conditional LanguageModels

> ![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fba8eaa2d-8c36-4f5b-b222-e60b091aa063%2FUntitled.png?table=block&id=6f689ac1-dfc7-45ac-9f72-5704d81f6dcf&cache=v2)
> Calculating the Probability of aSentence


> RNN is frequently used in language modeling since RNN can capture long-distance dependencies






