Tag: dying ReLU

Why it matters: Rectified Linear Unit (ReLU): the formula max(0,x), how it works, uses in deep learning, dying ReLU fixes, plus Leaky, PReLU, GELU, SwiGLU variants.