Setup guide for GPU acceleration on Apple Silicon M1 with PyTorch MPS backend and HuggingFace Transformers.

An approach that enables large-batch contrastive learning under memory constraints, similar to gradient accumulation.

Introduction to AutoML and hyperparameter optimization using Bayesian Optimization with Gaussian Process Regression.

Key considerations for model compression competitions, covering FLOPs, memory access cost, and speed optimization guidelines from ShuffleNet v2.

Overview of model compression techniques: NAS, network pruning, knowledge distillation, tensor decomposition, and quantization.

Ensemble methods for AI competitions: hard voting, soft voting, and weighted voting to improve model performance.

How gradient accumulation works in PyTorch for effective large-batch training on limited GPU memory.

Clarifying common mistakes in PyTorch training loops, validation ordering, and K-fold cross validation usage.

Overview of hyperparameter tuning methods including grid search, random search, and Bayesian optimization, with an introduction to Ray for parallel tuning.

Guide to multi-GPU training in PyTorch covering model parallelism, DataParallel, and DistributedDataParallel with code examples.

Introduction to generative models: probability distributions, independence assumptions, chain rule, and auto-regressive models.

Core concepts of the Transformer model: encoder self-attention, Query/Key/Value embeddings, and multi-head attention mechanics.

Discussion on how the relationship between model parameter count and generalization performance shifted with scaling law findings.

Practical CNN implementation in PyTorch: add_module, training loops, and batch normalization.

Key CNN architectures from ILSVRC: AlexNet, VGGNet, GoogLeNet, and ResNet, with analysis of receptive fields and 1x1 convolutions.

Why weight initialization matters in deep learning and why zero initialization should be avoided.

Convolution fundamentals: stride, padding, parameter counting, and 1x1 convolutions.

Practical comparison of SGD, Momentum, and Adam optimizers on function approximation with noisy data in PyTorch.

Deep learning optimization fundamentals: generalization, overfitting, cross-validation, bias-variance tradeoff, bootstrapping, bagging, and boosting.

Deep learning fundamentals: key components (data, model, loss, optimizer) and a brief history from AlexNet to GPT-3.

Introduction to neural networks covering linear regression, softmax classification, activation functions, and why deep layers are preferred.

Neural networks as function approximators: linear models, activation functions, multi-layer perceptrons, universal approximation theorem, and loss functions.

Practical PyTorch tips covering parameter initialization, model.eval(), tensor views, and the training loop.