Build · 2h 30m · ₹0
An end-to-end deep learning captioning pipeline pairing a frozen ResNet50 visual encoder with a custom LSTM decoder, comparing greedy decoding against length-normalized beam search.
What it does
The mechanics, data flow, and user interaction model behind Pixel_Info.
Upload an image, get back a natural-language caption. Under the hood: a frozen ResNet50 (ImageNet-pretrained) extracts a 2048-dimensional feature vector, which is projected down to 512 dimensions and fused with learned word embeddings inside an LSTM decoder to generate captions token by token. Two distinct decoding strategies are implemented and empirically evaluated — greedy decoding and length-normalized beam search (k=3) — rather than just shipping one and assuming superiority.
Technical Highlights
- Transfer learning pipeline via frozen ResNet50 vision backbone + custom linear projection into 512-dim LSTM hidden space
- Empirical comparative evaluation of Greedy Decoding vs. Length-Normalized Beam Search (k=3)
- Automated BLEU-1 through BLEU-4 quantitative scoring matrix with NLTK smoothing
- Production-grade training loop: Automatic Mixed Precision (AMP), ReduceLROnPlateau scheduling, and early stopping
- FastAPI inference backend with single-load model caching at startup + Next.js 15 / TypeScript interactive frontend
Why it matters
The architectural judgment, practical engineering decisions, and core problems solved.
This is a from-scratch implementation and evaluation of the classic CNN-encoder / RNN-decoder captioning architecture, conducted with genuine experimental rigor: strict image-level train/validation splitting (preventing data leakage across multi-caption datasets), systematic BLEU-1 through BLEU-4 benchmark scoring, and honest reporting of where beam search underperforms greedy decoding. Beam search produces 13% longer, more descriptive phrase structures at the trade-off of exact single-word overlap with reference sets — an insightful finding documented with real telemetry rather than cherry-picked metrics.
Automated alt-text and accessibility description generation for digital media libraries
Semantic image indexing and visual search cataloging for e-commerce platforms
Visual asset tagging and metadata extraction for content management systems
Educational baseline for studying encoder-decoder multimodal alignment and decoding heuristics
System architecture
End-to-end execution pipeline running across PyTorch, ResNet50, FastAPI, Next.js, NLTK.
Frozen ImageNet visual feature extraction generating 2048-dim representation vectors
Dimension reduction from 2048 to 512 dimensions with batch normalization and dropout
Autoregressive token generation combining image vector with learned word embeddings
Length-normalized cumulative probability search across top candidate sequence beams
Cached in-memory model inference serving real-time captions to a Next.js 15 interface
The path
Step-by-step implementation guide. Verbatim code snippets, configurations, and prompts.
Feature Extraction with Pretrained ResNet50 Backbone
Freeze early convolutional layers of ResNet50, remove the final classification head, and extract 2048-dimensional visual vectors from the global average pooling layer.
Verbatim Code / Config
class ResNetEncoder(nn.Module):
def __init__(self, embed_size=512):
super().__init__()
resnet = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
for param in resnet.parameters(): param.requires_grad = False
self.backbone = nn.Sequential(*list(resnet.children())[:-1])
self.projection = nn.Linear(2048, embed_size)Building the Word-Embedding LSTM Decoder
Construct the recurrent decoder initializing hidden states with projected image embeddings and sequencing word tokens with teacher forcing during training.
Verbatim Code / Config
class DecoderLSTM(nn.Module):
def __init__(self, embed_size, hidden_size, vocab_size, num_layers=1):
super().__init__()
self.embed = nn.Embedding(vocab_size, embed_size)
self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)
self.linear = nn.Linear(hidden_size, vocab_size)Implementing Length-Normalized Beam Search
Write custom beam search keeping top k=3 candidate hypotheses at each step, dividing log-probability sum by length penalty (length^alpha) to prevent favoring short phrases.
Verbatim Code / Config
def beam_search(encoder_out, k=3, max_len=20, alpha=0.7):
# Maintain top-k sequences scored by: sum(log_probs) / (len^alpha)
# Terminate beams hitting <EOS> token or max_len
return best_captionFastAPI Model Cache & Next.js 15 Interface
Wrap PyTorch weights in a singleton startup lifespan in FastAPI and connect an image dropzone UI built in Next.js 15 that visualizes greedy vs. beam search comparison.
Verbatim Code / Config
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.model = load_caption_model('weights/best_lstm.pt')
app.state.vocab = load_vocab('vocab.json')
yieldWhere it broke
The failure mode, root-cause breakdown, and resolution discovered during development.
The Tell
“Standard beam search without length normalization consistently produced clipped 2-3 word captions like 'a dog' instead of descriptive sentences.”
Why it failed
Because probabilities are strictly between 0 and 1, multiplying probabilities (or summing negative log probabilities) naturally penalizes longer sequences, causing unnormalized beam search to favor trivially short captions.
The Fix
Added length normalization with penalty factor alpha = 0.7: score = sum(log_p) / (len ** 0.7). This leveled the playing field for descriptive phrases, increasing average caption length by 13% with richer adjectives.
What it cost
₹0 to build and run permanently within verified free tiers.
| Service / Tool | Cost | Free Tier Limits |
|---|---|---|
| PyTorch & TorchVision | ₹0 | Open-source BSD-3-clause deep learning framework |
| FastAPI Inference Server | ₹0 | Open-source asynchronous Python backend |
| Next.js 15 & Tailwind v4 | ₹0 | Open-source React frontend on free Vercel hobby tier |
| Google Colab GPU / Local CUDA | ₹0 | Trained on free Colab T4 GPU instance |
| NLTK Library | ₹0 | Open-source BLEU evaluation metric suite |
Make it yours
Three concrete variations you can build and ship using this exact foundation.
- 01
Medical X-Ray Report Generator: Replaces natural images with chest radiographs to generate preliminary radiological finding summaries.
- 02
Automated E-Commerce Product Tagger: Generates descriptive SEO product titles from multi-angle catalog photography.
- 03
Audio Spectrogram Descriptor: Pairs CNN audio spectrogram encoders with LSTM decoders to describe ambient audio scenes in text.
Where next
Ready to ship Pixel_Info?
Review the architecture, clone the prompt and implementation steps, and deploy your live URL for ₹0.