AutoCap

AutoCap is an AI-powered multimodal data engine designed to automate the creation of high-quality, domain-specific image–caption datasets. The project tackles a major challenge in computer vision and multimodal AI: generic public datasets often fail to represent specialized environments like universities, hospitals, or industrial spaces, while manual data annotation is expensive and difficult to scale. AutoCap solves this by transforming raw images into semantically validated image–text pairs through an intelligent caption generation and evaluation pipeline.

At its core, AutoCap combines deep learning, transfer learning, and semantic evaluation into a full-stack application that functions as both a caption generation platform and a synthetic dataset creation engine. Users can upload single images or entire image batches, generate multiple candidate captions through vision-language models, rank captions using CLIP-based semantic similarity scoring, automatically flag low-quality or mismatched outputs, and export validated datasets in structured formats such as JSON or CSV. This makes the system useful not only for caption generation, but also for building training data for downstream AI applications such as robotics, visual understanding, and reinforcement learning systems.

The project explores and compares multiple image captioning architectures, including a CNN + LSTM baseline, an attention-enhanced CNN + RNN model, and a flagship CNN + Transformer architecture for higher-quality generation. It also incorporates domain adaptation, where models trained on large public datasets like Flickr8k, Flickr30k, and MS COCO are fine-tuned on smaller domain-specific datasets (such as university environments) to learn specialized vocabulary and context with minimal data. This approach turns AutoCap into more than a captioning system — it becomes a self-improving data generation pipeline.

A major innovation in the project is the semantic validation layer powered by CLIP embeddings, which measures image-caption alignment, filters unreliable outputs, and acts as automated quality control for dataset construction. Combined with multi-dimensional caption flagging, metadata generation, caption review tools, and downloadable curated datasets, the platform focuses heavily on data quality, transparency, and reproducibility. Beyond traditional BLEU, METEOR, and CIDEr metrics, the system also incorporates semantic evaluation and engineering performance analysis, giving it strong research and practical deployment value.

Built as a full-stack AI application using React + TypeScript, Spring Boot, FastAPI, PyTorch, Hugging Face, CLIP, and Supabase, AutoCap blends software engineering with applied AI research. The platform includes an interactive dashboard for inference, dataset exploration tools, searchable caption repositories, feedback and documentation modules, and a scalable architecture designed for future extension into broader multimodal data generation workflows.

AutoCap sits at the intersection of Computer Vision, Multimodal AI, MLOps, and Data-Centric AI, and represents both a research contribution and a practical software product — focused not just on generating captions, but on generating better data for intelligent systems.