2 min read
BigDataForge

BigDataForge is a RESTful API project designed to manage and process complex structured data in JSON format. Built with Golang, it leverages a distributed systems approach to ensure scalability and reliability, integrating technologies like Redis, Elasticsearch, and RabbitMQ for optimized data storage, indexing, and message queueing.

This project provides comprehensive support for CRUD operations, advanced conditional updates, and search capabilities. It demonstrates parent-child indexing and search using Elasticsearch, making it ideal for scenarios requiring hierarchical data relationships and efficient query execution.

Features:
  1. RESTful APIs: Flexible endpoints to handle structured JSON data.
  2. CRUD Operations: Create, read, update, delete, and patch capabilities with advanced validation.
  3. Elasticsearch Integration: Parent-child indexing and search for efficient hierarchical data retrieval.
  4. Queueing System: RabbitMQ for handling message queues and ensuring reliable data ingestion.
  5. Data Validation: Schema validation to ensure data integrity.
  6. Security: Authentication using GCP OAuth2.0
  7. Distributed System Architecture: Leveraging Redis for key-value storage and Elasticsearch for advanced indexing and search.