The Problem
Global procurement data is fragmented, unstructured, and buried across thousands of independent government portals. For businesses looking to bid on public contracts, finding relevant tenders in real-time was a tedious, manual process that often resulted in missed opportunities.
At Byteshake Ltd., we set out to build Tenderbase—a centralized platform that not only aggregated this data but made it instantly searchable through semantic meaning rather than just exact keyword matches.
The Architecture
The system required three distinct core components: a robust distributed scraping engine, a highly optimized OLAP database, and an AI semantic search layer.
- Data Ingestion Layer: Contributed to building a scalable web scraping cluster using Python. This system automatically crawls hundreds of portals daily, extracts raw text, cleanses it, and queues it for processing.
- Storage & Analytics (ClickHouse): Standard relational databases buckled under the weight of hundreds of thousands of large text blobs during complex text searches. I spearheaded the migration of the core analytical workload to ClickHouse, a columnar database perfectly suited for blazing-fast aggregations and filtering over massive datasets.
- AI Semantic Search (OpenAI / Vector Embeddings): To move beyond simple text matching, we utilized OpenAI to generate vector embeddings for tender descriptions. By integrating these embeddings, users can now search for "cloud infrastructure upgrades" and find tenders labeled as "server modernization" based on context.
- The API Backend: Built with CodeIgniter 4, the REST API serves as the lightweight glue holding the system together, securely exposing the data to our React frontend while managing user authentication and subscription billing.
Key Features & Challenges Overcome
- AI Natural Language Search: Developed an advanced search engine that takes raw human input, processes it through an AI API, and dynamically structures it into high-precision database filters.
- AI Audit & Preference Engine: Engineered automated AI audit report generation and a proprietary preference engine that instantly predicts a user's tender interests based solely on their business URL or description.
- Enterprise Integrations: Seamlessly integrated the platform with major third-party CRMs and tools including Salesforce, Monday.com, HubSpot, WhatsApp, and Google Sheets.
- Data Ingestion & Automation: Built a robust Python scraping cluster and automation pipeline that continuously crawls source websites and uploads cleansed data automatically without manual intervention.
- Interactive Gantt Charts: Designed and implemented dynamic Gantt charts for visualizing complex DPS (Dynamic Purchasing System) and Framework tenders, significantly improving user project planning.
- Query Optimization: Initial full-text searches were slow under heavy load. By optimizing ClickHouse indexes and caching, our team drastically reduced the 95th percentile query latency.
The Impact
Today, Tenderbase indexes over 500,000+ tenders and serves hundreds of enterprise clients. The transition to ClickHouse and the implementation of AI search improved tender discovery efficiency and enabled users to identify relevant opportunities more quickly.