Abhishekh SinghBuilding distributed systems that power customer success at scale
I architect the data infrastructure, search platforms, and AI systems behind Gainsight's enterprise SaaS, from pipeline engines processing billions of records daily, to AI-powered search with dual-LLM providers and an MCP server that lets Claude and ChatGPT query community knowledge natively.
Abhishekh Singh
Principal Engineer at Gainsight with 12+ years building production distributed systems across two product lines: Customer Success (CS) and Customer Communities (CC). Based in Hyderabad, India. B.Tech in Computer Science from BBD National Institute of Technology & Management, Lucknow.
On the CS side, I built the Data Highway pipeline engine, the graph-based permission system used by 7+ services, and the event processing platform with cross-region SQS failover. On the CC side, I architected the AI-native search platform: three LLM-powered features, a production MCP server exposing dozens of tools, and a federated search system indexing records from 5 sources.
My work sits at the intersection of distributed systems, data engineering, and AI, building the kind of infrastructure that other teams build their products on.
Platform Architecture
10+ production services across Java (Vert.x, Spring Boot) and Python: DAG orchestration, event-driven pipelines, graph-based RBAC, and real-time search
AI & MCP Integration
Production MCP server exposing dozens of tools for Claude/ChatGPT. Three AI features: ReplyBot (OpenAI + Gemini fallback), AI Answers Summary (SSE streaming), Smart Search Summary (Llama 3 70B)
Data at Scale
61B+ records processed daily, 25M+ search records indexed, 35+ SaaS connectors, dual-warehouse analytics (Postgres + Redshift), S3 Parquet data lake
Security-First Design
Graph-based RBAC with Neo4j, JWT with Redis-backed revocation, permission-scoped Algolia tokens, process-isolated task sandboxing
Building at Every Layer of the Stack
Systems I've Built & Architected
My Contributions to Gainsight's CustomerOS
Community Search & AI Platform
- Cross-provider LLM fallback: dedicated adapter per provider handles payload asymmetry, with automatic retry across providers even on client errors to prevent single-provider outages
- SSE streaming with thinking-phase deduplication: stateful stream processor filters duplicate reasoning blocks per model family while preserving final output fidelity
- Domain-driven prompt architecture: composable prompt traits (model, version, temperature, reasoning effort) enable per-feature model versioning and A/B testing without code duplication
- Dynamic rate limiting with zero-restart retuning: atomic Redis counters (no locks, no races), thresholds configurable at runtime so ops can retune without a deploy
- Zero-downtime full sync: bulk-index 25M+ records into temp Algolia index, atomic rename over production. No search outage during reindex
- Analytics data lake on S3 Parquet + Athena with tag expression consolidation reducing Algolia API calls by 90%+
Communities needed AI features such as suggested replies and summaries without being tied to one model vendor, plus a single search experience across several content sources.
Tens of millions of records across multiple content sources, with three AI features in production.
Keep the AI layer provider-agnostic: every feature talks to a common interface, with a fallback model ready if the primary has trouble.
A little more integration work for each provider, in exchange for resilience to any single vendor outage or change.
Features keep working through provider hiccups, and search stays available while content is re-indexed. Smarter query handling also cut search API usage substantially.
Customer Community MCP Server
- Full OAuth2.1 authorization flow with PKCE and dynamic client registration, listed on major AI assistant connector registries
- Multi-region deployment with edge-based request routing for geographic locality and failover
- Permission-scoped search tokens with access control baked at the token layer, not post-filtered
- Per-tenant rate limiting, connection lifecycle management, and graceful degradation on backend failures
- Dozens of tools spanning search, analytics, content moderation, user management, and education backends behind a single MCP interface
AI assistants needed a safe, standard way to query community knowledge for enterprise customers.
A large set of tools behind one standard interface, available across regions.
Treat security as a design input from the start: access rules apply at every layer instead of being added at the end.
More up-front design and testing for each tool, in exchange for customer-safe results by default.
One interface that the major AI assistants can connect to, with fair-use controls and graceful behaviour when a dependency is slow or unavailable.
Data Highway: DAG-Based Pipeline Engine
- Processes 61B+ records/day across 600+ enterprise tenants at 98.7% job success rate
- Multi-worker architecture: DAG Workers (Java/Vert.x) + Dynamic Workers (35 isolated connector JARs) + Python/Celery ML Workers + Apache Drill SQL engine
- 60+ custom Apache Drill UDFs for tenant-specific data transformations
- Horizontally-scalable cluster coordinated by Zookeeper leader election and distributed workflow orchestration
- Failure-injection tasks (OOM, disk exhaustion, FD leak) as production resilience probes
Enterprise customers keep their data in many different systems. Each one needs it brought together, transformed and loaded reliably.
Billions of records a day across hundreds of enterprise customers, with dozens of source connectors.
Model the work as a graph of tasks run by a pool of specialised workers, coordinated centrally so scheduling stays safe at scale.
Keeping connectors isolated from each other means more to package and deploy, in exchange for stopping one source’s problems from spreading.
High job success at that volume, flexible per-customer transformations, and regular resilience drills so failure handling is tested before it is needed.
User Permissions (RBAC)
- Graph-native model: resources, policies, groups, and manager chains as traversable edges, so a single graph traversal resolves access in milliseconds
- Targeted Redis invalidation: editing a sharing group invalidates only affected members × resources, not the full cache
- Row-level data enforcement: restricted data never leaves the database, access control enforced at SQL query-build time
- Permission SDK consumed by 7+ platform services: analytics, query federation, reporting, API gateway, data store, catalog, and cross-product bridge
- Heartbeat-based leader election for worker coordination with automatic cleanup of orphaned graph nodes
Every service on the platform needed one consistent answer to the question: can this person do this?
Used by many services across the platform.
Model permissions as relationships instead of flat rules, so an access question can be answered quickly with a single lookup.
Answers are cached for speed, so a change to sharing must refresh only what it actually affects.
Fast, consistent access decisions, and sensitive data filtered at the data layer before it is returned.
Usage Data Model (UDM)
- ~25 pluggable task builders: adding a new data source is a single builder class
- Dual-warehouse abstraction: PostgreSQL for small tenants, Redshift for high-volume OLAP, switched transparently
- Permissions injected at query-build time: restricted data never leaves the database
- Cross-system DAG submission to the pipeline engine with retry and backoff
- Secure multi-tenant data exchange with per-tenant credential isolation
Dashboards needed per-customer analytics across many data sources, for customers from small to very large.
Many data sources feeding per-customer analytics, with a pluggable way to add more.
Right-size storage for each customer: smaller and very large customers use different storage engines behind one interface.
Two storage paths to maintain, in exchange for the right cost and speed for each customer.
Adding a data source is a small, contained change, with access rules applied when queries are built and each customer’s data kept separate.
Event Processing
- Cross-region SQS failover: automatic switch to secondary queues on publish errors
- Self-healing delivery: dedicated recovery process detects and recovers orphaned delivery rows after worker crashes
- Blocked-tenant handling isolates noisy tenants from starving other tenants' event delivery
- Enrichment loop: events decorated with business context from the analytics data store before delivery
- Tiered architecture: Receiver → Router → Relayer → Broadcaster with purpose-matched runtimes
Platform services needed to publish an event once and have many subscribers receive it reliably, even when parts of the system fail.
A staged pipeline in which each stage scales independently.
Split delivery into small stages with the right tool for each, and make every stage safe to retry.
More moving parts than a single dispatcher, in exchange for keeping a failure contained to one stage.
Reliable delivery that recovers on its own after failures, and one busy customer cannot slow the rest down.
Audit Service
- Async write via event bus: HTTP 200 returns before DB insert completes
- Random-tenant retention cleanup with +10 day safety buffer
- Separate read/write DB pools to prevent analytics from starving ingestion
Config Server
- Upsert + append-only version history table in single transaction
- Database-enforced integrity with case-insensitive keys and FK cascades
- Redis-cached auth lookups with fail-fast startup ping
Projects I Build in the Open
GoatCounter Dashboard
- Two complete designs on one set of components: Nova (glass, aurora, tilting cards) and Classic (the original flat look), each in dark and light. A new visit opens in Nova dark, and remembering a choice is opt-in
- A visitors chart that draws itself, with a ghost line for the previous period, annotated spikes, a crosshair, and click-to-zoom into a day's 24 hours
- Countries on a WebGL globe (with a flat map fallback), browsers, systems and devices as donuts with real logos, and drill-down from referrers to pages, browsers to versions, and countries to regions
- When readers show up as a 3D skyline by weekday and hour, plus a calendar year with streaks and goal rings
- All your sites in one place: switch between the sites of a GoatCounter account, or narrow every card to a single page. A command palette, keyboard shortcuts and a refresh that clears the cache
- Rebuilt for speed and privacy: about 130 KB on first load instead of 970 KB, and usable in about 1 second instead of 6.4 on a throttled phone. No CDN, no analytics, and the key goes only to your own GoatCounter site
mcp-halflist
A recorded run of the real tool against a small sample server. Not a live connection.
Choose a command above to replay a recorded run.
- Five suites: handshake, tools, resources and prompts, plus an offline security scan for prompt injection, data exfiltration, cross-tool manipulation, suspicious encoding, and tool pinning to catch rug-pull changes
- OAuth2 PKCE with auto-discovery: provide the server URL and the tool discovers auth endpoints, opens a browser, then handles token exchange and caching automatically
- Debug logging with multi-layer request and response tracing (MCP operations, HTTP wire, SDK internals) and credential masking
- JUnit XML output for CI integration with GitHub Actions, GitLab, and Jenkins
- Config file (halflist.toml) with environment variable expansion, so CI pipelines and teams share the same testing configuration
- GitHub Action available: three lines of YAML to add MCP server testing to any workflow
Technologies & Tools
Languages & Frameworks
Cloud & Infrastructure
Data & Search
AI & Protocols
DevOps & Quality
Background
B.Tech, Computer Science & Engineering
BBD National Institute of Technology & Management, Lucknow (2010-2014) · 75.16%
Intermediate, Mathematics
Rastriya Inter College (2008-2010) · 73.6%
Languages
English (Full Professional) · Hindi (Native)
Awards & Leadership
Vice-Captain, Nihilentia. Selected to lead Nihilent's annual fest (Dec 2014). Successfully organized events and competitions.
IT-FIESTA 2011. Appreciation Certificate awarded by T. Ashok (Founder & CEO, Stag Software Pvt. Ltd.)
UTKARSH-13 & 14. Coordinated Technical Events at BBDNITM (2013 & 2014)
ENCORE-12. Honor & Excellence Certificate at IET Lucknow (Oct 2012)
Techsamagam-11. Certificate Of Merit, FORCE CIC Education (Nov 2011)
3rd Position, UP Board. Block-level topper in Jaunpur, High School Examinations (2007)
What Colleagues Say
"Abhishekh is an outstanding person, great to work with and a real self starter. He is very effective in team and testing related activities. He has strong and broad technical and automation expertise. Abhishekh is one of the top people I have met and worked with in my career."
"Abhishekh is very passionate and has great vision for his work. His focus keeps everything moving smoothly, he makes sure all the deadlines are met, and makes sure that whatever project he is working on meets the highest standards."
"Abhishekh successfully automated our application and single handedly managed all testing activities. He is extremely enthusiastic about his work which is infectious, makes it fun to learn. He always has a creative, positive outlook and he's good at organizing and bringing people together."
"I had the privilege of working with Abhishek on a couple of academic projects during our B.Tech. He is a highly organized, goal oriented, independent and hard working perfectionist. He has very good analytical and interpersonal skills. He is the most motivated and enthusiastic person I have ever met."
"Abhishek has tremendous knowledge of Testing, always there to help for problems related to testing. His rational approach of making complex topics very simpler by relating it with day to day examples was amazing. That was the beauty of his approach in System Testing and Integration Testing."
Next stepGot a hard scaling problem? Let's talk.