Refactor Plan: ORM to Query Builder Migration
1. Current State Assessment
- Hot-path module:
ProductRepository.js's getProductsWithDetails method, responsible for fetching a paginated list of products along with their category and supplier information. This method is called frequently by the public API /api/products. - Performance bottlenecks: Observed N+1 queries when fetching category and supplier details for each product, leading to high database connection usage and increased latency (average 300ms per request). The ORM (e.g., Sequelize) is generating multiple
SELECT statements or complex, unoptimized JOIN clauses. Excessive object hydration for large result sets also contributes to CPU overhead. - Current test coverage: The
ProductRepository module has 95% unit test coverage and 80% integration test coverage, primarily focused on functional correctness of data retrieval and persistence.
2. Target State Definition
- Proposed query builder solution: Knex.js, a SQL query builder for Node.js, will be used to construct raw SQL queries for the
getProductsWithDetails method. - Architectural changes: A new
ProductQueryBuilder.js module will be introduced, encapsulating the Knex.js logic for the hot path. The ProductRepository.js will be updated to delegate the getProductsWithDetails call to this new query builder, maintaining the existing repository interface. This keeps the data access layer clean and allows for future ORM/query builder coexistence. - Expected performance improvements: Anticipated reduction in query latency to under 100ms, achieved by crafting a single, optimized SQL query with explicit
LEFT JOINs and selecting only necessary columns. This will eliminate N+1 issues and reduce object hydration overhead.
3. Step-by-Step Migration Strategy
- Phase 1: Setup and Parallel Implementation
* Install Knex.js and its PostgreSQL adapter. Configure Knex with the existing database connection details. * Create ProductQueryBuilder.js and implement the optimized getProductsWithDetails logic using Knex. * Modify ProductRepository.js to include both the old ORM call and the new Knex call. Introduce a feature flag (useKnexProductDetails) to switch between them. * Develop new unit tests specifically for ProductQueryBuilder.js to verify its SQL generation and data mapping.
- Phase 2: Shadow Mode/Observability
* Deploy the new code with useKnexProductDetails set to false. * Add logging to ProductRepository.js to execute *both* the ORM and Knex paths for getProductsWithDetails for a small percentage of requests (e.g., 1%). Log the results and performance metrics (query duration, number of DB calls) from both paths without returning the Knex result to the user. * Monitor application logs and metrics for any discrepancies in returned data or errors from the Knex path.
- Phase 3: Incremental Rollout
* Enable useKnexProductDetails for 5% of internal users or a specific region. * Closely monitor application performance (latency, error rates, CPU, memory) and database metrics (query load, connection pool usage) for the affected service. * Conduct A/B testing if possible, comparing user experience metrics between ORM and Knex groups. * Gradually increase the percentage of traffic routed to the Knex path (e.g., 25%, 50%, 100%) while continuously monitoring.
- Phase 4: Full Cutover and ORM Removal
* Once monitoring confirms stability and performance gains, fully enable useKnexProductDetails for 100% of traffic. * Remove the deprecated ORM-based getProductsWithDetails implementation from ProductRepository.js and its associated tests. * Update relevant documentation (e.g., architecture diagrams, ProductRepository README) to reflect the change.
4. Testing Strategy
- Unit Tests: New tests for
ProductQueryBuilder.js to validate correct SQL generation, parameter binding, and result set mapping. Existing ProductRepository unit tests will be updated to use the new query builder. - Integration Tests: Existing integration tests for
ProductRepository will be run against the new Knex implementation to ensure data consistency and correct interaction with the PostgreSQL database. - Performance Tests:
* Establish a baseline using JMeter or k6 for the /api/products endpoint *before* migration. * Execute load tests *after* each phase of the incremental rollout to confirm expected latency reduction and increased throughput under various load conditions. * Run regression tests across other API endpoints to ensure no unintended performance degradation.
- End-to-End Tests: Automated browser tests or API tests covering critical user flows involving product listing will be executed to confirm no functional regressions.
- Observability: Enhanced logging for database queries (slow query logs), application-level metrics (request latency, error rates, database call counts), and custom dashboards in Prometheus/Grafana or Datadog to track the performance of the
getProductsWithDetails path specifically. Alerting configured for deviations from baseline.
5. Rollback Plan
- Trigger Conditions: Rollback will be initiated if:
* Error rates for /api/products increase by more than 0.5% compared to baseline. * Average latency for /api/products increases by more than 10% compared to baseline. * Database CPU utilization or connection pool exhaustion exceeds predefined thresholds. * Critical data inconsistencies are detected in production.
* Immediately disable the useKnexProductDetails feature flag, reverting all traffic to the previous ORM-based implementation. This is the primary and fastest rollback mechanism. * If the feature flag mechanism fails or is insufficient, redeploy the previous stable version of the service from CI/CD. * Data consistency: Since the migration only changes read paths, data consistency issues during rollback are minimal. Any potential issues would be related to incorrect data mapping, which should be caught in shadow mode. * Communication plan: Alert on-call engineers via PagerDuty. Communicate status updates to stakeholders (product, engineering leadership) via Slack and incident management tools.