Tag: database

  • Fulfillment – Daily Changelog – October 16, 2025

    Fulfillment – Daily Changelog – October 16, 2025

    Changelog – October 16, 2025

    Order Snapshot System Implementation

    Immutable Order Data Snapshots

    • Problem: Order line items lost critical data when variants were modified or deleted
    • Order #5405 showed incorrect sizes after Oct 10 variant data restoration
    • Line items depended on live variant data, which could change or disappear
    • No historical record of what was actually ordered/shipped
    • Solution: Implemented comprehensive snapshot system to capture immutable order data at creation time
    • Added new columns to mama_tried_order_line_items:
      • snapshot_product_data (JSONB) – Complete product details
      • snapshot_variant_data (JSONB) – Complete variant details including size, color, SKU
      • snapshot_pricing_data (JSONB) – Pricing breakdown (blank + print costs)
    • Captures data from:
      • Product details (title, type flags, QuickBooks ID)
      • Variant details (size, color, SKU, style info)
      • Pricing details (blank price, print price, total)
    • Implementation:
    • Modified order creation endpoint to capture snapshots
    • Updated line items endpoint to include snapshot data
    • Added TypeScript types for snapshot data structures
    • Created backfill script for existing orders
    • Database Migration: database_migrations/add_line_item_snapshots.sql
    • Files Modified:
    • src/app/api/orders/route.ts – Capture snapshots on order creation
    • src/app/api/orders/[id]/line-items/route.ts – Include snapshot data in responses
    • src/app/api/orders/[id]/route.ts – Return snapshot data
    • src/types/orders.ts – Added snapshot type definitions
    • src/lib/orderUtils.ts – Enhanced order type assignment
    • Documentation:
    • SNAPSHOT_IMPLEMENTATION_SUMMARY.md – Complete technical overview
    • SNAPSHOT_QUICKSTART.md – Quick reference guide
    • SNAPSHOT_TESTING_GUIDE.md – Testing procedures
    • Scripts Created:
    • scripts/backfill_order_snapshots.py – Backfill snapshots for existing orders
    • scripts/verify_snapshots.sql – Verify snapshot data integrity

    Raffle Sticker Order Type Fix

    • Problem: 176 raffle sticker orders not being classified correctly after variant data reset
    • Orders not appearing in raffle sticker filtered views
    • assignOrderType() function missing is_raffle_sticker field
    • Root Cause: Order type detection couldn’t identify raffle stickers without the flag
    • Solution:
    • Updated assignOrderType() to include is_raffle_sticker field from product data
    • Database fix applied to update 176 orders with correct raffle-sticker type
    • Removed erroneous variant associations
    • Files Modified: src/lib/orderUtils.ts
    • Impact: All raffle sticker orders now properly classified and filterable

    Create Reprint Functionality

    • Feature: Added ability to create reprint orders for incorrect shipments
    • Implementation:
    • New API endpoint: /api/orders/[id]/create-reprint
    • Creates new order with -REPRINT suffix
    • Copies customer info and allows variant selection
    • Marks as is_reprint=TRUE with original_order_id reference
    • Sets total price to $0.00 (no charge to customer)
    • UI Component: CreateReprintDialog for easy reprint creation
    • Files Created:
    • src/app/api/orders/[id]/create-reprint/route.ts
    • src/components/orders/create-reprint-dialog.tsx
    • Use Case: Created reprint order #5405-REPRINT for incorrect 2XL shipment

    Order #5405 Investigation & Fix

    • Investigation: Comprehensive analysis of variant data corruption impact
    • Verified all orders that shipped since Oct 10 variant restoration
    • Only 1 order (out of 80+ shipped) had incorrect size sent
    • Order #5405 Details:
    • Customer ordered 2 shirts in size L
    • Shipped 1 correct (L Solid Gold) + 1 incorrect (2XL Western Stacked instead of L)
    • Shopify showed correct L variant (41889779384456)
    • System shipped 2XL variant (41889779449992)
    • Resolution: Created reprint order #5405-REPRINT with correct L size
    • Documentation:
    • REAL_IMPACT_SUMMARY.md – Analysis of actual damage
    • ACTUAL_DAMAGE_ASSESSMENT.md – Detailed assessment
    • AFFECTED_ORDERS_SUMMARY.md – Complete order analysis
    • CUSTOMER_CONTACT_PLAN.md – Customer communication strategy
    • VARIANT_DATA_CORRUPTION_INVESTIGATION.md – Technical investigation
    • Scripts:
    • fix_order_variants.py – Tool to detect and fix variant mismatches
    • verify_order_variants.py – Verification script
    • investigate_actual_shipments.py – Analyze what was actually shipped
    • create_reprint_5405.sql – SQL for creating reprint order
    • show_5405_change.sql – Document variant changes

    Shipment Sync Optimization & Hang Prevention

    Shipment Processing Deadlock Fix

    • Problem: Shipment sync operations hanging and blocking other database operations
    • Mark-as-shipped endpoint had long-running transactions
    • Concurrent batch updates causing deadlocks
    • Multiple sync operations exhausting database connection pool
    • Root Cause:
    • No transaction timeouts on long operations
    • Batch updates lacking row-level locking
    • Redundant calls between sync endpoints
    • Each sync creating new database connections
    • Solutions Implemented:
    1. Transaction Timeout: Added 30-second timeout to mark-as-shipped endpoint
    2. Row-Level Locking: Implemented SELECT FOR UPDATE SKIP LOCKED for batch updates
    3. Streamlined Sync Flow:
      • sync-shopify endpoint now handles status updates inline instead of calling mark-as-shipped
      • sync-tracking endpoint handles status updates inline, reducing connections
    4. Server-Side Lock System:
      • New sync operation lock to prevent overlapping syncs
      • Lock check before starting bulk operations
      • Auto-release after timeout
    5. Connection Pool Optimization: Updated src/lib/db.ts with better pool settings
    • New API Endpoints:
    • /api/sync-operations/acquire-lock – Acquire exclusive sync lock
    • /api/sync-operations/check-lock – Check if sync is currently running
    • /api/sync-operations/release-lock – Manually release stuck locks
    • Files Modified:
    • src/app/api/shipments/mark-as-shipped/route.ts – Added timeout & locking
    • src/app/api/shipments/sync-shopify/route.ts – Inline status updates
    • src/app/api/shippingeasy/sync-tracking/route.ts – Inline status updates
    • src/components/shipments/BulkSyncTrackingButton.tsx – Lock checking
    • src/lib/db.ts – Connection pool optimization
    • src/lib/syncLock.ts – New lock management utility
    • src/utils/fetchWithTimeout.ts – Timeout utility
    • Documentation: SHIPMENT_SYNC_HANG_FIX_SUMMARY.md
    • Impact: Eliminated sync hangs, reduced connection exhaustion, improved concurrent operation handling

    Minor Enhancements

    Batch API Improvements

    • Enhanced: Batch endpoints now include proper error handling for variant assignments
    • Files Modified:
    • src/app/api/batches/[id]/bulk-weight/route.ts
    • src/app/api/batches/[id]/send-to-shippingeasy/route.ts
    • src/app/api/batches/[id]/sync-shippingeasy/route.ts

    UI Updates

    • Updated: Order list and detail pages to support new snapshot data
    • Files Modified:
    • src/app/(authenticated)/orders/[id]/page.tsx
    • src/app/(authenticated)/orders/page.tsx

    Auth & Middleware

    • Improved: Authentication handling and middleware logging
    • Files Modified:
    • src/lib/auth.ts
    • src/middleware.ts

    Testing & Verification

    Comprehensive Testing Tools

    • Created Python scripts for data verification:
    • fix_order_variants.py – 380 lines – Detect and fix variant issues
    • verify_order_variants.py – 330 lines – Verify order data integrity
    • investigate_actual_shipments.py – 365 lines – Analyze shipment history
    • check_what_i_changed.py – Track database changes
    • Created SQL verification scripts:
    • scripts/verify_snapshots.sql – Verify snapshot backfill
    • temp_line_items.sql – Temporary line item analysis

    Investigation Results

    • Output Files:
    • shipment_investigation_output.txt – Detailed shipment analysis
    • shipment_investigation_report.json – Structured investigation data
    • variant_verification_results.json – Variant verification results

    Database Backups

    • Created pre-change backup: pre_change_backup.dump (4.1 MB)
    • Backup taken before applying major snapshot system changes

    Summary Statistics

    • Total Changes: 8,501 lines added, 63 lines deleted (snapshot system commit)
    • Additional Changes: 616 lines added, 90 deleted (shipment sync commit)
    • New Files: 47 files created (documentation, scripts, components, API routes)
    • Orders Fixed: 176 raffle sticker orders reclassified
    • Orders Analyzed: 80+ shipped orders verified for data corruption
    • Actual Impact: 1 order shipped with wrong size (out of 80+)

    Deployment Status

    • Not Yet Deployed: Changes committed but awaiting production deployment
    • Reason: Awaiting thorough testing due to scope of changes
    • Next Deployment: Will include both snapshot system and shipment sync optimizations
  • Fulfillment – Daily Changelog – October 13, 2025

    Fulfillment – Daily Changelog – October 13, 2025

    Changelog – October 13, 2025

    UI Improvements

    Fixed Product Title Display on Orders Page

    • Problem: Orders were showing “(Default Title)” where size/variant information should be displayed
    • Root Cause Investigation:
    • Initially thought the issue was in item.title field showing “Default Title”
    • Database investigation revealed the actual problem: Shopify was storing “Default Title” in the variant.size field for raffle stickers and other items without actual sizes
    • This caused inconsistent display – some orders showed it (newer syncs with “Default Title” in size), others didn’t (older syncs with NULL/empty size)
    • Solution: Filter out “Default Title” from both product titles AND variant sizes across all order views
    • Check both item.title === 'Default Title' and item.variant.size === 'Default Title'
    • Only show variant information (parentheses) when there’s actually meaningful size or color data
    • Updated getVariantDisplay() helper function to exclude “Default Title” sizes
    • Files Modified:
    • src/app/(authenticated)/orders/page.tsx – Main orders listing page
    • src/app/(authenticated)/store-manager/orders/page.tsx – Store manager orders view
    • src/app/(authenticated)/admin-dashboard/page.tsx – Admin dashboard orders
    • src/app/(authenticated)/store-manager/orders/[id]/page.tsx – Store manager order details
    • src/app/(authenticated)/orders/[id]/page.tsx – Order details page (including getVariantDisplay function)
    • Impact: Clean display without Shopify’s placeholder text – items without sizes/colors show just the product name, items with actual variant data show it properly
    • Why It Started Suddenly: Shopify changed how they sync variants for products without sizes, now explicitly setting size to “Default Title” instead of NULL

    Temporarily Removed “Address Updated” Badge

    • Problem: Badge was showing up incorrectly on orders that didn’t have address changes
    • Root Cause: Known issue with address comparison logic in updateOrderShippingAddress function (documented in changelog_09192025.md)
    • Solution: Removed the badge display temporarily until the underlying comparison logic can be fixed
    • Files Modified: src/app/(authenticated)/orders/page.tsx
    • Impact: Eliminates false positive indicators that were requiring manual resets
    • Follow-up Required: Need to debug and fix the address comparison logic in updateOrderShippingAddress before re-enabling the badge

    Bug Fixes

    Fixed Database Pool Configuration TypeScript Error

    • Problem: TypeScript compilation error – min property not recognized in Pool configuration
    • Root Cause: The node-postgres pg library’s PoolConfig doesn’t support min (minimum pool size) or acquireTimeoutMillis properties
    • Solution: Removed invalid min and acquireTimeoutMillis properties from Pool configuration
    • Files Modified: src/lib/db.ts
    • Impact: TypeScript compilation now passes without errors, ready for deployment

    Fixed JSX Syntax Error in Store Manager Order Details

    • Problem: TypeScript parsing error due to missing closing brace in JSX return statement
    • Root Cause: When fixing the “Default Title” display issue, the closing brace for the map function’s return statement was accidentally removed
    • Solution: Properly structured the JSX with correct closing braces
    • Files Modified: src/app/(authenticated)/store-manager/orders/[id]/page.tsx
    • Impact: Build errors resolved, page renders correctly

    Fixed Unused Variables in Product Sync Tools

    • Problem: ESLint errors for unused confirmationText and setConfirmationText variables
    • Root Cause: Legacy state variables left over from previous confirmation dialog implementation
    • Solution: Removed unused state variables
    • Files Modified: src/components/products/product-sync-tools.tsx
    • Impact: Clean linting, no warnings

    New Features

    Shopify Product Price Change Webhooks

    • Purpose: Automatically track and notify when product variant prices change in Shopify
    • Implementation:
    • Added support for products/update webhook topic to existing webhook handler
    • Compares incoming variant prices with stored prices in database
    • Only triggers notifications when prices actually change (avoids false positives)
    • Handles new variants by storing prices without notification (first-time setup)
    • Updates prices in mama_tried_product_variants table after change detection
    • Email Notifications:
    • Sends one email per product with all variant price changes listed
    • Includes: store name, product title, old → new price for each variant, percentage change
    • Shows compare at price changes if applicable
    • Provides direct link to Shopify admin product page
    • Beautiful branded HTML email template with color-coded price increases/decreases
    • Files Created:
    • src/lib/emails/types/price-change.ts – TypeScript type definitions
    • src/lib/emails/templates/price-change.ts – HTML email template
    • src/lib/emails/senders/price-change.ts – Email sending logic
    • database_migrations/048_add_price_changes_alert_type.sql – Database migration
    • docs/shopify-price-change-webhooks.md – Comprehensive documentation
    • Files Modified:
    • src/app/api/webhooks/shopify/route.ts – Added products/update handler
    • Setup Required:
    1. Run database migration to add price_changes alert type
    2. Configure email recipients via Settings → Email Notifications
    3. Add products/update webhook in Shopify admin (URL: existing webhook endpoint)
    • Security: Uses same HMAC signature verification as existing order webhooks
    • Documentation: Full setup, testing, and troubleshooting guide in /docs/shopify-price-change-webhooks.md

    Added Manual Step Tracker to Raffle Sticker Batches

    • Problem: Raffle sticker batches didn’t have the manual status tracker that regular batches have, making it impossible to manually complete steps when needed
    • Solution: Added BatchStatusTerminal component to raffle sticker batch page
    • Implementation:
    • Uncommented BatchStatusTerminal import
    • Enabled batch stages query to fetch stage data
    • Added data formatting for stages and batch status
    • Placed component below shipping panel in right column
    • Files Modified: src/app/(authenticated)/batches/[id]/raffle-page.tsx
    • Impact: Raffle sticker batches now have the same manual tracker functionality as regular batches for manually completing steps when database timeouts or other issues occur

    Database Fixes

    Manually Updated Batch Status for Failed Completions

    • Problem: Batches 498, 506, and 511 didn’t get updated to “fulfilled” status, likely due to database timeouts/stalls during shipment sync
    • Root Cause: Database was getting stalled during ShippingEasy shipment sync operations
    • Solution: Manually updated database to set all affected batches to “fulfilled” status
    • Batches Updated:
    • Batch 498 (D251009-2) – raffle_stickers → fulfilled ✅
    • Batch 506 (D251009-6) – raffle_stickers → fulfilled ✅
    • Batch 511 (D251010-1) – raffle_stickers → fulfilled ✅
    • SQL Executed:
      UPDATE mama_tried_batches 
      SET batch_status = 'fulfilled', updated_at = CURRENT_TIMESTAMP 
      WHERE id IN (498, 506, 511);
    • Note: All affected batches were raffle_stickers type; batches before 498 were working normally

    Fixed Raffle Sticker Order Type Assignment

    • Problem: After variant data reset, raffle sticker orders weren’t being classified correctly and didn’t show up when filtering for “Raffle Sticker” orders
    • Root Cause:
    • assignOrderType() function wasn’t selecting is_raffle_sticker field from products table
    • Newer orders never got their order type assigned
    • Immediate Fix (Database):
    • Updated 176 orders (86 + 90) to have correct raffle-sticker order type
    • Removed 4 bogus variants created during variant reset from raffle sticker products
    • Cleared variant references from 112 raffle sticker line items
    • Raffle stickers are now simple products without variants (as intended)
    • Code Fix (Deploy Tomorrow):
    • Updated assignOrderType() in src/lib/orderUtils.ts to include is_raffle_sticker field
    • Files Modified: src/lib/orderUtils.ts
    • Impact: Raffle sticker orders now show up correctly in filtered views