From 247bc8b3af2cd4104d8b7993f0f6f624fc43a6d0 Mon Sep 17 00:00:00 2001 From: Arthur Belleville Date: Fri, 14 Nov 2025 09:14:25 +0100 Subject: [PATCH] Add docs --- docs/CLOUD_BUILD_ENV_CONFIG.md | 355 +++++++++++++++++++++++++++++++++ docs/CLOUD_BUILD_SETUP.md | 268 +++++++++++++++++++++++++ docs/DOCKER_BUILD.md | 163 +++++++++++++++ docs/DOCKER_FIX_SUMMARY.md | 145 ++++++++++++++ 4 files changed, 931 insertions(+) create mode 100644 docs/CLOUD_BUILD_ENV_CONFIG.md create mode 100644 docs/CLOUD_BUILD_SETUP.md create mode 100644 docs/DOCKER_BUILD.md create mode 100644 docs/DOCKER_FIX_SUMMARY.md diff --git a/docs/CLOUD_BUILD_ENV_CONFIG.md b/docs/CLOUD_BUILD_ENV_CONFIG.md new file mode 100644 index 0000000..963bcc7 --- /dev/null +++ b/docs/CLOUD_BUILD_ENV_CONFIG.md @@ -0,0 +1,355 @@ +# Cloud Build Environment Configuration + +This guide explains how to set up separate staging and production triggers with environment-specific configurations. + +## Overview + +The `cloudbuild.yaml` uses substitution variables to make it environment-agnostic. You'll create **two separate triggers** (one for staging, one for production) with different substitution variable values. + +**Note**: The Dockerfile uses a single final stage. Environment differentiation (staging vs production) is handled entirely through Cloud Run environment variables, not at build time. This simplifies the build process - the same Docker image can be used for both environments. + +## Architecture + +``` +┌─────────────────┐ ┌──────────────────────┐ +│ Git Branch │ │ Cloud Build Trigger │ +│ - main │─────▶│ (Production) │ +│ │ │ Variables: prod │ +└─────────────────┘ └──────────────────────┘ + +┌─────────────────┐ ┌──────────────────────┐ +│ Git Branch │ │ Cloud Build Trigger │ +│ - develop │─────▶│ (Staging) │ +│ │ │ Variables: staging │ +└─────────────────┘ └──────────────────────┘ +``` + +## Required Substitution Variables + +### Common Variables (Different per Environment) + +| Variable | Production Value | Staging Value | +| ----------------------- | ------------------------ | ---------------------------- | +| `$_NODE_ENV` | `production` | `staging` | +| `$_SERVICE_NAME` | `xtablo-api` | `xtablo-api-staging` | +| `$_SUPABASE_URL` | Production Supabase URL | Staging Supabase URL | +| `$_STREAM_CHAT_API_KEY` | Production key | Staging key | +| `$_EMAIL_USER` | `noreply@xtablo.com` | `staging@xtablo.com` | +| `$_EMAIL_CLIENT_ID` | Production OAuth client | Staging OAuth client | +| `$_R2_ACCOUNT_ID` | Production R2 account | Staging R2 account | +| `$_CORS_ORIGIN` | `https://app.xtablo.com` | `https://staging.xtablo.com` | +| `$_XTABLO_URL` | `https://app.xtablo.com` | `https://staging.xtablo.com` | +| `$_LOG_LEVEL` | `info` | `debug` | + +### Build Configuration (Same for Both) + +| Variable | Value | +| ----------------- | ---------------------------------- | +| `$_AR_PROJECT_ID` | Your GCP project ID | +| `$_AR_REPOSITORY` | `xtablo` (or your repository name) | + +### Secret Manager References (Different per Environment) + +Each environment should have its own secrets in Secret Manager: + +| Variable | Production Secret Name | Staging Secret Name | +| ------------------------------------- | --------------------------------- | ------------------------------------ | +| `$_SUPABASE_SERVICE_ROLE_KEY_SECRET` | `supabase-service-role-key-prod` | `supabase-service-role-key-staging` | +| `$_SUPABASE_CONNECTION_STRING_SECRET` | `supabase-connection-string-prod` | `supabase-connection-string-staging` | +| `$_SUPABASE_CA_CERT_SECRET` | `supabase-ca-cert-prod` | `supabase-ca-cert-staging` | +| `$_STREAM_CHAT_API_SECRET_SECRET` | `stream-chat-api-secret-prod` | `stream-chat-api-secret-staging` | +| `$_STRIPE_SECRET_KEY_SECRET` | `stripe-secret-key-prod` | `stripe-secret-key-staging` | +| `$_STRIPE_WEBHOOK_SECRET_SECRET` | `stripe-webhook-secret-prod` | `stripe-webhook-secret-staging` | +| `$_EMAIL_CLIENT_SECRET_SECRET` | `email-client-secret-prod` | `email-client-secret-staging` | +| `$_EMAIL_REFRESH_TOKEN_SECRET` | `email-refresh-token-prod` | `email-refresh-token-staging` | +| `$_R2_ACCESS_KEY_ID_SECRET` | `r2-access-key-id-prod` | `r2-access-key-id-staging` | +| `$_R2_SECRET_ACCESS_KEY_SECRET` | `r2-secret-access-key-prod` | `r2-secret-access-key-staging` | + +## Setup Instructions + +### Step 1: Create Secrets in Secret Manager + +#### Production Secrets + +```bash +# Supabase (Production) +echo -n "prod-service-role-key" | gcloud secrets create supabase-service-role-key-prod --data-file=- +echo -n "prod-connection-string" | gcloud secrets create supabase-connection-string-prod --data-file=- +echo -n "prod-ca-cert" | gcloud secrets create supabase-ca-cert-prod --data-file=- + +# Stream Chat (Production) +echo -n "prod-stream-secret" | gcloud secrets create stream-chat-api-secret-prod --data-file=- + +# Stripe (Production) +echo -n "prod-stripe-key" | gcloud secrets create stripe-secret-key-prod --data-file=- +echo -n "prod-webhook-secret" | gcloud secrets create stripe-webhook-secret-prod --data-file=- + +# Email (Production) +echo -n "prod-client-secret" | gcloud secrets create email-client-secret-prod --data-file=- +echo -n "prod-refresh-token" | gcloud secrets create email-refresh-token-prod --data-file=- + +# R2 (Production) +echo -n "prod-access-key-id" | gcloud secrets create r2-access-key-id-prod --data-file=- +echo -n "prod-secret-access-key" | gcloud secrets create r2-secret-access-key-prod --data-file=- +``` + +#### Staging Secrets + +```bash +# Supabase (Staging) +echo -n "staging-service-role-key" | gcloud secrets create supabase-service-role-key-staging --data-file=- +echo -n "staging-connection-string" | gcloud secrets create supabase-connection-string-staging --data-file=- +echo -n "staging-ca-cert" | gcloud secrets create supabase-ca-cert-staging --data-file=- + +# Stream Chat (Staging) +echo -n "staging-stream-secret" | gcloud secrets create stream-chat-api-secret-staging --data-file=- + +# Stripe (Staging - use test keys) +echo -n "staging-stripe-key" | gcloud secrets create stripe-secret-key-staging --data-file=- +echo -n "staging-webhook-secret" | gcloud secrets create stripe-webhook-secret-staging --data-file=- + +# Email (Staging) +echo -n "staging-client-secret" | gcloud secrets create email-client-secret-staging --data-file=- +echo -n "staging-refresh-token" | gcloud secrets create email-refresh-token-staging --data-file=- + +# R2 (Staging) +echo -n "staging-access-key-id" | gcloud secrets create r2-access-key-id-staging --data-file=- +echo -n "staging-secret-access-key" | gcloud secrets create r2-secret-access-key-staging --data-file=- +``` + +### Step 2: Grant Access to Secrets + +```bash +PROJECT_ID="your-project-id" +SERVICE_ACCOUNT="${PROJECT_ID}@appspot.gserviceaccount.com" + +# Grant access to all production secrets +for secret in \ + supabase-service-role-key-prod \ + supabase-connection-string-prod \ + supabase-ca-cert-prod \ + stream-chat-api-secret-prod \ + stripe-secret-key-prod \ + stripe-webhook-secret-prod \ + email-client-secret-prod \ + email-refresh-token-prod \ + r2-access-key-id-prod \ + r2-secret-access-key-prod +do + gcloud secrets add-iam-policy-binding $secret \ + --member="serviceAccount:${SERVICE_ACCOUNT}" \ + --role="roles/secretmanager.secretAccessor" +done + +# Grant access to all staging secrets +for secret in \ + supabase-service-role-key-staging \ + supabase-connection-string-staging \ + supabase-ca-cert-staging \ + stream-chat-api-secret-staging \ + stripe-secret-key-staging \ + stripe-webhook-secret-staging \ + email-client-secret-staging \ + email-refresh-token-staging \ + r2-access-key-id-staging \ + r2-secret-access-key-staging +do + gcloud secrets add-iam-policy-binding $secret \ + --member="serviceAccount:${SERVICE_ACCOUNT}" \ + --role="roles/secretmanager.secretAccessor" +done +``` + +### Step 3: Create Cloud Build Triggers + +#### Production Trigger + +Via gcloud CLI: + +```bash +gcloud builds triggers create github \ + --name="xtablo-api-production" \ + --repo-name="xtablo-source" \ + --repo-owner="your-github-org" \ + --branch-pattern="^main$" \ + --build-config="apps/api/cloudbuild.yaml" \ + --substitutions=' +_NODE_ENV=production, +_SERVICE_NAME=xtablo-api, +_AR_PROJECT_ID=your-project-id, +_AR_REPOSITORY=xtablo, +_SUPABASE_URL=https://your-prod-project.supabase.co, +_STREAM_CHAT_API_KEY=your-prod-stream-key, +_EMAIL_USER=noreply@xtablo.com, +_EMAIL_CLIENT_ID=your-prod-oauth-client-id, +_R2_ACCOUNT_ID=your-prod-r2-account, +_CORS_ORIGIN=https://app.xtablo.com, +_XTABLO_URL=https://app.xtablo.com, +_LOG_LEVEL=info, +_SUPABASE_SERVICE_ROLE_KEY_SECRET=supabase-service-role-key-prod, +_SUPABASE_CONNECTION_STRING_SECRET=supabase-connection-string-prod, +_SUPABASE_CA_CERT_SECRET=supabase-ca-cert-prod, +_STREAM_CHAT_API_SECRET_SECRET=stream-chat-api-secret-prod, +_STRIPE_SECRET_KEY_SECRET=stripe-secret-key-prod, +_STRIPE_WEBHOOK_SECRET_SECRET=stripe-webhook-secret-prod, +_EMAIL_CLIENT_SECRET_SECRET=email-client-secret-prod, +_EMAIL_REFRESH_TOKEN_SECRET=email-refresh-token-prod, +_R2_ACCESS_KEY_ID_SECRET=r2-access-key-id-prod, +_R2_SECRET_ACCESS_KEY_SECRET=r2-secret-access-key-prod +' +``` + +#### Staging Trigger + +Via gcloud CLI: + +```bash +gcloud builds triggers create github \ + --name="xtablo-api-staging" \ + --repo-name="xtablo-source" \ + --repo-owner="your-github-org" \ + --branch-pattern="^develop$" \ + --build-config="apps/api/cloudbuild.yaml" \ + --substitutions=' +_NODE_ENV=staging, +_SERVICE_NAME=xtablo-api-staging, +_AR_PROJECT_ID=your-project-id, +_AR_REPOSITORY=xtablo, +_SUPABASE_URL=https://your-staging-project.supabase.co, +_STREAM_CHAT_API_KEY=your-staging-stream-key, +_EMAIL_USER=staging@xtablo.com, +_EMAIL_CLIENT_ID=your-staging-oauth-client-id, +_R2_ACCOUNT_ID=your-staging-r2-account, +_CORS_ORIGIN=https://staging.xtablo.com, +_XTABLO_URL=https://staging.xtablo.com, +_LOG_LEVEL=debug, +_SUPABASE_SERVICE_ROLE_KEY_SECRET=supabase-service-role-key-staging, +_SUPABASE_CONNECTION_STRING_SECRET=supabase-connection-string-staging, +_SUPABASE_CA_CERT_SECRET=supabase-ca-cert-staging, +_STREAM_CHAT_API_SECRET_SECRET=stream-chat-api-secret-staging, +_STRIPE_SECRET_KEY_SECRET=stripe-secret-key-staging, +_STRIPE_WEBHOOK_SECRET_SECRET=stripe-webhook-secret-staging, +_EMAIL_CLIENT_SECRET_SECRET=email-client-secret-staging, +_EMAIL_REFRESH_TOKEN_SECRET=email-refresh-token-staging, +_R2_ACCESS_KEY_ID_SECRET=r2-access-key-id-staging, +_R2_SECRET_ACCESS_KEY_SECRET=r2-secret-access-key-staging +' +``` + +### Step 4: Configure via Console (Alternative) + +If you prefer using the Google Cloud Console: + +1. Go to **Cloud Build > Triggers** +2. Click **Create Trigger** +3. Configure: + + - **Name**: `xtablo-api-production` (or `xtablo-api-staging`) + - **Event**: Push to a branch + - **Repository**: Your GitHub repository + - **Branch**: `^main$` (or `^develop$` for staging) + - **Build configuration**: Cloud Build configuration file + - **Location**: `apps/api/cloudbuild.yaml` + +4. Scroll to **Substitution variables** and add all variables from the table above + +5. Save the trigger + +## Verification + +### Test Production Deployment + +```bash +# Trigger manually +gcloud builds triggers run xtablo-api-production --branch=main + +# Check logs +gcloud builds list --limit=5 +gcloud builds log BUILD_ID + +# Verify deployment +gcloud run services describe xtablo-api --region=europe-west1 +``` + +### Test Staging Deployment + +```bash +# Trigger manually +gcloud builds triggers run xtablo-api-staging --branch=develop + +# Check logs +gcloud builds list --limit=5 + +# Verify deployment +gcloud run services describe xtablo-api-staging --region=europe-west1 +``` + +### Verify Environment Variables + +```bash +# Production +gcloud run services describe xtablo-api --region=europe-west1 \ + --format="value(spec.template.spec.containers[0].env)" + +# Staging +gcloud run services describe xtablo-api-staging --region=europe-west1 \ + --format="value(spec.template.spec.containers[0].env)" +``` + +## Best Practices + +1. **Secret Naming**: Use `-prod` and `-staging` suffixes for secrets +2. **Service Names**: Use different names for each environment (`xtablo-api` vs `xtablo-api-staging`) +3. **Branches**: Deploy production from `main`, staging from `develop` +4. **Stripe Keys**: Use Stripe test keys in staging +5. **Supabase**: Use separate Supabase projects for staging and production +6. **Monitoring**: Set up different alert thresholds for staging vs production +7. **Resource Limits**: You may want lower limits for staging to save costs + +## Quick Reference + +### Environment Comparison + +| Aspect | Production | Staging | +| ------------------ | ---------------- | -------------------- | +| **Branch** | `main` | `develop` | +| **Service Name** | `xtablo-api` | `xtablo-api-staging` | +| **Domain** | `app.xtablo.com` | `staging.xtablo.com` | +| **Log Level** | `info` | `debug` | +| **Stripe** | Live keys | Test keys | +| **Min Instances** | 1 (optional) | 0 | +| **Secrets Suffix** | `-prod` | `-staging` | + +## Troubleshooting + +### "Secret not found" error + +- Ensure secret names in substitution variables match actual secret names +- Verify secrets exist: `gcloud secrets list | grep staging` or `grep prod` + +### Wrong environment variables in deployment + +- Check trigger substitution variables in Cloud Build console +- Verify the correct trigger is being used for the branch + +### Services pointing to wrong databases + +- Ensure `SUPABASE_URL` is different for staging/production +- Check that secret names include correct environment suffix + +## Example: Adding a New Environment Variable + +To add a new environment variable (e.g., `FEATURE_FLAG`): + +1. Add to `cloudbuild.yaml` in the `--set-env-vars` line: + + ```yaml + "NODE_ENV=$_NODE_ENV,...,FEATURE_FLAG=$_FEATURE_FLAG" + ``` + +2. Add to **both triggers** with different values: + + - Production: `_FEATURE_FLAG=false` + - Staging: `_FEATURE_FLAG=true` + +3. Update app code to read `process.env.FEATURE_FLAG` diff --git a/docs/CLOUD_BUILD_SETUP.md b/docs/CLOUD_BUILD_SETUP.md new file mode 100644 index 0000000..35c6016 --- /dev/null +++ b/docs/CLOUD_BUILD_SETUP.md @@ -0,0 +1,268 @@ +# Google Cloud Build Setup Guide + +## Overview + +This guide explains how to configure Google Cloud Build for automatic deployment of the XTablo API to Cloud Run. + +## Prerequisites + +1. Google Cloud Project with billing enabled +2. Cloud Build API enabled +3. Cloud Run API enabled +4. Secret Manager API enabled +5. Artifact Registry repository created + +## Required Substitution Variables + +The `cloudbuild.yaml` uses substitution variables that must be configured in your Cloud Build trigger. Here's what each variable is for: + +### Build Configuration Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `$_NODE_ENV` | Environment (staging/production) | `production` | +| `$_AR_PROJECT_ID` | Artifact Registry project ID | `your-project-id` | +| `$_AR_REPOSITORY` | Artifact Registry repository name | `xtablo` | +| `$_SERVICE_NAME` | Cloud Run service name | `xtablo-api` | + +### Application Environment Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `$_SUPABASE_URL` | Supabase project URL | `https://xxx.supabase.co` | +| `$_STREAM_CHAT_API_KEY` | Stream Chat API key | `your-stream-api-key` | +| `$_EMAIL_USER` | Email sender address | `noreply@xtablo.com` | +| `$_EMAIL_CLIENT_ID` | OAuth2 client ID for email | `your-client-id` | +| `$_R2_ACCOUNT_ID` | Cloudflare R2 account ID | `your-r2-account-id` | +| `$_CORS_ORIGIN` | CORS allowed origin | `https://app.xtablo.com` | +| `$_XTABLO_URL` | Frontend application URL | `https://app.xtablo.com` | + +## Setting Up Substitution Variables + +### Option 1: Via Google Cloud Console + +1. Go to Cloud Build > Triggers +2. Select your trigger (or create a new one) +3. Scroll to "Substitution variables" +4. Add each variable with its value: + ``` + _NODE_ENV = production + _AR_PROJECT_ID = your-project-id + _AR_REPOSITORY = xtablo + _SERVICE_NAME = xtablo-api + _SUPABASE_URL = https://your-project.supabase.co + _STREAM_CHAT_API_KEY = your-key + _EMAIL_USER = noreply@xtablo.com + _EMAIL_CLIENT_ID = your-client-id + _R2_ACCOUNT_ID = your-account-id + _CORS_ORIGIN = https://app.xtablo.com + _XTABLO_URL = https://app.xtablo.com + ``` + +### Option 2: Via gcloud CLI + +```bash +gcloud builds triggers create github \ + --name="xtablo-api-production" \ + --repo-name="xtablo-source" \ + --repo-owner="your-github-org" \ + --branch-pattern="^main$" \ + --build-config="apps/api/cloudbuild.yaml" \ + --substitutions=' + _NODE_ENV=production, + _AR_PROJECT_ID=your-project-id, + _AR_REPOSITORY=xtablo, + _SERVICE_NAME=xtablo-api, + _SUPABASE_URL=https://your-project.supabase.co, + _STREAM_CHAT_API_KEY=your-key, + _EMAIL_USER=noreply@xtablo.com, + _EMAIL_CLIENT_ID=your-client-id, + _R2_ACCOUNT_ID=your-account-id, + _CORS_ORIGIN=https://app.xtablo.com, + _XTABLO_URL=https://app.xtablo.com + ' +``` + +## Setting Up Secrets in Secret Manager + +The sensitive values (API keys, tokens, etc.) are stored in Google Cloud Secret Manager. Create these secrets: + +### Required Secrets + +```bash +# Supabase secrets +echo -n "your-service-role-key" | gcloud secrets create supabase-service-role-key --data-file=- +echo -n "your-connection-string" | gcloud secrets create supabase-connection-string --data-file=- +echo -n "your-ca-cert" | gcloud secrets create supabase-ca-cert --data-file=- + +# Stream Chat secret +echo -n "your-stream-secret" | gcloud secrets create stream-chat-api-secret --data-file=- + +# Stripe secrets +echo -n "your-stripe-key" | gcloud secrets create stripe-secret-key --data-file=- +echo -n "your-webhook-secret" | gcloud secrets create stripe-webhook-secret --data-file=- + +# Email secrets +echo -n "your-client-secret" | gcloud secrets create email-client-secret --data-file=- +echo -n "your-refresh-token" | gcloud secrets create email-refresh-token --data-file=- + +# R2 (Cloudflare) secrets +echo -n "your-access-key-id" | gcloud secrets create r2-access-key-id --data-file=- +echo -n "your-secret-access-key" | gcloud secrets create r2-secret-access-key --data-file=- +``` + +### Grant Cloud Run Access to Secrets + +```bash +# Get your Cloud Run service account +PROJECT_ID="your-project-id" +SERVICE_ACCOUNT="${PROJECT_ID}@appspot.gserviceaccount.com" + +# Grant access to each secret +for secret in \ + supabase-service-role-key \ + supabase-connection-string \ + supabase-ca-cert \ + stream-chat-api-secret \ + stripe-secret-key \ + stripe-webhook-secret \ + email-client-secret \ + email-refresh-token \ + r2-access-key-id \ + r2-secret-access-key +do + gcloud secrets add-iam-policy-binding $secret \ + --member="serviceAccount:${SERVICE_ACCOUNT}" \ + --role="roles/secretmanager.secretAccessor" +done +``` + +## Environment-Specific Configurations + +### Production Trigger + +```yaml +_NODE_ENV: production +_CORS_ORIGIN: https://app.xtablo.com +_XTABLO_URL: https://app.xtablo.com +``` + +### Staging Trigger + +```yaml +_NODE_ENV: staging +_CORS_ORIGIN: https://staging.xtablo.com +_XTABLO_URL: https://staging.xtablo.com +_SERVICE_NAME: xtablo-api-staging +``` + +## Verifying the Setup + +After creating your trigger and setting up the secrets: + +1. **Test the trigger manually:** + ```bash + gcloud builds triggers run xtablo-api-production --branch=main + ``` + +2. **Check the build logs:** + ```bash + gcloud builds list --limit=5 + gcloud builds log BUILD_ID + ``` + +3. **Verify the deployed service:** + ```bash + gcloud run services describe xtablo-api --region=europe-west1 + ``` + +4. **Check environment variables:** + ```bash + gcloud run services describe xtablo-api --region=europe-west1 --format="value(spec.template.spec.containers[0].env)" + ``` + +## Troubleshooting + +### Build Fails with "Missing Substitution Variable" +- Ensure all `$_VARIABLE` names are defined in your trigger +- Check for typos in variable names + +### Cloud Run Deployment Fails +- Verify the service account has necessary permissions +- Check that the Artifact Registry URL is correct +- Ensure the image was successfully pushed + +### Application Fails to Start +- Check Cloud Run logs: `gcloud run logs read --service=xtablo-api --region=europe-west1` +- Verify all secrets are accessible to the service account +- Check that environment variables are properly set + +### Secret Access Denied +- Ensure the Cloud Run service account has `roles/secretmanager.secretAccessor` for each secret +- Verify secrets exist: `gcloud secrets list` + +## Additional Configuration + +### Service Account Permissions + +Your Cloud Run service needs these roles: + +```bash +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:${SERVICE_ACCOUNT}" \ + --role="roles/secretmanager.secretAccessor" + +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:${SERVICE_ACCOUNT}" \ + --role="roles/cloudsql.client" # If using Cloud SQL +``` + +### Cloud Run Configuration Options + +You can add these to your `cloudbuild.yaml` deploy step: + +```yaml +- '--memory' +- '512Mi' +- '--cpu' +- '1' +- '--max-instances' +- '10' +- '--min-instances' +- '0' +- '--concurrency' +- '80' +- '--timeout' +- '300' +- '--allow-unauthenticated' # If your API is public +``` + +## Manual Deployment + +If you need to deploy manually without Cloud Build: + +```bash +# Build the image +docker build -f apps/api/Dockerfile --target production -t xtablo-api:latest . + +# Tag for Artifact Registry +docker tag xtablo-api:latest europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest + +# Push to Artifact Registry +docker push europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest + +# Deploy to Cloud Run +gcloud run deploy xtablo-api \ + --image=europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest \ + --region=europe-west1 \ + --set-env-vars="NODE_ENV=production,PORT=8080,..." \ + --update-secrets="SUPABASE_SERVICE_ROLE_KEY=supabase-service-role-key:latest,..." +``` + +## Support + +For more information: +- [Cloud Build Documentation](https://cloud.google.com/build/docs) +- [Cloud Run Documentation](https://cloud.google.com/run/docs) +- [Secret Manager Documentation](https://cloud.google.com/secret-manager/docs) + diff --git a/docs/DOCKER_BUILD.md b/docs/DOCKER_BUILD.md new file mode 100644 index 0000000..4af75cc --- /dev/null +++ b/docs/DOCKER_BUILD.md @@ -0,0 +1,163 @@ +# Docker Build Guide for XTablo API + +## Overview + +This Dockerfile is configured for a **pnpm monorepo** setup. It must be built from the repository root, not from the `apps/api` directory. + +## Build Commands + +### Production Build (default) + +```bash +cd /path/to/xtablo-source +docker build -f apps/api/Dockerfile -t xtablo-api:production . +``` + +### Staging Build + +```bash +cd /path/to/xtablo-source +docker build -f apps/api/Dockerfile --target staging -t xtablo-api:staging . +``` + +### Build Specific Stages (for testing) + +```bash +# Test dependencies stage +docker build -f apps/api/Dockerfile --target deps -t xtablo-api:deps . + +# Test build stage +docker build -f apps/api/Dockerfile --target build -t xtablo-api:build . + +# Test production dependencies +docker build -f apps/api/Dockerfile --target prod-deps -t xtablo-api:prod-deps . +``` + +## Running the Container + +### Basic Run + +```bash +docker run -p 8080:8080 \ + -e SUPABASE_URL=your_url \ + -e SUPABASE_SERVICE_ROLE_KEY=your_key \ + -e STREAM_API_KEY=your_key \ + -e STREAM_SECRET=your_secret \ + xtablo-api:production +``` + +### With Environment File + +```bash +docker run -p 8080:8080 --env-file .env.production xtablo-api:production +``` + +## Docker Compose Example + +```yaml +version: "3.8" + +services: + api: + build: + context: . + dockerfile: apps/api/Dockerfile + target: production + ports: + - "8080:8080" + environment: + - NODE_ENV=production + - SUPABASE_URL=${SUPABASE_URL} + - SUPABASE_SERVICE_ROLE_KEY=${SUPABASE_SERVICE_ROLE_KEY} + - STREAM_API_KEY=${STREAM_API_KEY} + - STREAM_SECRET=${STREAM_SECRET} + restart: unless-stopped +``` + +## Image Structure + +- **Base Image**: `node:18-alpine` +- **Package Manager**: pnpm (via corepack) +- **Working Directory**: `/app/apps/api` +- **User**: `nodejs` (non-root, UID 1001) +- **Port**: 8080 + +## Multi-Stage Build Details + +1. **base**: Sets up Node.js, pnpm, copies workspace files +2. **deps**: Installs all dependencies (including dev dependencies) +3. **build**: Compiles TypeScript to JavaScript +4. **prod-deps**: Installs only production dependencies +5. **staging**: Creates staging image with production dependencies +6. **production**: Creates production image (default target) + +## Important Notes + +- ⚠️ **Always build from the monorepo root**, not from `apps/api` +- Environment variables should be injected at runtime, not baked into the image +- The image uses pnpm's workspace features with proper symlink structure +- pnpm version: 10.19.0 (via corepack) +- Node version: 18.20.8 +- The `.dockerignore` file excludes local `dist` and `node_modules` to ensure clean builds + +## Verification Commands + +### Check image size + +```bash +docker images xtablo-api:production +``` + +### Inspect image + +```bash +docker run --rm --entrypoint sh xtablo-api:production -c "ls -la /app/apps/api" +``` + +### Test pnpm installation + +```bash +docker run --rm --entrypoint sh xtablo-api:production -c "pnpm --version" +``` + +### Check built files + +```bash +docker run --rm --entrypoint sh xtablo-api:production -c "ls -la /app/apps/api/dist" +``` + +## Troubleshooting + +### "pnpm-lock.yaml: not found" + +- Make sure you're building from the repository root +- Use `-f apps/api/Dockerfile` to specify the Dockerfile location + +### "Permission denied" + +- The container runs as a non-root user (`nodejs`) +- Ensure file permissions are set correctly (handled by the Dockerfile) + +### Dependencies not found + +- The monorepo structure requires the entire workspace for dependency resolution +- Make sure `pnpm-workspace.yaml` is present in the root + +## CI/CD Integration + +For Cloud Build or similar CI/CD systems, update your build configuration: + +```yaml +steps: + - name: "gcr.io/cloud-builders/docker" + args: + - "build" + - "-f" + - "apps/api/Dockerfile" + - "-t" + - "gcr.io/$PROJECT_ID/xtablo-api:$COMMIT_SHA" + - "-t" + - "gcr.io/$PROJECT_ID/xtablo-api:latest" + - "." + dir: "." # Build from root, not apps/api +``` diff --git a/docs/DOCKER_FIX_SUMMARY.md b/docs/DOCKER_FIX_SUMMARY.md new file mode 100644 index 0000000..503f8ac --- /dev/null +++ b/docs/DOCKER_FIX_SUMMARY.md @@ -0,0 +1,145 @@ +# Docker Build Fix Summary + +## Issue +When deploying to Google Cloud, the application failed with: +``` +Error: Cannot find module '/app/apps/api/dist/index.js' +``` + +## Root Causes + +### 1. Missing `.dockerignore` +- Local `dist/` folders were being copied into the Docker build context +- This caused nested `dist/dist/` structures and incorrect file locations + +### 2. TypeScript Compilation Structure +- `tsconfig.json` was missing `rootDir: "./src"` +- TypeScript compiled `src/index.ts` to `dist/src/index.js` instead of `dist/index.js` +- The start script expected `dist/index.js` + +### 3. pnpm Symlink Issues +- Docker's `COPY` command was not preserving pnpm's symlink structure +- `node_modules` were copied but symlinks to packages in `.pnpm/` were lost +- Node.js couldn't resolve packages like `@hono/node-server` + +## Solutions Implemented + +### 1. Created `.dockerignore` at Repository Root +``` +**/dist +**/node_modules +**/__tests__ +# ... and other build artifacts +``` + +This ensures Docker doesn't copy local build artifacts into the image. + +### 2. Updated `apps/api/tsconfig.json` +Added: +```json +{ + "compilerOptions": { + "rootDir": "./src", + // ... + }, + "include": ["src/**/*"] +} +``` + +This ensures TypeScript compiles `src/index.ts` → `dist/index.js` (not `dist/src/index.js`). + +### 3. Fixed Dockerfile Dependency Installation +Changed from copying `node_modules` to installing them fresh in the final stage: + +**Before:** +```dockerfile +COPY --from=prod-deps /app/node_modules ./node_modules +``` + +**After:** +```dockerfile +# Copy built files and workspace structure +COPY --from=build /app/apps/api/dist ./apps/api/dist +COPY --from=build /app/apps/api/package.json ./apps/api/package.json +COPY --from=prod-deps /app/packages ./packages + +# Install dependencies with proper symlinks +RUN pnpm install --frozen-lockfile --prod --filter @xtablo/api... +``` + +This ensures pnpm creates proper symlinks in `/app/apps/api/node_modules/` that point to packages in `/app/node_modules/.pnpm/`. + +### 4. Updated `cloudbuild.yaml` +Changed build command to run from monorepo root: + +**Before:** +```yaml +args: [ 'build', '--target', '$_NODE_ENV', '-t', '...', 'apps/api' ] +``` + +**After:** +```yaml +args: [ 'build', '-f', 'apps/api/Dockerfile', '--target', '$_NODE_ENV', '-t', '...', '.' ] +``` + +## Verification + +### Module Resolution Test +```bash +docker run --rm --entrypoint sh xtablo-api:production -c \ + "ls -la /app/apps/api/node_modules/@hono/" +``` + +Shows proper symlinks: +``` +lrwxrwxrwx node-server -> ../../../../node_modules/.pnpm/@hono+node-server@... +``` + +### Application Start Test +```bash +docker run --rm -e SUPABASE_URL=test -e SUPABASE_SERVICE_ROLE_KEY=test \ + xtablo-api:production +``` + +Application starts successfully (fails only on missing GCP credentials, not module resolution). + +## Final Image Details +- **Size**: 1GB (production) +- **Node.js**: 18.20.8 +- **pnpm**: 10.19.0 +- **User**: nodejs (non-root, UID 1001) +- **Working Directory**: `/app/apps/api` +- **Module Structure**: Proper pnpm workspace with symlinks + +## Files Modified + +1. `/apps/api/Dockerfile` - Updated multi-stage build +2. `/apps/api/tsconfig.json` - Added `rootDir` and `include` +3. `/apps/api/cloudbuild.yaml` - Fixed build context +4. `/.dockerignore` - Created to exclude build artifacts +5. `/apps/api/DOCKER_BUILD.md` - Added comprehensive documentation + +## Key Takeaways + +1. **Always use `.dockerignore`** to prevent local artifacts from contaminating Docker builds +2. **pnpm symlinks require special handling** - install dependencies in the final stage rather than copying +3. **TypeScript `rootDir` matters** - set it explicitly to control output structure +4. **Monorepo builds must run from root** - use `-f` flag to specify Dockerfile location + +## Testing in Production + +The image is now ready for deployment. All module resolution issues are fixed. Make sure to provide proper environment variables at runtime: + +```bash +docker run -p 8080:8080 \ + -e SUPABASE_URL=... \ + -e SUPABASE_SERVICE_ROLE_KEY=... \ + -e STREAM_API_KEY=... \ + -e STREAM_SECRET=... \ + # ... other env vars + xtablo-api:production +``` + +## Date +Fixed: November 13, 2024 +