This commit is contained in:
Arthur Belleville 2025-11-14 09:14:25 +01:00
parent 803c9ff391
commit 247bc8b3af
No known key found for this signature in database
4 changed files with 931 additions and 0 deletions

View file

@ -0,0 +1,355 @@
# Cloud Build Environment Configuration
This guide explains how to set up separate staging and production triggers with environment-specific configurations.
## Overview
The `cloudbuild.yaml` uses substitution variables to make it environment-agnostic. You'll create **two separate triggers** (one for staging, one for production) with different substitution variable values.
**Note**: The Dockerfile uses a single final stage. Environment differentiation (staging vs production) is handled entirely through Cloud Run environment variables, not at build time. This simplifies the build process - the same Docker image can be used for both environments.
## Architecture
```
┌─────────────────┐ ┌──────────────────────┐
│ Git Branch │ │ Cloud Build Trigger │
│ - main │─────▶│ (Production) │
│ │ │ Variables: prod │
└─────────────────┘ └──────────────────────┘
┌─────────────────┐ ┌──────────────────────┐
│ Git Branch │ │ Cloud Build Trigger │
│ - develop │─────▶│ (Staging) │
│ │ │ Variables: staging │
└─────────────────┘ └──────────────────────┘
```
## Required Substitution Variables
### Common Variables (Different per Environment)
| Variable | Production Value | Staging Value |
| ----------------------- | ------------------------ | ---------------------------- |
| `$_NODE_ENV` | `production` | `staging` |
| `$_SERVICE_NAME` | `xtablo-api` | `xtablo-api-staging` |
| `$_SUPABASE_URL` | Production Supabase URL | Staging Supabase URL |
| `$_STREAM_CHAT_API_KEY` | Production key | Staging key |
| `$_EMAIL_USER` | `noreply@xtablo.com` | `staging@xtablo.com` |
| `$_EMAIL_CLIENT_ID` | Production OAuth client | Staging OAuth client |
| `$_R2_ACCOUNT_ID` | Production R2 account | Staging R2 account |
| `$_CORS_ORIGIN` | `https://app.xtablo.com` | `https://staging.xtablo.com` |
| `$_XTABLO_URL` | `https://app.xtablo.com` | `https://staging.xtablo.com` |
| `$_LOG_LEVEL` | `info` | `debug` |
### Build Configuration (Same for Both)
| Variable | Value |
| ----------------- | ---------------------------------- |
| `$_AR_PROJECT_ID` | Your GCP project ID |
| `$_AR_REPOSITORY` | `xtablo` (or your repository name) |
### Secret Manager References (Different per Environment)
Each environment should have its own secrets in Secret Manager:
| Variable | Production Secret Name | Staging Secret Name |
| ------------------------------------- | --------------------------------- | ------------------------------------ |
| `$_SUPABASE_SERVICE_ROLE_KEY_SECRET` | `supabase-service-role-key-prod` | `supabase-service-role-key-staging` |
| `$_SUPABASE_CONNECTION_STRING_SECRET` | `supabase-connection-string-prod` | `supabase-connection-string-staging` |
| `$_SUPABASE_CA_CERT_SECRET` | `supabase-ca-cert-prod` | `supabase-ca-cert-staging` |
| `$_STREAM_CHAT_API_SECRET_SECRET` | `stream-chat-api-secret-prod` | `stream-chat-api-secret-staging` |
| `$_STRIPE_SECRET_KEY_SECRET` | `stripe-secret-key-prod` | `stripe-secret-key-staging` |
| `$_STRIPE_WEBHOOK_SECRET_SECRET` | `stripe-webhook-secret-prod` | `stripe-webhook-secret-staging` |
| `$_EMAIL_CLIENT_SECRET_SECRET` | `email-client-secret-prod` | `email-client-secret-staging` |
| `$_EMAIL_REFRESH_TOKEN_SECRET` | `email-refresh-token-prod` | `email-refresh-token-staging` |
| `$_R2_ACCESS_KEY_ID_SECRET` | `r2-access-key-id-prod` | `r2-access-key-id-staging` |
| `$_R2_SECRET_ACCESS_KEY_SECRET` | `r2-secret-access-key-prod` | `r2-secret-access-key-staging` |
## Setup Instructions
### Step 1: Create Secrets in Secret Manager
#### Production Secrets
```bash
# Supabase (Production)
echo -n "prod-service-role-key" | gcloud secrets create supabase-service-role-key-prod --data-file=-
echo -n "prod-connection-string" | gcloud secrets create supabase-connection-string-prod --data-file=-
echo -n "prod-ca-cert" | gcloud secrets create supabase-ca-cert-prod --data-file=-
# Stream Chat (Production)
echo -n "prod-stream-secret" | gcloud secrets create stream-chat-api-secret-prod --data-file=-
# Stripe (Production)
echo -n "prod-stripe-key" | gcloud secrets create stripe-secret-key-prod --data-file=-
echo -n "prod-webhook-secret" | gcloud secrets create stripe-webhook-secret-prod --data-file=-
# Email (Production)
echo -n "prod-client-secret" | gcloud secrets create email-client-secret-prod --data-file=-
echo -n "prod-refresh-token" | gcloud secrets create email-refresh-token-prod --data-file=-
# R2 (Production)
echo -n "prod-access-key-id" | gcloud secrets create r2-access-key-id-prod --data-file=-
echo -n "prod-secret-access-key" | gcloud secrets create r2-secret-access-key-prod --data-file=-
```
#### Staging Secrets
```bash
# Supabase (Staging)
echo -n "staging-service-role-key" | gcloud secrets create supabase-service-role-key-staging --data-file=-
echo -n "staging-connection-string" | gcloud secrets create supabase-connection-string-staging --data-file=-
echo -n "staging-ca-cert" | gcloud secrets create supabase-ca-cert-staging --data-file=-
# Stream Chat (Staging)
echo -n "staging-stream-secret" | gcloud secrets create stream-chat-api-secret-staging --data-file=-
# Stripe (Staging - use test keys)
echo -n "staging-stripe-key" | gcloud secrets create stripe-secret-key-staging --data-file=-
echo -n "staging-webhook-secret" | gcloud secrets create stripe-webhook-secret-staging --data-file=-
# Email (Staging)
echo -n "staging-client-secret" | gcloud secrets create email-client-secret-staging --data-file=-
echo -n "staging-refresh-token" | gcloud secrets create email-refresh-token-staging --data-file=-
# R2 (Staging)
echo -n "staging-access-key-id" | gcloud secrets create r2-access-key-id-staging --data-file=-
echo -n "staging-secret-access-key" | gcloud secrets create r2-secret-access-key-staging --data-file=-
```
### Step 2: Grant Access to Secrets
```bash
PROJECT_ID="your-project-id"
SERVICE_ACCOUNT="${PROJECT_ID}@appspot.gserviceaccount.com"
# Grant access to all production secrets
for secret in \
supabase-service-role-key-prod \
supabase-connection-string-prod \
supabase-ca-cert-prod \
stream-chat-api-secret-prod \
stripe-secret-key-prod \
stripe-webhook-secret-prod \
email-client-secret-prod \
email-refresh-token-prod \
r2-access-key-id-prod \
r2-secret-access-key-prod
do
gcloud secrets add-iam-policy-binding $secret \
--member="serviceAccount:${SERVICE_ACCOUNT}" \
--role="roles/secretmanager.secretAccessor"
done
# Grant access to all staging secrets
for secret in \
supabase-service-role-key-staging \
supabase-connection-string-staging \
supabase-ca-cert-staging \
stream-chat-api-secret-staging \
stripe-secret-key-staging \
stripe-webhook-secret-staging \
email-client-secret-staging \
email-refresh-token-staging \
r2-access-key-id-staging \
r2-secret-access-key-staging
do
gcloud secrets add-iam-policy-binding $secret \
--member="serviceAccount:${SERVICE_ACCOUNT}" \
--role="roles/secretmanager.secretAccessor"
done
```
### Step 3: Create Cloud Build Triggers
#### Production Trigger
Via gcloud CLI:
```bash
gcloud builds triggers create github \
--name="xtablo-api-production" \
--repo-name="xtablo-source" \
--repo-owner="your-github-org" \
--branch-pattern="^main$" \
--build-config="apps/api/cloudbuild.yaml" \
--substitutions='
_NODE_ENV=production,
_SERVICE_NAME=xtablo-api,
_AR_PROJECT_ID=your-project-id,
_AR_REPOSITORY=xtablo,
_SUPABASE_URL=https://your-prod-project.supabase.co,
_STREAM_CHAT_API_KEY=your-prod-stream-key,
_EMAIL_USER=noreply@xtablo.com,
_EMAIL_CLIENT_ID=your-prod-oauth-client-id,
_R2_ACCOUNT_ID=your-prod-r2-account,
_CORS_ORIGIN=https://app.xtablo.com,
_XTABLO_URL=https://app.xtablo.com,
_LOG_LEVEL=info,
_SUPABASE_SERVICE_ROLE_KEY_SECRET=supabase-service-role-key-prod,
_SUPABASE_CONNECTION_STRING_SECRET=supabase-connection-string-prod,
_SUPABASE_CA_CERT_SECRET=supabase-ca-cert-prod,
_STREAM_CHAT_API_SECRET_SECRET=stream-chat-api-secret-prod,
_STRIPE_SECRET_KEY_SECRET=stripe-secret-key-prod,
_STRIPE_WEBHOOK_SECRET_SECRET=stripe-webhook-secret-prod,
_EMAIL_CLIENT_SECRET_SECRET=email-client-secret-prod,
_EMAIL_REFRESH_TOKEN_SECRET=email-refresh-token-prod,
_R2_ACCESS_KEY_ID_SECRET=r2-access-key-id-prod,
_R2_SECRET_ACCESS_KEY_SECRET=r2-secret-access-key-prod
'
```
#### Staging Trigger
Via gcloud CLI:
```bash
gcloud builds triggers create github \
--name="xtablo-api-staging" \
--repo-name="xtablo-source" \
--repo-owner="your-github-org" \
--branch-pattern="^develop$" \
--build-config="apps/api/cloudbuild.yaml" \
--substitutions='
_NODE_ENV=staging,
_SERVICE_NAME=xtablo-api-staging,
_AR_PROJECT_ID=your-project-id,
_AR_REPOSITORY=xtablo,
_SUPABASE_URL=https://your-staging-project.supabase.co,
_STREAM_CHAT_API_KEY=your-staging-stream-key,
_EMAIL_USER=staging@xtablo.com,
_EMAIL_CLIENT_ID=your-staging-oauth-client-id,
_R2_ACCOUNT_ID=your-staging-r2-account,
_CORS_ORIGIN=https://staging.xtablo.com,
_XTABLO_URL=https://staging.xtablo.com,
_LOG_LEVEL=debug,
_SUPABASE_SERVICE_ROLE_KEY_SECRET=supabase-service-role-key-staging,
_SUPABASE_CONNECTION_STRING_SECRET=supabase-connection-string-staging,
_SUPABASE_CA_CERT_SECRET=supabase-ca-cert-staging,
_STREAM_CHAT_API_SECRET_SECRET=stream-chat-api-secret-staging,
_STRIPE_SECRET_KEY_SECRET=stripe-secret-key-staging,
_STRIPE_WEBHOOK_SECRET_SECRET=stripe-webhook-secret-staging,
_EMAIL_CLIENT_SECRET_SECRET=email-client-secret-staging,
_EMAIL_REFRESH_TOKEN_SECRET=email-refresh-token-staging,
_R2_ACCESS_KEY_ID_SECRET=r2-access-key-id-staging,
_R2_SECRET_ACCESS_KEY_SECRET=r2-secret-access-key-staging
'
```
### Step 4: Configure via Console (Alternative)
If you prefer using the Google Cloud Console:
1. Go to **Cloud Build > Triggers**
2. Click **Create Trigger**
3. Configure:
- **Name**: `xtablo-api-production` (or `xtablo-api-staging`)
- **Event**: Push to a branch
- **Repository**: Your GitHub repository
- **Branch**: `^main$` (or `^develop$` for staging)
- **Build configuration**: Cloud Build configuration file
- **Location**: `apps/api/cloudbuild.yaml`
4. Scroll to **Substitution variables** and add all variables from the table above
5. Save the trigger
## Verification
### Test Production Deployment
```bash
# Trigger manually
gcloud builds triggers run xtablo-api-production --branch=main
# Check logs
gcloud builds list --limit=5
gcloud builds log BUILD_ID
# Verify deployment
gcloud run services describe xtablo-api --region=europe-west1
```
### Test Staging Deployment
```bash
# Trigger manually
gcloud builds triggers run xtablo-api-staging --branch=develop
# Check logs
gcloud builds list --limit=5
# Verify deployment
gcloud run services describe xtablo-api-staging --region=europe-west1
```
### Verify Environment Variables
```bash
# Production
gcloud run services describe xtablo-api --region=europe-west1 \
--format="value(spec.template.spec.containers[0].env)"
# Staging
gcloud run services describe xtablo-api-staging --region=europe-west1 \
--format="value(spec.template.spec.containers[0].env)"
```
## Best Practices
1. **Secret Naming**: Use `-prod` and `-staging` suffixes for secrets
2. **Service Names**: Use different names for each environment (`xtablo-api` vs `xtablo-api-staging`)
3. **Branches**: Deploy production from `main`, staging from `develop`
4. **Stripe Keys**: Use Stripe test keys in staging
5. **Supabase**: Use separate Supabase projects for staging and production
6. **Monitoring**: Set up different alert thresholds for staging vs production
7. **Resource Limits**: You may want lower limits for staging to save costs
## Quick Reference
### Environment Comparison
| Aspect | Production | Staging |
| ------------------ | ---------------- | -------------------- |
| **Branch** | `main` | `develop` |
| **Service Name** | `xtablo-api` | `xtablo-api-staging` |
| **Domain** | `app.xtablo.com` | `staging.xtablo.com` |
| **Log Level** | `info` | `debug` |
| **Stripe** | Live keys | Test keys |
| **Min Instances** | 1 (optional) | 0 |
| **Secrets Suffix** | `-prod` | `-staging` |
## Troubleshooting
### "Secret not found" error
- Ensure secret names in substitution variables match actual secret names
- Verify secrets exist: `gcloud secrets list | grep staging` or `grep prod`
### Wrong environment variables in deployment
- Check trigger substitution variables in Cloud Build console
- Verify the correct trigger is being used for the branch
### Services pointing to wrong databases
- Ensure `SUPABASE_URL` is different for staging/production
- Check that secret names include correct environment suffix
## Example: Adding a New Environment Variable
To add a new environment variable (e.g., `FEATURE_FLAG`):
1. Add to `cloudbuild.yaml` in the `--set-env-vars` line:
```yaml
"NODE_ENV=$_NODE_ENV,...,FEATURE_FLAG=$_FEATURE_FLAG"
```
2. Add to **both triggers** with different values:
- Production: `_FEATURE_FLAG=false`
- Staging: `_FEATURE_FLAG=true`
3. Update app code to read `process.env.FEATURE_FLAG`

268
docs/CLOUD_BUILD_SETUP.md Normal file
View file

@ -0,0 +1,268 @@
# Google Cloud Build Setup Guide
## Overview
This guide explains how to configure Google Cloud Build for automatic deployment of the XTablo API to Cloud Run.
## Prerequisites
1. Google Cloud Project with billing enabled
2. Cloud Build API enabled
3. Cloud Run API enabled
4. Secret Manager API enabled
5. Artifact Registry repository created
## Required Substitution Variables
The `cloudbuild.yaml` uses substitution variables that must be configured in your Cloud Build trigger. Here's what each variable is for:
### Build Configuration Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `$_NODE_ENV` | Environment (staging/production) | `production` |
| `$_AR_PROJECT_ID` | Artifact Registry project ID | `your-project-id` |
| `$_AR_REPOSITORY` | Artifact Registry repository name | `xtablo` |
| `$_SERVICE_NAME` | Cloud Run service name | `xtablo-api` |
### Application Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `$_SUPABASE_URL` | Supabase project URL | `https://xxx.supabase.co` |
| `$_STREAM_CHAT_API_KEY` | Stream Chat API key | `your-stream-api-key` |
| `$_EMAIL_USER` | Email sender address | `noreply@xtablo.com` |
| `$_EMAIL_CLIENT_ID` | OAuth2 client ID for email | `your-client-id` |
| `$_R2_ACCOUNT_ID` | Cloudflare R2 account ID | `your-r2-account-id` |
| `$_CORS_ORIGIN` | CORS allowed origin | `https://app.xtablo.com` |
| `$_XTABLO_URL` | Frontend application URL | `https://app.xtablo.com` |
## Setting Up Substitution Variables
### Option 1: Via Google Cloud Console
1. Go to Cloud Build > Triggers
2. Select your trigger (or create a new one)
3. Scroll to "Substitution variables"
4. Add each variable with its value:
```
_NODE_ENV = production
_AR_PROJECT_ID = your-project-id
_AR_REPOSITORY = xtablo
_SERVICE_NAME = xtablo-api
_SUPABASE_URL = https://your-project.supabase.co
_STREAM_CHAT_API_KEY = your-key
_EMAIL_USER = noreply@xtablo.com
_EMAIL_CLIENT_ID = your-client-id
_R2_ACCOUNT_ID = your-account-id
_CORS_ORIGIN = https://app.xtablo.com
_XTABLO_URL = https://app.xtablo.com
```
### Option 2: Via gcloud CLI
```bash
gcloud builds triggers create github \
--name="xtablo-api-production" \
--repo-name="xtablo-source" \
--repo-owner="your-github-org" \
--branch-pattern="^main$" \
--build-config="apps/api/cloudbuild.yaml" \
--substitutions='
_NODE_ENV=production,
_AR_PROJECT_ID=your-project-id,
_AR_REPOSITORY=xtablo,
_SERVICE_NAME=xtablo-api,
_SUPABASE_URL=https://your-project.supabase.co,
_STREAM_CHAT_API_KEY=your-key,
_EMAIL_USER=noreply@xtablo.com,
_EMAIL_CLIENT_ID=your-client-id,
_R2_ACCOUNT_ID=your-account-id,
_CORS_ORIGIN=https://app.xtablo.com,
_XTABLO_URL=https://app.xtablo.com
'
```
## Setting Up Secrets in Secret Manager
The sensitive values (API keys, tokens, etc.) are stored in Google Cloud Secret Manager. Create these secrets:
### Required Secrets
```bash
# Supabase secrets
echo -n "your-service-role-key" | gcloud secrets create supabase-service-role-key --data-file=-
echo -n "your-connection-string" | gcloud secrets create supabase-connection-string --data-file=-
echo -n "your-ca-cert" | gcloud secrets create supabase-ca-cert --data-file=-
# Stream Chat secret
echo -n "your-stream-secret" | gcloud secrets create stream-chat-api-secret --data-file=-
# Stripe secrets
echo -n "your-stripe-key" | gcloud secrets create stripe-secret-key --data-file=-
echo -n "your-webhook-secret" | gcloud secrets create stripe-webhook-secret --data-file=-
# Email secrets
echo -n "your-client-secret" | gcloud secrets create email-client-secret --data-file=-
echo -n "your-refresh-token" | gcloud secrets create email-refresh-token --data-file=-
# R2 (Cloudflare) secrets
echo -n "your-access-key-id" | gcloud secrets create r2-access-key-id --data-file=-
echo -n "your-secret-access-key" | gcloud secrets create r2-secret-access-key --data-file=-
```
### Grant Cloud Run Access to Secrets
```bash
# Get your Cloud Run service account
PROJECT_ID="your-project-id"
SERVICE_ACCOUNT="${PROJECT_ID}@appspot.gserviceaccount.com"
# Grant access to each secret
for secret in \
supabase-service-role-key \
supabase-connection-string \
supabase-ca-cert \
stream-chat-api-secret \
stripe-secret-key \
stripe-webhook-secret \
email-client-secret \
email-refresh-token \
r2-access-key-id \
r2-secret-access-key
do
gcloud secrets add-iam-policy-binding $secret \
--member="serviceAccount:${SERVICE_ACCOUNT}" \
--role="roles/secretmanager.secretAccessor"
done
```
## Environment-Specific Configurations
### Production Trigger
```yaml
_NODE_ENV: production
_CORS_ORIGIN: https://app.xtablo.com
_XTABLO_URL: https://app.xtablo.com
```
### Staging Trigger
```yaml
_NODE_ENV: staging
_CORS_ORIGIN: https://staging.xtablo.com
_XTABLO_URL: https://staging.xtablo.com
_SERVICE_NAME: xtablo-api-staging
```
## Verifying the Setup
After creating your trigger and setting up the secrets:
1. **Test the trigger manually:**
```bash
gcloud builds triggers run xtablo-api-production --branch=main
```
2. **Check the build logs:**
```bash
gcloud builds list --limit=5
gcloud builds log BUILD_ID
```
3. **Verify the deployed service:**
```bash
gcloud run services describe xtablo-api --region=europe-west1
```
4. **Check environment variables:**
```bash
gcloud run services describe xtablo-api --region=europe-west1 --format="value(spec.template.spec.containers[0].env)"
```
## Troubleshooting
### Build Fails with "Missing Substitution Variable"
- Ensure all `$_VARIABLE` names are defined in your trigger
- Check for typos in variable names
### Cloud Run Deployment Fails
- Verify the service account has necessary permissions
- Check that the Artifact Registry URL is correct
- Ensure the image was successfully pushed
### Application Fails to Start
- Check Cloud Run logs: `gcloud run logs read --service=xtablo-api --region=europe-west1`
- Verify all secrets are accessible to the service account
- Check that environment variables are properly set
### Secret Access Denied
- Ensure the Cloud Run service account has `roles/secretmanager.secretAccessor` for each secret
- Verify secrets exist: `gcloud secrets list`
## Additional Configuration
### Service Account Permissions
Your Cloud Run service needs these roles:
```bash
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:${SERVICE_ACCOUNT}" \
--role="roles/secretmanager.secretAccessor"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:${SERVICE_ACCOUNT}" \
--role="roles/cloudsql.client" # If using Cloud SQL
```
### Cloud Run Configuration Options
You can add these to your `cloudbuild.yaml` deploy step:
```yaml
- '--memory'
- '512Mi'
- '--cpu'
- '1'
- '--max-instances'
- '10'
- '--min-instances'
- '0'
- '--concurrency'
- '80'
- '--timeout'
- '300'
- '--allow-unauthenticated' # If your API is public
```
## Manual Deployment
If you need to deploy manually without Cloud Build:
```bash
# Build the image
docker build -f apps/api/Dockerfile --target production -t xtablo-api:latest .
# Tag for Artifact Registry
docker tag xtablo-api:latest europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest
# Push to Artifact Registry
docker push europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest
# Deploy to Cloud Run
gcloud run deploy xtablo-api \
--image=europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest \
--region=europe-west1 \
--set-env-vars="NODE_ENV=production,PORT=8080,..." \
--update-secrets="SUPABASE_SERVICE_ROLE_KEY=supabase-service-role-key:latest,..."
```
## Support
For more information:
- [Cloud Build Documentation](https://cloud.google.com/build/docs)
- [Cloud Run Documentation](https://cloud.google.com/run/docs)
- [Secret Manager Documentation](https://cloud.google.com/secret-manager/docs)

163
docs/DOCKER_BUILD.md Normal file
View file

@ -0,0 +1,163 @@
# Docker Build Guide for XTablo API
## Overview
This Dockerfile is configured for a **pnpm monorepo** setup. It must be built from the repository root, not from the `apps/api` directory.
## Build Commands
### Production Build (default)
```bash
cd /path/to/xtablo-source
docker build -f apps/api/Dockerfile -t xtablo-api:production .
```
### Staging Build
```bash
cd /path/to/xtablo-source
docker build -f apps/api/Dockerfile --target staging -t xtablo-api:staging .
```
### Build Specific Stages (for testing)
```bash
# Test dependencies stage
docker build -f apps/api/Dockerfile --target deps -t xtablo-api:deps .
# Test build stage
docker build -f apps/api/Dockerfile --target build -t xtablo-api:build .
# Test production dependencies
docker build -f apps/api/Dockerfile --target prod-deps -t xtablo-api:prod-deps .
```
## Running the Container
### Basic Run
```bash
docker run -p 8080:8080 \
-e SUPABASE_URL=your_url \
-e SUPABASE_SERVICE_ROLE_KEY=your_key \
-e STREAM_API_KEY=your_key \
-e STREAM_SECRET=your_secret \
xtablo-api:production
```
### With Environment File
```bash
docker run -p 8080:8080 --env-file .env.production xtablo-api:production
```
## Docker Compose Example
```yaml
version: "3.8"
services:
api:
build:
context: .
dockerfile: apps/api/Dockerfile
target: production
ports:
- "8080:8080"
environment:
- NODE_ENV=production
- SUPABASE_URL=${SUPABASE_URL}
- SUPABASE_SERVICE_ROLE_KEY=${SUPABASE_SERVICE_ROLE_KEY}
- STREAM_API_KEY=${STREAM_API_KEY}
- STREAM_SECRET=${STREAM_SECRET}
restart: unless-stopped
```
## Image Structure
- **Base Image**: `node:18-alpine`
- **Package Manager**: pnpm (via corepack)
- **Working Directory**: `/app/apps/api`
- **User**: `nodejs` (non-root, UID 1001)
- **Port**: 8080
## Multi-Stage Build Details
1. **base**: Sets up Node.js, pnpm, copies workspace files
2. **deps**: Installs all dependencies (including dev dependencies)
3. **build**: Compiles TypeScript to JavaScript
4. **prod-deps**: Installs only production dependencies
5. **staging**: Creates staging image with production dependencies
6. **production**: Creates production image (default target)
## Important Notes
- ⚠️ **Always build from the monorepo root**, not from `apps/api`
- Environment variables should be injected at runtime, not baked into the image
- The image uses pnpm's workspace features with proper symlink structure
- pnpm version: 10.19.0 (via corepack)
- Node version: 18.20.8
- The `.dockerignore` file excludes local `dist` and `node_modules` to ensure clean builds
## Verification Commands
### Check image size
```bash
docker images xtablo-api:production
```
### Inspect image
```bash
docker run --rm --entrypoint sh xtablo-api:production -c "ls -la /app/apps/api"
```
### Test pnpm installation
```bash
docker run --rm --entrypoint sh xtablo-api:production -c "pnpm --version"
```
### Check built files
```bash
docker run --rm --entrypoint sh xtablo-api:production -c "ls -la /app/apps/api/dist"
```
## Troubleshooting
### "pnpm-lock.yaml: not found"
- Make sure you're building from the repository root
- Use `-f apps/api/Dockerfile` to specify the Dockerfile location
### "Permission denied"
- The container runs as a non-root user (`nodejs`)
- Ensure file permissions are set correctly (handled by the Dockerfile)
### Dependencies not found
- The monorepo structure requires the entire workspace for dependency resolution
- Make sure `pnpm-workspace.yaml` is present in the root
## CI/CD Integration
For Cloud Build or similar CI/CD systems, update your build configuration:
```yaml
steps:
- name: "gcr.io/cloud-builders/docker"
args:
- "build"
- "-f"
- "apps/api/Dockerfile"
- "-t"
- "gcr.io/$PROJECT_ID/xtablo-api:$COMMIT_SHA"
- "-t"
- "gcr.io/$PROJECT_ID/xtablo-api:latest"
- "."
dir: "." # Build from root, not apps/api
```

145
docs/DOCKER_FIX_SUMMARY.md Normal file
View file

@ -0,0 +1,145 @@
# Docker Build Fix Summary
## Issue
When deploying to Google Cloud, the application failed with:
```
Error: Cannot find module '/app/apps/api/dist/index.js'
```
## Root Causes
### 1. Missing `.dockerignore`
- Local `dist/` folders were being copied into the Docker build context
- This caused nested `dist/dist/` structures and incorrect file locations
### 2. TypeScript Compilation Structure
- `tsconfig.json` was missing `rootDir: "./src"`
- TypeScript compiled `src/index.ts` to `dist/src/index.js` instead of `dist/index.js`
- The start script expected `dist/index.js`
### 3. pnpm Symlink Issues
- Docker's `COPY` command was not preserving pnpm's symlink structure
- `node_modules` were copied but symlinks to packages in `.pnpm/` were lost
- Node.js couldn't resolve packages like `@hono/node-server`
## Solutions Implemented
### 1. Created `.dockerignore` at Repository Root
```
**/dist
**/node_modules
**/__tests__
# ... and other build artifacts
```
This ensures Docker doesn't copy local build artifacts into the image.
### 2. Updated `apps/api/tsconfig.json`
Added:
```json
{
"compilerOptions": {
"rootDir": "./src",
// ...
},
"include": ["src/**/*"]
}
```
This ensures TypeScript compiles `src/index.ts``dist/index.js` (not `dist/src/index.js`).
### 3. Fixed Dockerfile Dependency Installation
Changed from copying `node_modules` to installing them fresh in the final stage:
**Before:**
```dockerfile
COPY --from=prod-deps /app/node_modules ./node_modules
```
**After:**
```dockerfile
# Copy built files and workspace structure
COPY --from=build /app/apps/api/dist ./apps/api/dist
COPY --from=build /app/apps/api/package.json ./apps/api/package.json
COPY --from=prod-deps /app/packages ./packages
# Install dependencies with proper symlinks
RUN pnpm install --frozen-lockfile --prod --filter @xtablo/api...
```
This ensures pnpm creates proper symlinks in `/app/apps/api/node_modules/` that point to packages in `/app/node_modules/.pnpm/`.
### 4. Updated `cloudbuild.yaml`
Changed build command to run from monorepo root:
**Before:**
```yaml
args: [ 'build', '--target', '$_NODE_ENV', '-t', '...', 'apps/api' ]
```
**After:**
```yaml
args: [ 'build', '-f', 'apps/api/Dockerfile', '--target', '$_NODE_ENV', '-t', '...', '.' ]
```
## Verification
### Module Resolution Test
```bash
docker run --rm --entrypoint sh xtablo-api:production -c \
"ls -la /app/apps/api/node_modules/@hono/"
```
Shows proper symlinks:
```
lrwxrwxrwx node-server -> ../../../../node_modules/.pnpm/@hono+node-server@...
```
### Application Start Test
```bash
docker run --rm -e SUPABASE_URL=test -e SUPABASE_SERVICE_ROLE_KEY=test \
xtablo-api:production
```
Application starts successfully (fails only on missing GCP credentials, not module resolution).
## Final Image Details
- **Size**: 1GB (production)
- **Node.js**: 18.20.8
- **pnpm**: 10.19.0
- **User**: nodejs (non-root, UID 1001)
- **Working Directory**: `/app/apps/api`
- **Module Structure**: Proper pnpm workspace with symlinks
## Files Modified
1. `/apps/api/Dockerfile` - Updated multi-stage build
2. `/apps/api/tsconfig.json` - Added `rootDir` and `include`
3. `/apps/api/cloudbuild.yaml` - Fixed build context
4. `/.dockerignore` - Created to exclude build artifacts
5. `/apps/api/DOCKER_BUILD.md` - Added comprehensive documentation
## Key Takeaways
1. **Always use `.dockerignore`** to prevent local artifacts from contaminating Docker builds
2. **pnpm symlinks require special handling** - install dependencies in the final stage rather than copying
3. **TypeScript `rootDir` matters** - set it explicitly to control output structure
4. **Monorepo builds must run from root** - use `-f` flag to specify Dockerfile location
## Testing in Production
The image is now ready for deployment. All module resolution issues are fixed. Make sure to provide proper environment variables at runtime:
```bash
docker run -p 8080:8080 \
-e SUPABASE_URL=... \
-e SUPABASE_SERVICE_ROLE_KEY=... \
-e STREAM_API_KEY=... \
-e STREAM_SECRET=... \
# ... other env vars
xtablo-api:production
```
## Date
Fixed: November 13, 2024