Add docs
This commit is contained in:
parent
803c9ff391
commit
247bc8b3af
4 changed files with 931 additions and 0 deletions
355
docs/CLOUD_BUILD_ENV_CONFIG.md
Normal file
355
docs/CLOUD_BUILD_ENV_CONFIG.md
Normal file
|
|
@ -0,0 +1,355 @@
|
|||
# Cloud Build Environment Configuration
|
||||
|
||||
This guide explains how to set up separate staging and production triggers with environment-specific configurations.
|
||||
|
||||
## Overview
|
||||
|
||||
The `cloudbuild.yaml` uses substitution variables to make it environment-agnostic. You'll create **two separate triggers** (one for staging, one for production) with different substitution variable values.
|
||||
|
||||
**Note**: The Dockerfile uses a single final stage. Environment differentiation (staging vs production) is handled entirely through Cloud Run environment variables, not at build time. This simplifies the build process - the same Docker image can be used for both environments.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────────┐
|
||||
│ Git Branch │ │ Cloud Build Trigger │
|
||||
│ - main │─────▶│ (Production) │
|
||||
│ │ │ Variables: prod │
|
||||
└─────────────────┘ └──────────────────────┘
|
||||
|
||||
┌─────────────────┐ ┌──────────────────────┐
|
||||
│ Git Branch │ │ Cloud Build Trigger │
|
||||
│ - develop │─────▶│ (Staging) │
|
||||
│ │ │ Variables: staging │
|
||||
└─────────────────┘ └──────────────────────┘
|
||||
```
|
||||
|
||||
## Required Substitution Variables
|
||||
|
||||
### Common Variables (Different per Environment)
|
||||
|
||||
| Variable | Production Value | Staging Value |
|
||||
| ----------------------- | ------------------------ | ---------------------------- |
|
||||
| `$_NODE_ENV` | `production` | `staging` |
|
||||
| `$_SERVICE_NAME` | `xtablo-api` | `xtablo-api-staging` |
|
||||
| `$_SUPABASE_URL` | Production Supabase URL | Staging Supabase URL |
|
||||
| `$_STREAM_CHAT_API_KEY` | Production key | Staging key |
|
||||
| `$_EMAIL_USER` | `noreply@xtablo.com` | `staging@xtablo.com` |
|
||||
| `$_EMAIL_CLIENT_ID` | Production OAuth client | Staging OAuth client |
|
||||
| `$_R2_ACCOUNT_ID` | Production R2 account | Staging R2 account |
|
||||
| `$_CORS_ORIGIN` | `https://app.xtablo.com` | `https://staging.xtablo.com` |
|
||||
| `$_XTABLO_URL` | `https://app.xtablo.com` | `https://staging.xtablo.com` |
|
||||
| `$_LOG_LEVEL` | `info` | `debug` |
|
||||
|
||||
### Build Configuration (Same for Both)
|
||||
|
||||
| Variable | Value |
|
||||
| ----------------- | ---------------------------------- |
|
||||
| `$_AR_PROJECT_ID` | Your GCP project ID |
|
||||
| `$_AR_REPOSITORY` | `xtablo` (or your repository name) |
|
||||
|
||||
### Secret Manager References (Different per Environment)
|
||||
|
||||
Each environment should have its own secrets in Secret Manager:
|
||||
|
||||
| Variable | Production Secret Name | Staging Secret Name |
|
||||
| ------------------------------------- | --------------------------------- | ------------------------------------ |
|
||||
| `$_SUPABASE_SERVICE_ROLE_KEY_SECRET` | `supabase-service-role-key-prod` | `supabase-service-role-key-staging` |
|
||||
| `$_SUPABASE_CONNECTION_STRING_SECRET` | `supabase-connection-string-prod` | `supabase-connection-string-staging` |
|
||||
| `$_SUPABASE_CA_CERT_SECRET` | `supabase-ca-cert-prod` | `supabase-ca-cert-staging` |
|
||||
| `$_STREAM_CHAT_API_SECRET_SECRET` | `stream-chat-api-secret-prod` | `stream-chat-api-secret-staging` |
|
||||
| `$_STRIPE_SECRET_KEY_SECRET` | `stripe-secret-key-prod` | `stripe-secret-key-staging` |
|
||||
| `$_STRIPE_WEBHOOK_SECRET_SECRET` | `stripe-webhook-secret-prod` | `stripe-webhook-secret-staging` |
|
||||
| `$_EMAIL_CLIENT_SECRET_SECRET` | `email-client-secret-prod` | `email-client-secret-staging` |
|
||||
| `$_EMAIL_REFRESH_TOKEN_SECRET` | `email-refresh-token-prod` | `email-refresh-token-staging` |
|
||||
| `$_R2_ACCESS_KEY_ID_SECRET` | `r2-access-key-id-prod` | `r2-access-key-id-staging` |
|
||||
| `$_R2_SECRET_ACCESS_KEY_SECRET` | `r2-secret-access-key-prod` | `r2-secret-access-key-staging` |
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### Step 1: Create Secrets in Secret Manager
|
||||
|
||||
#### Production Secrets
|
||||
|
||||
```bash
|
||||
# Supabase (Production)
|
||||
echo -n "prod-service-role-key" | gcloud secrets create supabase-service-role-key-prod --data-file=-
|
||||
echo -n "prod-connection-string" | gcloud secrets create supabase-connection-string-prod --data-file=-
|
||||
echo -n "prod-ca-cert" | gcloud secrets create supabase-ca-cert-prod --data-file=-
|
||||
|
||||
# Stream Chat (Production)
|
||||
echo -n "prod-stream-secret" | gcloud secrets create stream-chat-api-secret-prod --data-file=-
|
||||
|
||||
# Stripe (Production)
|
||||
echo -n "prod-stripe-key" | gcloud secrets create stripe-secret-key-prod --data-file=-
|
||||
echo -n "prod-webhook-secret" | gcloud secrets create stripe-webhook-secret-prod --data-file=-
|
||||
|
||||
# Email (Production)
|
||||
echo -n "prod-client-secret" | gcloud secrets create email-client-secret-prod --data-file=-
|
||||
echo -n "prod-refresh-token" | gcloud secrets create email-refresh-token-prod --data-file=-
|
||||
|
||||
# R2 (Production)
|
||||
echo -n "prod-access-key-id" | gcloud secrets create r2-access-key-id-prod --data-file=-
|
||||
echo -n "prod-secret-access-key" | gcloud secrets create r2-secret-access-key-prod --data-file=-
|
||||
```
|
||||
|
||||
#### Staging Secrets
|
||||
|
||||
```bash
|
||||
# Supabase (Staging)
|
||||
echo -n "staging-service-role-key" | gcloud secrets create supabase-service-role-key-staging --data-file=-
|
||||
echo -n "staging-connection-string" | gcloud secrets create supabase-connection-string-staging --data-file=-
|
||||
echo -n "staging-ca-cert" | gcloud secrets create supabase-ca-cert-staging --data-file=-
|
||||
|
||||
# Stream Chat (Staging)
|
||||
echo -n "staging-stream-secret" | gcloud secrets create stream-chat-api-secret-staging --data-file=-
|
||||
|
||||
# Stripe (Staging - use test keys)
|
||||
echo -n "staging-stripe-key" | gcloud secrets create stripe-secret-key-staging --data-file=-
|
||||
echo -n "staging-webhook-secret" | gcloud secrets create stripe-webhook-secret-staging --data-file=-
|
||||
|
||||
# Email (Staging)
|
||||
echo -n "staging-client-secret" | gcloud secrets create email-client-secret-staging --data-file=-
|
||||
echo -n "staging-refresh-token" | gcloud secrets create email-refresh-token-staging --data-file=-
|
||||
|
||||
# R2 (Staging)
|
||||
echo -n "staging-access-key-id" | gcloud secrets create r2-access-key-id-staging --data-file=-
|
||||
echo -n "staging-secret-access-key" | gcloud secrets create r2-secret-access-key-staging --data-file=-
|
||||
```
|
||||
|
||||
### Step 2: Grant Access to Secrets
|
||||
|
||||
```bash
|
||||
PROJECT_ID="your-project-id"
|
||||
SERVICE_ACCOUNT="${PROJECT_ID}@appspot.gserviceaccount.com"
|
||||
|
||||
# Grant access to all production secrets
|
||||
for secret in \
|
||||
supabase-service-role-key-prod \
|
||||
supabase-connection-string-prod \
|
||||
supabase-ca-cert-prod \
|
||||
stream-chat-api-secret-prod \
|
||||
stripe-secret-key-prod \
|
||||
stripe-webhook-secret-prod \
|
||||
email-client-secret-prod \
|
||||
email-refresh-token-prod \
|
||||
r2-access-key-id-prod \
|
||||
r2-secret-access-key-prod
|
||||
do
|
||||
gcloud secrets add-iam-policy-binding $secret \
|
||||
--member="serviceAccount:${SERVICE_ACCOUNT}" \
|
||||
--role="roles/secretmanager.secretAccessor"
|
||||
done
|
||||
|
||||
# Grant access to all staging secrets
|
||||
for secret in \
|
||||
supabase-service-role-key-staging \
|
||||
supabase-connection-string-staging \
|
||||
supabase-ca-cert-staging \
|
||||
stream-chat-api-secret-staging \
|
||||
stripe-secret-key-staging \
|
||||
stripe-webhook-secret-staging \
|
||||
email-client-secret-staging \
|
||||
email-refresh-token-staging \
|
||||
r2-access-key-id-staging \
|
||||
r2-secret-access-key-staging
|
||||
do
|
||||
gcloud secrets add-iam-policy-binding $secret \
|
||||
--member="serviceAccount:${SERVICE_ACCOUNT}" \
|
||||
--role="roles/secretmanager.secretAccessor"
|
||||
done
|
||||
```
|
||||
|
||||
### Step 3: Create Cloud Build Triggers
|
||||
|
||||
#### Production Trigger
|
||||
|
||||
Via gcloud CLI:
|
||||
|
||||
```bash
|
||||
gcloud builds triggers create github \
|
||||
--name="xtablo-api-production" \
|
||||
--repo-name="xtablo-source" \
|
||||
--repo-owner="your-github-org" \
|
||||
--branch-pattern="^main$" \
|
||||
--build-config="apps/api/cloudbuild.yaml" \
|
||||
--substitutions='
|
||||
_NODE_ENV=production,
|
||||
_SERVICE_NAME=xtablo-api,
|
||||
_AR_PROJECT_ID=your-project-id,
|
||||
_AR_REPOSITORY=xtablo,
|
||||
_SUPABASE_URL=https://your-prod-project.supabase.co,
|
||||
_STREAM_CHAT_API_KEY=your-prod-stream-key,
|
||||
_EMAIL_USER=noreply@xtablo.com,
|
||||
_EMAIL_CLIENT_ID=your-prod-oauth-client-id,
|
||||
_R2_ACCOUNT_ID=your-prod-r2-account,
|
||||
_CORS_ORIGIN=https://app.xtablo.com,
|
||||
_XTABLO_URL=https://app.xtablo.com,
|
||||
_LOG_LEVEL=info,
|
||||
_SUPABASE_SERVICE_ROLE_KEY_SECRET=supabase-service-role-key-prod,
|
||||
_SUPABASE_CONNECTION_STRING_SECRET=supabase-connection-string-prod,
|
||||
_SUPABASE_CA_CERT_SECRET=supabase-ca-cert-prod,
|
||||
_STREAM_CHAT_API_SECRET_SECRET=stream-chat-api-secret-prod,
|
||||
_STRIPE_SECRET_KEY_SECRET=stripe-secret-key-prod,
|
||||
_STRIPE_WEBHOOK_SECRET_SECRET=stripe-webhook-secret-prod,
|
||||
_EMAIL_CLIENT_SECRET_SECRET=email-client-secret-prod,
|
||||
_EMAIL_REFRESH_TOKEN_SECRET=email-refresh-token-prod,
|
||||
_R2_ACCESS_KEY_ID_SECRET=r2-access-key-id-prod,
|
||||
_R2_SECRET_ACCESS_KEY_SECRET=r2-secret-access-key-prod
|
||||
'
|
||||
```
|
||||
|
||||
#### Staging Trigger
|
||||
|
||||
Via gcloud CLI:
|
||||
|
||||
```bash
|
||||
gcloud builds triggers create github \
|
||||
--name="xtablo-api-staging" \
|
||||
--repo-name="xtablo-source" \
|
||||
--repo-owner="your-github-org" \
|
||||
--branch-pattern="^develop$" \
|
||||
--build-config="apps/api/cloudbuild.yaml" \
|
||||
--substitutions='
|
||||
_NODE_ENV=staging,
|
||||
_SERVICE_NAME=xtablo-api-staging,
|
||||
_AR_PROJECT_ID=your-project-id,
|
||||
_AR_REPOSITORY=xtablo,
|
||||
_SUPABASE_URL=https://your-staging-project.supabase.co,
|
||||
_STREAM_CHAT_API_KEY=your-staging-stream-key,
|
||||
_EMAIL_USER=staging@xtablo.com,
|
||||
_EMAIL_CLIENT_ID=your-staging-oauth-client-id,
|
||||
_R2_ACCOUNT_ID=your-staging-r2-account,
|
||||
_CORS_ORIGIN=https://staging.xtablo.com,
|
||||
_XTABLO_URL=https://staging.xtablo.com,
|
||||
_LOG_LEVEL=debug,
|
||||
_SUPABASE_SERVICE_ROLE_KEY_SECRET=supabase-service-role-key-staging,
|
||||
_SUPABASE_CONNECTION_STRING_SECRET=supabase-connection-string-staging,
|
||||
_SUPABASE_CA_CERT_SECRET=supabase-ca-cert-staging,
|
||||
_STREAM_CHAT_API_SECRET_SECRET=stream-chat-api-secret-staging,
|
||||
_STRIPE_SECRET_KEY_SECRET=stripe-secret-key-staging,
|
||||
_STRIPE_WEBHOOK_SECRET_SECRET=stripe-webhook-secret-staging,
|
||||
_EMAIL_CLIENT_SECRET_SECRET=email-client-secret-staging,
|
||||
_EMAIL_REFRESH_TOKEN_SECRET=email-refresh-token-staging,
|
||||
_R2_ACCESS_KEY_ID_SECRET=r2-access-key-id-staging,
|
||||
_R2_SECRET_ACCESS_KEY_SECRET=r2-secret-access-key-staging
|
||||
'
|
||||
```
|
||||
|
||||
### Step 4: Configure via Console (Alternative)
|
||||
|
||||
If you prefer using the Google Cloud Console:
|
||||
|
||||
1. Go to **Cloud Build > Triggers**
|
||||
2. Click **Create Trigger**
|
||||
3. Configure:
|
||||
|
||||
- **Name**: `xtablo-api-production` (or `xtablo-api-staging`)
|
||||
- **Event**: Push to a branch
|
||||
- **Repository**: Your GitHub repository
|
||||
- **Branch**: `^main$` (or `^develop$` for staging)
|
||||
- **Build configuration**: Cloud Build configuration file
|
||||
- **Location**: `apps/api/cloudbuild.yaml`
|
||||
|
||||
4. Scroll to **Substitution variables** and add all variables from the table above
|
||||
|
||||
5. Save the trigger
|
||||
|
||||
## Verification
|
||||
|
||||
### Test Production Deployment
|
||||
|
||||
```bash
|
||||
# Trigger manually
|
||||
gcloud builds triggers run xtablo-api-production --branch=main
|
||||
|
||||
# Check logs
|
||||
gcloud builds list --limit=5
|
||||
gcloud builds log BUILD_ID
|
||||
|
||||
# Verify deployment
|
||||
gcloud run services describe xtablo-api --region=europe-west1
|
||||
```
|
||||
|
||||
### Test Staging Deployment
|
||||
|
||||
```bash
|
||||
# Trigger manually
|
||||
gcloud builds triggers run xtablo-api-staging --branch=develop
|
||||
|
||||
# Check logs
|
||||
gcloud builds list --limit=5
|
||||
|
||||
# Verify deployment
|
||||
gcloud run services describe xtablo-api-staging --region=europe-west1
|
||||
```
|
||||
|
||||
### Verify Environment Variables
|
||||
|
||||
```bash
|
||||
# Production
|
||||
gcloud run services describe xtablo-api --region=europe-west1 \
|
||||
--format="value(spec.template.spec.containers[0].env)"
|
||||
|
||||
# Staging
|
||||
gcloud run services describe xtablo-api-staging --region=europe-west1 \
|
||||
--format="value(spec.template.spec.containers[0].env)"
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Secret Naming**: Use `-prod` and `-staging` suffixes for secrets
|
||||
2. **Service Names**: Use different names for each environment (`xtablo-api` vs `xtablo-api-staging`)
|
||||
3. **Branches**: Deploy production from `main`, staging from `develop`
|
||||
4. **Stripe Keys**: Use Stripe test keys in staging
|
||||
5. **Supabase**: Use separate Supabase projects for staging and production
|
||||
6. **Monitoring**: Set up different alert thresholds for staging vs production
|
||||
7. **Resource Limits**: You may want lower limits for staging to save costs
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Environment Comparison
|
||||
|
||||
| Aspect | Production | Staging |
|
||||
| ------------------ | ---------------- | -------------------- |
|
||||
| **Branch** | `main` | `develop` |
|
||||
| **Service Name** | `xtablo-api` | `xtablo-api-staging` |
|
||||
| **Domain** | `app.xtablo.com` | `staging.xtablo.com` |
|
||||
| **Log Level** | `info` | `debug` |
|
||||
| **Stripe** | Live keys | Test keys |
|
||||
| **Min Instances** | 1 (optional) | 0 |
|
||||
| **Secrets Suffix** | `-prod` | `-staging` |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Secret not found" error
|
||||
|
||||
- Ensure secret names in substitution variables match actual secret names
|
||||
- Verify secrets exist: `gcloud secrets list | grep staging` or `grep prod`
|
||||
|
||||
### Wrong environment variables in deployment
|
||||
|
||||
- Check trigger substitution variables in Cloud Build console
|
||||
- Verify the correct trigger is being used for the branch
|
||||
|
||||
### Services pointing to wrong databases
|
||||
|
||||
- Ensure `SUPABASE_URL` is different for staging/production
|
||||
- Check that secret names include correct environment suffix
|
||||
|
||||
## Example: Adding a New Environment Variable
|
||||
|
||||
To add a new environment variable (e.g., `FEATURE_FLAG`):
|
||||
|
||||
1. Add to `cloudbuild.yaml` in the `--set-env-vars` line:
|
||||
|
||||
```yaml
|
||||
"NODE_ENV=$_NODE_ENV,...,FEATURE_FLAG=$_FEATURE_FLAG"
|
||||
```
|
||||
|
||||
2. Add to **both triggers** with different values:
|
||||
|
||||
- Production: `_FEATURE_FLAG=false`
|
||||
- Staging: `_FEATURE_FLAG=true`
|
||||
|
||||
3. Update app code to read `process.env.FEATURE_FLAG`
|
||||
268
docs/CLOUD_BUILD_SETUP.md
Normal file
268
docs/CLOUD_BUILD_SETUP.md
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
# Google Cloud Build Setup Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide explains how to configure Google Cloud Build for automatic deployment of the XTablo API to Cloud Run.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Google Cloud Project with billing enabled
|
||||
2. Cloud Build API enabled
|
||||
3. Cloud Run API enabled
|
||||
4. Secret Manager API enabled
|
||||
5. Artifact Registry repository created
|
||||
|
||||
## Required Substitution Variables
|
||||
|
||||
The `cloudbuild.yaml` uses substitution variables that must be configured in your Cloud Build trigger. Here's what each variable is for:
|
||||
|
||||
### Build Configuration Variables
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `$_NODE_ENV` | Environment (staging/production) | `production` |
|
||||
| `$_AR_PROJECT_ID` | Artifact Registry project ID | `your-project-id` |
|
||||
| `$_AR_REPOSITORY` | Artifact Registry repository name | `xtablo` |
|
||||
| `$_SERVICE_NAME` | Cloud Run service name | `xtablo-api` |
|
||||
|
||||
### Application Environment Variables
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `$_SUPABASE_URL` | Supabase project URL | `https://xxx.supabase.co` |
|
||||
| `$_STREAM_CHAT_API_KEY` | Stream Chat API key | `your-stream-api-key` |
|
||||
| `$_EMAIL_USER` | Email sender address | `noreply@xtablo.com` |
|
||||
| `$_EMAIL_CLIENT_ID` | OAuth2 client ID for email | `your-client-id` |
|
||||
| `$_R2_ACCOUNT_ID` | Cloudflare R2 account ID | `your-r2-account-id` |
|
||||
| `$_CORS_ORIGIN` | CORS allowed origin | `https://app.xtablo.com` |
|
||||
| `$_XTABLO_URL` | Frontend application URL | `https://app.xtablo.com` |
|
||||
|
||||
## Setting Up Substitution Variables
|
||||
|
||||
### Option 1: Via Google Cloud Console
|
||||
|
||||
1. Go to Cloud Build > Triggers
|
||||
2. Select your trigger (or create a new one)
|
||||
3. Scroll to "Substitution variables"
|
||||
4. Add each variable with its value:
|
||||
```
|
||||
_NODE_ENV = production
|
||||
_AR_PROJECT_ID = your-project-id
|
||||
_AR_REPOSITORY = xtablo
|
||||
_SERVICE_NAME = xtablo-api
|
||||
_SUPABASE_URL = https://your-project.supabase.co
|
||||
_STREAM_CHAT_API_KEY = your-key
|
||||
_EMAIL_USER = noreply@xtablo.com
|
||||
_EMAIL_CLIENT_ID = your-client-id
|
||||
_R2_ACCOUNT_ID = your-account-id
|
||||
_CORS_ORIGIN = https://app.xtablo.com
|
||||
_XTABLO_URL = https://app.xtablo.com
|
||||
```
|
||||
|
||||
### Option 2: Via gcloud CLI
|
||||
|
||||
```bash
|
||||
gcloud builds triggers create github \
|
||||
--name="xtablo-api-production" \
|
||||
--repo-name="xtablo-source" \
|
||||
--repo-owner="your-github-org" \
|
||||
--branch-pattern="^main$" \
|
||||
--build-config="apps/api/cloudbuild.yaml" \
|
||||
--substitutions='
|
||||
_NODE_ENV=production,
|
||||
_AR_PROJECT_ID=your-project-id,
|
||||
_AR_REPOSITORY=xtablo,
|
||||
_SERVICE_NAME=xtablo-api,
|
||||
_SUPABASE_URL=https://your-project.supabase.co,
|
||||
_STREAM_CHAT_API_KEY=your-key,
|
||||
_EMAIL_USER=noreply@xtablo.com,
|
||||
_EMAIL_CLIENT_ID=your-client-id,
|
||||
_R2_ACCOUNT_ID=your-account-id,
|
||||
_CORS_ORIGIN=https://app.xtablo.com,
|
||||
_XTABLO_URL=https://app.xtablo.com
|
||||
'
|
||||
```
|
||||
|
||||
## Setting Up Secrets in Secret Manager
|
||||
|
||||
The sensitive values (API keys, tokens, etc.) are stored in Google Cloud Secret Manager. Create these secrets:
|
||||
|
||||
### Required Secrets
|
||||
|
||||
```bash
|
||||
# Supabase secrets
|
||||
echo -n "your-service-role-key" | gcloud secrets create supabase-service-role-key --data-file=-
|
||||
echo -n "your-connection-string" | gcloud secrets create supabase-connection-string --data-file=-
|
||||
echo -n "your-ca-cert" | gcloud secrets create supabase-ca-cert --data-file=-
|
||||
|
||||
# Stream Chat secret
|
||||
echo -n "your-stream-secret" | gcloud secrets create stream-chat-api-secret --data-file=-
|
||||
|
||||
# Stripe secrets
|
||||
echo -n "your-stripe-key" | gcloud secrets create stripe-secret-key --data-file=-
|
||||
echo -n "your-webhook-secret" | gcloud secrets create stripe-webhook-secret --data-file=-
|
||||
|
||||
# Email secrets
|
||||
echo -n "your-client-secret" | gcloud secrets create email-client-secret --data-file=-
|
||||
echo -n "your-refresh-token" | gcloud secrets create email-refresh-token --data-file=-
|
||||
|
||||
# R2 (Cloudflare) secrets
|
||||
echo -n "your-access-key-id" | gcloud secrets create r2-access-key-id --data-file=-
|
||||
echo -n "your-secret-access-key" | gcloud secrets create r2-secret-access-key --data-file=-
|
||||
```
|
||||
|
||||
### Grant Cloud Run Access to Secrets
|
||||
|
||||
```bash
|
||||
# Get your Cloud Run service account
|
||||
PROJECT_ID="your-project-id"
|
||||
SERVICE_ACCOUNT="${PROJECT_ID}@appspot.gserviceaccount.com"
|
||||
|
||||
# Grant access to each secret
|
||||
for secret in \
|
||||
supabase-service-role-key \
|
||||
supabase-connection-string \
|
||||
supabase-ca-cert \
|
||||
stream-chat-api-secret \
|
||||
stripe-secret-key \
|
||||
stripe-webhook-secret \
|
||||
email-client-secret \
|
||||
email-refresh-token \
|
||||
r2-access-key-id \
|
||||
r2-secret-access-key
|
||||
do
|
||||
gcloud secrets add-iam-policy-binding $secret \
|
||||
--member="serviceAccount:${SERVICE_ACCOUNT}" \
|
||||
--role="roles/secretmanager.secretAccessor"
|
||||
done
|
||||
```
|
||||
|
||||
## Environment-Specific Configurations
|
||||
|
||||
### Production Trigger
|
||||
|
||||
```yaml
|
||||
_NODE_ENV: production
|
||||
_CORS_ORIGIN: https://app.xtablo.com
|
||||
_XTABLO_URL: https://app.xtablo.com
|
||||
```
|
||||
|
||||
### Staging Trigger
|
||||
|
||||
```yaml
|
||||
_NODE_ENV: staging
|
||||
_CORS_ORIGIN: https://staging.xtablo.com
|
||||
_XTABLO_URL: https://staging.xtablo.com
|
||||
_SERVICE_NAME: xtablo-api-staging
|
||||
```
|
||||
|
||||
## Verifying the Setup
|
||||
|
||||
After creating your trigger and setting up the secrets:
|
||||
|
||||
1. **Test the trigger manually:**
|
||||
```bash
|
||||
gcloud builds triggers run xtablo-api-production --branch=main
|
||||
```
|
||||
|
||||
2. **Check the build logs:**
|
||||
```bash
|
||||
gcloud builds list --limit=5
|
||||
gcloud builds log BUILD_ID
|
||||
```
|
||||
|
||||
3. **Verify the deployed service:**
|
||||
```bash
|
||||
gcloud run services describe xtablo-api --region=europe-west1
|
||||
```
|
||||
|
||||
4. **Check environment variables:**
|
||||
```bash
|
||||
gcloud run services describe xtablo-api --region=europe-west1 --format="value(spec.template.spec.containers[0].env)"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Build Fails with "Missing Substitution Variable"
|
||||
- Ensure all `$_VARIABLE` names are defined in your trigger
|
||||
- Check for typos in variable names
|
||||
|
||||
### Cloud Run Deployment Fails
|
||||
- Verify the service account has necessary permissions
|
||||
- Check that the Artifact Registry URL is correct
|
||||
- Ensure the image was successfully pushed
|
||||
|
||||
### Application Fails to Start
|
||||
- Check Cloud Run logs: `gcloud run logs read --service=xtablo-api --region=europe-west1`
|
||||
- Verify all secrets are accessible to the service account
|
||||
- Check that environment variables are properly set
|
||||
|
||||
### Secret Access Denied
|
||||
- Ensure the Cloud Run service account has `roles/secretmanager.secretAccessor` for each secret
|
||||
- Verify secrets exist: `gcloud secrets list`
|
||||
|
||||
## Additional Configuration
|
||||
|
||||
### Service Account Permissions
|
||||
|
||||
Your Cloud Run service needs these roles:
|
||||
|
||||
```bash
|
||||
gcloud projects add-iam-policy-binding $PROJECT_ID \
|
||||
--member="serviceAccount:${SERVICE_ACCOUNT}" \
|
||||
--role="roles/secretmanager.secretAccessor"
|
||||
|
||||
gcloud projects add-iam-policy-binding $PROJECT_ID \
|
||||
--member="serviceAccount:${SERVICE_ACCOUNT}" \
|
||||
--role="roles/cloudsql.client" # If using Cloud SQL
|
||||
```
|
||||
|
||||
### Cloud Run Configuration Options
|
||||
|
||||
You can add these to your `cloudbuild.yaml` deploy step:
|
||||
|
||||
```yaml
|
||||
- '--memory'
|
||||
- '512Mi'
|
||||
- '--cpu'
|
||||
- '1'
|
||||
- '--max-instances'
|
||||
- '10'
|
||||
- '--min-instances'
|
||||
- '0'
|
||||
- '--concurrency'
|
||||
- '80'
|
||||
- '--timeout'
|
||||
- '300'
|
||||
- '--allow-unauthenticated' # If your API is public
|
||||
```
|
||||
|
||||
## Manual Deployment
|
||||
|
||||
If you need to deploy manually without Cloud Build:
|
||||
|
||||
```bash
|
||||
# Build the image
|
||||
docker build -f apps/api/Dockerfile --target production -t xtablo-api:latest .
|
||||
|
||||
# Tag for Artifact Registry
|
||||
docker tag xtablo-api:latest europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest
|
||||
|
||||
# Push to Artifact Registry
|
||||
docker push europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest
|
||||
|
||||
# Deploy to Cloud Run
|
||||
gcloud run deploy xtablo-api \
|
||||
--image=europe-west1-docker.pkg.dev/PROJECT_ID/xtablo/xtablo-api:latest \
|
||||
--region=europe-west1 \
|
||||
--set-env-vars="NODE_ENV=production,PORT=8080,..." \
|
||||
--update-secrets="SUPABASE_SERVICE_ROLE_KEY=supabase-service-role-key:latest,..."
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For more information:
|
||||
- [Cloud Build Documentation](https://cloud.google.com/build/docs)
|
||||
- [Cloud Run Documentation](https://cloud.google.com/run/docs)
|
||||
- [Secret Manager Documentation](https://cloud.google.com/secret-manager/docs)
|
||||
|
||||
163
docs/DOCKER_BUILD.md
Normal file
163
docs/DOCKER_BUILD.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
# Docker Build Guide for XTablo API
|
||||
|
||||
## Overview
|
||||
|
||||
This Dockerfile is configured for a **pnpm monorepo** setup. It must be built from the repository root, not from the `apps/api` directory.
|
||||
|
||||
## Build Commands
|
||||
|
||||
### Production Build (default)
|
||||
|
||||
```bash
|
||||
cd /path/to/xtablo-source
|
||||
docker build -f apps/api/Dockerfile -t xtablo-api:production .
|
||||
```
|
||||
|
||||
### Staging Build
|
||||
|
||||
```bash
|
||||
cd /path/to/xtablo-source
|
||||
docker build -f apps/api/Dockerfile --target staging -t xtablo-api:staging .
|
||||
```
|
||||
|
||||
### Build Specific Stages (for testing)
|
||||
|
||||
```bash
|
||||
# Test dependencies stage
|
||||
docker build -f apps/api/Dockerfile --target deps -t xtablo-api:deps .
|
||||
|
||||
# Test build stage
|
||||
docker build -f apps/api/Dockerfile --target build -t xtablo-api:build .
|
||||
|
||||
# Test production dependencies
|
||||
docker build -f apps/api/Dockerfile --target prod-deps -t xtablo-api:prod-deps .
|
||||
```
|
||||
|
||||
## Running the Container
|
||||
|
||||
### Basic Run
|
||||
|
||||
```bash
|
||||
docker run -p 8080:8080 \
|
||||
-e SUPABASE_URL=your_url \
|
||||
-e SUPABASE_SERVICE_ROLE_KEY=your_key \
|
||||
-e STREAM_API_KEY=your_key \
|
||||
-e STREAM_SECRET=your_secret \
|
||||
xtablo-api:production
|
||||
```
|
||||
|
||||
### With Environment File
|
||||
|
||||
```bash
|
||||
docker run -p 8080:8080 --env-file .env.production xtablo-api:production
|
||||
```
|
||||
|
||||
## Docker Compose Example
|
||||
|
||||
```yaml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: apps/api/Dockerfile
|
||||
target: production
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
- SUPABASE_URL=${SUPABASE_URL}
|
||||
- SUPABASE_SERVICE_ROLE_KEY=${SUPABASE_SERVICE_ROLE_KEY}
|
||||
- STREAM_API_KEY=${STREAM_API_KEY}
|
||||
- STREAM_SECRET=${STREAM_SECRET}
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
## Image Structure
|
||||
|
||||
- **Base Image**: `node:18-alpine`
|
||||
- **Package Manager**: pnpm (via corepack)
|
||||
- **Working Directory**: `/app/apps/api`
|
||||
- **User**: `nodejs` (non-root, UID 1001)
|
||||
- **Port**: 8080
|
||||
|
||||
## Multi-Stage Build Details
|
||||
|
||||
1. **base**: Sets up Node.js, pnpm, copies workspace files
|
||||
2. **deps**: Installs all dependencies (including dev dependencies)
|
||||
3. **build**: Compiles TypeScript to JavaScript
|
||||
4. **prod-deps**: Installs only production dependencies
|
||||
5. **staging**: Creates staging image with production dependencies
|
||||
6. **production**: Creates production image (default target)
|
||||
|
||||
## Important Notes
|
||||
|
||||
- ⚠️ **Always build from the monorepo root**, not from `apps/api`
|
||||
- Environment variables should be injected at runtime, not baked into the image
|
||||
- The image uses pnpm's workspace features with proper symlink structure
|
||||
- pnpm version: 10.19.0 (via corepack)
|
||||
- Node version: 18.20.8
|
||||
- The `.dockerignore` file excludes local `dist` and `node_modules` to ensure clean builds
|
||||
|
||||
## Verification Commands
|
||||
|
||||
### Check image size
|
||||
|
||||
```bash
|
||||
docker images xtablo-api:production
|
||||
```
|
||||
|
||||
### Inspect image
|
||||
|
||||
```bash
|
||||
docker run --rm --entrypoint sh xtablo-api:production -c "ls -la /app/apps/api"
|
||||
```
|
||||
|
||||
### Test pnpm installation
|
||||
|
||||
```bash
|
||||
docker run --rm --entrypoint sh xtablo-api:production -c "pnpm --version"
|
||||
```
|
||||
|
||||
### Check built files
|
||||
|
||||
```bash
|
||||
docker run --rm --entrypoint sh xtablo-api:production -c "ls -la /app/apps/api/dist"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "pnpm-lock.yaml: not found"
|
||||
|
||||
- Make sure you're building from the repository root
|
||||
- Use `-f apps/api/Dockerfile` to specify the Dockerfile location
|
||||
|
||||
### "Permission denied"
|
||||
|
||||
- The container runs as a non-root user (`nodejs`)
|
||||
- Ensure file permissions are set correctly (handled by the Dockerfile)
|
||||
|
||||
### Dependencies not found
|
||||
|
||||
- The monorepo structure requires the entire workspace for dependency resolution
|
||||
- Make sure `pnpm-workspace.yaml` is present in the root
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
For Cloud Build or similar CI/CD systems, update your build configuration:
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
- name: "gcr.io/cloud-builders/docker"
|
||||
args:
|
||||
- "build"
|
||||
- "-f"
|
||||
- "apps/api/Dockerfile"
|
||||
- "-t"
|
||||
- "gcr.io/$PROJECT_ID/xtablo-api:$COMMIT_SHA"
|
||||
- "-t"
|
||||
- "gcr.io/$PROJECT_ID/xtablo-api:latest"
|
||||
- "."
|
||||
dir: "." # Build from root, not apps/api
|
||||
```
|
||||
145
docs/DOCKER_FIX_SUMMARY.md
Normal file
145
docs/DOCKER_FIX_SUMMARY.md
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
# Docker Build Fix Summary
|
||||
|
||||
## Issue
|
||||
When deploying to Google Cloud, the application failed with:
|
||||
```
|
||||
Error: Cannot find module '/app/apps/api/dist/index.js'
|
||||
```
|
||||
|
||||
## Root Causes
|
||||
|
||||
### 1. Missing `.dockerignore`
|
||||
- Local `dist/` folders were being copied into the Docker build context
|
||||
- This caused nested `dist/dist/` structures and incorrect file locations
|
||||
|
||||
### 2. TypeScript Compilation Structure
|
||||
- `tsconfig.json` was missing `rootDir: "./src"`
|
||||
- TypeScript compiled `src/index.ts` to `dist/src/index.js` instead of `dist/index.js`
|
||||
- The start script expected `dist/index.js`
|
||||
|
||||
### 3. pnpm Symlink Issues
|
||||
- Docker's `COPY` command was not preserving pnpm's symlink structure
|
||||
- `node_modules` were copied but symlinks to packages in `.pnpm/` were lost
|
||||
- Node.js couldn't resolve packages like `@hono/node-server`
|
||||
|
||||
## Solutions Implemented
|
||||
|
||||
### 1. Created `.dockerignore` at Repository Root
|
||||
```
|
||||
**/dist
|
||||
**/node_modules
|
||||
**/__tests__
|
||||
# ... and other build artifacts
|
||||
```
|
||||
|
||||
This ensures Docker doesn't copy local build artifacts into the image.
|
||||
|
||||
### 2. Updated `apps/api/tsconfig.json`
|
||||
Added:
|
||||
```json
|
||||
{
|
||||
"compilerOptions": {
|
||||
"rootDir": "./src",
|
||||
// ...
|
||||
},
|
||||
"include": ["src/**/*"]
|
||||
}
|
||||
```
|
||||
|
||||
This ensures TypeScript compiles `src/index.ts` → `dist/index.js` (not `dist/src/index.js`).
|
||||
|
||||
### 3. Fixed Dockerfile Dependency Installation
|
||||
Changed from copying `node_modules` to installing them fresh in the final stage:
|
||||
|
||||
**Before:**
|
||||
```dockerfile
|
||||
COPY --from=prod-deps /app/node_modules ./node_modules
|
||||
```
|
||||
|
||||
**After:**
|
||||
```dockerfile
|
||||
# Copy built files and workspace structure
|
||||
COPY --from=build /app/apps/api/dist ./apps/api/dist
|
||||
COPY --from=build /app/apps/api/package.json ./apps/api/package.json
|
||||
COPY --from=prod-deps /app/packages ./packages
|
||||
|
||||
# Install dependencies with proper symlinks
|
||||
RUN pnpm install --frozen-lockfile --prod --filter @xtablo/api...
|
||||
```
|
||||
|
||||
This ensures pnpm creates proper symlinks in `/app/apps/api/node_modules/` that point to packages in `/app/node_modules/.pnpm/`.
|
||||
|
||||
### 4. Updated `cloudbuild.yaml`
|
||||
Changed build command to run from monorepo root:
|
||||
|
||||
**Before:**
|
||||
```yaml
|
||||
args: [ 'build', '--target', '$_NODE_ENV', '-t', '...', 'apps/api' ]
|
||||
```
|
||||
|
||||
**After:**
|
||||
```yaml
|
||||
args: [ 'build', '-f', 'apps/api/Dockerfile', '--target', '$_NODE_ENV', '-t', '...', '.' ]
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
### Module Resolution Test
|
||||
```bash
|
||||
docker run --rm --entrypoint sh xtablo-api:production -c \
|
||||
"ls -la /app/apps/api/node_modules/@hono/"
|
||||
```
|
||||
|
||||
Shows proper symlinks:
|
||||
```
|
||||
lrwxrwxrwx node-server -> ../../../../node_modules/.pnpm/@hono+node-server@...
|
||||
```
|
||||
|
||||
### Application Start Test
|
||||
```bash
|
||||
docker run --rm -e SUPABASE_URL=test -e SUPABASE_SERVICE_ROLE_KEY=test \
|
||||
xtablo-api:production
|
||||
```
|
||||
|
||||
Application starts successfully (fails only on missing GCP credentials, not module resolution).
|
||||
|
||||
## Final Image Details
|
||||
- **Size**: 1GB (production)
|
||||
- **Node.js**: 18.20.8
|
||||
- **pnpm**: 10.19.0
|
||||
- **User**: nodejs (non-root, UID 1001)
|
||||
- **Working Directory**: `/app/apps/api`
|
||||
- **Module Structure**: Proper pnpm workspace with symlinks
|
||||
|
||||
## Files Modified
|
||||
|
||||
1. `/apps/api/Dockerfile` - Updated multi-stage build
|
||||
2. `/apps/api/tsconfig.json` - Added `rootDir` and `include`
|
||||
3. `/apps/api/cloudbuild.yaml` - Fixed build context
|
||||
4. `/.dockerignore` - Created to exclude build artifacts
|
||||
5. `/apps/api/DOCKER_BUILD.md` - Added comprehensive documentation
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. **Always use `.dockerignore`** to prevent local artifacts from contaminating Docker builds
|
||||
2. **pnpm symlinks require special handling** - install dependencies in the final stage rather than copying
|
||||
3. **TypeScript `rootDir` matters** - set it explicitly to control output structure
|
||||
4. **Monorepo builds must run from root** - use `-f` flag to specify Dockerfile location
|
||||
|
||||
## Testing in Production
|
||||
|
||||
The image is now ready for deployment. All module resolution issues are fixed. Make sure to provide proper environment variables at runtime:
|
||||
|
||||
```bash
|
||||
docker run -p 8080:8080 \
|
||||
-e SUPABASE_URL=... \
|
||||
-e SUPABASE_SERVICE_ROLE_KEY=... \
|
||||
-e STREAM_API_KEY=... \
|
||||
-e STREAM_SECRET=... \
|
||||
# ... other env vars
|
||||
xtablo-api:production
|
||||
```
|
||||
|
||||
## Date
|
||||
Fixed: November 13, 2024
|
||||
|
||||
Loading…
Reference in a new issue