Workflows

Real-world scenarios and step-by-step guides for common txpark workflows.

Quick Testing Workflow

Test a feature branch quickly with automatic cleanup.

Scenario

You're developing a new feature and want to quickly test it on a real testnet that automatically cleans itself up after testing.

Steps

1. Generate sequencer configuration (first time only):

   ./scripts/setup-sequencer.sh quick-test
   

2. Deploy testnet from your feature branch:

   txpark deploy quick-test --sequencer --build-branch feature/my-feature --lifetime 2h
   

3. Monitor deployment:

   # Watch status until all pods are ready
   txpark status quick-test --watch
   

4. View logs to verify functionality:

   # Follow EVM sequencer logs
   txpark logs quick-test evm-sequencer --follow
   

5. Test your feature:

   # Access the testnet through its endpoints
   curl https://quick-test.txpark.nomadic-labs.com/rpc \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
   

6. Automatic cleanup:

7. Testnet automatically deletes after 2 hours
8. Or manually destroy: txpark destroy quick-test

Expected Time

• Setup: 5 minutes (first time)
• Deployment: 10-15 minutes
• Testing: Variable
• Cleanup: Automatic

---

Development Workflow

Iterative development with sequential testnet deployments.

Scenario

You're actively developing and want to test multiple iterations of your code throughout the day.

Steps

1. List current testnets:

   txpark list --owner $USER
   

2. Deploy first iteration:

   txpark deploy dev-iter1 --sequencer --build-branch feature/iteration-1 --lifetime 4h
   

3. After making changes, deploy next iteration:

   # Previous testnet still running for comparison
   txpark deploy dev-iter2 --sequencer --build-branch feature/iteration-2 --lifetime 4h
   

4. Compare behavior between iterations:

   # Terminal 1: Watch iter1 logs
   txpark logs dev-iter1 evm-sequencer --follow
   
   # Terminal 2: Watch iter2 logs
   txpark logs dev-iter2 evm-sequencer --follow
   

5. Clean up old iterations:

   # Manual cleanup of older testnets
   txpark destroy dev-iter1
   

Tips

• Use consistent naming: dev-iter1, dev-iter2, etc.
• Keep lifetime at 4h to prevent accumulation
• List frequently to track active testnets
• Compare endpoints side-by-side in browser

---

Load Testing Workflow

Stress test with high-performance resources.

Scenario

You need to perform load testing to validate performance under heavy traffic.

Steps

1. Deploy high-performance testnet:

   ./scripts/setup-sequencer.sh load-test
   
   txpark deploy load-test \
     --sequencer \
     --build-branch master \
     --profile high-performance \
     --lifetime 8h
   

2. Verify resource allocation:

   txpark shell load-test evm-sequencer
   # Inside pod:
   df -h  # Check storage
   free -h  # Check memory
   

3. Run load tests:

   # Example with k6 or your preferred tool
   k6 run --vus 100 --duration 1h \
     --env RPC_URL=https://load-test.txpark.nomadic-labs.com/rpc \
     load-test-script.js
   

4. Monitor during testing:

   # Terminal 1: Watch pod status
   txpark status load-test --watch
   
   # Terminal 2: Follow logs
   txpark logs load-test evm-sequencer --follow
   
   # Terminal 3: Check resource usage
   kubectl top pods -n testnet-load-test
   

5. Collect metrics:

   # Export results
   txpark logs load-test evm-sequencer --timestamps > load-test-results.log
   
   # Check Blockscout for on-chain metrics
   open https://load-test-blockscout.txpark.nomadic-labs.com/
   

6. Clean up:

   txpark destroy load-test
   

Key Considerations

• Use high-performance profile for adequate resources
• Set longer lifetime (8h) to complete tests
• Monitor CPU and memory usage throughout
• Save logs and metrics for analysis

---

Long-Running Staging Environment

Create a permanent staging environment for extended testing.

Scenario

You need a stable staging environment that persists across days for integration testing.

Steps

1. Deploy permanent testnet:

   ./scripts/setup-sequencer.sh staging
   
   txpark deploy staging \
     --sequencer \
     --build-branch stable-branch \
     --profile high-performance \
     --permanent
   

2. Verify deployment:

   txpark status staging
   

3. Share endpoints with team:

   Dashboard: https://staging.txpark.nomadic-labs.com/
   RPC: https://staging.txpark.nomadic-labs.com/rpc
   Explorer: https://staging-blockscout.txpark.nomadic-labs.com/
   

4. Periodic upgrades:

   # When stable-branch is updated
   txpark upgrade staging --build-branch stable-branch
   
   # Monitor upgrade
   txpark logs staging evm-sequencer --follow
   

5. Regular health checks:

   # Add to cron or CI
   txpark status staging | grep "STATUS.*Running" || alert-team
   

6. When no longer needed:

   # Permanent testnets require manual cleanup
   txpark destroy staging
   

Maintenance Schedule

• Daily: Check status, review logs
• Weekly: Verify all endpoints accessible
• Monthly: Consider upgrading to latest stable
• As needed: Destroy and redeploy if issues arise

---

Debugging Failed Deployments

Troubleshoot deployment issues systematically.

Scenario

Your testnet deployment failed or pods aren't becoming ready.

Steps

1. Check overall status:

   txpark status my-testnet
   

2. Identify failing pods: Look for pods with status other than "Running" or "0/N" ready count.

3. Check pod events:

   kubectl get events -n testnet-my-testnet --sort-by='.lastTimestamp'
   

4. View logs from failing component:

   # If EVM sequencer is failing
   txpark logs my-testnet evm-sequencer --tail 200
   

5. Check init containers (if pod is stuck in Init state):

   # Check kernel builder logs
   txpark logs my-testnet kernel-builder
   
   # Check rollup originator logs
   txpark logs my-testnet rollup-originator
   

6. Inspect pod directly:

   kubectl describe pod -n testnet-my-testnet -l component=evm-sequencer
   

7. Common issues and fixes:

Pipeline failure:

# Check GitLab pipeline status
# Visit: https://gitlab.com/tezos/tezos/-/pipelines

# Redeploy with working branch
txpark destroy my-testnet
txpark deploy my-testnet --sequencer --build-branch master

Missing configuration:

# Verify kernel_setup.yml exists
ls -la output/kernel_setup.yml

# Regenerate if missing
./scripts/setup-sequencer.sh my-testnet

Resource constraints:

# Check node resources
kubectl top nodes

# Try default profile instead
txpark destroy my-testnet
txpark deploy my-testnet --sequencer --build-branch master --profile default

1. Interactive debugging:

   # Shell into pod (if running)
   txpark shell my-testnet evm-sequencer
   
   # Check binary versions
   /octez-binaries/octez-evm-node --version
   
   # Check network connectivity
   curl http://l1-node:8732/chains/main/blocks/head
   

---

Managing Multiple Testnets

Efficiently manage several concurrent testnets.

Scenario

You're working on multiple features and need to track several active testnets.

Steps

1. Naming convention:

   # Use descriptive names
   txpark deploy feature-authentication --sequencer --build-branch feat/auth
   txpark deploy feature-payments --sequencer --build-branch feat/payments
   txpark deploy bugfix-memory-leak --sequencer --build-branch fix/memory
   

2. List and filter:

   # View all your testnets
   txpark list --owner $USER
   
   # Sort by expiration to see what needs attention
   txpark list --sort-by expiration
   
   # Find expired testnets
   txpark list --expired
   

3. Batch status checks:

   # Create a monitoring script
   cat > check-testnets.sh <<'EOF'
   #!/bin/bash
   for testnet in feature-authentication feature-payments bugfix-memory-leak; do
     echo "=== $testnet ==="
     txpark status $testnet | head -n 10
     echo
   done
   EOF
   chmod +x check-testnets.sh
   ./check-testnets.sh
   

4. Batch cleanup:

   # Clean up old testnets
   txpark list --output json | \
     jq -r '.[] | select(.Owner == env.USER and .Status == "Expired") | .Name' | \
     xargs -I {} txpark destroy {} --force
   

5. Document active testnets:

   # Generate markdown table
   echo "# Active Testnets"
   echo "| Name | Purpose | Expires | URL |"
   echo "|------|---------|---------|-----|"
   txpark list --owner $USER --output json | \
     jq -r '.[] | "| \(.Name) | \(.Purpose) | \(.ExpiresAt) | https://\(.Name).txpark.nomadic-labs.com/ |"'
   

Organization Tips

• Naming: Use <type>-<description> format (feature-, bugfix-, test-)
• Lifetime: Keep default 1h for experiments, 4-8h for active work
• Cleanup: Run daily cleanup of expired testnets
• Documentation: Maintain a shared document with active testnets

---

CI/CD Integration

Automate testnet deployment in CI pipelines.

Scenario

You want to automatically deploy testnets for testing in your CI/CD pipeline.

Example GitLab CI Configuration

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy-testnet
  - integration-test
  - cleanup

variables:
  TESTNET_NAME: "ci-$CI_PIPELINE_ID"
  TESTNET_LIFETIME: "2h"

deploy_testnet:
  stage: deploy-testnet
  image: google/cloud-sdk:alpine
  before_script:
    - gcloud auth activate-service-account --key-file=$GCP_SERVICE_ACCOUNT_KEY
    - gcloud container clusters get-credentials txpark-cluster --region europe-west1
    - curl -LO https://gitlab.com/tezos-infra/evm/txpark/-/releases/permalink/latest/downloads/txpark-linux-amd64
    - chmod +x txpark-linux-amd64
    - mv txpark-linux-amd64 /usr/local/bin/txpark
  script:
    - ./scripts/setup-sequencer.sh $TESTNET_NAME
    - txpark deploy $TESTNET_NAME --sequencer --build-branch $CI_COMMIT_REF_NAME --lifetime $TESTNET_LIFETIME
    - txpark status $TESTNET_NAME
  artifacts:
    reports:
      dotenv: testnet.env
  environment:
    name: testnet-$CI_PIPELINE_ID
    on_stop: cleanup_testnet

integration_tests:
  stage: integration-test
  dependencies:
    - deploy_testnet
  script:
    - export RPC_URL="https://${TESTNET_NAME}.txpark.nomadic-labs.com/rpc"
    - npm run integration-tests
  artifacts:
    reports:
      junit: test-results.xml

cleanup_testnet:
  stage: cleanup
  image: google/cloud-sdk:alpine
  when: always
  script:
    - txpark destroy $TESTNET_NAME --force
  environment:
    name: testnet-$CI_PIPELINE_ID
    action: stop

GitHub Actions Example

# .github/workflows/integration-test.yml
name: Integration Tests

on:
  pull_request:
    branches: [main]

jobs:
  integration-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup GCloud
        uses: google-github-actions/setup-gcloud@v1
        with:
          service_account_key: ${{ secrets.GCP_SA_KEY }}
          project_id: nl-tezos-x-alpha-infra

      - name: Configure kubectl
        run: |
          gcloud container clusters get-credentials txpark-cluster \
            --region europe-west1

      - name: Install txpark
        run: |
          curl -LO https://gitlab.com/tezos-infra/evm/txpark/-/releases/permalink/latest/downloads/txpark-linux-amd64
          chmod +x txpark-linux-amd64
          sudo mv txpark-linux-amd64 /usr/local/bin/txpark

      - name: Deploy testnet
        id: deploy
        env:
          TESTNET_NAME: "gh-${{ github.run_id }}"
        run: |
          ./scripts/setup-sequencer.sh $TESTNET_NAME
          txpark deploy $TESTNET_NAME \
            --sequencer \
            --build-branch ${{ github.head_ref }} \
            --lifetime 2h
          echo "testnet_name=$TESTNET_NAME" >> $GITHUB_OUTPUT

      - name: Run integration tests
        env:
          RPC_URL: "https://${{ steps.deploy.outputs.testnet_name }}.txpark.nomadic-labs.com/rpc"
        run: npm run integration-tests

      - name: Cleanup
        if: always()
        run: |
          txpark destroy ${{ steps.deploy.outputs.testnet_name }} --force

Best Practices for CI Integration

1. Use unique names: Include pipeline/run ID in testnet name
2. Set appropriate lifetime: 2-4h to cover test duration with buffer
3. Always cleanup: Use always() or when: always to ensure cleanup
4. Capture logs on failure: Save testnet logs as artifacts
5. Environment variables: Export testnet URLs for test access
6. Parallel testing: Deploy separate testnets for parallel jobs

---

Emergency Procedures

Handle critical situations with testnets.

Testnet Consuming Too Many Resources

1. Identify the testnet:

   kubectl top pods --all-namespaces | grep testnet | sort -k3 -h
   

2. Check what's running:

   txpark list --sort-by age
   

3. Immediate action:

   # Destroy resource-heavy testnet
   txpark destroy problematic-testnet --force
   

Mass Cleanup Required

1. List all testnets:

   txpark list --output json > testnets-backup.json
   

2. Destroy expired testnets:

   txpark list --expired --output json | \
     jq -r '.[].Name' | \
     xargs -I {} txpark destroy {} --force
   

3. Destroy old testnets (older than 7 days):

   txpark list --older-than 7d --output json | \
     jq -r '.[].Name' | \
     xargs -I {} txpark destroy {} --force
   

Stuck Namespace Deletion

If a testnet namespace won't delete:

1. Check for finalizers:

   kubectl get namespace testnet-stuck -o yaml
   

2. Remove finalizers:

   kubectl patch namespace testnet-stuck -p '{"metadata":{"finalizers":[]}}' --type=merge
   

3. Force delete:

   kubectl delete namespace testnet-stuck --force --grace-period=0
   

---

Next Steps

• Review Best Practices for optimization tips
• Check Commands for detailed command reference
• See Development to contribute improvements