Best Practices¶

Recommendations and guidelines for effective testnet management with txpark.

Naming Conventions¶

Choose clear, descriptive names for your testnets to improve organization and team collaboration.

Recommended Patterns¶

Feature development:

feature-<description>      # feature-authentication
feat-<ticket-id>           # feat-L2-1234

Bug fixes:

bugfix-<description>       # bugfix-memory-leak
fix-<ticket-id>            # fix-L2-5678

Testing:

test-<purpose>             # test-load, test-integration
qa-<version>               # qa-v1.2.3

Personal experiments:

<username>-<description>   # alice-experiment
dev-<purpose>              # dev-prototype

CI/CD:

ci-<pipeline-id>           # ci-123456
gh-<run-id>                # gh-987654321

Naming Rules¶

✅ Do: - Use lowercase letters, numbers, and hyphens only - Keep names under 30 characters for readability - Include context (purpose, owner, or ticket ID) - Use consistent patterns across your team

❌ Don't: - Use underscores (not Kubernetes-friendly) - Use generic names like "test" or "my-testnet" - Include sensitive information in names - Create names longer than 253 characters (K8s limit)

Examples¶

# Good names
txpark deploy feature-auth-improvements --lifetime 4h
txpark deploy alice-debugging --lifetime 2h
txpark deploy ci-$CI_PIPELINE_ID --lifetime 1h

# Poor names
txpark deploy test  # Too generic
txpark deploy my_testnet  # Underscores not allowed
txpark deploy AliceTestingAuthenticationFeature  # Too long, capitals

Lifetime Management¶

Choose appropriate lifetimes to balance resource usage and convenience.

Lifetime Guidelines¶

Use Case	Recommended Lifetime	Rationale
Quick testing	1-2h	Fast iteration, automatic cleanup
Active development	4-8h	Covers working day
Integration testing	8-12h	Overnight test runs
Staging/Demo	Permanent	Long-term stability
CI/CD	2h	Covers test execution + buffer
Load testing	8h	Extended test duration

Setting Lifetimes¶

Ephemeral testnets (auto-cleanup):

txpark deploy quick-test --lifetime 1h       # 1 hour
txpark deploy daily-dev --lifetime 8h        # 8 hours
txpark deploy integration --lifetime 1d      # 1 day

Permanent testnets (manual cleanup):

txpark deploy staging --permanent
txpark deploy demo --permanent

Monitoring Expiration¶

Check expiration times regularly:

# View all testnets sorted by expiration
txpark list --sort-by expiration

# Find expiring soon
txpark list --output json | \
  jq '.[] | select(.ExpiresAt != null and .ExpiresAt < (now + 3600))'

Cleanup Strategy¶

Daily cleanup:

# Remove expired testnets
txpark list --expired --output json | \
  jq -r '.[].Name' | \
  xargs -I {} txpark destroy {} --force

Weekly audit:

# Review all permanent testnets
txpark list --output json | \
  jq '.[] | select(.Permanent == true) | {Name, Owner, Age}'

Resource Optimization¶

Use appropriate resource profiles to balance performance and cost.

Profile Selection¶

Default profile (recommended for most use cases): - 8Gi data storage, 50Gi rollup storage - 2 CPU cores, 4Gi memory - Suitable for: Development, testing, CI/CD

High-performance profile (use sparingly): - 16Gi data storage, 100Gi rollup storage - 2 CPU cores, 16Gi memory - Suitable for: Load testing, staging, production-like environments

When to Use High-Performance¶

✅ Use high-performance for: - Load testing with significant traffic - Long-running staging environments - Stress testing edge cases - Demonstrations requiring high availability

❌ Don't use high-performance for: - Quick feature testing - CI/CD pipelines - Personal experiments - Short-lived testnets

Cost Considerations¶

# Default profile: Lower resource usage
txpark deploy dev-test --profile default --lifetime 4h

# High-performance: Use only when needed
txpark deploy load-test --profile high-performance --lifetime 8h

# Permanent testnets: Monitor resource usage
txpark deploy staging --permanent --profile high-performance

Resource Monitoring¶

Check actual resource usage:

# View resource consumption
kubectl top pods -n testnet-my-testnet

# Check storage usage
txpark shell my-testnet evm-sequencer
df -h | grep -E "data|rollup"

Security Considerations¶

Protect your testnets and sensitive data.

Secrets Management¶

Don't commit secrets:

# ❌ Never commit these files
output/*.env
output/kernel_setup.yml
.env files

Use GitLab CLI authentication:

# ✅ Authenticate with glab (recommended)
glab auth login

# Verify authentication
glab auth status

The glab CLI handles authentication securely and automatically refreshes tokens.

Network Security¶

Ingress is HTTPS-only: - All testnets use automatic SSL certificates - HTTP traffic is redirected to HTTPS - TLS certificates auto-renew via cert-manager

RPC endpoint protection:

# Consider rate limiting for public-facing testnets
# (Configured at infrastructure level)

Access Control¶

Kubernetes RBAC: - Ensure proper kubectl permissions - Use separate service accounts for CI/CD - Limit namespace access appropriately

Testnet isolation: - Each testnet runs in its own namespace - Resources are isolated via Kubernetes network policies - No cross-testnet communication by default

Monitoring and Observability¶

Track testnet health and debug issues effectively.

Status Monitoring¶

Regular checks:

# Quick status
txpark status my-testnet

# Continuous monitoring
txpark status my-testnet --watch

# Batch monitoring
for testnet in $(txpark list --owner $USER --output json | jq -r '.[].Name'); do
  echo "=== $testnet ==="
  txpark status $testnet | head -n 15
done

Log Management¶

Effective log viewing:

# Follow logs in real-time
txpark logs my-testnet evm-sequencer --follow

# Search for errors
txpark logs my-testnet evm-sequencer --tail 500 | grep -i error

# Export logs for analysis
txpark logs my-testnet evm-sequencer --timestamps --tail 1000 > testnet-logs.txt

Multi-component monitoring:

# Terminal 1: EVM sequencer
txpark logs my-testnet evm-sequencer --follow

# Terminal 2: Smart rollup node
txpark logs my-testnet smart-rollup-node --follow

# Terminal 3: L1 node
txpark logs my-testnet l1-node --follow

Health Checks¶

Automated health checks:

#!/bin/bash
# check-testnet-health.sh

TESTNET=$1

# Check pod status
if ! txpark status $TESTNET | grep -q "Running.*2/2"; then
  echo "ERROR: Pods not ready"
  exit 1
fi

# Check RPC endpoint
RPC_URL="https://${TESTNET}.txpark.nomadic-labs.com/rpc"
if ! curl -sf "$RPC_URL" -X POST \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' > /dev/null; then
  echo "ERROR: RPC endpoint not responding"
  exit 1
fi

echo "✓ Testnet healthy"

Alerting¶

Set up alerts for permanent testnets:

# Add to monitoring system (e.g., Prometheus, Datadog)
# Alert on:
# - Pod restarts > 3
# - Pod not ready > 10 minutes
# - High CPU/memory usage
# - RPC endpoint errors

Team Collaboration¶

Work effectively with shared testnet infrastructure.

Communication¶

Document active testnets:

# Active Testnets - Team Dev

| Name | Owner | Purpose | Expires | URL |
|------|-------|---------|---------|-----|
| staging | team | Staging environment | Never | [Link](https://staging.txpark.nomadic-labs.com) |
| feature-auth | alice | Authentication work | 2026-03-20 18:00 | [Link](https://feature-auth.txpark.nomadic-labs.com) |
| bugfix-leak | bob | Memory leak investigation | 2026-03-19 22:00 | [Link](https://bugfix-leak.txpark.nomadic-labs.com) |

Share testnet information:

# Generate shareable status
txpark status my-testnet | head -n 30 > testnet-info.txt

# Create team dashboard
txpark list --output json > team-testnets.json

Ownership¶

Clear ownership:

# Namespaces are tagged with owner
# Use --owner flag to filter your testnets
txpark list --owner alice

# Respect others' testnets
# Don't destroy testnets you don't own without asking

Shared Testnets¶

For permanent shared testnets:

# Use generic names without personal identifiers
txpark deploy staging --permanent  # ✅
txpark deploy alice-staging --permanent  # ❌

# Document in team wiki
# Assign a DRI (Directly Responsible Individual)
# Set up shared access credentials

Troubleshooting Tips¶

Common issues and solutions.

Deployment Failures¶

Pod stuck in Init state:

# Check init container logs
txpark logs my-testnet kernel-builder
txpark logs my-testnet rollup-originator

# Common causes:
# - Missing kernel_setup.yml
# - GitLab pipeline still running
# - Insufficient permissions

Pod CrashLoopBackOff:

# View recent logs
txpark logs my-testnet evm-sequencer --tail 200

# Check for:
# - Configuration errors
# - Binary compatibility issues
# - Resource constraints

ImagePullBackOff:

# Check image availability
kubectl describe pod -n testnet-my-testnet -l component=evm-sequencer

# Common causes:
# - GitLab pipeline failed
# - Invalid pipeline ID
# - Missing CI artifacts

Performance Issues¶

Slow RPC responses:

# Check resource usage
kubectl top pods -n testnet-my-testnet

# Consider upgrading to high-performance profile
txpark destroy slow-testnet
txpark deploy slow-testnet --profile high-performance

High restart count:

# Identify problematic pod
txpark status my-testnet | grep -i restart

# Check logs for crash reason
txpark logs my-testnet evm-sequencer --tail 500

# May indicate:
# - Insufficient memory (OOMKilled)
# - Application bugs
# - Network issues

Cleanup Issues¶

Namespace stuck in Terminating:

# Check for finalizers
kubectl get namespace testnet-stuck -o yaml

# Remove finalizers if safe
kubectl patch namespace testnet-stuck -p '{"metadata":{"finalizers":[]}}' --type=merge

# Force delete
kubectl delete namespace testnet-stuck --force --grace-period=0

PVC not releasing:

# Check PVC status
kubectl get pvc -n testnet-my-testnet

# Delete manually if needed
kubectl delete pvc -n testnet-my-testnet --all

CI/CD Best Practices¶

Optimize testnet usage in automated pipelines.

Pipeline Design¶

Efficient resource usage:

# Use default profile for CI
txpark deploy ci-$CI_PIPELINE_ID --profile default

# Set appropriate lifetime
txpark deploy ci-$CI_PIPELINE_ID --lifetime 2h

# Always cleanup
txpark destroy ci-$CI_PIPELINE_ID --force

Parallel testing:

# Deploy separate testnets for parallel jobs
job1:
  script:
    - txpark deploy ci-${CI_PIPELINE_ID}-job1

job2:
  script:
    - txpark deploy ci-${CI_PIPELINE_ID}-job2

Error Handling¶

Capture logs on failure:

after_script:
  - |
    if [ "$CI_JOB_STATUS" == "failed" ]; then
      txpark logs ci-$CI_PIPELINE_ID evm-sequencer > testnet-logs.txt
      txpark status ci-$CI_PIPELINE_ID > testnet-status.txt
    fi
artifacts:
  when: on_failure
  paths:
    - testnet-logs.txt
    - testnet-status.txt

Retry logic:

# Retry deployment on transient failures
for i in {1..3}; do
  if txpark deploy ci-$CI_PIPELINE_ID; then
    break
  fi
  echo "Deployment attempt $i failed, retrying..."
  sleep 30
done

Cost Optimization¶

Share testnets across jobs:

deploy_testnet:
  stage: deploy
  script:
    - txpark deploy ci-$CI_PIPELINE_ID

integration_test_1:
  stage: test
  dependencies: [deploy_testnet]
  script:
    - run-tests-against ci-$CI_PIPELINE_ID

integration_test_2:
  stage: test
  dependencies: [deploy_testnet]
  script:
    - run-tests-against ci-$CI_PIPELINE_ID

cleanup:
  stage: cleanup
  script:
    - txpark destroy ci-$CI_PIPELINE_ID --force

Performance Optimization¶

Maximize testnet efficiency.

Binary Management¶

Use branch builds wisely:

# For stable testing, use specific tags
txpark deploy stable-test --build-branch v19.0

# For development, use feature branches
txpark deploy dev-test --build-branch feature/my-feature

# Reuse pipeline IDs when possible to avoid rebuilding

Storage Management¶

Monitor disk usage:

# Check storage consumption
txpark shell my-testnet evm-sequencer
df -h | grep -E "data|rollup"

# For long-running testnets, consider periodic restarts
txpark upgrade my-testnet --build-branch master  # Preserves data

Network Optimization¶

Use local endpoints when possible:

# From within the cluster
http://evm-sequencer:8545/rpc

# From outside (through ingress)
https://my-testnet.txpark.nomadic-labs.com/rpc

Documentation¶

Keep your testnet usage well-documented.

Team Wiki¶

Maintain a team wiki page with: - Active permanent testnets and their purpose - Naming conventions - Deployment procedures - Common issues and solutions - Contact information for testnet owners

Code Comments¶

#!/bin/bash
# deploy-integration-testnet.sh
#
# Purpose: Deploy testnet for integration testing
# Owner: DevOps team
# Lifetime: 8h (covers overnight test run)
# Resources: High-performance (required for load tests)
#
# Prerequisites:
# - Run setup-sequencer.sh first
# - Authenticate with GitLab: glab auth login
#
# Usage:
#   ./deploy-integration-testnet.sh <testnet-name>

TESTNET_NAME=${1:-integration-$(date +%s)}
./scripts/setup-sequencer.sh $TESTNET_NAME
txpark deploy $TESTNET_NAME \
  --sequencer \
  --build-branch master \
  --profile high-performance \
  --lifetime 8h

Runbooks¶

Create runbooks for common scenarios: - Deploying staging environment - Emergency testnet cleanup - Investigating performance issues - Upgrading permanent testnets

Next Steps¶

Try the Workflows for hands-on examples
Review Commands for detailed command reference
Check Development to contribute improvements