An application that feels fast with ten users can behave very differently with ten thousand. Code paths that were never stressed start to queue. Database connection pools run dry. Autoscaling reacts a few minutes too late. The first people to find out are usually your customers.
The symptoms are familiar:
Slow API responses and rising latency
High CPU and memory utilization on application servers
Database connection exhaustion
Request timeouts and HTTP 5xx errors
Services that become unresponsive during traffic spikes
Performance testing is cheapest when it happens before a major product launch, a marketing campaign, a seasonal peak or an infrastructure change, not after.
This guide shows how Artillery load testing helps engineering teams simulate realistic traffic, measure how an AWS-hosted application responds, and turn the results into measurable improvements. We cover test design, result interpretation, correlation with Amazon CloudWatch metrics, common bottlenecks, and CI/CD integration.
What Is Load Testing?
Load testing is one member of a family of application performance testing techniques. Each answers a different question:
Test typeQuestion it answersTypical useLoad testingDoes the system meet its goals under expected traffic?Pre-release validation, baselinesStress testingWhere does it break, and how does it fail?Capacity planning, failure-mode discoverySpike testingCan it absorb a sudden surge and recover?Campaigns, flash sales, notificationsSoak testingDoes it degrade over hours (leaks, pool drift)?Memory leaks, connection leaks, log growthScalability testingHow does performance change as traffic or resources grow?Autoscaling validation, right-sizing
Most teams should start with load tests against a baseline, then add spike and soak tests as the system matures. Stress tests are valuable but should be run deliberately, in an isolated environment.
Tech Team covers the latest in technology, offering helpful articles, practical guides, and expert insights to keep readers informed about digital trends, tools, and innovation.
Product strategyCloud architectureDevOps deliverySoftware support
Send us a message
Fill out the form below and we'll get back to you soon.
is a developer-focused, open-source load and performance testing tool. Tests are described declaratively in YAML, with JavaScript available when you need custom logic.
Relevant capabilities include:
YAML-based test configuration that lives in your repository
HTTP and API testing
WebSocket testing
Scenario-based virtual user behavior (multi-step journeys, not isolated requests)
CLI execution with JSON output and console reporting
CI/CD friendliness, including threshold checks
Optional Artillery Cloud for reporting and collaboration, plus distributed execution options
Note: Feature availability differs between Artillery versions, and some cloud-based capabilities depend on plan. The open-source CLI covers the local, scripted workflow in this article; confirm plan-dependent features in the official docs before relying on them.
The key advantage over a simple request blaster is the scenario model: Artillery simulates users who log in, browse, search and submit, with realistic pauses between steps. That is much closer to how production traffic behaves.
Understanding the Components of an Artillery Test
An Artillery test file has two main sections: config (how to run) and scenarios (what virtual users do).
ElementPurposeconfigGlobal test settingstargetBase URL of the application or API under testphasesTraffic generation pattern over timevariablesValues reused in scenariospluginsOptional extensions (e.g. expect, ensure)processorA JavaScript file with custom functionsscenariosNamed user journeysflowOrdered steps a virtual user performs
Key terms
Arrival rate: how many new virtual users start each second within a phase.
Duration: how long a phase lasts.
Virtual user (VU): one simulated user running a scenario from start to finish.
Requests per second (RPS): the rate of HTTP requests actually sent.
Response latency: the time from sending a request to receiving the response.
Virtual users and RPS are related but not interchangeable. If an arrival rate is 10 users per second and each scenario makes five requests, you generate roughly 50 requests per second, ignoring think time, retries and failures. Always decide which quantity your requirement is expressed in.
Setting Up Artillery
1. Install a supported Node.js version
Check the Artillery installation docs for the Node.js version currently supported, then install it with your preferred version manager.
2. Create a dedicated directory and project
mkdir perf-tests && cd perf-tests
npm init -y
3. Install Artillery as a project dependency
npm install --save-dev artillery
Artillery can also be installed globally, but a project-level install is better for teams. Pinning the version (via package-lock.json or an exact version in package.json) means a test run next month uses the same engine as today, so results stay comparable and CI runs are repeatable.
4. Verify the installation
npx artillery --version
5. Create and run your first test
npx artillery run load-test.yml
Creating Your First Artillery Load Test
The example below targets a fictional SaaS product API. api.example.com is illustrative. Replace it with an environment you own and are authorized to test. Never point load tests at public third-party websites.
target sets the base URL; scenario URLs are relative to it.
Warm-up phase sends a trickle of traffic so caches, connection pools and JIT compilation are not measured cold.
Ramp-up phase raises the arrival rate from 2 to 20 users per second (rampTo), revealing the point where behavior starts to change.
Sustained phase holds 20 users per second for five minutes to see steady-state behavior.
plugins: expect enables response assertions. A response with the wrong status or content type is counted as a failure.
ensure defines pass/fail criteria: here, p95 latency under 800 ms and an error rate below 1%. If a threshold is breached, the CLI exits with a non-zero code, which CI can act on. The 800 ms and 1% figures are examples; derive yours from business requirements.
scenarios / name gives the journey a readable name in reports.
Designing Realistic User Scenarios
Hammering one endpoint tells you little. Real users arrive, authenticate, look around, and occasionally write data. Your test should reflect that mix.
Consider this journey for an illustrative SaaS dashboard or e-commerce app:
Think time and pacing.think pauses a virtual user between steps. Without it, a "user" fires requests as fast as the network allows, which is rarely how people behave and tends to overstate load per user.
Mixed journeys.weight splits virtual users across scenarios: here roughly 70% browse and 30% place test orders. Base your weights on production analytics where possible.
Correlation.capture extracts a value (a product ID) from one response and reuses it in the next request, as a real client would.
Dynamic data. The CSV payload supplies varied search terms so you do not exercise one cached query repeatedly. Repeating identical requests can produce flattering cache-hit results that do not represent real use.
Test data preparation. Seed the staging database with a representative volume and distribution of data. An empty or tiny database hides slow queries.
Authentication safely. Do not hard-code tokens in YAML. The example reads TEST_API_TOKEN from an environment variable via $processEnvironment. Use a dedicated, low-privilege test account or short-lived token; store it in your CI secret store; and make sure reports and logs do not print headers. Never reuse production credentials.
Running Load Tests
Local execution and saving output
export TEST_API_TOKEN="<your-staging-test-token>"
npx artillery run scenarios.yml --output results/baseline.json
The --output flag writes a JSON report you can archive, diff or feed into other tooling.
Different traffic profiles
Define environments in the YAML and select one at runtime:
npx artillery run scenarios.yml --environment smoke
npx artillery run scenarios.yml --environment load --output results/load-run-01.json
Comparing baseline and later runs
Keep each run's JSON output, name files consistently, and record what changed (code version, instance size, data volume). A result is only meaningful relative to a comparable baseline.
Start small, then scale up
Begin with a low arrival rate, confirm scripts and assertions work, then increase gradually. Debugging a broken script at high volume wastes time and money.
Warning: High-volume tests can overload the target, generate significant cloud costs (compute, data transfer, logs) and degrade service for real users. Do not run load tests against production without explicit authorization, agreed limits, monitoring, and a way to stop the test immediately.
Understanding Artillery Test Results
Artillery prints a summary at the end of a run and in periodic intervals during it. The metrics that matter most:
Request count and rate (e.g. http.requests, http.request_rate)
HTTP status codes (e.g. http.codes.200, http.codes.500)
Successful and failed requests / virtual users (vusers.completed, vusers.failed)
Response time summary: min, max, median, p95, p99
Error rate (failed requests as a share of total)
Throughput (completed work per second, not just requests sent)
Exact metric names can vary by version; confirm them in the docs and in your own output.
Why averages mislead
An average blends fast and slow responses. If 95% of requests are quick and 5% take several seconds, the mean can still look acceptable while one in twenty users has a poor experience. Percentiles describe that tail: p95 means 95% of requests were faster than this value.
Illustrative interpretation (hypothetical values, not measured results)
All numbers above are fabricated for explanation only.
Reading the table: at 20 users per second the mean (180 ms) looks fine, but p95 has tripled versus baseline and p99 is more than four times higher. Something is queuing for a minority of requests. By 40 users per second the median barely moves while the tail explodes and errors appear, a classic signature of a shared resource saturating (a connection pool, a CPU limit or a slow query) rather than uniform slowness.
This is the moment to look at infrastructure metrics.
Integrating Artillery with AWS Monitoring
Artillery tells you what users experience. AWS metrics tell you why. Run both side by side and line up timestamps: note when each phase started and ended, then look at the same window in CloudWatch.
Workflow: Artillery generates traffic; CloudWatch collects metrics from each layer; you correlate them by time window.
Reminder: Many of these metrics require configuration. Memory on EC2, for example, needs the CloudWatch agent; application metrics need instrumentation; container-level insight may need Container Insights. Enable and verify them before test day.
Amazon CloudWatch
Look at CPU, network traffic, application logs, request latency, error metrics, custom application metrics (queue depth, cache hit ratio, pool usage) and alarms. Create a CloudWatch dashboard dedicated to load tests so every run is viewed through the same lens. Alarms double as a safety net that tells you if the test is hurting something it should not.
Amazon ECS and AWS Fargate
Watch service CPU and memory utilization, running task count versus desired count, service autoscaling activity, deployment health, and task restarts or failures. Tasks restarting under load often point to out-of-memory kills or failing health checks.
Amazon EC2
For EC2-based workloads, check CPU utilization (including burstable credit balance on T-family instances), network throughput, disk performance and any instance-level limits.
Amazon RDS
Monitor database connections, CPU, read and write latency, database load (Performance Insights, if enabled), storage performance, and how close you are to the maximum connection count.
Application Load Balancer
Track request count, target response time, healthy and unhealthy host counts, and HTTP 4xx/5xx counts. Distinguish ELB-generated 5xx errors (e.g. no healthy targets) from target-generated ones (your application failing).
Worked example: correlating latency with infrastructure (hypothetical)
Suppose the table above reflects your run. You line up the timestamps and observe:
ObservationLayerp95 rises sharply 4 minutes into the 20 users/s phaseArtilleryALB target response time rises at the same momentALBECS CPU stays around 40%, memory stableECSRDS connection count flattens at the configured maximumRDSRDS CPU moderate, but one query dominates database loadRDS
CPU headroom on the application tier rules out "add more tasks" as the first fix. Connections pinned at the limit and one dominant query point to connection pool exhaustion and a slow query. Scaling the ECS service might even worsen things by opening more connections. This is exactly why correlation matters: the symptom is in the API, the cause is in the database.
Use the signals from your results and AWS metrics to match symptoms to causes.
1. Application server CPU saturationIndicates: latency rising with load; CPU near its limit. Inspect: ECS/EC2 CPU, run queue, profiling. Fix: optimize hot code paths, then scale horizontally.
4. Connection pool exhaustionIndicates: latency plateau, timeouts, errors while CPU is idle. Inspect: RDS connections, pool metrics, wait times. Fix: tune pool size per instance against the database limit; consider connection pooling (e.g. RDS Proxy) where suitable.
5. Inefficient API endpointsIndicates: one scenario step dominates latency. Inspect: per-endpoint timing, tracing. Fix: paginate, reduce payload size, avoid over-fetching, batch data access.
7. Excessive external API callsIndicates: latency tied to third-party response times. Inspect: outbound call duration and error rates. Fix: cache results, add timeouts and circuit breakers, move calls to background jobs. Mock or stub third parties during load tests unless you have permission to test them.
8. Network latencyIndicates: consistently high baseline even at low load. Inspect: load generator location, cross-region/AZ traffic, NAT or bandwidth limits. Fix: co-locate components, reduce chatty calls, use CloudFront for cacheable content.
9. Inadequate autoscaling configurationIndicates: latency spikes during ramps that recover later. Inspect: scaling events versus load timeline, cooldowns, scaling metric. Fix: adjust target tracking thresholds, minimum capacity, and warm-up settings; validate with spike tests.
10. Poorly configured load balancer or application timeoutsIndicates: 502/504 errors, unhealthy targets, mismatched idle timeouts. Inspect: ALB 5xx metrics, health check settings, keep-alive and timeout values. Fix: align ALB, application and upstream timeouts; tune health checks.
Scaling is not the only answer. Throwing more instances at an inefficient query or an undersized connection pool rarely solves the underlying problem and increases cost. Investigate the application first, then scale deliberately.
Improving AWS Application Performance After Testing
Once you know the bottleneck, apply targeted changes:
Right-size compute: match CPU/memory to measured demand, not guesses.
ECS service autoscaling and EC2 Auto Scaling (where applicable) tuned from observed behavior.
Database query optimization and index review based on actual slow queries.
Connection management: correct pool sizes, timeouts, and proxying where helpful.
Caching with Amazon ElastiCache for repeated reads where appropriate.
CloudFront for static and cacheable content.
Load balancing configuration and health check tuning.
Fewer unnecessary API calls from clients and between services.
Code-level optimization of hot paths.
Background processing: move slow, non-interactive work (emails, report generation) to queues and workers.
Validate every change. Re-run the same scenario, with comparable data volume and traffic profile, and compare against your saved baseline. If you change three things at once, you will not know which one helped.
Integrating Artillery into CI/CD
Performance regressions are easier to fix when they are caught close to the change that caused them. A typical workflow:
Developer opens a pull request.
Unit and integration tests run.
The application is deployed to a staging environment.
Artillery runs a controlled performance test.
Results are evaluated against defined thresholds.
The pipeline passes or fails.
Approved changes continue through the release process.
Example GitHub Actions workflow (simplified)
# .github/workflows/performance-smoke.yml (illustrative)
name: Performance smoke test
on:
workflow_dispatch:
push:
branches: [main]
jobs:
artillery-smoke:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22" # confirm the version supported by your Artillery release
cache: npm
- name: Install dependencies (pinned via lockfile)
run: npm ci
- name: Run Artillery smoke test against staging
env:
TEST_API_TOKEN: ${{ secrets.STAGING_TEST_API_TOKEN }}
run: |
mkdir -p results
npx artillery run scenarios.yml \
--environment smoke \
--output results/smoke.json
- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: artillery-results
path: results/
retention-days: 14
If the ensure thresholds in your YAML are breached, the Artillery command exits non-zero and the job fails.
Practical guidance
Store scripts in the repository alongside the code they test.
Pin Artillery's version through your lockfile.
Keep secrets in GitHub Secrets (or your CI's secret store), never in YAML or logs.
Target a controlled staging environment, not production.
Set thresholds such as p95 latency and maximum error rate, informed by business requirements.
Publish or retain reports as build artifacts for trend comparison.
Do not run expensive tests on every small PR.
Smoke test vs. full load test
A performance smoke test is short and light (one to two minutes at low rate). It catches obvious regressions, such as a new endpoint that is suddenly ten times slower, and is cheap enough to run frequently. A full load or stress test is longer, heavier and costs more, so run it on a schedule (nightly or weekly), before releases, or on demand before major events.
Best Practices for Effective Load Testing
Define performance objectives first, such as target p95 latency and acceptable error rate.
Establish a baseline before changing anything.
Use realistic user journeys built from real usage patterns.
Test in an isolated or controlled environment so other traffic does not skew results.
Increase traffic gradually.
Monitor infrastructure during the test, not just the API.
Use representative test data in volume and distribution.
Avoid testing third-party systems without permission.
Repeat tests under comparable conditions (same data, region, instance sizes).
Track p95 and p99, not just averages.
Set thresholds based on business requirements.
Document findings and corrective actions.
Re-test after every significant optimization.
There is no universal latency threshold. A checkout API, an internal reporting tool and a search-as-you-type endpoint have very different tolerances. Let user expectations and business impact set your targets.
These practices fit naturally with broader engineering quality gates. For how Qbits approaches testing and maintainability, see Software Development Standards at Qbits.
Common Load-Testing Mistakes to Avoid
MistakeWhy it hurtsBetter approachUnrealistic traffic patternsResults do not predict productionModel journeys, think time and weights from analyticsTesting only one endpointMisses interactions between componentsUse multi-step, mixed scenariosIgnoring database and infrastructure metricsYou see the symptom, not the causeMonitor ALB, ECS/EC2, RDS during every runRunning excessive traffic too earlyWastes time and can destabilize environmentsStart small and ramp graduallyUsing production without authorization or safeguardsRisks real customer impact and costTest in staging; if production is unavoidable, get approval, limits and a kill switchTreating average latency as the only metricHides tail latencyTrack median, p95, p99 and error rateIgnoring failed requestsA fast error is still an errorAssert on status codes and alert on error rateForgetting authentication and test dataTests fail or hit unrealistic pathsPrepare accounts, tokens and seeded data firstComparing tests under different conditionsDifferences are not attributableKeep environment, data and profile consistentAssuming autoscaling fixes everythingScale cannot fix bad queries or pool limitsDiagnose the bottleneck, then scale where appropriate
How Qbits Helps Businesses Build Performance-Ready Applications
Qbits is a product-first software and cloud engineering partner. Performance work sits where application design and infrastructure meet, which is where many teams need an outside perspective. Our services can support organizations with:
Application and API engineering
AWS infrastructure design
Performance and scalability planning
CI/CD automation
Cloud monitoring and observability
Container orchestration with ECS or EKS
Database and infrastructure optimization
Production support and performance tuning
In practice, that can mean assessing your application architecture, establishing performance baselines, designing a testing strategy, investigating bottlenecks, and improving infrastructure based on measured results rather than assumptions. You can learn more about our approach to Cloud & Platform Engineering or visit the Qbits homepage.
Conclusion
Load testing is an ongoing engineering practice, not a one-time pre-launch checkbox. As your product, data and traffic evolve, so should your tests.
Artillery load testing gives teams a practical, scriptable way to simulate realistic traffic, measure how applications behave, and find performance problems before users do. But the numbers from a test run only become useful when you combine them with application logs, AWS infrastructure metrics and sound engineering analysis. Percentiles show where users are hurting; CloudWatch, ECS, RDS and ALB metrics show why.
Software Development Standards at Qbits: How We Build Clean, Secure and Maintainable Software
Software Development Standards at Qbits: How We Build Clean, Secure and Maintainable Software Building software that works is only the beginning. At Qbits, we...