Performance Optimization

Learn how to optimize NestLens for minimal performance impact on your application.

Performance Overview

These numbers come from two benchmarks in the repository — run them on your own hardware rather than trusting the tables. Both were measured on Node 25, Apple Silicon, with the default configuration and the default in-memory storage.

One request at a time

npm run benchmark:

	measured
Added request latency (empty endpoint)	p50 0.025 ms, p99 0.18 ms
10,000 entries held in memory	2.3 MB
Write throughput, memory storage	~1,500,000 entries/second
Write throughput, SQLite storage	~23,000 entries/second

The latency figure is the difference between the same application with and without NestLens, over 2,000 requests after a warm-up.

Under concurrency

npm run benchmark:load runs the application in a process of its own and drives 32 concurrent connections at it, which is the question that matters if you are leaving NestLens on:

	GET, no body	POST, 2.5 KB body
CPU without NestLens	27 ms / 1,000 requests	50 ms / 1,000 requests
CPU with NestLens, defaults	55 ms / 1,000 requests	87 ms / 1,000 requests
Throughput cost	~32%	~18%
Idle CPU, dashboard open, nothing arriving	~0.5%

So roughly 28 microseconds of CPU per request, plus what the payload costs to copy. On a service handling 200 requests a second that is about 0.6% of one core.

Watchers that wrap a library — queries, cache, mail — cost in proportion to how often that library is called, so an application making twenty queries per request pays twenty times the per-entry cost rather than the per-request one.

Two things dominate beyond that: the storage driver you choose (SQLite is roughly sixty times slower to write than memory, and Redis depends on your network) and how much payload you record. maxBodySize, sampling and the filter hook are the levers.

:::note Improved in 0.10.0 The figure above was 200 ms / 1,000 requests before 0.10.0 — 7× what it is now. Three things were responsible and none of them were the price of the feature:

in-memory eviction sorted every key on every save once the entry cap was reached, which is the steady state of a capped storage — 32% of the whole process's CPU,
the collector's masker re-derived the answer for every field name on every entry instead of remembering it,
every request read process.memoryUsage() twice, for a figure that is not meaningful under concurrency (see captureMemory).

If you measured NestLens before 0.10.0 and put it aside, the numbers you got are not the numbers now. :::

Serving the dashboard

The dashboard is a static bundle served by NestLens itself, so it is worth knowing what it costs the application hosting it:

Files are read from disk once and kept in memory. Serving them is the only work per request; the event loop of your application is not blocked reading ~1 MB off disk every time somebody opens the dashboard.
Fingerprinted assets (assets/*) are sent with Cache-Control: public, max-age=31536000, immutable, so a browser that has loaded the dashboard once re-fetches nothing until you upgrade NestLens.
index.html is sent with Cache-Control: no-cache. It carries the mount point injected per request and points at the current bundle, so it is revalidated every time — that is what makes the immutable assets safe.
Scripts, stylesheets and other text assets are compressed on the way out, in brotli or gzip depending on what the browser asked for. NestLens writes its own responses so that your global interceptors cannot rewrite them, which also means nothing else in your pipeline compresses them — this does not depend on your application having compression middleware installed.

Measured on the shipped bundle:

uncompressed brotli
First load (index.html + vendor + app + CSS) 287 KB 79 KB
Opening a log, job, cache or similar entry 95 KB 26 KB
Opening a query entry (includes the SQL formatter) 344 KB 90 KB

Each file is compressed once per process and then reused, and the work runs on zlib's thread pool rather than the event loop. Already-compressed formats (PNG, WOFF2, ICO) and bodies under 1 KB are sent as they are — compressing them costs CPU and produces the same size or slightly larger.

Responses carry Vary: Accept-Encoding, so a shared cache in front of your application stores one entry per encoding instead of serving a brotli body to a client that cannot read it.
Each page and each entry detail view is a separate chunk, fetched when it is first needed. Opening a log entry no longer downloads the views for GraphQL, mail, models and the other seventeen types you did not open.

The SQL formatter is the one large exception: it is 230 KB of the query detail view, and it runs on first render because that view opens formatted. It is downloaded only when you open a query entry, and then cached like every other fingerprinted asset.

	uncompressed	brotli
First load (`index.html` + vendor + app + CSS)	287 KB	79 KB
Opening a log, job, cache or similar entry	95 KB	26 KB
Opening a query entry (includes the SQL formatter)	344 KB	90 KB

None of this applies when the dashboard is disabled: no files are read and nothing is cached.

Buffer Configuration

The collector uses buffering to minimize database writes.

Buffer Settings

// In CollectorService (hard-coded constants)
private readonly BUFFER_SIZE = 100;           // Entries before flush
private readonly FLUSH_INTERVAL = 1000;       // 1 second
private readonly MAX_BUFFERED_ENTRIES = 1000; // Ceiling while storage is down

When storage is slower than the traffic

One batch is on its way to the storage at a time. A flush that starts while another is still going waits for it and then writes whatever accumulated in the meantime; the interval timer and the buffer's own threshold skip rather than queue, because the flush in flight will take what they would have.

That keeps MAX_BUFFERED_ENTRIES meaningful. Every caller used to start its own write, so entries already in flight were outside the ceiling — measured at thirty concurrent writes and three thousand entries in flight against a store taking 300 ms a batch, three times the cap and growing with the lag.

At a rate the storage can keep up with, nothing is dropped and nothing waits: a thousand entries a second against that same 300 ms store wrote all three thousand. Past that the ceiling below decides what survives.

When storage stops answering

Entries that could not be written are kept and retried, but only up to MAX_BUFFERED_ENTRIES — past that the oldest are dropped and counted. NestLens runs inside your process, and its data is disposable in a way your memory is not.

While storage is failing, flushing moves entirely to the interval timer: an entry arriving during an outage is buffered and returns immediately rather than waiting on a write that is going to fail. One error is logged when the outage starts, and one line when storage answers again, reporting how many entries were dropped in between.

:::note Not configurable BUFFER_SIZE and FLUSH_INTERVAL are private readonly constants on CollectorService. They are not exposed through NestLensModule.forRoot(...) and cannot be set via configuration. There is no config option for them today. :::

Changing Buffer Behavior

The only way to change these values is to subclass CollectorService and provide your subclass for the collector token. Because the fields are private readonly, you must redeclare them in the subclass (and re-implement any flush logic that reads them):

// Requires extending CollectorService and overriding the relevant fields/methods.
class OptimizedCollector extends CollectorService {
  protected readonly BUFFER_SIZE = 500;      // Larger buffer
  protected readonly FLUSH_INTERVAL = 5000;  // Flush every 5 seconds
}

Trade-offs:

Larger Buffer: Less frequent writes, more memory usage
Smaller Buffer: More frequent writes, less memory usage

Flush Strategy

Configure when buffered entries are written:

async collect(type: EntryType, payload: any) {
  this.buffer.push({ type, payload });

  // Immediate flush for critical entries
  if (type === 'exception') {
    await this.flush();
    return;
  }

  // Buffer others until full
  if (this.buffer.length >= this.BUFFER_SIZE) {
    await this.flush();
  }
}

Database Optimization

Indexing Strategy

Create indexes for common queries:

-- SQLite (default)
CREATE INDEX idx_type ON entries(type);
CREATE INDEX idx_created_at ON entries(createdAt);
CREATE INDEX idx_request_id ON entries(requestId);
CREATE INDEX idx_type_created ON entries(type, createdAt);

-- Compound index for filtered queries
CREATE INDEX idx_type_status ON entries(type, json_extract(payload, '$.statusCode'));

Connection Pooling

Use connection pooling for better performance:

// For custom storage backends
const pool = new Pool({
  max: 20,              // Maximum connections
  min: 5,               // Minimum connections
  idleTimeoutMillis: 30000,
});

Batch Operations

Use batch inserts instead of individual saves:

// GOOD - Batch insert
async saveBatch(entries: Entry[]): Promise<Entry[]> {
  const placeholders = entries.map(() => '(?, ?, ?, ?)').join(',');
  const values = entries.flatMap(e => [e.type, JSON.stringify(e.payload), e.requestId, e.createdAt]);

  await this.db.run(
    `INSERT INTO entries (type, payload, requestId, createdAt) VALUES ${placeholders}`,
    values
  );
}

// BAD - Individual inserts
for (const entry of entries) {
  await this.save(entry);
}

Pruning Optimization

Configure aggressive pruning to keep database small.

Optimized Pruning Config

NestLensModule.forRoot({
  pruning: {
    enabled: true,
    maxAge: 6,          // Keep only 6 hours
    interval: 15,       // Prune every 15 minutes
  },
})

Type-Specific Pruning

Implement custom pruning per entry type:

class CustomPruningService extends PruningService {
  async prune(): Promise<void> {
    // Keep exceptions longer (24 hours)
    await this.storage.pruneByType('exception', new Date(Date.now() - 24 * 60 * 60 * 1000));

    // Keep requests shorter (1 hour)
    await this.storage.pruneByType('request', new Date(Date.now() - 1 * 60 * 60 * 1000));

    // Keep logs very short (15 minutes)
    await this.storage.pruneByType('log', new Date(Date.now() - 15 * 60 * 1000));
  }
}

Vacuum Database

Periodically vacuum SQLite database:

@Cron('0 2 * * *') // Daily at 2 AM
async vacuumDatabase() {
  if (this.storage instanceof SqliteStorage) {
    await this.storage.run('VACUUM');
    this.logger.log('Database vacuumed');
  }
}

Watcher Optimization

Disable Unused Watchers

Only enable watchers you need:

NestLensModule.forRoot({
  watchers: {
    request: true,      // Essential
    exception: true,    // Essential
    query: true,        // Important

    // Disable everything else
    log: false,
    cache: false,
    event: false,
    job: false,
    schedule: false,
    mail: false,
    httpClient: false,
    redis: false,
    model: false,
    notification: false,
    view: false,
    command: false,
    gate: false,
    batch: false,
    dump: false,
  },
})

Optimize Query Watcher

NestLensModule.forRoot({
  watchers: {
    query: {
      enabled: true,
      slowThreshold: 500,     // Higher threshold = fewer entries
      ignorePatterns: [
        /^SELECT.*FROM sqlite_/,  // Ignore system tables
        /^PRAGMA/,                 // Ignore pragmas
        /^EXPLAIN/,                // Ignore explains
      ],
    },
  },
})

Optimize Request Watcher

NestLensModule.forRoot({
  watchers: {
    request: {
      enabled: true,
      captureBody: false,         // Disable body capture
      captureResponse: false,      // Disable response capture
      captureSession: false,       // Disable session capture
      maxBodySize: 0,              // No body capture
      ignorePaths: [
        '/health',
        '/metrics',
        '/favicon.ico',
        '/static/*',
      ],
    },
  },
})

Passing a settings block never turns a watcher off — only false or { enabled: false } does. (Before 0.10.0 it did, silently: any configured watcher recorded nothing at all.)

Request Watcher Memory

captureMemory records how much the heap grew across the handler. It is off by default and should usually stay off:

watchers: {
  request: { captureMemory: true },   // default: false
}

The figure is process.memoryUsage().heapUsed read either side of the handler. Under concurrency the heap is shared with every other request in flight and a garbage collection may run between the two readings, so the number is mostly noise: measured on an endpoint returning {ok: true}, it ranged from -570 KB to +671 KB and came out negative once in thirty. Reading it twice per request cost about 2.5% of the process's CPU.

Turn it on where the application handles one thing at a time — a local reproduction, a worker — and the number means something.

Shutting Down

The last thing NestLens does on shutdown is flush whatever is still buffered. That flush has a three-second deadline, after which the application finishes shutting down without it.

The deadline exists because a storage that has stopped answering does not fail the flush — it never returns. Awaiting it meant app.close() never resolved, SIGTERM did nothing, and the process waited for whatever eventually killed it. An unreachable Redis was enough to leave a rolling deploy hanging.

Measured against a storage whose save never settles:

storage	`app.close()`
healthy	1 ms
throwing	302 ms
hanging	never (now: 3 s)

A normal flush is milliseconds, so the deadline only ever applies when storage is already failing — and entries it will not accept were not going to be kept either way.

The production stance, in one setting

Recording only what failed takes five settings that are only correct together, and the ordering below is the part nobody guesses. preset: 'failures-only' is those settings:

NestLensModule.forRoot({
  preset: 'failures-only',
  storage: { driver: 'redis', redis: { url: process.env.REDIS_URL, db: 1 } },
})

It sets sampling.rate: 0 with the failing types in sampling.always, a filter that narrows those types to their failures, and turns off GraphQL response capture and resolver tracing. Anything you write beside it wins — except filter, which composes: under a preset, a filter of your own means "the failures, and this too".

Measured with npm run benchmark:load, 32 concurrent connections:

	GET /ping	POST /order (2.5 KB body)	RSS at rest
without NestLens	34,573 req/s	17,937 req/s	126 MB
defaults	21,242 req/s	14,732 req/s	308 MB
`failures-only`	24,475 req/s	18,241 req/s	182 MB

The rest of this section is what the preset is made of.

Sampling

NestLens records everything by default, which is the point of it. When that is more than you want to pay for, sampling records a fraction of traffic instead:

NestLensModule.forRoot({
  sampling: {
    rate: 0.1,              // one request in ten
    always: ['exception'],  // exceptions regardless — this is the default
  },
})

The decision is made per request, from its id, so a request and everything recorded under it — its queries, cache reads, logs and outgoing calls — are kept together or dropped together. A detail page is therefore always complete; you get fewer requests, not partial ones.

It costs a hash rather than a callback, and it runs before the entry is masked or buffered, so a dropped entry costs almost nothing. Use filter instead when the rule depends on what is inside the entry:

	use
"record a tenth of traffic"	`sampling`
"record only 4xx and 5xx"	`filter`
"record everything for one customer"	`filter`, or a monitored tag

rate: 0 records nothing except what always names — exceptions only, which is a reasonable way to run in production if you mostly want the error pages.

:::note What sampling does not save The decision is made in the collector, and a watcher has already built the payload by the time it is asked. With captureResponse: true, a GraphQL response is serialized and walked for every operation, including ones sampling then discards. To stop paying for the capture itself, use the watcher's own graphql.samplingRate, which is applied before anything is read. :::

:::caution Sampling runs first, and filter only sees what survives it The order is sampling → filter → mask → buffer. An entry sampling dropped never reaches filter, so the two narrow rather than combine: "only failed GraphQL operations" needs the type kept by sampling and then narrowed by the filter.

NestLensModule.forRoot({
  sampling: {
    rate: 0.1,
    // Kept whatever the rate says — `filter` decides which of them to write.
    always: ['exception', 'graphql'],
  },
  filter: (entry) =>
    entry.type !== 'graphql' ? true : entry.payload.hasErrors === true,
})

Naming a type in filter alone does nothing for a type sampling is dropping. :::

Entry Filtering Performance

Use Efficient Filters

// GOOD - Fast checks
filter: (entry) => {
  if (entry.type === 'request') {
    return entry.payload.statusCode >= 400;
  }
  return true;
}

// BAD - Expensive operations
filter: async (entry) => {
  if (entry.type === 'request') {
    // Database lookup on every entry - SLOW!
    const user = await db.findUser(entry.payload.userId);
    return user.trackingEnabled;
  }
  return true;
}

Cache Filter Results

const filterCache = new Map<string, boolean>();

filter: (entry) => {
  const key = `${entry.type}:${entry.payload.path}`;

  if (filterCache.has(key)) {
    return filterCache.get(key);
  }

  const shouldCollect = expensiveFilterLogic(entry);
  filterCache.set(key, shouldCollect);

  return shouldCollect;
}

Use Batch Filters

Batch filtering is more efficient than per-entry:

// GOOD - Process batch
filterBatch: (entries) => {
  // Process all at once
  return entries.filter(e => e.type !== 'log' || e.payload.level === 'error');
}

// LESS EFFICIENT - Per-entry
filter: (entry) => {
  return entry.type !== 'log' || entry.payload.level === 'error';
}

Memory Management

Monitor Memory Usage

setInterval(() => {
  const usage = process.memoryUsage();

  if (usage.heapUsed > 500 * 1024 * 1024) { // 500MB
    logger.warn('High memory usage, flushing buffers');
    collector.flush();
  }
}, 60000);

Limit Payload Size

filter: (entry) => {
  // Truncate large payloads
  if (entry.type === 'request' && entry.payload.body) {
    const bodyStr = JSON.stringify(entry.payload.body);

    if (bodyStr.length > 10000) { // 10KB
      entry.payload.body = {
        _truncated: true,
        _size: bodyStr.length,
      };
    }
  }

  return true;
}

Clear Old Data Aggressively

NestLensModule.forRoot({
  pruning: {
    enabled: true,
    maxAge: 1,          // 1 hour only
    interval: 10,       // Prune every 10 minutes
  },
})

CPU Optimization

Minimize JSON Operations

// GOOD - Avoid unnecessary parsing
async save(entry: Entry): Promise<Entry> {
  const payloadStr = JSON.stringify(entry.payload);
  await this.db.run(
    'INSERT INTO entries (type, payload) VALUES (?, ?)',
    [entry.type, payloadStr]
  );
}

// BAD - Multiple JSON operations
async save(entry: Entry): Promise<Entry> {
  const temp = JSON.parse(JSON.stringify(entry)); // Unnecessary
  const payloadStr = JSON.stringify(temp.payload);
  // ...
}

Use Async Operations

Keep operations non-blocking:

// GOOD - Async
async collect(type: EntryType, payload: any) {
  this.buffer.push({ type, payload });

  if (this.buffer.length >= this.BUFFER_SIZE) {
    // Non-blocking flush
    this.flush().catch(err => logger.error(err));
  }
}

// BAD - Blocking
collect(type: EntryType, payload: any) {
  this.buffer.push({ type, payload });

  if (this.buffer.length >= this.BUFFER_SIZE) {
    // Blocks until complete
    this.flushSync();
  }
}

Network Optimization

Compress Large Entries

import { gzip } from 'zlib';
import { promisify } from 'util';

const gzipAsync = promisify(gzip);

async save(entry: Entry): Promise<Entry> {
  let payload = JSON.stringify(entry.payload);

  // Compress if large
  if (payload.length > 50000) {
    const compressed = await gzipAsync(payload);
    payload = compressed.toString('base64');
    entry.compressed = true;
  }

  // Save compressed payload
  await this.db.save({ ...entry, payload });
}

Batch API Requests

If using external storage:

// GOOD - Batch requests
async saveBatch(entries: Entry[]): Promise<Entry[]> {
  return this.api.post('/entries/batch', { entries });
}

// BAD - Individual requests
for (const entry of entries) {
  await this.api.post('/entries', entry);
}

Production Optimizations

Complete Production Config

NestLensModule.forRoot({
  // Minimal watchers
  watchers: {
    request: {
      enabled: true,
      captureBody: false,
      captureResponse: false,
      ignorePaths: ['/health', '/metrics'],
    },
    exception: true,
    // All others disabled
  },

  // Aggressive pruning
  pruning: {
    enabled: true,
    maxAge: 1,       // 1 hour
    interval: 15,    // Every 15 minutes
  },

  // Efficient filtering
  filter: (entry) => {
    // Only errors in production
    if (entry.type === 'request') {
      return entry.payload.statusCode >= 500;
    }
    return entry.type === 'exception';
  },
})

Disable in Production

The safest optimization:

NestLensModule.forRoot({
  enabled: process.env.NODE_ENV !== 'production',
})

Benchmarking

Measure NestLens Impact

// Without NestLens
const start = Date.now();
for (let i = 0; i < 1000; i++) {
  await makeRequest();
}
const baseline = Date.now() - start;

// With NestLens
const startWithNestLens = Date.now();
for (let i = 0; i < 1000; i++) {
  await makeRequest();
}
const withNestLens = Date.now() - startWithNestLens;

const overhead = ((withNestLens - baseline) / baseline) * 100;
console.log(`NestLens overhead: ${overhead.toFixed(2)}%`);

Load Testing

# Use artillery or ab for load testing
artillery quick --count 10 -n 100 http://localhost:3000/api/users

# Monitor performance
node --inspect index.js

Performance Monitoring

Add Metrics

@Injectable()
export class PerformanceMonitor {
  private metrics = {
    entriesCollected: 0,
    entriesFlushed: 0,
    flushDuration: [],
    bufferSize: 0,
  };

  trackCollection() {
    this.metrics.entriesCollected++;
  }

  trackFlush(duration: number, count: number) {
    this.metrics.entriesFlushed += count;
    this.metrics.flushDuration.push(duration);
  }

  getMetrics() {
    return {
      ...this.metrics,
      avgFlushDuration: avg(this.metrics.flushDuration),
      entriesPerSecond: this.metrics.entriesCollected / uptime(),
    };
  }
}

Dashboard Integration

Create a metrics endpoint:

@Controller('admin')
export class MetricsController {
  @Get('nestlens/metrics')
  async getMetrics() {
    return {
      // { pending, capacity, dropped } — `pending` near `capacity` means
      // storage is slower than collection, and `dropped` is what that has
      // already cost.
      buffer: collector.getBufferSize(),
      storageSize: await storage.getStorageStats(),
      performance: performanceMonitor.getMetrics(),
    };
  }
}

Best Practices

1. Start with Defaults

Begin with default settings, then optimize if needed.

2. Measure Before Optimizing

Profile your application to identify actual bottlenecks.

3. Test Changes

Benchmark before and after optimization changes.

4. Monitor Production

Track NestLens impact in production metrics.

5. Disable if Needed

Don't hesitate to disable NestLens in production if performance is critical.

Troubleshooting

High Memory Usage

Reduce buffer size
Enable aggressive pruning
Disable body/response capture
Add entry filtering

Slow Response Times

Disable unused watchers
Use async collection only
Optimize filter functions
Reduce payload capture

Database Growth

Enable pruning
Reduce maxAge
Filter more aggressively
Implement type-specific retention

Next Steps

Create Custom Watchers
Implement Custom Storage
Configure Entry Filtering

Performance Overview​

One request at a time​

Under concurrency​

Serving the dashboard​

Buffer Configuration​

Buffer Settings​

When storage is slower than the traffic​

When storage stops answering​

Changing Buffer Behavior​

Flush Strategy​

Database Optimization​

Indexing Strategy​

Connection Pooling​

Batch Operations​

Pruning Optimization​

Optimized Pruning Config​

Type-Specific Pruning​

Vacuum Database​

Watcher Optimization​

Disable Unused Watchers​

Optimize Query Watcher​

Optimize Request Watcher​

Request Watcher Memory​

Shutting Down​

The production stance, in one setting​

Sampling​

Entry Filtering Performance​

Use Efficient Filters​

Cache Filter Results​

Use Batch Filters​

Memory Management​

Monitor Memory Usage​

Limit Payload Size​

Clear Old Data Aggressively​

CPU Optimization​

Minimize JSON Operations​

Use Async Operations​

Network Optimization​

Compress Large Entries​

Batch API Requests​

Production Optimizations​

Complete Production Config​

Disable in Production​

Benchmarking​

Measure NestLens Impact​

Load Testing​

Performance Monitoring​

Add Metrics​

Dashboard Integration​

Best Practices​

1. Start with Defaults​

2. Measure Before Optimizing​

3. Test Changes​

4. Monitor Production​

5. Disable if Needed​

Troubleshooting​

High Memory Usage​

Slow Response Times​

Database Growth​

Next Steps​