· hands on

Use TypeScript generators for cleaner programming

Generators help fetch and process data from APIs more granular, improving code modularity and user-friendliness. This tutorial teaches you how to use TypeScript generators for cleaner programming using a real-world example.

Contents

Response Schema Definition

The schema for the API response is defined using Zod, which ensures that the fetched data matches the expected structure. This is useful for validating the response:

import { z } from 'zod';
 
// Schema to validate the API response
const ResponseSchema = z.object({
  ticker: z.string(),
  queryCount: z.number(),
  resultsCount: z.number(),
  adjusted: z.boolean(),
  results: z.array(
    z.object({
      v: z.number(),
      vw: z.number(),
      o: z.number(),
      c: z.number(),
      h: z.number(),
      l: z.number(),
      t: z.number(),
      n: z.number(),
    })
  ),
  status: z.string(),
  request_id: z.string(),
  count: z.number(),
  next_url: z.string().optional(),
});
 
type ResponseType = z.infer<typeof ResponseSchema>;

Pay attention to the optional next_url. It serves as our indicator of whether an API endpoint contains additional results. We will keep fetching more data as long as there is a URL for the next batches, but once it becomes undefined, we will cease querying the endpoint.

Tip: You can easily generate Zod schemas using the JSON to Zod Schema converter from "transform.tools" or the json-to-zod npm package.

Traditional Asynchronous Function

In the traditional approach, we use an asynchronous function in TypeScript that handles the fetching, parsing, and looping through the paginated results:

async function fetchData(address: string, apiKey: string): Promise<ResponseType> {
  const url = new URL(address);
  url.searchParams.append('apiKey', apiKey);
  url.searchParams.append('limit', '200');
  const response = await fetch(url);
  const payload = await response.json();
  const parsed = ResponseSchema.parse(payload);
  return parsed;
}
 
const apiKey = 'top-secret';
const resource = 'https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2023-01-09/2023-01-09';
 
let next_url: string | undefined = resource;
while (next_url) {
  const chunk = await fetchData(next_url, apiKey);
  console.log(chunk.results.length, chunk.next_url);
  next_url = chunk.next_url;
}

The fetchData function holds the business logic to query data, while dealing with the next URL is tackled outside this function. As a result, the executing code must be aware of the internal details of the endpoint. This leads to poor encapsulation and makes it harder to manage for more complex asynchronous flows.

Generator Function Approach

In the generator approach, we create an async generator function to yield the results, making it easier to handle each chunk of data separately:

async function* fetchData(address: string, apiKey: string): AsyncGenerator<ResponseType> {
  let url: URL | undefined = new URL(address);
  while (url) {
    url.searchParams.append('apiKey', apiKey);
    url.searchParams.append('limit', '200');
    const response = await fetch(url);
    const payload = await response.json();
    const parsed = ResponseSchema.parse(payload);
    yield parsed;
    parsed.next_url ? (url = new URL(parsed.next_url)) : (url = undefined);
  }
}
 
const apiKey = 'top-secret';
const resource = 'https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2023-01-09/2023-01-09';
 
const dataGenerator = fetchData(resource, apiKey);
for await (const chunk of dataGenerator) {
  console.log(chunk.results.length, chunk.next_url);
}

The generator also separates data fetching logic (keeping track of the url) from the iteration logic (for await...of loop). This approach makes the code more modular and user-friendly, as the consuming code no longer needs to update the next_url.

It's even possible to return each response individually instead of yielding parsed data in batches of 200. This way, each call to yield will provide a single result item:

for (const result of parsed.results) {
  yield result;
}

Be aware that you need to adjust the returned result type accordingly:

type ResponseType = z.infer<typeof ResponseSchema>;
type ResultType = ResponseType['results'][0];
 
async function* fetchData(address: string, apiKey: string): AsyncGenerator<ResultType> {
  // ...
}

Final Result

Below is the complete TypeScript code that defines an async generator function to fetch data from a REST API, parses the response, and yields each individual result item. This approach provides fine-grained control over processing each piece of data:

import { z } from 'zod';
 
const ResponseSchema = z.object({
  ticker: z.string(),
  queryCount: z.number(),
  resultsCount: z.number(),
  adjusted: z.boolean(),
  results: z.array(
    z.object({
      v: z.number(),
      vw: z.number(),
      o: z.number(),
      c: z.number(),
      h: z.number(),
      l: z.number(),
      t: z.number(),
      n: z.number(),
    })
  ),
  status: z.string(),
  request_id: z.string(),
  count: z.number(),
  next_url: z.string().optional(),
});
 
type ResponseType = z.infer<typeof ResponseSchema>;
type ResultType = ResponseType['results'][0];
 
async function* fetchData(address: string, apiKey: string): AsyncGenerator<ResultType> {
  let url: URL | undefined = new URL(address);
  while (url) {
    url.searchParams.append('apiKey', apiKey);
    url.searchParams.append('limit', '200');
    const response = await fetch(url);
    const payload = await response.json();
    const parsed = ResponseSchema.parse(payload);
    for (const result of parsed.results) {
      yield result;
    }
    parsed.next_url ? (url = new URL(parsed.next_url)) : (url = undefined);
  }
}
 
const apiKey = 'top-secret';
const resource = 'https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2023-01-09/2023-01-09';
 
const dataGenerator = fetchData(resource, apiKey);
for await (const result of dataGenerator) {
  console.log(result);
}

Video Tutorial

Back to Blog