Fetch every page of a result • specmill

Many APIs return a long result in pages. Some generated wrappers fetch all of those pages for you in one call. This guide is for the person calling such a wrapper. The maintainer who turns paging on for an operation should read configuration instead. The examples use a hypothetical client with a paging wrapper named get_items().

How to tell a wrapper pages

A paging wrapper has two extra arguments at the end of its signature, and its help page documents both:

get_items(kind, pageNumber = 1L, all_pages = TRUE, max_pages = 100L)

pageNumber is the API’s own page argument. all_pages and max_pages are added by the wrapper and control the paging. A wrapper without all_pages makes one request, as it always did.

The basic call

items <- get_items('books')

The wrapper requests page 1, then page 2, and so on, one request after another, and stops when a page comes back empty. You get the pages combined into one value, in page order: one table if the client turns responses into tables, otherwise one list of records. When the client builds tables, one page (all_pages = FALSE) and all pages are the same kind of table, so code that works on one page keeps working on all of them. Without tables, a single page comes back as the API sent it, and the combined call returns one list of records.

Start somewhere else, or fetch one page

The page you give is the page the wrapper starts from:

get_items('books', pageNumber = 3)                   # pages 3, 4, 5, ... to the end
get_items('books', pageNumber = 3, all_pages = FALSE) # page 3 only

all_pages = FALSE is exactly the wrapper without paging: one request, the plain result, nothing added to it. Use it to look at the shape of the data before you fetch everything.

Limit how much you fetch

max_pages is the most pages one call will request (the default is shown in the help page). Reaching it stops the call with a warning, because there may be more data:

items <- get_items('books', max_pages = 10)
#> Warning message:
#> Pagination truncated at max_pages

Each page is one request to the API, so a large max_pages on a big result can mean many requests. Keep it small while you explore, and watch any rate limits the API documents.

Find out why it stopped

By default, the result carries a record of the paging in attr(, 'pagination'). A wrapper generated with operation policy pagination: {metadata: false} omits this attribute while retaining limit warnings and errors:

attr(items, 'pagination')
#> $requests
#> [1] 4
#> $item_count
#> [1] 7
#> $stop_reason
#> [1] "empty_page"

`stop_reason`	Meaning	Is there more data?
`empty_page`	A page came back empty. This is the normal end.	No
`completed`	The client’s own rule said the last page had arrived (for example a `last` flag in the response).	No
`max_pages`	You hit `max_pages`.	Possibly: raise the limit or continue from a later page

If a wrapper always stops at max_pages, its pages never come back empty, for example because the API wraps the records in an envelope such as {"data": [...]}. Tell the client’s maintainer; the fix is a setting, not something to work around in your code.

requests is the number of pages requested, including the empty page when one ended the call, and item_count is how many rows or records came back. Check for truncation like this:

paging <- attr(items, 'pagination')
if (identical(paging$stop_reason, 'max_pages')) {
  # The call started at page 1, so the next page is the one after the pages it fetched.
  more <- get_items('books', pageNumber = 1L + paging$requests)
}

The record is only added when all_pages = TRUE. It can be lost if you modify the result with a function that drops attributes, so read it right after the call.

What a table looks like

When the client returns tables, the column types are worked out once over all the pages together, exactly as if the API had sent every row in one response. A column that holds "F" on one page and "MF" on another stays text, and an identifier such as "007" is treated the same on every page. See calling a wrapper with many inputs for the same rule applied to vectors of inputs.

When something goes wrong

If any page fails, the call stops with that error and returns nothing, so you never get a silently incomplete result. Problems that affect the whole call, such as a missing credential, are raised before the first request.

A dry run (the client’s dry-run option) prepares the first page’s request and returns it without sending anything.

Paging and many inputs do not combine

A wrapper either pages or takes a vector of inputs; it does not do both. To page through several kinds, loop yourself:

results <- lapply(c('books', 'films'), get_items)

For maintainers

Hooks run once per page, and an optional post_combine hook runs once on the combined result. A pre_request hook must not change the page argument; see hooks.

Reviewed cursor paging

Generated wrappers accept pagination.mode: cursor for a string query/path input, or an explicit client input used by a complete request binding. Use the client input for a POST facade whose request mapping places its cursor in the body. No response field names or cursor locations are inferred.

operations:
  GET /items:
    pagination:
      mode: cursor
      parameter: cursor
      next_cursor: cursor_next
      completed: cursor_done
      items: cursor_items
      metadata: false
      error_policy: partial
      warn_errors: false
      on_limit: cursor_limit

These reviewed client functions describe an envelope explicitly:

cursor_next <- function(page) page$pagination$nextCursor
cursor_done <- function(page, state) identical(page$pagination$hasNext, FALSE)
cursor_items <- function(page) api_request_table(page$results)
cursor_limit <- function(state, limit) {
  warning(paste('Pagination stopped at', limit, 'pages'), call. = FALSE)
}

The next cursor is opaque: NULL or an empty string ends iteration. Repeated nonempty cursors, NA, non-string values and multi-entry cursors abort before another request. A NULL initial GET cursor and an empty initial POST cursor retain their public defaults. all_pages = FALSE sends one request and applies items to that response. Automatic paging retains each raw page for completed and next_cursor, and combines the extracted items. Table extraction runs inside the batching context so api_request_table() can build types over all pages. A dry run returns the first prepared request.

error_policy defaults to stop; partial retains successful pages and empty discards them after a request error. warn_errors defaults to TRUE. Credential and other specmill_call_error conditions always abort. Extraction and cursor callback failures abort as well; successful-response formatter failures inside a client response policy can use the documented call-error class when they must remain hard errors.

Limit warnings default to the existing pager warning. An on_limit(state, limit) client function replaces that warning when the successful page count reaches the limit, including completion on that last page. Set warn_limits: false to suppress limit warnings. The callback owns any service-specific warning text. Existing page-mode wrappers and their defaults remain supported. Client helpers must include the current pager adapter and pagination companion before adopting these controls; generated clients do not need specmill at runtime.

stop_on_empty defaults to TRUE. Set pagination.stop_on_empty: false when an empty collection can still advertise a next page. The reviewed completion and next-cursor callbacks then determine continuation; cursor validation, repeat detection and max_pages still apply. This does not override stop_on_short in the standalone companion.