Turning paging into a lazy sequence with Clojure tranducers

Christoph Frick - ofnir.net — Thu, 05 Oct 2017 22:00:00 GMT

When dealing with APIs we are often confronted with some sort of paging to access lists. Yet if we need to access all of the data, it would be nice to have them as a lazy sequence.

This can be done nicely with transducers:

(defn page-seq
  "Flat sequence of items loaded via the given page loading fn"
  [get-page-fn]
  (eduction
   (map get-page-fn)
   (take-while seq)
   cat
   (range)))

Add some tests:

(def pages
  (into [] (partition-all 20 (range 56))))

(defn get-page
  [page]
  (println "Loading page" page)
  (nth pages page ()))

(page-seq get-page)
;Loading page 0
;Loading page 1
;Loading page 2
;Loading page 3
; => (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ... 54 55)

Now see, if that is really lazy:

(take 10 (page-seq get-page))
;Loading page 0
;Loading page 1
; => (0 1 2 3 4 5 6 7 8 9)

Well, no. We should only see Loading page 0 and not the one for page 1. The reason for this is chunking. Clojure internally loads itself chunks of 32. So side effects will be problematic. But for just lazily loading the whole lot having some mismatch in the used paging and the chunking is good enough for now.

Christoph Frick - ofnir.net - Open Source Consultant, Software Developer and System Administrator (Posts about Transducer)

Turning paging into a lazy sequence with Clojure tranducers