Turning paging into a lazy sequence with Clojure tranducers
When dealing with APIs we are often confronted with some sort of paging to access lists. Yet if we need to access all of the data, it would be nice to have them as a lazy sequence.
This can be done nicely with transducers:
(defn page-seq "Flat sequence of items loaded via the given page loading fn" [get-page-fn] (eduction (map get-page-fn) (take-while seq) cat (range)))
Add some tests:
(def pages (into [] (partition-all 20 (range 56)))) (defn get-page [page] (println "Loading page" page) (nth pages page ())) (page-seq get-page) ;Loading page 0 ;Loading page 1 ;Loading page 2 ;Loading page 3 ; => (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ... 54 55)
Now see, if that is really lazy:
Well, no. We should only see Loading page 0
and not the one for page
1
. The reason for this is chunking. Clojure internally loads itself
chunks of 32. So side effects will be problematic. But for just lazily
loading the whole lot having some mismatch in the used paging and the chunking
is good enough for now.