{"id":353,"date":"2014-12-15T19:16:21","date_gmt":"2014-12-15T18:16:21","guid":{"rendered":"http:\/\/joost.vunderink.net\/blog\/?p=353"},"modified":"2014-12-15T19:16:21","modified_gmt":"2014-12-15T18:16:21","slug":"processing-an-array-of-promises-sequentially-in-node-js","status":"publish","type":"post","link":"https:\/\/joost.vunderink.net\/blog\/2014\/12\/15\/processing-an-array-of-promises-sequentially-in-node-js\/","title":{"rendered":"Processing an array of promises sequentially in node.js"},"content":{"rendered":"

This post describes how you can perform a sequence of promises sequentially – one after another – using Kriskowal’s Q library<\/a> in node.js<\/a>. If you’re just interested in how to do this, and not in the other examples, scroll down to the last code snippet.<\/p>\n

Imagine you have an array of filenames, and you want to upload those\u00a0files to a server. Normally, you’d just fire off uploads asynchronously and wait for all of them to finish. However, what if the server you upload to has restrictions, for example maximum 1 concurrent upload, or a bandwidth limit?<\/p>\n

In this post, we’ll be using the Q library to create and process promises, and log4js<\/a> to get log lines with timestamps easily. First we create an uploader.js<\/span>\u00a0 module with\u00a0a function uploadFile<\/span>\u00a0\u00a0that takes a filename, and uploads it to a server. For demonstration purposes, this function doesn’t actually upload a file, but simply waits for a random time, and then fulfills the promise.<\/p>\n

var Q = require('q');\r\nvar log4js = require('log4js');\r\nvar logger = log4js.getLogger('uploader');\r\n\r\nexports.uploadFile = function(filename) {\r\n    var deferred = Q.defer();\r\n    Q.fcall(function() {\r\n        var delay = Math.random() * 4000 + 3000;\r\n        logger.info(\"Starting upload: \" + filename);\r\n        setTimeout(function() {\r\n            logger.info(\"Completed upload: \" + filename);\r\n            return deferred.resolve();\r\n        }, delay)\r\n    });\r\n    return deferred.promise;\r\n}\r\n<\/pre>\n
The following code uploads a single file:<\/p>\n
var log4js = require('log4js');\r\nvar logger = log4js.getLogger('upload-example-1');\r\nvar uploader = require('.\/uploader');\r\n\r\nvar filename = 'file1.jpg';\r\n\r\nuploader.uploadFile(filename)\r\n    .then(function(result) {\r\n        logger.info(\"The file has been uploaded.\");\r\n    })\r\n    .catch(function(error) {\r\n        logger.error(error);\r\n    });\r\n<\/pre>\n
The output of this script is:<\/p>\n
[14:15:28.128] [INFO] uploader - Starting upload: file1.jpg\r\n[14:15:32.169] [INFO] uploader - Completed upload: file1.jpg\r\n[14:15:32.170] [INFO] upload-example-1 - The file has been uploaded.<\/pre>\n
That’s uploading a single file. This is how you use Q to upload multiple files in parallel:<\/p>\n
var Q = require('q');\r\nvar log4js = require('log4js');\r\nvar logger = log4js.getLogger('upload-example-2');\r\nvar uploader = require('.\/uploader');\r\n\r\nvar filenames = ['file1.jpg', 'file2.txt', 'file3.pdf'];\r\nvar promises = filenames.map(uploader.uploadFile);\r\n\r\nQ.allSettled(promises)\r\n    .then(function(results) {\r\n        logger.info(\"All files uploaded. Results:\");\r\n        logger.info(results.map(function(result) { return result.state }));\r\n    })\r\n    .catch(function(error) {\r\n        logger.error(error);\r\n    });\r\n<\/pre>\n
Here, we have an array of filenames, which we turn into an array of promises by using the map<\/span>\u00a0\u00a0method. After that, we use Q.allSettled()<\/span>\u00a0\u00a0to wait until all promises have either been fulfilled or rejected. We don’t use Q.all()<\/span>\u00a0\u00a0here, because that would stop processing as soon as one of the promises is rejected.<\/p>\n
This leads to the following output:<\/p>\n
[14:49:09.598] [INFO] uploader - Starting upload: file1.jpg\r\n[14:49:09.601] [INFO] uploader - Starting upload: file2.txt\r\n[14:49:09.602] [INFO] uploader - Starting upload: file3.pdf\r\n[14:49:13.788] [INFO] uploader - Completed upload: file3.pdf\r\n[14:49:14.489] [INFO] uploader - Completed upload: file2.txt\r\n[14:49:15.014] [INFO] uploader - Completed upload: file1.jpg\r\n[14:49:15.014] [INFO] upload-example-2 - All files uploaded. Results:\r\n[14:49:15.014] [INFO] upload-example-2 - [ 'fulfilled', 'fulfilled', 'fulfilled' ]<\/pre>\n
As you can see, the three uploads are started immediately after each other, and run in parallel.<\/p>\n
To turn an array of filenames into a sequentially processed array of promises, we can use reduce<\/span>\u00a0. If you’ve never used reduce<\/span>\u00a0\u00a0before (here is the documentation<\/a>), this will look a bit weird.<\/p>\n
var Q = require('q');\r\nvar log4js = require('log4js');\r\nvar logger = log4js.getLogger('upload-example-3');\r\nvar uploader = require('.\/uploader');\r\n\r\nvar filenames = ['file1.jpg', 'file2.txt', 'file3.pdf'];\r\n\r\nvar lastPromise = filenames.reduce(function(promise, filename) {\r\n    return promise.then(function() {\r\n        return uploader.uploadFile(filename);\r\n    });\r\n}, Q.resolve())\r\n\r\nlastPromise\r\n  .then(function() {\r\n    logger.info(\"All files uploaded.\");\r\n  })\r\n  .catch(function(error) {\r\n    logger.error(error);\r\n  });\r\n\r\n<\/pre>\n
Let’s go through this. We\u00a0call the filenames.reduce()<\/span>\u00a0\u00a0with two arguments, a callback and an initial value. Because we passed a second argument to\u00a0reduce(), it will now call the given callback for each element in filenames<\/span>\u00a0. The callback gets two arguments. The first time, the first argument is the initial value and the second argument is the first element of filenames<\/span>\u00a0. Each next time, the first argument is the return value of the previous time the callback was called, and the second argument is the next element of filenames<\/span>\u00a0.<\/p>\n
In other words, the first argument of the callback is always a promise, and the second is an array element. We use an “empty promise”, Q.resolve()<\/span>\u00a0, as “seed” for this chain.<\/p>\n
Using this code, each next step in the reduce()<\/span>\u00a0\u00a0chain is only called when the previous step has been completed, as can be seen in the output:<\/p>\n
[14:22:35.935] [INFO] uploader - Starting upload: file1.jpg\r\n[14:22:39.814] [INFO] uploader - Completed upload: file1.jpg\r\n[14:22:39.815] [INFO] uploader - Starting upload: file2.txt\r\n[14:22:45.293] [INFO] uploader - Completed upload: file2.txt\r\n[14:22:45.293] [INFO] uploader - Starting upload: file3.pdf\r\n[14:22:48.657] [INFO] uploader - Completed upload: file3.pdf\r\n[14:22:48.658] [INFO] upload-test-3 - All files uploaded.<\/pre>\n
The code turns out to do\u00a0exactly what we want. The\u00a0above\u00a0reduce()<\/span>\u00a0\u00a0solution can be used to perform all kinds of promises sequentially, by inserting the right code in the callback to reduce()<\/span>\u00a0.<\/p>\n
However, there is one thing to add, which is error handling. What if one of the promises is rejected? Let’s try that. We’ll make a failing-uploader.js<\/span>\u00a0, which we’ll rig to\u00a0fail sometimes.<\/p>\n
var Q = require('q');\r\nvar log4js = require('log4js');\r\nvar logger = log4js.getLogger('failing-uploader');\r\n\r\nexports.uploadFile = function(filename) {\r\n    var deferred = Q.defer();\r\n    Q.fcall(function() {\r\n        var delay = Math.random() * 4000 + 3000;\r\n        logger.info(\"Starting upload: \" + filename);\r\n        setTimeout(function() {\r\n            if (filename === 'file2.txt') {\r\n                logger.error(\"Timeout while uploading: \" + filename);\r\n                return deferred.reject(\"Timeout while uploading: \" + filename);\r\n            }\r\n            else {\r\n                logger.info(\"Completed upload: \" + filename);\r\n                return deferred.resolve();\r\n            }\r\n        }, delay)\r\n    });\r\n    return deferred.promise;\r\n}\r\n<\/pre>\n
It turns out that an error stops the entire chain. When we modify Example 3 by changing require(‘uploader’)<\/span>\u00a0\u00a0to require(‘failing-uploader’)<\/span>\u00a0, and then running it, we get:<\/p>\n
[18:04:47.576] [INFO] failing-uploader - Starting upload: file1.jpg\r\n[18:04:53.896] [INFO] failing-uploader - Completed upload: file1.jpg\r\n[18:04:53.903] [INFO] failing-uploader - Starting upload: file2.txt\r\n[18:04:59.701] [ERROR] failing-uploader - Timeout while uploading: file2.txt\r\n[18:04:59.702] [ERROR] file-upload - Not all files uploaded: Timeout while uploading: file2.txt<\/pre>\n
This might be what you want, or maybe you want to just register the error while continuing uploading the other files. In that case, you need to modify the callback to reduce()<\/span>\u00a0, for example like this:<\/p>\n
var Q = require('q');\r\nvar log4js = require('log4js');\r\nvar logger = log4js.getLogger('upload-example-4');\r\nvar uploader = require('.\/failing-uploader');\r\n\r\nvar filenames = ['file1.jpg', 'file2.txt', 'file3.pdf'];\r\nvar results = [];\r\n\r\nvar lastPromise = filenames.reduce(function(promise, filename) {\r\n    return promise.then(function() {\r\n        results.push(true);\r\n        return uploader.uploadFile(filename);\r\n    })\r\n    .catch(function(error) {\r\n        results.push(false);\r\n        logger.error(\"Caught an error but continuing with the other uploads.\");\r\n    });\r\n}, Q.resolve());\r\n\r\nlastPromise\r\n    .then(function() {\r\n        \/\/ Remove the first result, which is <true> returned by\r\n        \/\/ the seed promise Q.resolve().\r\n        \/\/ This is a clumsy way of storing and retrieving the results.\r\n        \/\/ Suggestions for improvement welcome!\r\n        results.splice(0, 1);\r\n        logger.info(\"All files uploaded. Results:\");\r\n        logger.info(results);\r\n    })\r\n    .catch(function(error) {\r\n        logger.error(\"Not all files uploaded: \" + error);\r\n    });\r\n<\/pre>\n
 <\/p>\n
 <\/p>\n
This will catch the rejection a level deeper, so the chain can continue. The output of this code is:<\/p>\n
[18:15:55.659] [INFO] failing-uploader - Starting upload: file1.jpg\r\n[18:15:59.883] [INFO] failing-uploader - Completed upload: file1.jpg\r\n[18:15:59.884] [INFO] failing-uploader - Starting upload: file2.txt\r\n[18:16:05.279] [ERROR] failing-uploader - Timeout while uploading: file2.txt\r\n[18:16:05.279] [ERROR] file-upload - Caught an error but continuing with the other uploads.\r\n[18:16:05.279] [INFO] failing-uploader - Starting upload: file3.pdf\r\n[18:16:10.600] [INFO] failing-uploader - Completed upload: file3.pdf\r\n[18:16:10.601] [INFO] file-upload - All files uploaded. Results:\r\n[18:16:10.601] [INFO] file-upload - [ true, false, true ]<\/pre>\n
And indeed, the second upload fails, but the chain continues, and at the end, you know what succeeded and what failed.<\/p>\n
Our promises have been processed sequentially, with error catching, and continuing on errors anyway.<\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
This post describes how you can perform a sequence of promises sequentially – one after another – using Kriskowal’s Q library in node.js. If you’re just interested in how to do this, and not in the other examples, scroll down … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[176,177,10],"tags":[189,188,184,226,182,178,179,183,181,180,187,186,185],"class_list":["post-353","post","type-post","status-publish","format-standard","hentry","category-javascript","category-node-js","category-software-development","tag-array-of-promises","tag-javascript-2","tag-library","tag-node-js","tag-nodejs","tag-promise","tag-promises","tag-q","tag-sequential","tag-serial","tag-upload","tag-upload-file","tag-uploading-files-sequentially"],"_links":{"self":[{"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/posts\/353"}],"collection":[{"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/comments?post=353"}],"version-history":[{"count":17,"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/posts\/353\/revisions"}],"predecessor-version":[{"id":370,"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/posts\/353\/revisions\/370"}],"wp:attachment":[{"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/media?parent=353"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/categories?post=353"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/joost.vunderink.net\/blog\/wp-json\/wp\/v2\/tags?post=353"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}