Stack Exchange
Fastest way to split object into smaller objects in Javascript?
I have a big object like below containing 1mil props.
{ 
"Name1":{"some":"object"},
"Name2":{"some":"object"},
"Name1000000":{"some":"object"}
}

I want to split this object into N parts so I wrote the following code


var bigObject = {
  "Name1": {
    "some": "object"
  },
  "Name2": {
    "some": "object"
  },
  "Name1000000": {
    "some": "object"
  }
};

const names = Object.keys(bigObject);
const partsCount = 4;
const parts = names
  .reduce((acc, name, idx) => {
    const reduceIndex = idx % partsCount;
    if (acc[reduceIndex] == null) {
      acc[reduceIndex] = {};
    }
    acc[reduceIndex][name] = request.body[name];
    return acc;
  }, new Array(Math.min(partsCount, names.length)));



While this works, the problem is with performance. Currently it's taking 1.2 to 1.5 seconds!! Is there any performant way to write this code? My expectation is that this should be hardly double digit milliseconds for new processors running at ~3 GHz. Is my expectation wrong?
Update:
I somewhat don't understand why people are looking for solutions outside of the given problem/question by asking "why" and "what" you are doing with this code. some have assumed it to be database use case. There are literally 1000s of use cases, anyway mine is IOT sensor data ingestion where max of 1 million sensors data is aggregated into a bulk REST call which needs to be processed, now here comes second issue as soon as i reveal the use-case people will try to optimize that, so let me be clear i cannot change anything apart for the code shown above Its already a parsed object now i have to divide it and process it.

You can always try it with a for loop and benchmark both methods.

I replaced reduce with a for loop and got 1.2 minimum of the range said above..vs i am expecting a 1millions object split 4 way to be done in like 99ms

Are you asking how split JSON or a JavaScript object? bigObject seems to be a JavaScript object, not JSON.

I assume from the code in the question above its evident i know how to split JSON, i am asking how to do it in performant way.. I am specifically talking of the JS Object, but you have any performant way to do it for JSON its just a method away so do tell me

If you want to split JSON, you should consider using a custom parser. If you want to split a JavaScript object, the problem is unrelated to JSON. I don't think parsing the whole JSON into a JavaScript object is always the most performant way. The parser has to read and interpret the JSON data. It can already split it.

can you tell me what is the custom parser that can do this speedily..? or you asking to write one for myself?

I'm asking if you want to split JSON or if you want to split a JavaScript object. If you want the most performant way to split JSON, you probably need a custom parser. If you want to split a JavaScript object, this question is unrelated to JSON.

I have a parsed JSON object (ie: js object) in memory so the parsing time is somewhere else not in the 1.2 seconds now i want to split this into smaller JS objects is there a speedier way to do it? but if you know of a much faster(les than 1.2 sec) parser to do this let me know i can always use stringified version of this..

I would try to write the code in a compiled language and run it from Node.js.

passing data time from and to this language will add up and i think will be equal to what i am getting...(gut feel)

reduce is written in a compiled language and it is much faster than a pure for loop. Don't trust your gut feel. You don't pass anything anywhere. You write a module in C++ and import it as shared library into the Node.js process.

what didn't understand reduce and for are c or c++ on V8 whats the difference between than, instead i think reduce does a lot more than for not sure what you are suggesting

I'm saying that logic in a script language is much slower than logic in a compiled language. You want the most performant solution and according to my experience you get much better performance with a compiled language. reduce does a lot more. If you want to do the same with pure JavaScript and a pure for loop you pay the price in performance. Logic in C++ is faster than logic in JavaScript.

@jabaa "reduce is written in a compiled language and it is much faster than a pure for loop." - no. reduce is (currently) written in Torque, which may be compiled, but it's actually more complicated than that. And that makes it no faster than a JIT-compiled for loop.

@Laukik What are you trying to achieve here? Why do you have a million properties in an object? Why do you want to split that into arbitrary subsets (given that objects are unordered)?

Is "why" really relevant? The reason i want to do this is cause i want to run parallel processing on it on N threads, i dont care of the order as long as they are complete object..

@Laukik Yes, it matters for us to see the big picture, otherwise we cannot make appropriate suggestions (see e.g. below the idea to distribute the splitting on multiple threads). So where does the object come from, how long does it take to originally create (parse?) it? And how do pass the parts into the worker threads?

@Bergi This is IOT data use case where sensor data in already aggregated into one giant object post this i have probability to change stuff. What i am asking is can this code be optimised further... i donot understand how worker threads need any more context than I have got a object which has 1 mil props i need to divide it equally between N threads, how its parsed who created the object in memory should be irrelevant is my guess here

@Laukik I guess you should try to use an array instead of an object if possible, it's much easier to manipulate. How the object was created in memory is relevant, since in the optimal solution you would avoid doing that entirely and only create the small(er) objects in each of the worker threads to begin with. In your case, it sounds like you cannot change the HTTP call that you need to serve; I would suggest you look into a streaming JSON parser and then distribute the individual properties onto the workers.

@Bergi i think there is one answer down below which converts it into array and does the same.. but its not fast enough, I am running this on MAC intel which runs on 3GHz and 25GB/sec ram speed this object hardly ocupies 100MB in RAM not sure what take this much time.

@Laukik I did not mean to convert the object into an array, but to have the http client send an array in the first place. But if you cannot change that, as I said, I would look into a streaming JSON parser.

Testing if (acc[reduceIndex] == null) in every iteration seems inefficient, and since it sometimes accesses properties that don't exist, might considerably slow down your code. Try to pre-populate the array:
const parts = names.reduce((acc, name, idx) => {
  acc[idx % partsCount][name] = request.body[name];
  return acc;
}, Array.from({length: Math.min(partsCount, names.length)}, () => ({})));


This takes 700ms, this may be the winner amongst all answers yet.

But my approach here is to use Object.entries to divide the object into an array of its properties.  Then divide those properties into N chunks using Array.slice, and then reassemble each chunk into an object using Object.fromEntries and finally push each object into an array of parts.
function partitionProperties(obj, partsCount) {
  const entries = Object.entries(obj);
  const parts = [];
  for (let i = 1, a = 0, b = 0; i <= partsCount; ++i) {
    a = b;
    b = i / partsCount * entries.length | 0;
    parts.push(Object.fromEntries(entries.slice(a, b)));
  }
  return parts;
}

It's important to acknowledge that, while this does divide the object into parts, it doesn't shuffle/arrange the properties into the same groups that you did.  You shuffled/distributed them into groups one at a time like the way you would deal cards: [{a, d, g}, {b, e, h}, {c, f, i}]  whereas this just partitions them [{a, b, c}, {d, e, f}, {g, h, i}].  But it's not clear that you were attached to the order or grouping.
I also don't know if this is the fastest, but it is relatively simple and straightforward, which has its own advantages.

This takes 1.5 seconds so worse than code in the problem statement... But thanks for help

I tried a lot of optimisation for the problem code right from sticking to one hidden class type in for v8 etc, profiling it i cannot take it down below 600ms
It seems Object.keys/Object.entries takes roughly 300ms, you can get some savings by pre-populating the parts array with 250000 props object but in total it was not under 100ms as i was originally thinking.
So i tried the array method as suggested in comments(change the bigObject to array of Objects) as code given below and i was able to split the whole thing in 73ms
Conclusion: Since i am not sure if i can change the original format to the new array format, but 1 thing is very clear stick to array in JS for any elements greater than 1K, I had my reasons to use object so that i don't get repeated prop names, but then its 73ms i guess merging them on server side would not be that bad i guess.
function main(names) {
    let parts = Array.from({ length: Math.min(partsCount, names.length) }, _ => []);
    for (let index = 0; index < names.length; index++) {
        parts[index % partsCount].push(names[index])
    }
    return parts;
}


const bigArray = Array.from({ length: 1000000 }, (_, idx) => ({ "tag": `Name${idx}`, "value": idx, "timestamp": date, "qualityh": idx, "qualityl": idx }));
const partsCount = 4;
console.time("B");
main(bigArray);
console.timeEnd("B");


You also might want to try a Map if you need unique names, it might perform better than an object when large.

Yes but its serialised version is JSON object for REST apis. which is what i was using.
