Use output documents of a first aggregation pipeline as input for a second aggregation pipeline

[yannispgs]

Hi !

I am trying to run an aggregation pipeline on one of my collection that has multiple stages. The purpose of it is to get an array of data, and then, depending on a condition, running (or not) another aggregation pipeline with this transformed data.

We already have an aggregation pipeline who does that by having a $facet as last stage, with one variable metrics that store all the documents as they are, and another variable tests that store the result of the second aggregation pipeline. This $facet pipeline only runs if certain conditions are met. That way, if the conditions are met, the second pipeline is run and we retrieve the results in both variables metrics and tests. Otherwise, we just return the output in the metrics variable.

The issue is that $facet load all the documents of every subpipeline in a single variable. My first pipeline can return dozens of thousands of results for some apps, thus triggering the error about the max size a single document can have (16MB).

To resolve that, I tried to run 2 aggregation pipelines sequentially, but I don't want to use a "temporary collection" to store the results of the first pipeline to put as input in the 2nd one (with $out or $merge). However, I did not find another way to do it with Mongoose. The $documents stage could work, but it is not supported by Mongoose...

Is there soemone who managed to achieve something similar ?

[vkarpov15]

What makes you think the $documents stage isn't supported by Mongoose? You should be able to use any MongoDB aggregation pipeline.

Another potential alternative would be to use $lookup if you're getting a bunch of data from a different collection and transforming it via a pipeline.

[yannispgs]

I get the same MongoServerError with my real usecase, and even in this minimal example :

const mongoose = require('mongoose');
require('dotenv').config();

const User = require('./models/User');

(async () => {
  // const conn1 = mongoose.createConnection(process.env.MONGO_URI1);
  await mongoose.connect(process.env.MONGOSTRING);

  console.log(await User.aggregate([
    {
      $documents: [
        { hello: 'world' },
      ]
    },
    {
      $match: {
        hello: 'world',
      }
    }
  ]));
})();

An almost-empty User Model is enough. I always get this error: MongoServerError: '$documents' can only be run with {aggregate: 1}.

I can make an Aggregation pipeline work with a $documents as first step only if I run it at a Database-level (as specified in the MongoDB $documents doc) with db.aggregate([]) in mongosh.

Is there a way to run Aggregations at the Database-level with Mongoose (maybe by creating the Aggregate with the unrecommended new Aggregate() method) ? It seems there is a way with the mongodb node Driver, but I don't want to use 2 different drivers on the same project.

[yannispgs]

If the $documents aggregation pipeline stage is supposed to work, what would be a minimal example to make it work ?

[yannispgs]

@vkarpov15 Here is a simple reproduction repository. Just need to yarn install and run node index.js (if you have a MongoDB instance locally setup)

[vkarpov15]

@yannispgs run mongoose.connection.aggregate() not TestModel.aggregate. MongoDB only supports $documents in db-level aggregate, not collection level aggregate.

[yannispgs]

That's what I was looking for. I could not find any information on 'database-level aggregation' examples.
Thanks for your answer