Yesterday a University of Maryland research team posted something different into r/PromptEngineering than the usual side-project flex. No landing page, no “look what I built” energy, just a straight ask for testers willing to break something in a controlled setting. u/LeoXzz, a researcher on the team, is running a study on whether their new observability tool for multi-agent systems actually helps people debug LangGraph and LangChain pipelines, or whether it just adds another dashboard nobody needed.
Here’s the twist. This isn’t a “come try my cool tool” post dressed up as research. It’s the other way around: real academic research that happens to be testing a tool. This Redditor said it straight: “‘No, it doesn’t’ is a perfectly good finding.” I’ll be honest, seeing that line from someone testing their own tool made me sit up a little. Most builders in this subreddit want you sold before you’ve even opened the tab, and this one is openly fine with the study proving the tool useless. That’s a different incentive structure than a normal launch post, and it changes how much you can trust what comes out the other end. A founder needs you to say yes. A researcher just needs an honest answer, even an inconvenient one.
Anyone who’s actually run a multi-agent pipeline knows why observability here is such a pain point. One agent calls another, that one kicks off a tool call, something loops back three steps later, and by the time something breaks you’re staring at a wall of logs trying to reconstruct what talked to what. Print statements and sticky notes only get you so far once there are five agents in the loop instead of one script. You end up scrolling through timestamps trying to figure out whether agent three actually received the output agent two thought it sent, or whether the whole thing silently retried and nobody noticed. That’s the exact failure mode that eats an afternoon, and it’s the one this study is aimed at.
The setup itself is easy to picture once you frame it that way. You work through a multi-agent pipeline the normal way first, whatever your current debugging process looks like. Then you run the same kind of task again with the tool in front of you. If you’ve used LangSmith or Langfuse, the shape is familiar: same traces and runs, just a different lens on what your agents are actually doing at each hop. One commenter on the thread said the study fit their own LangGraph work almost too well, tracking game data and player stats across multiple agents for sports infographics, which is exactly the kind of messy multi-agent setup this tool is built to make legible.
Here’s how the original poster laid out the whole study, step by step:
- 🧪 Fill out the roughly 2-minute screener to see if you qualify.
- 🔍 Get verified through GitHub or LinkedIn before they’ll schedule you. They’re checking real profiles, not just an email address.
- 🎥 Join a 75-minute Zoom session, recorded, think-aloud, working through structured tasks on a multi-agent pipeline.
- 📆 Spend about a week running the tool against your own actual LangGraph project, sending quick async feedback as you go.
- 🗣️ Close it out with a 30-minute follow-up interview.
Finish all three parts and you get a $150 gift card for your time.
Pro tip: don’t apply unless you’ve already got a real LangGraph project sitting around. The week-of-use portion needs something you can actually plug the tool into, not a toy repo you spin up the morning of. If you’ve got recent LangGraph or LangChain work sitting on a public GitHub, that’s exactly the profile this contributor’s team is looking for, and it makes the identity verification step painless too. Even a scrappy weekend project counts, as long as it’s got real multi-agent calls happening in it and not just a single chain pretending to be a pipeline.
Second pro tip: treat the think-aloud Zoom session like a real debugging pairing session. Say what confuses you out loud, even the dumb stuff you’d normally keep to yourself. That’s the actual data the researcher is after, not a polished performance for the recording. If a trace view confuses you for ten seconds before it clicks, say that out loud in the moment instead of smoothing it over after the fact. Researchers can’t code for a hesitation you never voiced.
Third pro tip: this is IRB-approved academic work, not a growth hack dressed up as a survey. That’s worth something. It means someone is accountable for how your data gets used, and questions get answered by an actual university researcher in the comments, not a support bot. It also explains the slower, heavier process, three parts instead of a one-click signup, since the study needs real behavioral data, not a five-star rating on an app store page. That extra friction is annoying if you just want to try a new tool, but it’s exactly what makes the eventual findings worth trusting.
Most tool builders never run a study like this before shipping. Actual users, actual money, actual willingness to publish “this didn’t move the needle” as a real result. Worth a look even if you’re just curious how a real observability tool for multi-agent systems gets road-tested before anyone tries to sell it to you. Head over to the original Reddit thread for the screener link and to see what other LangGraph and LangChain devs are already asking in the comments.
Frequently Asked Questions
Q: Do I need public GitHub repos to participate?
No, your repos can be private. The screener form asks for your GitHub to verify your background, but you can mention if your work is in private repos and just describe your LangGraph experience. If you’re unsure what they need, email zxu169@umd.edu.
Q: Can I take breaks during the 75-minute Zoom session, especially if I have accessibility needs?
Absolutely. This is IRB-approved academic research, and the team should be prepared to accommodate breaks and accessibility requests. Mention your needs when you schedule, or reach out to the researchers beforehand to arrange adjustments.
Q: What if I can’t complete all three parts of the study?
The $150 compensation is for completing the full study: the initial 75-minute session, a week of independent use, and the 30-minute follow-up. If your schedule is uncertain, ask about flexibility during the screener, but expect participation to be required for full compensation.
Paid UMD research study: help us test a new observability tool for multi-agent systems (LangGraph/LangChain devs, 75-min session)
by u/LeoXzz in PromptEngineering