Great to hear you’ve all had varying levels of success with Treejack.

Harry – you’re half right – we designed Treejack primarily for unmoderated testing. But, Treejack was born from moderated, in-person testing. The very first prototype was a stack of paper with a human “computer”!

As mentioned by others, Treejack and in fact all our tools, are always going to be supplementary to in-person testing – if you can afford the luxury of doing so. Think of it as the difference between running a survey or an interview. Interviews will always give you much richer insights, but surveys are a lot quicker and less time consuming.

Karl – your point about ensuring people don’t hop too much into a reflective mode is a critical one. Ensuring that participants don’t spend more time thinking than in the “real world” is central to the results’ validity. In any type of study, but particularly moderated sessions, “test conditions” are invariably introduced so reducing other forms of bias becomes critical.

There is also a randomisation feature in Treejack that no only mixes the order but can also present only a certain subset of tasks to present a participant. Use this to reduce the learning effect, as well as fewer tasks per participant.

Lastly, we are considering running a Treejack Q&A session. Anyone who is interested, just email me.

Thanks,
Sam

I’ve just completed a Treejack study as well, and I had several tasks which, in summary, look very successful.

However in post session questioning it became apparent that there was a process of either deduction or a click and hope for the best strategy at play.

I found participants didn’t understand several labels which they clicked on, leading to successful conclusions to their task; but these were chosen because other choices made even less sense.

The Treejack summary says this is a success, but I would say otherwise

I used a slightly different method to Nick – having users complete a series of 12 – 15 tasks before stopping and then going back over tasks which I thought required some more investigation. I didn’t want users to hop too much from an investigatory frame of mind, to a reflective mode between every task as I thought that this might lead them to be more and more analytical, or careful with each subsequent task.

Much of the value of Treejack lies in the time it saves you by building a prototype and capturing journeys, summarising data etc.

However, the results of an unmoderated treejack report could be misinterpreted without the qualitative component.

If you’re not present when participants are rumbling through your IA, you’ll be blind to false positives and could easily miss key insights.

E.g. if you had a dreaded ‘miscellaneous’ category and many participants ‘sucessfully’ found things within it, the treejack report would show that this is a strong part of the navigation.

Like many of these online tools, they can tell you what happened, but not why, and it’s the why that helps us.

How much value do you think you’d have got out of the study if you’d have gone down the unmoderated, remote route?