Holly Hester-Reilly is the founder and CEO of H2R Product Science, where she combines more than a decade of experience in technology companies with years of experience in scientific research to bring her clients a rigorous, research driven approach to developing their growth strategy and coaching their product teams.
In this conversation, Holly gives a masterclass addressing one of the biggest challenges we face: how to build a team and organization that is deeply connected to the people they seek to serve on an ongoing basis. In product management parlance, this is called continuous discovery or dual-track product discovery.
Even if you don’t work in a tech company or in product management, there is a lot of actionable and principled wisdom here that I believe is going to make an immediate and lasting impact on how you and your organization make the contribution you seek to make.
SCROLL BELOW FOR LINKS AND SHOW NOTES
- Twitter @h2rproductsci
In this episode, we mentioned the following resources:
- Holly’s (@h2rproductsci) company — H2R Product Science
- K Anders Ericsson – Peak (book)
- Deliberate Practice
- Marty Cagan — book, blog, trainings
- Related ENLIVEN episodes
- Outcomes-based roadmap
- Agile software development
- Mythical Man Month
- The Cathedral and the Bazaar
- Thanks for the Feedback
- Fight Song – Rachel Platten
- Physical (song) – Dua Lipa
- Empathy mapping
- Impostor syndrome
Transcripts may contain a few typos. With some episodes lasting 2+ hours, it can be difficult to catch minor errors. Enjoy!
Andrew 00:02:56 Holly, welcome to the show. Thank you so much for taking the time, be here and share this conversation with me.
Holly 00:03:04 Yes. Well, thank you so much, Andrew. I’m happy to be here.
Andrew 00:03:07 Yeah, this is going to be really fun. And as I was saying before we hit record, I’ve become a big fan of your work and in particular, your podcast, which we’ll, I’m sure we’ll jam about a little bit later in the conversation over the last six months. Uh, so it’s, it’s such a treat for me to get to have you come on the show. So really thrilled to have you here. One thing I always like to start a little bit more on the personal side and I came across that you would used to be competitive figure skater, and I was hoping you could tell me, like, is that, is that something you still do? Or like, how does that, does that play a role in your life? Still? Tell me about that.
Holly 00:03:33 Oh my God. I’m good. Good research job, Andrew. So yeah, I grew up, uh, in Massachusetts. And, uh, when I was a kid, when I was a younger kid, I had done like gymnastics and dance and then, Kristi Yamaguchi won the Olympics. And I just watched that and I was like, Oh my God, that was like the most amazing sport ever like, mom, please take me to the rink. I want to go skating Krissy. I’m a Gucci, when’s the Olympics. You know, she like goes Disneyland and tries to like eat or metal, you know, like this thing that would figure skaters to, I don’t know. And, uh, and I was like, take me to the rink. I want to try it. And I was, I was nine at the time, which is actually, They would, they would call that old to start competitive figure skating.
Holly 00:04:16 But because I had been a gymnast before, You know, I was, I had been stretching and practicing all sorts of limber things and athletic things. And so, So I was able to transition pretty easily. And by the time I was like 10, like within a year later I had quit every other sport and was just like only figure skating. That’s my thing. And I was skating like five days a week. By the time I was like 10 and, You know, like fast forward, By the time I got to eighth grade, I actually changed schools so that I could, Be close to the rank I was training at. And I trained at this rank at the time in marble Massachusetts, where we had a bunch of Olympians. So like I grew up with Olympians around me every day. So we had, uh, Elliot Kula who was the first man to land a quadruple jump in the Olympics from Russia. Cause there was this Russian coach Tatyana Tarasova um,
Andrew 00:05:02 Can you, without, without you were just in like the little American Mecca of figure skating basically.
Holly 00:05:07 Well, there’s several of them, but I was in one of them. I mean like Massachusetts at the time, like Nancy Kerrigan lived in Massachusetts and there were these coaches on the Cape, That, you know, coach these other Olympians, there were many like Olympians who had grown up in Massachusetts. And so it was like one of the places where, you know, becoming a world level figure skater seemed like a totally like not totally normal, but like something that you see people around you doing. And you’re like, Oh, I could do that. And then there’s lots of rings of practice that there’s lots of, you know, Local ranks and then they feed into the regional ranks and then, then they feed into like national training centers and, And it was at the peak of the sport. So most of my time training, Michelle Kwan was just like the dominant force. And I, excuse me, I loved Michelle Kwan. She was my favorite because not only was she a beautiful skater, but she also was a really great sport. And in the sport of figure skating, they teach you a lot about, or they, or they judge you a lot on whether you are a good sport, whether you present yourself properly off the ice and whether that they would want to represent the country. Oh yeah.
Andrew 00:06:10 I didn’t know. So there’s, there’s as much focus on like what you do off the ice as there is what you do on the ice.
Holly 00:06:15 Yeah. So what I would say is like, if you haven’t seen it, Tanya, and you’re curious about what figure skating is, will be like, go see I, Tonya, it’s real, it’s the most real portrayal of the figure skating world I’ve ever seen. And, And it’s, uh, it’s a, it’s a dark movie,
Andrew 00:06:35 Like it’s dark, but it’s real.
Holly 00:06:37 So this was, this was like, I mean transformative for me and I, to be honest to this day, it’s a part of who I am and it, and, You know, I went through my own sort of love affair and then, and then betrayal from the sport and didn’t do it for a while. And then, you know, as an adult came back to it and figured out how I, How I could just enjoy it, just enjoy what I can do and not accept what I can’t do and be fine with that. And now I have this great relationship with it where I use it for athletic, you know, I just leave it for exercise. And, I love to go to the bank and feel the way the ice feels on my feet. It feels like going home. And it’s one of the things that I’ve missed. Like I was skating this winter. I, I love when winter comes because I can go skate outside of New York city where there’s so many beautiful outdoor rinks and now COVID so,
Andrew 00:07:22 But I think a lot of people would probably have this experience where, you know, we’ve watched skaters on TV and you’re like, wow, it looks so they made like, they make it look so graceful and so easy, but I know it’s anything, but, and I’m curious, what does it actually feel like? Like, I dunno, I don’t know if it’s triple axle or I don’t know, move that I know, but to do a jump and spin, like, like what does that actually feel like and what does it take to do that?
Holly 00:07:45 Oh my God, it takes falling all the time. Like this is one of the things like when I was like, I have all these lessons from skating that I do apply to, you know, my day to day life, these days. And one of them is like, I literally, I’m not afraid of failure because I had to fall like a hundred times a day, every day for, and I was competitive for about seven years. I retired at, Just before I turned 17 and a lot of false, like 10,000 balls, you know what I mean?
Andrew 00:08:10 Thousands and thousands of balls. Lots of cold, hard ice.
Holly 00:08:15 Yeah. Like at one point I dropped my tailbone and then I like, could barely walk, like definitely. Yeah. Lots of, you know, lots of bruises, lots of like falling in cold puddles and then being like, well, we’re still here. We got to keep going, like just going through it. And so it takes, it just takes like tons and tons of practice, but then it also takes like, I mean, it takes the dedication, which for those of us who are in the middle of it, like we kind of all take it for granted. I think that like, of course we’re all dedicated. We’re here every day and like, we’re doing this, but once you kind of step outside and like leave the skating world, you watch what other people do. And you’re like, Oh, okay. I see why they said I was dedicated
Andrew 00:08:48 Big, wait, wait, other people don’t get up at 5:00 AM on Saturday to go skate. What the hell?
Holly 00:08:53 Exactly. They’re not all at the, by, Like 7:30 AM for a ballet class. And then, uh, you know, get on the ice and spend their morning like falling on their butt and the coal.
Andrew 00:09:03 That’s not what they do.
Holly 00:09:06 There’s so many things I’ve taken from it. And, But on the other side, like, it feels amazing when you learn a new trick. So you learn a new skill. Like you learn how to do a different kind of span or a different kind of jump than you’ve done before. For me, My, uh, the, the place that I got to was in my final years, I was training with these coaches, Mark Mitchell and Peter Johanson who teach the skin called with Boston and, and, Peter was an Olympian and Mark just barely missed the Olympics for the U S and, Training with them was just an honor. It was amazing. I’m so grateful that I got to train with them. And when I was working with them, I started landing double axles and, Landed a triple. And that’s the point at which it starts to actually feel like you’re like what you kind of imagined. It would be like, like when you do like a single jump, it doesn’t really feel like you’re spinning in the air. Like, cause the spin is so short when you do a double jump, like, it feels a little bit like you’re spinning in the air, but like, it’s also kind of quick, but once you start doing double axle and triples, then you’re like, yeah, no, I totally just jumped in the air, like spun a bunch of times and landed on like,
Andrew 00:10:04 Right. And I’m totally cool with that. Just be like a badass
Holly 00:10:08 It’s. Yeah. It’s kind of intoxicating.
Andrew 00:10:11 That is amazing. I love that. Are you, Are you familiar at all with, uh, Anders, Ericsson? No, I’m not. No. Does the term deliberate practice, are you familiar with that word?
Holly 00:10:21 I mean, I can imagine, but I don’t know the I’m not used to it, so enlighten me.
Andrew 00:10:25 So this is the guy who’s considered, I think pretty much he’s like the global expert on expertise. So he’s the person who like Malcolm Gladwell made popularized the 10,000 hour rule, which has since been like, okay, he kind of presented it wrong, but yeah, Andrew’s Eric K Anders Ericsson. He’s the person whose research Gladwell was referencing. And so he’s the, he’s the person who has done, I don’t know, decades at this point of research on basically how do humans get great at stuff? And yeah. So deliberate practices is a term he coined to describe a very particular kind of practice, which is what you just described, doing many, many, many hours of, of, you know, you’re at your, you’re working at your limit, you’re at your edge. It’s like constant feedback and it’s tough and it’s gritty. And like a lot of days it sucks and you just have to keep pushing and it’s like, you know, it’s so exhausting. Cause you’re so at your edge, but that’s actually the pathway towards mastery. And so it seems like one of the things that you acquired or developed rather, it’s probably a more accurate term without maybe realizing you did was the capacity to do that kind of work to do that kind of practice for a long time.
Holly 00:11:31 Yeah, I do think I did. And it’s something I’ve reflected on, you know, in the past, basically since I started training other people in product management, And going and working with people in business because frankly having grown up in that environment, it was surprising to me how uncomfortable with failure so many people are and how uncomfortable with like that, that uncomfortable place. Like it’s not, I’m not going to, like, I love this barrier. Riley has this phrase about like getting comfortable with being uncomfortable. Right. And like, that’s the thing. Like, I, I, I was trained from a young age to believe that if I wasn’t uncomfortable while I was practicing something I wasn’t growing and that wasn’t worth the effort. And You know, so now I, yeah, I did. I’m so grateful, you know, and I mean, I was always grateful. I knew there were a lot of elements of it that I, that I was just very privileged to be able to do and to have a part in. And, I just like, I can’t, I can’t stress enough how it’s affected my mindset and the way that I act as a professional, because, I’m used to being in that uncomfortable edge spot and that’s, that’s the place where you either fall or you land it, but at least you tried, you know?
Andrew 00:12:39 Yeah. It’s like that, that edge it’s like home-based for you now. Yeah, exactly. That’s, that’s so cool. I want to shift gears now and start to talk about product and product discovery and a lot of the stuff you’re working on for those who aren’t familiar with that term, uh, particularly the, the, the framing of it is continuous discovery. How would you explain that to someone who’s not familiar with that term? What does that, what does that mean?
Holly 00:12:57 Yeah, so continuous product discovery, That is basically the practice of continually doing, uh, doing some kind of research, every single sprint every single week. At least more frequently than, you know, just trying to make it as frequent as you can. I think the ultimate goal is that it’s every week, If teams aren’t there that’s okay. But, Doing it regularly in small batches so that, You’re continually testing, you know, what is it that I should be building and what are the nuances of how I should build it to make sure that it works for our user and our buyer and our customers. And then every time that you have, Important decisions to make, you can turn to the research to answer those decisions in near real time. So basically, This is the big difference between doing continuous product discovery and just doing product discovery in batches is that, You, uh, you actually can say to yourself, well, what is the biggest decision I have to make in the next two weeks?
Holly 00:13:57 That’s what I should do my research on in the next two weeks. And that way you don’t have to, you don’t have to guess when, if you’re doing batch as a contrast, if you’re doing a batch set of research, like you’re doing some big ethnography, You know, once a year or maybe even once a quarter, At the beginning of it, you’re guessing what are all the things I think I need to know? And then, You know, you get a bunch of helpful stuff and you start making decisions based off of it. But once you’re like, say a month, two months down, I guarantee you, you’re gonna come across questions that are decisions you have to make where your ethnography does not give you the answers you needed. And so this is, you know, if you stretch it out and you do it continuously, you’ll, you’ll have that at your fingertips.
Andrew 00:14:35 Yeah. So this, this is one of those things where I became familiar with the term continuous discovery a couple years ago. And it sounds like you and I both had a similar, uh, powerful experience, uh, from a man named Marty Cagan. So he’s someone who plays a huge role in my own career arc, which I didn’t even fully realize until a couple of years ago. But what was that experience for you? And let’s use that as a jumping off point.
Holly 00:14:58 Oh man. Well, um,
Andrew 00:15:00 I got out to Marty by the way, father of all this stuff,
Holly 00:15:03 Shout out to Marty. So for me, I, uh, I first was influenced by Marty. And I actually, I would say, I didn’t even know it at the time, but when I went, The first high growth startup I was in is a company called media math. It’s an ad tech company here in New York city. And, When I got the job at media math, uh, I was somewhere around like the fifth product manager there and,
Andrew 00:15:27 And it was like 150 ish people when you started, I think,
Holly 00:15:29 Yeah, exactly. I think I was like number one 43. Okay. So, uh, what I didn’t know is that, That they had been influenced heavily by Marty, Going back years. And so, uh, I had chosen them, Out of the, or the options I had largely because on, this is going to sound funny at this point in time, but largely because they practiced the scrum. And at the time I had read about agile and scrum and I had for at this point in time, I, I had never been in a company larger than 10 people, Doing stuff.
Andrew 00:16:05 What, sorry, just really quick for anyone who’s not familiar with the terms from what is that?
Holly 00:16:08 Oh yeah. So scrum is a version of agile software development, With a particular flavor of it where you have, uh, certain rules you follow, like you have sprint at the beginning of the sprint, you have a sprint planning meeting. At the end of the sprint, you have a, a, a, they call it a tip in the literature it’s called a sprint review, but most teams just do the sprint demo. And, uh, you have daily standups. And so there’s a couple of key elements that are involved in this, but basically it’s a particular architecture for, The design of how you develop software. And, One of the other things that you and I haven’t mentioned this call yet is, My actual, Training before I moved into software is as a chemical engineer. And so when I started learning about methodologies for developing software, I always sort of like applied this process development, Mindset to it and looked at like what makes sense to me. So at the point in time, when I got to media mass, I had read a lot of books from, Safari, which is a, the Tim O’Reilly’s online. Uh, there was a lot of the media online. And so I had read, like, I remember the cathedral and the bizarre, the mythical man month, Like agile software, product management, you know, various things like that,
Andrew 00:17:20 The foundational kind of classics there.
Holly 00:17:21 Exactly. Yeah. That’s, you know, that’s how I got my, you know, my, uh, what I’ll call my honorary computer science background. But I, So I had, I came in with this philosophy that we were supposed to be regular, that the reason why the work was split into sprints was so that we could inspect and adapt and figure out what made the most sense for, for customers, not just what made the most sense for engineers. What I didn’t know at the time is because I’d been at some small companies where it was really easy for me to influence everything. I didn’t know how rare that was, but, uh, it actually was really rare. And when I picked a medium math, I picked it because, They already had moved to scrum. And the year at this time, I want to say it was 2012 and not, I mean, all the other companies I’d interviewed with that were startups in New York were not doing scrum.
Holly 00:18:10 They were doing agile, but it was a much more, Well, they weren’t necessarily all doing agile. They basically didn’t have as much structure around their process. So I share all of that because I’m, before I even ever met Marty Cagan for the first time he was already influencing what was going on around me, because the company I was at was, Trying to be, uh, practicing, you know, true agile with, With dual track sprints, To some degree, uh, now I will add to some degree because, You know, we definitely upped it, but the fact that there was even the appetite there, you know, there was not a baseline assumption that we should be planning things out annually, Is actually something that I think, uh, you know, I think we can attribute back to having been influenced by Marty from the beginning.
Holly 00:18:53 He had, Trained, I think the COO at the time and, and such, And the, the head of product, a guy named Dave . He was the VP of product. When I got there, he had all of his new product managers go to Martins training. So within my first year there, I went out to Marty’s training and, And I was just like, this is fantastic. I think one thing I hear a lot is a lot of people go to the training and they’re like, Oh my God, I’ve just learned everything I did I’ve been doing is wrong. Um,
Andrew 00:19:19 Yeah, when I went, I had a similar, I talked to a lot of people who were just like, they had their head in their hands and they’re like, Oh God.
Holly 00:19:24 Yeah, exactly. I didn’t have that. I had more, like, here’s all the things that I like. Here’s so many more things I could be doing. That would be really great. Like, let me also build this. Let me also build that. Let me also build that
Andrew 00:19:37 As a quick aside for anyone who just listened to me say that that is in no way to say that going to Marty’s training is a bad idea. It’s amazing. And if you ever get the chance to go, you should absolutely go, Oh, God thing was just people realizing there’s such a better way available. And they have been, they weren’t aware of it. Yeah.
Holly 00:19:53 And to add onto that, It’s specifically, if you already believe in product discovery, but your colleagues don’t, or you’re constantly fighting other people about it, get them to come with you, bring the CTO or the VP of engineering or whatever. So that they can hear it from him directly. That would be, it’s a really good way to level up your team.
Andrew 00:20:11 Totally. So, okay. So you’re at media math and you’re without realizing it, you sort of, you’re swimming in a pool. That’s already framed into this type of working, right. If this is already in the DNA of the organization. So, Let’s for people who aren’t familiar with us and haven’t had the privilege to go to something like Marty’s training, What does that actually look like mean if I was, you know, a fly on the wall and I’m watching the way you were doing it at media math and the way you’re doing it now, what would I see? That’s different than the way let’s just say a typical company is approaching doing their product research.
Holly 00:20:43 I’m going to start with the basics that you see in most, most places that are trying to practice agile, which is you’re going to see the things that I talked about with the scrRight? You’re going to see sprint planning meetings. The thing that to me is a big indicator of whether this company is doing any form of continuous product discovery is, is what’s actually happening in those meetings. What’s actually happening in the sprint planning. What’s actually happening in the retrospective, you know, are they actually practicing a sprint demo and what’s happening in the demo or how are they communicating? What’s been worked on and what’s happening with their learning. So the thing that is really key is that, In a company like this, uh, when the development team gets together and meets with the product manager and the designer and anybody, you know, there, if they have a scrum master or a project manager, what have you released manager when they get together and meet with those people?
Holly 00:21:33 Every sprint, they’re not just following some plan that’s been laid out for months. That’s the big difference they’re actually getting together and saying, Hey, what is the most effective thing for us to do next? And in the most well-run companies, there is a thing that’s been laid out for months. And that’s the strategy. The strategy has been laid out. The vision has been laid out, but the specifics of what the software team is going to do are not laid out months in advance. And they’re definitely not laid out like six quarter six. You know, it’s not like an annual roadmap, right. Would actually features on the roadmap. What I mean by that, just to dive a little deeper is you would get into a meeting. And let’s say, uh, in my time at media math, the first place where we were starting to negotiate, what was coming up next was not the sprint planning, but rather the backlog grooming meeting, which comes before sprint planning.
Holly 00:22:20 So we would go to this meeting, this backlog grooming meeting, I’ll get into a minute. There’s actually one step before that we were doing too, but you get into this backlog grooming meeting. And as the product manager, you’re at that meeting, communicating to the engineers at this point in time, our best understanding of what’s the most important work for us to do next is this set of things. Now let’s talk about how we can go about breaking that into pieces, writing stories, for edit your acceptance criteria, You know, clearing up what it means to do this set of things. And in this meeting, we’ll just start to, we’ll just make sure that we’ve got enough of that scoped out that we have, you know, a sprints worth of work to do from it finished for disability, like two weeks on average. Yes, exactly.
Holly 00:22:57 So at medium aspirins for two weeks, I will say at this point in my life, I definitely do a lot of one week sprints. I think those are awesome when possible. Whether they’re possible, depends a lot on the organization design and the size of the organization. One of the other things I will add here is like when I was at media math and other companies I’ve been at too, We also would have a step either before or after backlog grooming where the product managers were meeting with other product managers. So basically some, some point along the way, which often gets missed in larger organizations where we’re actually thinking up and saying, here’s the chunk of work I was thinking my team would do in the next sprint, or that we’re going to start breaking apart into pieces. This is what it entails. How does it jive with what you all are working on? Does it step on any toes? Does it break anything? Does it go towards the same strategy? Are we going to align really well? That’s something that we would also do every sprint, not like once a quarter. And these are things that I’ve seen done, you know, once a quarter or less at some other companies I’ve worked with since then. And so I need to call out that that’s a, that’s a piece of it too.
Andrew 00:23:55 It sounds like one of the underlying principles in what you’re describing so far is this shift from sort of big heavyweight processes that happen once in a while to sort of really lightweight things that are just happening all the time. What do you think about that?
Holly 00:24:07 Yeah, I totally agree. I think that that is where companies need to go. I think, uh, I was blessed at medium asset. I’d never saw the heavyweight process there in the first place. It started to come in later as the company grew and they were trying to figure out how they work as a larger once we were up to like 800 people, there started to be more of those. And, You know, that definitely affected the way the company operated, but most companies that I work with it is a shift. It’s a shift from that to something else. And that’s really scary for, especially for the executives and the people who are used to reporting plans. And that’s one of the things that I think a lot of product people miss when they are trying to push, if you’re trying to be the change agent, who’s like, no, we need to do continuous product discovery. You know, I think Marty says in one of his articles about, Roadmaps, uh, that you, you like in the alternative to roadmaps, I think is the article. We can say that a feature based roadmap, isn’t helping the product team and it’s getting us into all sorts of trouble, but we also can’t just like, just say that and then like drop Mike and leave the room, right?
Andrew 00:25:06 Whether you like them or not, that do exist for some reason. So,
Holly 00:25:09 So you have to go to the underlying reasons and figure out what about that is giving you comfort or stability or, you know, value real value to the people who are consuming that roadmap, which is typically the executives and the, maybe the board, Sometimes the customers or their prospective sales. And how can you still meet those needs without committing to features ahead of time. And that’s, uh, that’s, that’s what we do on my teams I work with now, Is we do that with an outcomes based roadmap, with a clear strategy, with a clear vision, you know, selling the vision, selling the strategy, not selling the features.
Andrew 00:25:42 It seems like, cause I’ve been, I’ve been, uh, the person in the organization I work with now for a couple of years sort of being that change agent and pushing things and saying, Hey, there’s a better way we can do this. Like, there’s nothing necessarily wrong with what we’re doing now, but look, what else is available? Look how much better we could be doing these things and how much more value we can be creating. And I’m curious, you know, one of the, one of the mistakes I made was thinking that any sort of change like that would happen quickly, which just, I could not have been more wrong about that in, in every possible way. That has been extremely humbling for me. But one of the things I’m curious about is, is so if anyone, you know, people listening to this, they’re going to be walking out of this conversation, walking out of listening to this conversation with a lot of really cool ideas by the time we’re done. And in terms of how they go about implementing them, how would you recommend they like frame that for themselves mentally and psychologically? Like, are there levels to this? Like where do they start? And, you know, cause I think it’s easy to sneak, you see the city on the Hill and you’re like, Oh great. We’ll just go there. And then it doesn’t work. So how do, how do we actually like step our way there and what, what does that look like? Yeah. Oh, that’s an actual, an
Holly 00:26:44 Awesome question. So I do actually see there as being levels to this because I’ve been teaching people how to transition to con to continuous product discovery for so long that I’ve now kind of got like a I’ve realized patterns and what I recommend they do next. Let me start with, Starting small is always good. Because it is gonna take a long time. It’s not going to be a fast change, even if you are lucky enough to convince high level people that you need to make these changes, which often is one of the hard parts you still need to account for the fact that everybody within the organization, who’s going to learn to work this way, needs time to adjust to the changes. You know, it’s, it’s never going to be fast. It’s definitely going to be on the order of months at the shortest.
Holly 00:27:25 And in many cases, it’s going to be a process of continuously getting better at this and not feeling like it’s really optimized for more, more, much more than that, even. So as far as steps, I actually have put a lot of thought into this and I’ve developed something. I call the product science success path. So I have, Laid out five steps where the first step is, Basically a team that’s doing agile product development. They’re delivering software in a at least somewhat continuous fashion. You know, the most ideal version is that they’re actually pushing code multiple times a day to production and they’re using, you know, feature flags and all these things. But even if they’re just pushing it every sprint, at least that that’s step one. So if agile is sort of table stakes here, exactly. Like if you’re not even practicing, like if your team cannot deliver software, at least once a sprint, then you’ve got to fix that first.
Holly 00:28:10 You can work on doing continuous discovery, but like you’re gonna, it’s not going to have a lot of impact because your team is delivering software. So rarely they can’t pivot very well. So that’s like table stakes. Once you’ve got that, then the next step up is doing some form of continuous research, but it might not yet be regularly impacting the work that you do. Like a lot of times teams will start with like, let’s just get faster and more frequent. And how often we talk to customers or how often we look at data. And one of the things that we often see at this stage is that teams that do one or the other, but not both. So, A really important thing to keep in mind is that continuous product discovery is both quantitative and qualitative. It is not only one or the other.
Andrew 00:28:52 There’s a great webinar that you all did recently that we’ll link to in the show notes about how to basically how to do this kind of continuous rapid research and discovery, especially in uncertain weird times, like we’re in right now during the COVID quarantines globally. And I think the way you explained it in there was that one way of thinking about like the difference between qualitative and quantitative was that qualitative, was it sort of it’s, it’s an answers why things happen and how they happen. Whereas quantitative is a way of answering questions of like how much is happening and how many are of a thing are happening.
Holly 00:29:23 Yeah, exactly. Exactly. One thing I want to take a step back and say, is that, In terms of moving from one thing to the next, I kind of jumped over a step, which is there are some teams that do agile product development and don’t even do, Staged, bashed discovery. Like, you know, I don’t know if you’ve talked to teams like this, but, Basically the way I would describe that as like a team that’s doing agile product development and maybe like strategy is set somewhere else in the company entirely. And like the research, like the product manager, isn’t actually talking to customers. So, I do have a step in between there, which, I basically call like a basic level of product discovery practitioners. And at that step, They’re doing that discovery at least, but it might be, but it’s probably in a batch and not continuous.
Andrew 00:30:05 Got it. So it seems like the shift there is, so it’s table-stakes is, is you gotta be doing some form of agile and being able to release release product on an ongoing basis without, without the sort of big bang release. And it seems like then there’s another, another kind of you table-stakes factor here, which is that if your product teams are really just sort of, I think that the somewhat, Pejorative term for it is like a feature factory where someone else has done all the thinking and you’re just delivering a then, then that is, it seems like it’s the next table-stakes shift. Is that, am I getting that right?
Holly 00:30:35 Yeah. And basically if you’re, if you’re currently a feature factory as you just put it, And you’re not doing any of the thinking yourself, then you’re going to find it really hard to try to go straight from that to continuous, both qualitative and quantitative. It’s a big step just to learn how to talk to customers and how to interpret the research. We, when we work with teams, we, We do take it easy, like, right. Like I don’t mean like slacking off. I just mean, you know, one step at a time first, first we’ve got to make sure you know how to do some research. And then after we’ve got you doing research, we can talk about how we make it filter into the work you’re doing regularly. And, and one of the reasons why we do that is, uh, I’ll tell you, I have worked with teams that are feature factories and are trying, and they’ve got a product manager who’s trying to transition to continuous discovery. And a lot of times, There’s this huge learning curve for the engineers to come to this realization that their job isn’t only to deliver the features that have been asked of them. Yup. Yeah. It’s a tough one.
Andrew 00:31:33 Tell me a story about how about when you’ve seen an engineer go through that transition? Like what, what got that engineer over the hurdle? What made it cool?
Holly 00:31:41 Yeah. So the most common way that I’ve seen this actually click is, is to be honest at a company that’s in trouble. So when the company isn’t, Just, you know, everything’s growing and we’re all excited, but instead, Maybe the product itself, the performance is in trouble in terms of what impact it drives to the business. And the reason why the Exxon was commonplace for me to see this is because that’s the place where the engineer actually starts to fear for their job. And if the engineers are seeing that, Hey, if we don’t make this have a bigger impact on the business, then we’re not going to be able to work on this product anymore. That’s when they start to realize that they do actually care what they’re building. So most often I’ve seen that on, you know, something where it’s like, maybe it’s either a startup within an enterprise or it’s, A new product team, a team that’s launching a new product.
Holly 00:32:30 And so that team is in the riskier stage of like, is this product going to survive? And they’re either worried about their actual job or they’re worried about having to go back to the main product. Isn’t as exciting for them as the new one. I would say I’ve also seen it happen, but it’s harder in like I’m dying industries. So like media, you know, Like newspapers. Yeah, exactly. You know, being a part of that, being a part of something that’s been hasn’t yet really been disrupted by, Modern tech. But where they do see that that is, Coming. And the way that I’ve seen that transformation happen is basically, uh, in courses that I’ve taught. I actually, uh, you know, you mentioned at some point something about, well, the way we’re doing it now isn’t necessarily, you know, bad or wrong, but there’s a better way. I, after many, many experiences, not convincing people, Got to a place where I no longer will say that. I will say, Nope, this is wrong. I’m going to tell you,
Andrew 00:33:29 I love it. I’m gonna do it your way. Now, if this, if this works better, I’m stealing your playbook.
Holly 00:33:33 I mean, it’s, it’s emotionally difficult for everybody, but I will tell them failure stories, Because I’ve seen that I’ve lived through them and if they’re not scared, it’s probably because they haven’t yet. And the whole point of me being there to train is to bring the painful experiences that I’ve had to the classroom so that the other people can learn before going through them all themselves. Sure. So typically the next, most successful way for me to convince an engineer, who’s like, no, I really just want to shift features. I just want to ship code. Can you just measure me on the number of commits I make?
Andrew 00:34:03 Just let me just, let me leave me alone. Let me just put my headphones on and code. Okay.
Holly 00:34:06 Exactly. Is for me to really empathize with them to start to say, look, I went to engineering school too. I, I like to sit with a spreadsheet or, you know, like a doc and just work on it and not be interrupted and not have to think about the other things. But at the end of the day, the reason I’m doing that is to make an impact on the company. And here’s some stories I have from my past about times when that was missed and, and the whole company died. So my really big story around that is that I was actually in, towards the rest babies R us consulting, For a couple of months around when they filed for bankruptcy. So before they filed, they were looking, you know, like inside the company, we all knew the numbers we saw what was happening. Year over year, we were looking at e-commerce versus, you know, brick and mortar sales and Amazon was just eating their lunch entirely. Yep.
Andrew 00:34:54 So, so there was definitely an incentive to change. Like people were feeling the pain,
Holly 00:34:58 People were feeling the pain. But even with that, there were a lot of engineers there. They had only just moved to agile. I started working with them with January of 2017 and they had just moved to agile, like the fall before. And I say, move to agile, like generously,
Andrew 00:35:13 They had an intention to move to LA.
Holly 00:35:15 Right. Or they thought they had moved to agile. But the thing, the big, the big thing that I, that I mean is, There’s a lot of companies out there that just break the work into chunks, but they’re still planning a year in advance and that’s not true agile. Um,
Andrew 00:35:31 Can we just also call out if anyone here is working in an enterprise that got, what’s it called a safe, the scaled agile framework, that’s bullshit. That is not actually agile and you should go running for the Hills or screaming or something.
Holly 00:35:41 Yeah, totally. No, if you’re in that situation, you should start doing some research on how either, how to convince people or what’s gonna happen if at all, doesn’t work out and maybe start dusting off your resume. Yeah, exactly. Sadly it right. But it’s true. I wish that wasn’t true. But yeah,
Andrew 00:35:57 Toys R us, they had an intention to move to agile. Everyone sees the writing on the wall, then what happened?
Holly 00:36:02 So at the time that I started working with them, they had brought in a bunch of people who knew product discovery. They had someone from Amazon, someone from Google, someone from Netflix, like they literally brought in some big guns in the product world. And, you know, we’re looking to add the dual track, agile, You know, meaning doing product discovery and product delivery at the same time, To the, what they worked. But, What I saw was that, uh, it takes a lot longer, you know, to actually get that in place than the amount of time that they had left before they were going to die. One of the challenges was that the, The engineering teams were we’re used to not having to question what they’d been told to do and just being left to work on it for multiple quarters. And they didn’t want to change because it was very stressful for them.
Holly 00:36:48 You know, it was very stressful for somebody to be saying, well, actually, we’re going to need to really work as a team and inspect every sprint, whether the things we’re working on next are the most useful things. They thought they were doing that because they were doing sprint planning, they were doing, You know, backlog grooming, et cetera. But, The thing is that they were also lacking an overall strategy that it was all tying under. That would be what ties it together beyond just, we think we should make our software more modern. And so, uh, I was trying to convince the people I was working with there that, you know, we had to do this continuous process. We had, That we would, we would need to be doing research to understand what the strategy should be. And we need to be ready to, Sort of pivot as we came in with results on that.
Holly 00:37:34 Cause we didn’t have a strategy yet. We would do research at the same time for both the strategy and the tactical, which is something I’ve done with several teams. But it only been successful out with really high performing teams because it’s, it’s, there’s so much ambiguity. Most teams are not comfortable with that much ambiguity. By the time I was done with my contract there, I could, I basically totally saw the writing on the wall because I said, they’re not able to make this transition. They’re not able to work this way. They’re all too uncomfortable. They’re pushing back. They just want to, you know, do what they just want someone to tell them what to do and to do it. And then this isn’t going to save the ship.
Andrew 00:38:08 Yep. Yeah. I think it’s ironic that the scaled agile framework is, is a, the acronym is used where it is safe because it’s like, it’s like a, it’s like a Binkie right. It’s like an executive Binky in the enterprise. They, they feel it makes them feel safe because they’re like, look, we did something agile, but if you actually look at it in no way, shape or form, is that actually adhering to the principles of the agile manifesto? It’s just, it’s, it’s just a, I don’t know, a Mirage maybe.
Holly 00:38:30 Yeah. A very complicated one. Like,
Andrew 00:38:32 Yeah, I don’t even get it. I just know I’ve looked at enough to be like, no, this now. Yeah,
Holly 00:38:37 Yeah. The last time I was somewhere, I was actually at a growth startup that was, that had sent people to training for that. Cause they thought that they would help them with how they scale. And I was like, Oh, pit in my stomach.
Andrew 00:38:48 I was like, Oh, this is going to suck.
Holly 00:38:53 Literally told that story to other people. And that also helps them out because then I was like, look, the company did close and everybody got laid off.
Andrew 00:39:00 I think you’re right. In the sense that any sort of, you know, two of the, two of the main, uh, pillars that we explore in the show, right, are our product development and service development, but just like the, the development of what we put in the world to make a difference. And then the organizational design, like how do we build an organization that is extraordinary and et cetera. Uh, and one of the things that’s become really obvious is that, uh, it’s ideal if you’re going, if you want to make a change in an organization that there is some like motivating factor, right? There’s usually some pain, frankly, to be really blunt about it, Is usually what it takes. And so one of the things I’ve struggled with at times with some of my own work and, and you know, other people I talk to is they’re like, yeah, but we’re comfortable, right?
Andrew 00:39:37 Like there’s nothing really wrong. So there isn’t that incentive to change. It. It’s actually much harder in that case to change one of the ways that I found to be effective. And this is like, like blocking and tackling line level work. Like this is not dealing with the executives. This is like your one-on-one with someone on the engineering team. And they’re like, but why I have found in with many of the engineers I’ve worked with that one way to do it is like, even if they love that at the end of the day, they still, the thing they hate is inefficiency. And I’m like, think about this. You could spend all that time. Like working really hard. You’d come up with like the best engineered thing ever, but it’s wrong. It’s, it’s not that, not that you built it wrong, it’s that we built the wrong thing. And they’re like, Oh, and the, and that idea that like, they would have engineers hate inefficiency. And that is like the most personal form of professional inefficiency for an engineer. Yeah. So that’s one, one button you can put anyone listening to this can push that. I have found to be fairly effective, even with people who have done it a certain way for well over a decade. So that’s maybe one other place to look.
Holly 00:40:34 Yeah, I totally agree. And another thing I can say there is you might even ask them, when was the last time you built the wrong thing? If they’ve been in an enterprise for a while, I guarantee you, they have.
Andrew 00:40:44 Yep. Yeah. For sure. And you can tell, like, if you’ve been, I have, I have plenty of personal stories of, you know, we thought we had it right. And Oh God, we were so wrong and we always did months. Yeah. That is not a, not a pleasant experience. Okay. So let’s, let’s, let’s zoom back out here to the path you were laying out. So we had two table stakes things. We had agile, like real agile, and we had, That a team is able to do any sort of continuous discovery process and do their own thinking as well as their own delivering. Uh, and then what was the step after that?
Holly 00:41:15 So, So actually I, uh, I broke that into the agile and then the some kind of process of discovery and then the continuous making it continuous. So making it to the third one, And I call that the continuous product improvers stage after that, the next level is what I call, High impact. Uh, so the high impact experimenters at that stage, not only are they doing continuous product discovery, but they’re actually, Finding that they understand they’ve, they’ve gotten good enough at this. And this is something that like, it’s very, you kind of need to do it for a while to get to the place where this, where you’re, you need to deliberately practice it to get to the place where not only are you doing research, but you’re actually coming up with, with accurate conclusions from that research. So the way that I say, you know, how do you know if you’re at this stage is if you commonly find that once you release software to your users, your data shows that they use it in the ways that your discovery research suggested they would examples confirming your hypothesis, basically.
Holly 00:42:15 Exactly. Yeah. So a really great example of, you know, being as this place where the experimentation is high impact is that when I was working at Shutterstock, And we were launching a new product called Shutterstock editor, we had spent several months of, of agile development and continuous discovery and delivery, where we were talking to customers, every sprint, as well as delivering work, every sprint, which was all behind a feature flag for only beta users. And when we got the first, the first major milestone, which we said, this is the place at which, We feel comfortable doing some marketing around this, we made it the open beta. So the point at which anybody could join without having to be specially invited, we had made it as small as we possibly could. So it was only for social media managers and marketing managers who were posting to social media that just needed to easily crop images.
Holly 00:43:01 Now this might sound really basic, but actually it’s, It was a real pain point because most of these people were opening up Photoshop, which as if any of you have ever opened up Photoshop, you know, can actually take like several minutes and navigating, you know, the difficult interface just to crop an image, to be the right size for a social media posting. So our first launch was just a cropping and filtering so that you could post a social media, but we knew that people were going to want texts to be able to put on top of it. And we launched without that. And, Because we knew enough people will get value from it that it would be worth launching. And one relaunched, the number one most requested feature was to add techs. And this was actually an instance of, okay, I know that my experimentation was working because we knew what to expect.
Holly 00:43:45 We knew nothing surprised us about that launch. You know, for the actual product development team, it was, this was what we knew was going to happen. The amount of people we thought were gonna use it used it, they loved it, the people they had this particular profile and the people who weren’t getting enough value from it to use it yet, we’re asking for the thing that we were working on already. And so that, you know, it was like great. At this point in time, you’ve got really well functioning, continuous discovery and delivery on this one, This one product, And the, the, there were a couple of teams involved in making that product, but that, that set of teams, there is one more step to what I call the product science success path. And that last step is actually where you go from being high impact experimenters to being high growth product leaders.
Holly 00:44:26 And the key difference there is that now, Not only are your, Are, is your team knowing what is likely to happen with this release with when you launch software and what’s going on, but you’ve got the confidence and the trust of the stakeholders around you, and you’re actually spreading out that way of working to teams around you. So you might be a nucleus in your organization of change. You might be the team that’s doing continuous discovery and some other teams that are sort of getting going on it. And then the other teams that aren’t trying it yet. But you are doing it publicly for the company and you’re doing it in a, let’s call it a politically savvy way that, That makes people around you feel comfortable and the inspired rather than just want to fight.
Andrew 00:45:13 All right. So we’ve got the sort of five step path here, which I love that you laid that out because I think it really can help the person, you know, the listener can really find themselves at wherever they are on that path and say, okay, I just got to focus on getting to the next step. Right. I don’t have to worry about getting to perfect, but just what’s the next step, which I think is such a useful way because this making a transition like this, it sounds so simple, but it’s huge. And it’s, it’s like deceptive, it’s deceptive and it’s complexity and it’s, it’s really, it’s not just a change. It’s like an actual, full blown transformation in how you think about and do your work. So there’s so much depth in here and I wanna, I wanna, I wanna zoom in really quick on the shifting to a more agile kind of way of working, right?
Andrew 00:45:56 So like moving into a, uh, that step of moving into kind of continuous research and delivery where, you know, you, at least people aren’t, let’s say they’re onboard with agile. They thought, yep, cool. That makes sense. Let’s do that. And they are, you have the, whatever the authority or the setup or whatever you want to call it, to be able to do your own thinking and discovery. And you’re not totally beholden to someone else, you know, some strategy doc that was written by someone somewhere far away, right? So you have that ability or how do you apply this in a scenario where the work being done is really like what the problem you’re trying to solve is really nebulous. And it seems like a long range thing. So let me give you a concrete example. The last major product I worked on that, uh, I’ve just recently transitioned off of there’s a lot of problems.
Andrew 00:46:38 Cause we were at the edge of this, that nobody had solved. And I, by nobody, I mean, like it wasn’t even the academic literature, like no one, as far as we knew in the world knew how to solve these problems. And so some of the team, the people on the, the more of the data science and machine learning team, our side of the product, they were, they felt like they were needing long cycles because they’re like, look, you gave us this problem to solve. We don’t know how to solve it. No one knows how to solve it. So we’re like basically in the lab trying to figure out how to solve that problem. And they’re their sort of counter argument was like, I can’t tell you that this is going to be done in a sprint or like, I can’t, I can’t give you any certainty about when this is done. And so I’m curious, how do we bring, because machine learning and problems like this are going to become more and more a part of your average product in the world over time, how do we bring this idea of continuous discovery into a space like that that has a lot more of this almost R D type flavor to it. Does that make sense?
Holly 00:47:33 That’s a really good question. So I think there’s a, there’s a couple of, sort of nuances to what you’re describing. Also as you described that to me, I’m really curious, did this work? Yeah, it works awesome. That’s fantastic. It sounds like
Andrew 00:47:46 We launched it last year. Uh, it is so I can tell you more about it offline, but yeah, it’s, it works. It’s live, it’s in the world brutally hard, but we got it to work.
Holly 00:47:56 That sounds like something against a lot of odds. So that’s really awesome.
Andrew 00:48:00 I sometimes I’m still surprised.
Holly 00:48:03 So, so the way, so anytime we’re working with something that’s more nebulous where, you know, you go to the engineers and you’re like, we’d like to do this. And they’re like, I have no idea whether that can be done or how long it will take. Then our job as the product person is to do a couple of things. So the first thing is to understand why, like, why are they not sure? What are the details around this that, you know, they don’t have clarity on, What parts of this are making them on shore. And the way that I usually go about that is to do a premortem analysis, a premortem risk analysis, where we’re asking them to tell us where we say, let’s jump forward a year and say that this initiative has failed. Now what are the reasons it’s failed?
Holly 00:48:42 And so particularly when it’s something where the engineers are saying, I can’t give you an answer for this. You’re asking them to tell you, you’re giving them a safe space to give you all the reasons why they think that this is impossible or very, very hard to do. And you want them to be specific about it because you need to collaborate with them on which of those things are actually critical to the success of your product, because there may be things in there that aren’t, and your job is to be helping them to find the problem in a, as crisp of a way as you can. So, okay. So you need to get a sense of that. So that’s the first thing. Then the second thing is to actually see if there’s any more defining of the problem that you can do. So can you make the problem broken into smaller chunks?
Holly 00:49:24 Is there any piece of this that could be usable to somebody sooner, Than other pieces? So for example, like with Shutterstock editor, People were many people that we spoke to would have said, this isn’t valuable for me yet, but there was some group of people for which it was valuable. So let’s do that piece first. The third thing is to recognize which of the decisions you and the team are gonna make are two way doors versus one way doors. And so that’s you explain what that means? Yeah, absolutely. So, This is actually a terminology that I got from like Jeff Bezos, his shareholder letter. So that’s where a lot of people hear about this, but the idea is, Can you go back on that decision and how expensive is it to do so, so if you make this decision now, and then you work for two more months on this, And then you realize that it was the wrong decision. Cause you’ve got more research. Is it something that’s easy for you to change? Or is it something that now you’ve got two months of ripping something out and another two months to lay the new foundation?
Andrew 00:50:19 So sort of what’s the cost of being wrong, basically.
Holly 00:50:21 Exactly. Yeah. What is the cost of being wrong about this decision, the higher, the cost of being wrong? The more it is that you do research and discovery to come up with the answer for that decision. So typically the way that I’ll work on something like this with a team when we’re doing continuous discovery, Is we’ll work with the engineers to lay out to say, we’ve got some big nebulous problem of the things they have to work on. Which ones of them are two way doors, which ones are one way doors? How, how high is the cost of change if that decision was wrong. And then typically we’ll find some kind of piece of what they have to do that we either know that there is a two way door and, Is a foundational thing that has to be built anyways. And we’ll be like, okay, you get started on that piece while we do some more of the background research for the one way doors, or if there isn’t something like that, then typically it’s because the thing is so cutting edge, like you were describing that there’s something that the engineers just need time to work on.
Holly 00:51:17 And if that’s the case, then we’ll say, okay, you know, still within the context of that, what are the pieces of this that, you know, what does that look like? Like what does that, what does their learning plan look like for trying to find an answer to this and other parts along the way where we can be helpful with product definition or problem definition, or, You know, scope and how can we lay that out so that we can be bringing in full back to you?
Andrew 00:51:39 Got it. So that makes a lot of sense. I love that pre-mortem idea. That’s a really interesting, uh, way to do it. So one of things I’m curious about, I, in prepping for this, I heard you, I listened to something else you taught where you talked about high impact experimentation, and I think the way you framed it, there was that what makes experimentation high impact is prioritizing the riskiest things first and are the biggest questions that, you know, like basically what could kill this and let’s go over that first. So how do you, how does that integrate with what you just said about like the, the two way doors, the one way doors?
Holly 00:52:10 Yeah. So there’s a there’s I guess the one thing I should be super clear about is that there’s what are we prioritizing for research and what are we prioritizing for the engineers to be working on? And so often those are at odds. So we’re prioritizing for research, the riskiest things, but then we’re saying the engineers are going to work on the least risky things while we get the research in place before they do the risky things. Does that make sense?
Andrew 00:52:32 Okay. I think what you’re saying is in the, so, so zooming out to frame this for myself and the listener we’re operating in a dual track model. What that means is track one is we’ve got continuously discovering continuous discovery, right? So we’re continuously researching and discovering what do we need to build? And what’s the right thing to make. And then the other track in parallel is continuously delivering whatever we’ve validated is worth building. Right? And so with that context, you’re saying, I think that in the discovery track, we should be prioritizing the riskiest things first and say, okay, this could kill it. We got to go figure this out first. And then in the delivery track, we should be prioritizing the things that are the least risky. Is that what you’re saying?
Holly 00:53:12 So that is what I said, but it’s, it’s not, let me sort of refine that in the situation when you’re at the beginning of a very nebulous project with a bunch of unknowns, then you, Need, you absolutely need to have the researchers prioritizing the riskiest thing. First, what the engineers prioritize depends on whether the risk is feasibility risk or a different kind of risk. So if it’s a feasibility risk, which means that the engineers are not sure if they could possibly build this ever, then they need to prioritize that because they’re the only ones who can answer that.
Andrew 00:53:45 And is that in the discovery track or the delivery truck?
Holly 00:53:47 That’s a great question. I, yeah, it’s kind of both at that point. So if you have major feasibility risks, like you are doing something cutting edge, as you were describing in your case, and you do not know whether it’s possible for your engineers to do it or not, then they do need to prioritize that. Cause you better figure out before you do the rest of your work, whether you should even do this at all, because whether this is even a feasible product, right. In my experience, that’s a rare case. It’s an exciting case. I’m super thrilled for you, that you had a case of that. And it’s awesome when it happens. Here’s a couple of gray hairs too. I’m sure. Yeah. But it’s, it’s not the most common situation that the feasibility risk is the biggest risk. And so that’s why my, my answer may have not been as clear because it depends really on whether it’s that the other risks and Marty Cagan talks about this, there’s the value, uh, value, risk, there’s the viability risk and there’s the usability risk.
Holly 00:54:39 And so when we do pre-mortem risk analysis, we ask people to, we, we typically get the whole team together, including designers and other people. And we do all of those risks at once. And then we elevate from there to see what is the biggest kind of risk and what are the biggest risks. And so then I guess the thing is, is that if there’s big feasibility brass risks, you’re going to go one way where your engineers are going to start on the high risk, the high feasibility risk things. If there are not big feasibility risks, then your engineers should start on the things that have low other risks. So low value, risk, low usability risk, low viability risk, because those are the things that you know, are no matter what else you learn most likely to be needed.
Andrew 00:55:16 Got it. Okay. So if we, if we know something’s buildable and we’ve basically, if, if the question, if we’ve already answered the questions about like, does someone want this and need this and can we make it, then we should go after whatever, whatever appears to be most valuable and doable from there. That makes sense. Okay. I like that. Thank you. Cause that, that, that is a question that I, uh, beat my head against the wall on for literally years.
Holly 00:55:39 Oh, you came out on the right side of it. So
Andrew 00:55:41 I, I, I, I think I hope we did. I don’t know. You’re never, you’re never quite sure, but we got through it one way or the other, so I appreciate you putting that one to rest for me. So my psychology can let go of it. That’s so perfect. So I’m curious, uh, just riffing on that example, you know, we we’ve been framing all of this sort of in a lot of explicitly, but implicitly been framing all of this in a, uh, software, uh, primarily software oriented tech company, not a, particularly like a growth company, a growth startup or a, uh, you know, a well established tech company, like, like a Facebook or Google, Spotify, Netflix, whatever. How do you see? Cause you’re, you’re working with people in all sorts of spaces. How do you see this is, this is, this is everything we’re talking about here? Is it the same across those industries? Like, does this stuff translate perfectly to say hardware or to a company that would not historically have considered itself a tech company?
Holly 00:56:31 Oh boy. No, not exactly.
Andrew 00:56:35 I asked that because a subsegment of the, of the listeners of this, uh, of the, of the show are in tech, but in a whole other segment are people who their entry to this was more around like organizational design and, Like a real impact focus. Like how do we build something in the world that solves real world problems, not impact, not in the tech impact way, but like impacting the, like, solving an important problem with, And so they may not be in that situation that you and I are so familiar with. So that’s why message.
Holly 00:57:01 That’s awesome. So, so the truth is the reason why I said, no, it was just because the hurdles are different, but the principles are absolutely applicable in all cases. In fact, I had a conversation with someone recently about how they might, Help people apply this in nonprofit works for impact analysis. Oh, fascinating. So it absolutely, it does not have to be a software company. I myself have applied it for, I mean, I used it for the working with portfolio school, Which is a school I’ve used it, In other venues, uh, the thing that’s really different is what, what hurdles you’re going to face and how people around you are going to respond, You know, because of what they’re used to and what they experience.
Andrew 00:57:37 Okay. So, so tell me about the last time you worked through this process in a non tech context. What was that like?
Holly 00:57:43 Well, I basically, I mean the type, the, so we would literally say we’re starting a new initiative with an organization. That’s a non tech organization. We would still do a premortem risk analysis, but we might have some differences in what types of risk we’re emphasizing that we want to be brainstorming around. And in many cases I still will use the same words. I’ll still use value, risk am, but you know, it’s not about a software user. It’s about whoever uses the service, I’ll still use, Usability risk, but it’s about service usability or, you know, whatever other types of things are running and viability, viability risk, I should stop to clarify. That’s the one where it’s like, Hey, if we, if we made this thing for the customers and it cost us to do it, like, is it a viable business? And so that’s still, is this a viable operation where even if it’s a nonprofit and then feasibility is like operational as opposed to software.
Holly 00:58:32 Okay. So those are kind of the, the shifts that I make, if I’m doing this for a non-software, Organization or group. But then those, the, the core principles, they’re all still apply. So we’ll still map out, Hey, what are the, you know, if we’re a year in the future and this has failed, what do we, think’s gone wrong, everybody brainstorm what’s gone wrong. Let’s, let’s map out how important these risks are. Then let’s, Let’s figure out sort of what I would call a dual track discovery and operations. So it might be program development, it might be operations, but, you know, it’s that side of things, instead of it being about what your engineers are doing, it might be about your program managers or your, You know, people on the ground who are, who are giving services. But then the research side is actually still the same.
Holly 00:59:12 You’re still, it’s just that, you know, usability testing isn’t with a, with a computer and somebody sitting at a desk. Right. Like, But other than that, you’re actually still doing that. And at the end of the day, measuring, or, or trying to understand the usability of a, like a nonprofit service or an education or a school, Or a, like a signup flow or an application flow was one of the things I did. So we did, What does the process look like for someone who’s exploring, whether they want to go to the school, how does a person in that process, Experience, you know, the information they’re getting along the way and the help they’re getting towards the application and the, the, you know, signup, And you know, you can still do a lot of the same things. It’s just that it’s not all about the software.
Andrew 00:59:57 What is it that gets like, if someone’s bought into the idea, right. And they see the path, they’re like, cool, what actually trips it up. Like when they are going to go down this path, like, what are the things that get in the way of this actually happening? Whether that’s like political, if that’s like psychological, like worldviews or just some underlying assumptions that they didn’t even realize they had, but what, what trips someone up who wants to actually go down this path?
Holly 01:00:18 Yeah. There’s a couple key things. The first thing that I see tripping people up is poorly done research, making either the practitioner themself or the stakeholders think that research isn’t worthwhile.
Andrew 01:00:34 Hmm.
Holly 01:00:35 That definitely makes it hard to sell that we should do more of it and more continuously. And it’s the, the challenge there is that if a company hasn’t been doing a lot of research where they haven’t been doing a lot of continuous research, the first couple of rounds of research they do, if they don’t have a coach helping them, it’s probably going to be poorly done.
Andrew 01:00:59 Yeah. First time you do anything. You know, most people aren’t good at things. The first time they do them
Holly 01:01:03 And you don’t exactly know why you’re not good at it, or you don’t know what’s wrong about what you did or where you went, you know, where you made a wrong call, but the way that you learn that is a, if you don’t have a coach helping you with it, or some kind of guide is by going through the whole motion of then shipping the thing and seeing whether customers did what you thought they did with it. And if you, you know, if they don’t do what you expected, then guess what your research was not high quality in some form or another, Which is normal when you’re first starting, but then you have to assess why and figure that out and, you know, do retrospectives on it and then get better at how you do the research in the first place. And that’s, that’s a whole, that’s a whole Avenue.
Holly 01:01:42 Another common, uh, challenge to this is structural and organizational. So basically, you know, if the people who are listening are in a position to make changes to the organizational design and the way that things are managed, then great, you can start making these changes and then you’re gonna, You need to do it slow and then communicate with people and all these change management things, but you can do it. But a lot of times the person who’s looking to make these changes doesn’t have enough authority to change something. So some of the, some of the areas, for example, that people will often have trouble is around budgeting, like annual budgeting and the way that a company manages their investment, different projects. If your company invest in projects on an annual basis, based on some, you know, crappy, uh, guests that what’s going to happen with that thing that you might build in a year, then you’re going to hit some more trouble with trying to transition to this. And then you have to spend more time on the stakeholder management and understanding of how do we do this. And you also have to do a certain amount of projecting anyways, you know, just saying, well, I know that I don’t know enough, and this is going to get more clear, but here’s the back of the envelope estimations, just so that this can be compared with other things, apples to apples.
Andrew 01:02:51 I’m finding myself in a situation right now where I’m being asked to lead an effort to come up with a product, but I’m not being presented the problem. It’s like, Hey, we want you to build us this kind of system. And so there’s some little red flags going off in my brain, like, wait a minute, like this is validated. Is this the right thing to build, et cetera. So I’m curious about when you have someone like, well, let’s just take me as an example, but I’m sure this is a common case you run into. How would you, how do you coach someone or advise someone to deal with it where they are being presented with build us an X, as opposed to solve this problem? And like, how do you, what, what should, what should I do about this? Basically they’re using me as a proxy for many other people.
Holly 01:03:31 Yeah, absolutely. So super common problem. Right. And not, not what we wish we were facing as product managers, but something that we face often. The, the key thing that I typically do is say, look, we can put this in, in multiple streams at once. So one stream is going to be, Let’s assume for this, for this side of our brain, that we think that they validated it and we are just going to go forward. What would we do? What would that look like? Start mapping that out, start thinking about the delivery. What would we do exactly? What would we do if we were going to do that? Right. And then the other stream is going to be no, actually when we put our whole brain together, we’re like, what, why are you telling us we should just build X? So what would we do if we were given a Greenfield and allowed to explore?
Holly 01:04:18 And basically what we want to do is like, we’ll do the premortem risk analysis on the product we were asked to build, but then it’s going to come up with risks. That’ll make us think about areas that we want to explore in general. We’ll do discovery interviews, Trying to see if the product we were asked to build solves problems. And the way that we’ll go about that is typically like, well, have people use anything like this in the past? When do they typically use it? What problems were they? You know, what led them to use it? Why did they switch to that, et cetera. And then start to get a sense of what problems there are that this thing could solve. And then sometimes you’ll get to a place where you’re like, okay, it wasn’t very clear strategy. Wish they’d given me more, but you know, there’s something here I can do and I need to nudge it in the right direction. That’s the most common case to be honest. It’s less common that you get to that. And you’re like, okay, actually, this is just totally announced starter. We should kill it. But that does happen too.
Andrew 01:05:11 I would say in this case, like there’s a, there, there, but I’m not sure where it is.
Holly 01:05:14 Exactly. Yeah. So if there’s a, there, there, but you’re not sure it is where you really want to do is go back to the, The two way door, one way door, uh, concept. So you want to say, okay, if I’m not sure where there there is, Then what are the things I think it might be? And all of those things are there elements of this that we would need to build for all of them, if there are great, that’s not so risky. Start building that piece while you do the bigger picture research. If there are, are no elements like that, then, well, are there elements, is there anything where there’s like a two, two, these two versus those two decisions? If there is, then maybe you can just ask the questions that answered that basically you’ve got, if you’re given something where they say build X and you think there’s a, there, there, but you’re not quite sure what it is. You probably have a ton of decisions to make and not enough information about them. So what you need to focus on is prioritizing which decisions to make one.
Andrew 01:06:12 So, so last question on this topic, then we’ll start to, we’ll start to wrap up and shift to some rapid fire questions, but I started thinking about it in the context of research, but I could see it applying more broadly. How should someone listening to this? How would you advise them to decide what feedback to pay attention to and listen to and what feedback to set aside
Holly 01:06:30 Feedback that they’re getting from the firm? Anything.
Andrew 01:06:34 Yeah. So it could be, it could be user feedback in your research, or it could be also like, you know, Hey, you had a one on one with your boss and they gave you this feedback or your, your boyfriend gave you this feedback or whatever. How do you think, is there, is there a way that you would advise people to, to step through deciding when to listen to feedback and take it on versus set it aside?
Holly 01:06:51 Yeah, I mean, to be honest, This actually applies to going back to the beginning when I was talking about like, uh, teaching my kids to understand authorial, but not to respect it blindly. You can’t just respond to every feedback piece you get or every piece of information you get, just because somebody gave it to you, you have to, Figure out which ones are worth it. And the biggest way that I go about doing that is to start by saying, well, what is the source? Is if we’re talking about customer feedback, which customer said it, and is that customer representative of a body of users that I want to be increasing satisfaction for. So, you know, tie it back to your company strategy, tie it back to right now, we’re working on this segment of users. Is that customer from that segment.
Holly 01:07:38 Okay, great. Let me think about what they said was that customer, not from that segment. Well then let me put it down over here in a place where I say we should know that we’ve got these potential problems with this other segment, but they’re not part of our strategy. So we’re going to let it go. Then if the customer is part of the segment that you’re focusing on for your strategy, then the next question becomes, what is the basis of the feedback? Is it, you know, did they literally just use something and tell you what their experience was? Did they tell you how they felt about something that’s a new level up above, you know, what it was just to use it itself? Did they tell you that they can’t do something? Is it a blocker or did they tell you that they’re frustrated by something?
Holly 01:08:19 Or are they just telling you? I think it would be great if you did X, Y, Z. So you really want to take that in stride. And I think the key element that I typically apply there is my understanding of how humans understand their own behavior. If the customer is giving me feedback based on something they’ve done recently in their own past, I’m going to take it as more important, Than if they’re giving me feedback and they’re telling it to me based on what they think they want in their future. And that’s just because humans aren’t as good at predicting their future. So if they tell it to me that way, I will ask them to tell me about their recent experiences that affected that or why they think that and try to get to that place. And if they can’t articulate that well, but I’m going to put that in the bucket of, well, this feedback is probably something that it’s good for us to be aware of, but we’re not going to act on it right now.
Andrew 01:09:09 I love that you went and made that clear. It reminds me a little bit of a, there’s a book that I want an a former, an earlier guest on the show. Christina Woodkey recommended, uh, there’s a book she called a reference that she referenced called. Thanks for the feedback that I’ve been meaning to, to read that I think is about this exact question. So just kind of came back to mind is perhaps something, a good resource on this, on this topic.
Holly 01:09:29 Awesome. Well, if you do read that book, let me know. Cause I haven’t read it yet. So I’ll, I’ll take your cliff notes.
Andrew 01:09:34 Okay, perfect. Perfect.
Holly 01:09:36 I do. I do love Christina Woodkey so, you know, if you recommended them.
Andrew 01:09:39 Yeah. Christine is great. We’re gonna shift into the sort of last segment of the conversation here with some rapid fire questions. Again, these are short, but your answers don’t have to be riff. However, however, you please. So the first one is what are you, what are you either currently or recently, have you read, watched, listened to that really impacted you or just stood out to you?
Holly 01:10:01 Honestly? I’ve been listening to a lot of music and I, because I’ve been going through some changes in my life and I’ve needed it to help. Uh, it’s helped me stay upbeat. Uh, there’s a song called fight song. I think it’s by Rachel Platten. And that’s really impacted me. It’s like my Anthem for right now. Um,
Andrew 01:10:24 Is it like you get up in the morning and you put your blast that song and it’s like your get up and go with it.
Holly 01:10:27 I mean, I wish, but no, I can’t blast music with my kids, but if my kids aren’t here, then yes,
Andrew 01:10:33 Yeah. Put the headphones on. Just,
Holly 01:10:35 Yeah, no, it’s, it’s more like the song I play. As soon as the kids are away,
Andrew 01:10:41 It’d be like, all right, let’s do this.
Holly 01:10:43 Yes. And then I’m like, all right, I’m ready. Ready.
Andrew 01:10:47 I love that. I love it. There’s a similar, I’ll share some of my answers to these questions as well as we go. But there’s a song that right before quarantine started, I was at the gym and they played this song by, I think I’m going to say, I’m going to butcher this name of the artist, but I think it’s do a Lipa it’s called physical. And it is just the most like upbeat jam that if I’m having like a day or whatever, that song just gets me, like, I’m like, okay, let’s do this. Let’s go. It’s like, it’s like my current pump up song on my, yeah. And we’ll put that, we’ll put all this stuff in the show notes do for the listeners. So definitely check it out. Cool. So what is it, if you think about recent memory and that could be, you know, that could be the last week. It could be the last two years, just wherever it fits for you, but what is a small change that you’ve made that has had an outsized impact in your life or your work, but just small change, big impact getting outside?
Holly 01:11:41 I think, uh, it’s something that I didn’t put as much stock into, you know, people always say it’s really good to get outside, but you kind of think it’s one of those things. You’re like, ah, yeah, sure. It’s good to get outside. Like, it’s good to, you know, take your vitamins. But I feel so much better when I get some fresh air every day. And so I feel like just getting out for a walk, you know, Even if, uh, or like right now it’s like, it might just be a walk to the grocery store to buy groceries, but I think that’s really has a big impact cause it affects my mood. I know the days when I haven’t been doing it and also just exercise in general, like I’ve been doing more exercise while I was doing more exercise before quarantine and it has a really big effect.
Andrew 01:12:27 Yeah, for sure. Uh, so I guess one bit of context in case anyone’s listening to this in the far future, we’re recording this in April, 2020 during the global pandemic of COVID-19. So when we’re referring to quarantine, that’s what we’re talking about. Everyone is basically supposed to stay home right now. So that’s what we mean. What actually one resource I’ll recommend that has been, I guess my answer to this question, but specifically during warranty, the episode that actually just came out on Tuesday is with an awesome guy named Derek mills. Who’s the founder of a company called glow, uh, which is a, it’s my favorite of all of the, uh, like yoga apps out there. There are many, uh, but I love this one and which I did not think I was going to say. But that doing that I’ve just found is even a few minutes, every morning has made such a difference in my, just how my day goes.
Andrew 01:13:13 And it’s something I could do. Like even 10 minutes in the morning at home, I don’t have to go anywhere and it’s been great. So shout out to that. Definitely something to, uh, to try if you haven’t I’ll it out. Yeah, for sure. Especially, I definitely, it helps with the sanity when it’s hard to go outside or we’re not supposed to go anywhere. You know, all of us product manager types, we love us some lists. We love us some mental models. We love us some frameworks. Are there any, Mental models that you, you find to be extremely useful and that you lean on a lot? Yes.
Holly 01:13:42 I wish that I, because I I’m still getting used to the role of being a teacher. I mean, I love teaching, but, I wish that I could turn and be like, Oh, it’s so, and so’s mental model. I love it. It’s great. Here’s what it is. Cause I just like giving shout outs to other people. But honestly the thing that comes to mind that I use the most is one of my own, I call it the framework and I use it for doing an assessment of, of a product opportunity. Oomph is U M P F and basically it’s a, it’s a really lightweight assessment, When you’re trying to decide between different opportunities for, for improving a product. So for example, I recently used it with a client who is like, okay, we’ve got phase one out, we’ve got, you know, we’ve shipped, All these units, we’ve got all these users.
Holly 01:14:25 We want to figure out what’s going to give us the biggest growth for the next phase. And so we did this, we say, okay, you as for users. And we always start with the user because you gotta be in the mindset of the user. And, and I I’m really adamant about that because even though you could technically do these things in any order, I always want people to do user first because that makes them less likely to get confused about what a user will really do. Whereas if you start with like the business model side, it’s easy to be like, well, and then he was just, we’ll do this because it’ll make my business work. And you’re like, yeah, except they won’t do that.
Holly 01:14:55 So we start with the user and we map out who, which users this is for and what their characteristics are like, What kind of pains they have and what desired outcomes they’re going for. And then we go for M market and that’s really the place where we’re scaling up from qualitative to quantitative and saying, how many people are like that user? What kind of data points might we have that will help us when we’re comparing this against other assessments? How do we know the size of the unmet need in the market? Then we do P and in this case, that’s, Product. And so basically that’s a place where usually at this point in time, my clients will already have some ideas of what they might be doing in these different directions. And so like, just describe it. What does it look like?
Holly 01:15:30 What would it mean to do it? You know, what criteria do you have around it? And then, Ideally describe in terms of what outcomes it drives for your customer, more than what, Features that involves. But there might be some things where you’re like, Oh, I know I need to build a system to support X and we don’t have that system today. And then the last one FSPs ability. And so that’s basically this idea you’re going back to earlier, like how big is the feasibility risk? And if the feasibility is well understood, how, what is the general size of it? And all of these are meant to be more back of an envelope type of assessment, not, you know, not rigorous scientific assessment. And then you do that for a couple of opportunities at a time, or maybe maybe three or five even. And that way you have just this really basic framework, that’s really easy to do and lightweight that you can use to help you kind of assess okay. Of these different opportunities. I think we better start researching this one or these two.
Andrew 01:16:18 I love that. I love that. I’m totally gonna start using that one thing I’ve noticed just from my own experience and, you know, outing myself here. And in many people that I’ve worked with or met in our general field is a lot of, uh, I’d say a lot of us are recovering perfectionist and, uh, have, have grappled with I’ve long grappled with imposter syndrome, which is, I think a rampant psychological phenomenon. And whether you identify or whether you personally resonate with that or not, I’m curious. What, if you have any advice for people who resonate with that, what they can do to, to deal with that and move forward in a healthy way?
Holly 01:16:56 Oh my goodness. Yes. I have lots of experience with that. But it doesn’t bother me anywhere near what it did 10 years ago. And I think it’s exposure therapy. Like for me, it’s practicing not being perfect and being okay with it. It’s practicing feeling like an imposter and being okay with it. It’s really practicing being in that deliberate practice where you’re uncomfortable, but you keep going and then one day you look back and you’re like, I’m not uncomfortable anymore. Huh? That’s interesting.
Andrew 01:17:29 Yeah. Right on down. It reminds me of a, an exercise that a friend gave me once. That was, uh, he called it like a 30 day of 30, 30 days of discomfort challenge or something, something like that. And basically it was to take this on and everyday you had to do something that made you, I had to do something that may be uncomfortable. It could be anything, anything, it could be tiny. It could be big didn’t matter. But the key bit I in reflecting on it was, Before I did the thing, I had to write down what I was afraid of. And then after I did the thing, I had to write down what actually happened. And over a month of doing that, you know, it becomes very obvious how much, almost none of it, none of the bad things I was afraid of happening. None, literally none.
Holly 01:18:09 Yeah. Yeah. I’m a big believer in that as well. Like do the thing that you’re scared of, You know, like continually be like, well, what am I scared of? Well, why am I scared of that? Let me go do that. And uh, I had listened to a podcast, Sometime in the fall where someone was saying that she did that every day for a year. And then it was just like the most amazing year. Like by the end of it, she’s just like a changed person. And I thought to myself, like, I get that, that makes sense to me, I’m going to, I’m going to do this. I’m going to do things that I’m scared of. And it’s, it goes back to that feeling of flying like you do, if you do things you’re scared of regularly. And you’re like, well, why am I scared of this?
Holly 01:18:45 How bad would that be? You know, like it, you start to, that’s what the exposure is, the exposure to the failure or the, or the exposure to the what, what, what actually is the outcome that, you know, maybe isn’t even what you were scared of. And the more of that you get, the more your, you kind of toughen up your skin and you’re like, Oh yeah, I know. You know what, I’m not perfect. And that’s okay. You know? And the way I’m going to grow is I’m going to be just able to try things that I fail at and be okay when I fail.
Andrew 01:19:12 Yeah. Yeah. Well, I think that’s sort of the perfect place to wrap up this, this, what I hope is a first conversation, we’ve come full circle back to, you know, the ability to be in the discomfort, which I think is a through line of all of this and makes a real difference. So before we, before we wrap up Holly, is there any, any asks you have with the audience? Uh, what would you, do you have any, any request of the listener? What would you ask them?
Holly 01:19:33 Oh, well, I feel like I should just go with that theme of ask them to make themselves uncomfortable, to think about, you know, what, what in their lives they’re not doing, just because they’re scared. And, but it’s something that they’re thinking about doing and you know, why, why are they letting the fear get in the way, what would be the worst that would happen? And could they, you know, work towards that even with small steps? Yeah. I mean, related to that, I, you know, one of the things that I do is I coach, I used to coach skating and, And I’ve also done, you know, basically every time that I got my skill to a level where I was good enough to teach others, I was like, I’m going to help others with this. And so, uh, you know, if anybody’s interested in chatting about it, especially if you are listening to this anytime near, when we’ve recorded it, You know, currently, uh, during the quarantine, everyone’s like, I just want to talk to people. So, I’m totally open for some conversations, if anyone wants to, To chat with me. And, uh, I would be honored to hear what people are scared of and help them overcome it.
Andrew 01:20:32 Oh, I love that. That’s so generous of you. Thank you. And if people listening want to reach out, get in touch with you, where can people find you online and your work and if they want to engage with
Holly 01:20:41 Yeah. So, My website is H to our product science.com and my Twitter handle, which is the social media on the most is HQR product side ends in sci. Cause that was as long as it could be. And then, uh, you can also find me on LinkedIn as well. And you can email [email protected] and I will definitely respond. And, If, uh, you know, if you want to actually chat, you can find a link to just book a session, a book, a free chat with me in the contact page of my website. I think. So I’m happy to just talk with people at this point in time. I’m really interested in just getting to know where people are at in their personal products, management journeys or, Or other sort of, uh, you know, journeys to being bad-ass is.
Andrew 01:21:25 Yes, absolutely. So if you’re listening to this and this is all interesting to you and you think you could, you want to learn more, definitely reach out to Holly. She’s an amazing resource on all of this, as you can. No doubt tell by this conversation. So Holly, thank you so much for taking some time and hanging out with me. This has been a real pleasure.
Holly 01:21:40 Thank you so much, Andrew. It has been a real pleasure. Totally the fantastic experience I was hoping it would be. So thank you.