I ran a panel with experts from manufacturing, shipping / distribution, computation, and a social initiative at scale to expose the complexity, logistics, and engineering behind successful large projects. I’m Ruth Grace Wong, Site Reliability Engineer at Pinterest, and my panelists were:

Hayley Cashdollar, Manager of Automation at Proterra

Glenn Matlin, Data Scientist at LendUp

Theresa Condor, Vice President of Corporate Development and Board Member at Spire Global

Diane Gillespie, Emerita Professor at University of Washington Bothell

Here is the transcript of the video, edited for readability.

Helen Toner: I’m going to introduce the panel which is Logistics at Scale. Obviously if you want to do things in the world it helps to do those things with other people. If you want to do big things, there’s going to be a lot of logistics issues that you will need to solve: how to get things done and how to make things run smoothly. I’m sure once you’ve worked in an organization that isn’t taking care of its logistics you will never again underestimate how great it is to have people taking care of logistics. That’s what this is going to be about. It’s moderated by Ruth Grace Wong who is a Site Reliability Engineer at Pinterest and Ruth is going to take it from here and introduce the rest of the panelists.

Ruth Grace Wong: Hi, my name is Ruth and we’ve got some extremely talented individuals here. On the far end we’ve got Hayley Cashdollar. She works for Proterra, designing and building a factory to make batteries for zero emission transit buses. Next to her we’ve got Glenn Matlin who is a data scientist at LendUp which offers loans. Glenn works on machine learning driven underwriting and advanced product analytics so he’s our software computation representative. Then we’ve got Theresa Condor who’s the VP of corporate development of Spire Global which monitors 90 percent of global trade with satellites. She’s an industry veteran for distribution and supply chain. Finally we’ve got Diane Gillespie who is a full-time volunteer for Tostan, an award-winning nonprofit that has brought non-formal human rights-based education to hundreds of communities in North and West Africa. We have six questions for the panel and then we will be taking audience questions. The first question is: Tell me briefly about how you got to the role that you’re in now.

Hayley Cashdollar: Thank you for hosting us. I am a mechanical engineer by training. I went to the University of Colorado for my bachelor’s degree and from there I went straight to Tesla Motors where I worked on powertrain manufacturing engineering. I designed and built equipment that made the rotor and the cooling systems for the Model S. I then found that I really liked the mechanical design on that equipment but didn’t know anything about the software, and so I went back to grad school to get another degree in mechanical engineering with with an emphasis on controls engineering. In my current role I explore both the software and the hardware side of manufacturing problems.

Glenn Matlin: My background was really more about in econometrics and I came from Florida where we saw firsthand examples of bad financial lending in 2008. The subprime lending market cratered all of Florida in terms of employment and the economy. Having studied that, I worked my way out towards San Francisco because I was very interested in trying to use my skills to build better financial services. Most of my work was in analytics. A lot of the self-trained self-taught got me out here so I’m much less of an academic on my team than others. I tend to come from a do it yourself and work your way up kind of perspective on things.

Theresa Condor: Hi everyone. I’ve been with Spire Global basically since we started the company in 2012. I’m not technically a co-founder but have been doing everything from the beginning with the co-founding team. I’m not technically in the space sector. Our company Spire builds satellites: we put them up into orbit, listen to the earth and collect various types of data and then sell the data as a service. I’m not traditionally someone coming from the space sector. I’m not even an engineer. What interests me about what we do at Spire is how you can use the data to have an impact on big problems to help businesses be more efficient. I think for the purposes of this particular conference the things that I’m interested in are our ability, from our data, to identify illegal fishing that takes place on a global basis, to monitor global trade patterns, to collect a unique type of weather information from our satellites in places like Africa and over the oceans where you have no other type of infrastructure that can do it.

Diane Gillespie: Thanks again for hosting. I’m in awe of all of you. I’m Diane Gillespie and I’m from the University of Washington, the Bothell campus. It’s a very interdisciplinary, exciting place to be. I retired to help Tostan which is a non-profit program operating now in six countries in West and North Africa. It is a three year non formal human rights-based education program that works in resource-poor communities. It’s probably best known for the one of the outcomes of the program: that communities decide to abandon female genital cutting. The founder of this organization, Molly Melching, was recognized by the Skoll foundation, and they did their first book called However Long The Night on how she came to create an organization that has had this kind of impact. My role as a volunteer is to work in monitoring and evaluation. I also help with the training center.

Ruth Grace Wong: Amazing, thanks everyone. So when you’re doing something at scale there’s a huge opportunity for success but there’s also a huge opportunity for failure. Do you have any principles that you follow in scaling up, and also what’s the biggest mistake you’ve ever made?

Hayley Cashdollar: Thanks for the question. There are so many mistakes that it is hard to choose from. I have found it very helpful to lean on and completely trust the people around me because if you are scaling a project you cannot do it by yourself. That comes from hiring the right people to help you but then putting your faith 100% in those people and letting them do the job that you hired them to do. One of the mistakes that we made as we rapidly scaled our manufacturing facility was overlooking ergonomics on one of our systems and it’s been a real pain to go back and try to figure out how we can help avoid injuring our workers on one particular piece of equipment. If you are scaling really quickly it is really easy to overlook that kind of problem and I wish that we had taken a little bit more time to evaluate that particular part of that project. I think that comes with a lot of hardware scaling anytime you’re manufacturing.

Glenn Matlin: When you’re operating at scale I have found that really small decisions turn out to have a gigantic impact down the line. They’re the kind of decisions you never really end up thinking are important and they’re the kind of decisions you make just because they seem so mundane. I found that the principle of really thinking slowly about what you’re doing but being very fast and deliberate about it has been very useful. Taking the time to really consider all those small decisions and how they might impact you down the line is important. It lets you design and deal with edge cases. It lets you deal with situations where you get unexpected behavior from people using your service or your software. There are so many small decisions that end up becoming really impactful later on. One of our co-founders, Jake, is our current CTO. When he started, the company just decided to put floating-point storage for the value of our loans. I imagine this is like a really small thing to people but it turns out that when you’re doing compliance for regulatory bodies you need to calculate every single cent perfectly and you can’t really do that with imprecise floating calculations. That small decision five years later is a huge problem we had to go back and fix. To this day I think Jake would be like no, that was a mistake and it was headache but we now really think really tight about every little decision we make in our architecture because it pays off down the line.

Theresa Condor: I think fundamentally a principle that’s important to us in looking at scale is to make sure that you have a problem and a solution that you should scale. This means that it’s it’s big enough and it’s going to impact enough people but at the same time you can you can do it without having to move in lockstep with the same number of people that are trying to implement it. What I mean is that if you’re you’re having an impact and you have to keep hiring the exact same number of people, that’s not something that is going to scale, or you have to question whether you have the right solution. So you really want to have as wide a gap as possible in terms of the number of people that you impact and the number of people that it actually takes to to affect that. I think that’s fundamentally the the big problems that we want to tackle with our satellite constellation, and we’re talking about the the 1 billion people that rely almost exclusively on fish for their protein source and the other 4 billion people that do eat fish in some way for their nutrition, 7 billion people on the planet that are all impacted by weather to some extent and that continuing on. So big problems are the kind that really should scale and if you need to hire too many people in order to do that, it just won’t work. People, for us, is really the biggest thing that we worry about and probably the biggest decisions that we’ve made that have not gone well is hiring the wrong people or good people at the wrong time.

Diane Gillespie: So the principle I think for Tostan has been fidelity to the curriculum it’s created into its interactive pedagogy. There’s a lot of pressure to cut back especially the work in social norm change. People want to do it quicker and faster. The integrity of the curriculum and the interactive way in which the curriculum is taught is really critical when you start to scale up. When you do that there’s lots of pressures to fudge it. The biggest mistake Tostan made — I’m not sure Molly Melching would say this — is that it went to Somalia to implement its program. Its national headquarters are in Dakar, Senegal and it looked like a perfect setting because it’s in Somalia the worst type of female genital cutting is practiced and people beg Tostan to come. But it was a challenge in many ways because of the distance and the training but mostly there was no government and so the people who were working to train the Somalians in the program were actually arrested. It looked like a perfect setting but the kind of unforeseen details that arose in the actual implementation made it just too difficult to continue the implementation.

Ruth Grace Wong: Wow, that sounds like a very difficult situation. The next question is: I work by day as a site reliability engineer and I find that when we’re at scale I have to start writing systems to automate things and then you start having to write systems to manage the systems. What’s something that you find that you have to do in a completely different way at scale compared to not at scale?

Glenn Matlin: One of the big things that we have to really plan for is those edge cases that I was talking about, mainly because of the fact that we have to report and be held accountable by numerous government regulatory bodies at the federal and state level. Oftentimes when you’re in an environment where you’re filled with people, around people who are like really energetic and they want to move fast and they want to do big things and they think they’re really going to help people, we also have to make sure that we really think about every little decision we make, mainly because those edge cases can cause problems if we’re not thinking carefully about what we do. Financial reporting is an incredibly sensitive topic for a lot of people, and the financial services access that these people have is really their livelihood. So when we do our work we really think about them and we really focus on what we’re doing for them. If we make a mistake with that edge case somebody is not going to get a loan, they are not going to make rent or they might lose access to their home or vehicle. So our work is very important because we really think about the person at the end of it and making those mistakes is very costly for them.

Hayley Cashdollar: As a part of building a factory my entire existence is because we want to scale. My first year at Proterra I worked closely with a team of engineers where we built all of our prototype packs exclusively by hand and exclusively by engineers and that is not something that can scale. I’d say those are probably the most expensive battery packs ever made. A team of about 15 people, all engineers, building by hand for many, many hours. As we look to scale any sort of automation project, one of the things that we have to do is consider that you have to approach the problem in a completely different way. So you can’t use all the lessons learned on a manual hardware build and apply those to an automated hardware build because machines work differently than people do. Sometimes that makes it a lot easier and sometimes that makes it a lot harder. In order to bridge that gap and tackle those problems quickly I’ve found that it is important to do as much process prototyping as possible to prove that in transition from a manual to automated process, you aren’t overlooking something big that’s going to end up delaying a project. That’s been really key for Proterra as we’ve scaled our factory.

Theresa Condor: So for us this is very particular to our industry but it’s building satellites in a different way. Traditionally you’re building one satellite you’re spending many years on it and it’s kind of a bespoke design. For us we’re doing many satellites. We have 40 that are in orbit right now and we’re getting up to a steady state of 100, so that means we’re building on average one satellite every single week. So you definitely have to do things differently. A lot of that is is is related to doing it almost in an assembly-line way. The very first satellite that we built was here in San Francisco and we did it in what was basically a conference room with three people. It took many hours and we were frantic to get it out to the to the airport within half an hour so that it could make it to the rocket on time. Now it’s a completely different story where we have a whole manufacturing team. The MRP system that we use, the documentation of what is done, so we’re not relying on the tribal knowledge of those three people that built the first couple of satellites, so that it can actually be done day in day out no matter who is building it, that everything is put together the right way. The team that we ended up hiring to do the manufacturing is again, it’s not necessarily the engineers who designed it which is what it was initially. We’ve got someone who came in from Blackberry doing high volume manufacturing. The team is not necessarily engineers who actually are putting all the parts together. It’s a very different way of doing satellites that allows us to get to such a large number of them and then of course it’s really all about the data and what you do with the data.

Diane Gillespie: Tostan works in 22 different languages so you can imagine the scale up from Wolof which is the language they taught in Senegal. So when they expanded to Guinea Bissau you have Portuguese, well you can — I won’t name them all. So all of the material, because they work in resource-poor communities that are essentially illiterate, you can’t hand people textbooks. The program eventually over the three years does some work in literacy and numeracy but you have to create new classroom materials in the language because Tostan teaches in national languages. Funders don’t appreciate that that expense has to be built into the process. I would also say that we had to totally change our M&E system because, as you say, the sort of indigenous knowledge of the people who first worked in the program was not easily translatable to the new countries that Tostan went into, and we also need a system that can standardize across countries that have some cultural variation. So we have a grant from the Gates Foundation that’s very interesting in terms of helping us do a monitoring and evaluation system that really gives programmed feedback country-to-country. Allows us to ask really important questions about why, Mali, for example is not — the attitudes there aren’t changing as fast as in another country

Ruth Grace Wong: Wow very interesting. For large projects you need a lot of people, so what necessary partnerships have you had to make in your work? Examples include partnering with government or outsourcing part of your project to another company or even relying on local people in the field. So how do you decide when to do something in-house versus relying on a partnership?

Glenn Matlin: Yeah when we’re thinking about if we’re going to build it or buy it, we have to really decide long-term: does it make sense for us to really invest all this time into programming it, having people on staff to maintain it long term, and then dealing with at the updates that we’re going to need to do inevitably? When we make that build or buy decision there are a few things we think about, the first of which is: is it our core competency? LendUp and the data science team at LendUp is really focused on one thing: making machine learning models to do better underwriting, get people more approvals with lower loss rates and pass that savings on to them. So we focus on that. As a result, a lot of times other data science teams at big companies might build their own BI solutions or visualizations or their own internal infrastructure. You have to decide; we can’t necessarily do that. We can pull something off the shelf because there are tons of BI solutions or we can use Airflow or something else from another company that’s open source. Oftentimes we try to just figure out, is this something we are really good at, and if it is we’re going to keep doing it. Other times though especially because of the whole regulatory — I have to always talk about compliance — we really try to partner and work with the people that are monitoring us. We invite them in. We are very different than the Wells Fargos and Citi Banks of the world in that we really want them to come in, investigate us, look at us, help us understand that how we are doing things is how they expect us to stay compliant. We want to stay in that safe harbor area and really respect the rules because it’s very important for us. In that vein we really try to partner with the state and federal agencies that do monitor us and help us understand what they’re looking for when we do business.

Hayley Cashdollar: I love your answer because I cannot echo enough that you have to think about what your core competency is and think about what your business does. For example, at Proterra we make electric buses and that necessitates making batteries and to make batteries you have to make a factory and that may be full of robots. We do not make robots. If we were to hire a bunch of people and invest in a team, which we could do — we could make a team of people that’s incredible at designing pieces of equipment. But then what would they do after they’re done? So it’s better for us on projects like that that are outside of what our business model is set up to achieve to hire outside help for those. A lot of my job is actually co-designing with outside integrators and working with them to help us accomplish our automation needs on the manufacturing floor. We have a small team working on automation. Instead of the about 60 engineers that it took for these projects, only two of them work at Proterra. And I think that thinking about your business and the direction that you’re going is really important when you’re making those decisions.

Theresa Condor: So I guess I’ll say something that maybe is slightly controversial and I would say that partnerships are tricky. I think it’s something where both sides need to have skin in the game and often that means monetary skin in the game in some way. Otherwise I think there’s a long term chance that the partnership will not be successful and potentially even cause more damage. We build our satellites. We initially started thinking that we could use a lot of commercial off-the-shelf components, tap into the supply chain. What we want to focus on is the data and what you do with the data not per se the satellite constellation. What we ended up finding out is that we had more to lose about the parts when the parts are not delivered on time, not delivered to spec, do not work once there in space, than any of our suppliers did. So we ended up bringing pretty much everything in-house. We do the whole design of the spacecraft. We do all of the sub-system components ourselves. We integrate everything ourselves. We test everything ourselves. We do the whole chain minus the rocket launch. So in that sense we’re not building rockets, but we’re doing everything else. And that was a lesson for us in where do partnerships work and where do they not and how do you have to structure it. I would say the same thing can also be the case on the distribution side if you’re working with resellers or you’re partnering with international organizations or NGOs or any other type of organization. They can be powerful for sure they can also be a time sink if there’s not a very clear understanding of alignment but also skin in the game, I would say.

Diane Gillespie: Tostan partners with NGOs a lot but I think its’ most valued partnership is with the social mobilizers who actually grow out of the educational experience they have and become, what we call, the agents of organized diffusion. They spread what they’ve learned and they’re very — when you do development from the inside out — they know how to work with their family members, they know how to work with their extended family members. So you have them leading their own development. That partnership has led to a couple of interesting developments. Tostan leaves in place what they call a community management committee, and the really dynamic ones have begun federating, and they are now independent NGOs that get their own money. The other thing that’s happening is that, especially women are running for political office on human rights platforms. They’re able now to affect policy at the local region. So I would say that the most valuable partnership Tostan really has is with the people who are emerging out of the classes that are taught.

Ruth Grace Wong: That’s incredible. I really like how I can ask the same question and then get different responses based on the context. The second last question is: when doing projects at a large scale important aspects of the work can be lost, for example product quality in manufacturing, or accommodating all your users in software, or even retaining the personal relationships that drive change in a social initiative. What aspects of your work are easier at scale and what aspects are the most challenging to do well at scale?

Glenn Matlin: I think I’ve kind of harped on the hard parts. It’s edge cases it’s dealing with the fact that when you’ve got a million people, a billion dollars churning through your platform, something’s going to go wrong; it’s inevitable. Instead, I just want to harp on the parts that get easier, because I think that’s really the benefit of the field. When you do things that scale in my field, you get a lot more data. And a lot more information is powerful because you need that information to make good inferences and good predictions. From the machine learning side, more data just means you have better predictions. We can get better loss rates, better approval rates. Again, we pass the savings on to the customer. It’s fantastic to just have a huge amount of data churning through your platform. As another benefit from the data science side, when you’ve got a lot of people and a lot of data suddenly you get to try some something new, try experimentation. It’s hard when you’ve got a small amount of customers to try anything new. You’ve got to retain them and you’ve got to make sure you’re getting it right just for them, but when you’ve got room to play with, suddenly you’re finding yourself able to test new things. Just one small example from my work is that we’re finding that just adjusting the default amount of a loan by $25 can drastically change repayment rates. It turns out that for people that we are lending to, that $25, that default that they normally run with because they’re not tending to engage with the full site, makes a huge difference in repayment rates. It’s small things like that that can really go a long way.

Hayley Cashdollar: For us one of the challenges of scaling has been maintaining our quality, despite scaling. I have found that you have to communicate what you need from a quality standpoint in the right way. It’s not just the binary, this is good, this is bad, it’s why. And communicating that why can be very challenging in a technical environment when you’re speaking to somebody with a completely different background than you. That’s been a challenge that I’ve been working on with my team a lot. The great thing about scaling is that things go so fast. Since April my company has made 30 times the number of batteries that we made in the prior year and it is so great to have that information available and get more buses out on the road

Theresa Condor: I would echo that the great thing about scaling is that things go so fast, and you’re just constantly surprised where you can be. We started the beginning of the summer at 20 satellites and now we’re mid-August and we have 40 satellites. We just did three launches in the past two months and that’s just neat and exciting. The difficult side is to go back to a little bit what I was talking about in one of the earlier questions, which is people. That is the thing that we worry about the most, and what we find the most challenging. When you’re still a smaller company and you’re the ten people, you can manage the culture. You have everyone coming in and they care about it, and they feel special because they’re part of building something early on. That’s so difficult to do as you grow. I mean, it’s something everybody talks about in Silicon Valley of how do you do that as you add more people in it. I would say it really is a challenge to make sure that everybody is staying excited and driving towards the bigger picture when you don’t necessarily know everyone and you’re not touching and talking with them directly. Sometimes it feels like we spend more time almost dealing with internal things around culture and engagement and keeping everyone moving in the same direction than we spend with external related things. Sometimes that’s frustrating but that’s humans and humans are messy and at the end of the day that’s just as important as the external facing side.

Diane Gillespie: So I would put it in terms of training. That’s the challenge when you’re scaling up, especially if you’re going to have fidelity to both the philosophy and the product and your terms, but in terms of the educational program. Training is a great challenge because you have to train facilitators in all of these different countries, but once the program starts the energy level that’s produced is just, as they’re saying, really phenomenal. Now with Tostan’s programs, 7,000 communities have come together in public settings to declare their intention to abandon female genital cutting and child marriage, and they are celebrations. These are events that are planned by the people who come, the politicians come, everyone is involved in this. It’s really a new social norm that is beginning. So once it is at scale it’s just amazing

Ruth Grace Wong: That’s really cool. Okay so the last question is the one that are most curious about. Clearly you all do a lot of work. Many of you are doing startups or otherwise doing work with a high burnout rate. What personal sacrifices have you made for your work and do you have any advice for the audience about burnout and prioritizing what’s important to you?

Glenn Matlin: Yeah, I am looking forward to this question as well. This month I am taking my company’s credit card to scale from thousands to hundreds of thousands. I am preparing my first international travel conference and workshops. I’ve never travelled before, will certainly have to navigate that. I’m moving apartments at the same time all this month. If you live in San Francisco you understand what a total nightmare that is. I’m also trying to figure out how to be 30 at the same time. So this month has been very very busy and I have no time. So the truth is I’m being held together mostly with coffee and glue. At any given moment I might fall apart, but the truth is the motivation of the enormity of what we’re about to do with the credit card product really keeps me going. How to deal with it. I’m a very happy guy. I’m very optimistic. I am also very stoic about these kinds of things. My suggestions are prepare to fail. You’re going to fail a lot. Everyone’s going to tell you’re wrong. You’re going to have to get ready for things to go badly. You’re going to have to prepare for that and learn how to pick yourself up a lot. Ultimately it’s not about succeeding on every go. It really doesn’t work that way. My mentor has pretty much said repeatedly: we gain success through repeated brutal and awful failure. That is how machine learning works best, and it turns out to do it well so it’s a good philosophy to adapt when you’re doing something new and difficult.

Hayley Cashdollar: As far as personal burnout goes, I met you (Ruth) in the midst of one of the most challenging times of my career which was earlier this year where we had less than a month until our factory launched. It was a time where every single project I ever worked on was coming together and it was very challenging and time-consuming. One of the tools that I found was leaning on the people around me and helping get the people that I was working with as committed to those projects as I was, because then when I was low there’s somebody to lean on and bring me back up. That is good for the times where I was working very hard and long hours, but I also intend to lead a cyclical life and I will have four months where I work non-stop, and I will plan that I have time to not work as hard, like right now where I actually will leave work on time for dinner sometimes. I think planning in advance that I will have a cyclical life and know that I have will have no social obligations from January to April but then months with travel and time with friends after that it makes those really hard pushes easy to get through because you know what’s next.

Theresa Condor: I think the reality is that burnout is a very real thing. I know there have been some pretty publicized debates recently in Silicon Valley about whether you can have it all and be successful and have balance, or whether you really do need to almost work yourself to death in order to accomplish something big. At least in our mind and our experience with our company is that sometimes if you want to do something that is very challenging and that you think will have a big impact and hasn’t been done before the reality is you just have to run and work harder and faster than anyone else and I think that’s a very real thing for us. We have gone all-in on this. My husband actually also works with our company. He’s the CEO and a co-founder, so we’ve been building this organization together. I was pregnant with our daughter when we were fundraising together for series A and when she was a month old he came home and said, well I think we have to move to Scotland where we’re opening the manufacturing plant for the satellites. So we moved to Scotland as soon as she got her vaccinations and then from there it’s just been, everywhere I go she travels with me, and I have babysitters and we’re registered in daycares around the world, and so it’s just we’re all in. It’s a constant whirlwind of managing everything. I think for us it is work that we’re doing together as a family, and I think it would be very very difficult if you have someone with that level of focus and the spouse or partner is not involved in it and really has a hard time understanding why things need to be a certain way and so it’s made it all in. I mean, we took that risk we have no side job or stable income outside of the company that we’ve created and we’re bouncing around between continents and our whole life is in this thing, but at the same time I think that’s what’s made it possible as well.

Diane Gillespie: It’s interesting that you’ve asked this question. I’m not the entrepreneur, actually Molly Melching is my sister so I’ve seen firsthand burnout. However she’s not burned out when she’s close to the people, to the source. She gets burned out by bureaucracy, management kinds of issues. But Ashoka and the Skoll foundation have recognized that social entrepreneurs have a high burnout rate, and they have designed retreats — she’s going through one of them right now — where people have a chance to sit back and do reflection and they can’t talk about work. This has been transformative. I really recommend that people think about scheduling a time for reflection with others who are doing similar kind of work. I mean certainly you can reflect alone, a lot of people do meditation, but it’s actually being with like-minded people who are experiencing similar kinds of pressures and frustrations to develop yourself outside of the boundaries of your work life.

Ruth Grace Wong: Amazing. So that was the last question I have. We will be here for office hours until about five, five ten if you want to talk to any of the panelists and do we have time for audience questions?

Helen Toner: I think rather than taking audience questions for everyone we should just thank our panel now and then you can come up and ask questions. The next talks will be in here at 5:15 which will be a round of lightning talks, so let’s thank our panel.

[Applause]

Thanks to Alan Kalbfleish at Pascal Press for helping me edit this transcript, and Elizabeth Van Nostrand of AcesoUnderGlass.com for facilitating bringing Diane to the panel