Launching and Scaling Data Science Teams: DS Manager Playbook

Best practice playbook for building and managing your data science team

This post is intended to be a best practices guide to managing a data science team. This section picks up from my summary on Launching and Scaling Data Science Teams, and contains some overlap with the Data Science IC playbook. I am targeting a data scientist who has had success as an individual contributor, and is currently struggling with classic managerial questions: how do I scale, hire, evaluate, communicate? Within data science, this is an even tougher problem given the massive differences in the backgrounds of qualified candidates. This post is for people asking the question: “I’ve had success as a data science IC and gained responsibilities. I’ve been tasked with growing and leading a team at ________. Now what?”

The four sentence summary: Continuously align your org structure. Maximize your team’s knowledge transfer. Be intentional with your prioritization process. Emphasize the minimum sophistication of a solution.

Continuously Align Your Org Structure

Of all the sentences in my playbook four sentences summaries, this is by far the most general/poorly defined/opaque. My apologies. There is a reason, however: figuring out how to set up a data science team is really hard; there’s no settled “best practices”, it’s highly dependent on company and employees, and it’s a complete given that whatever setup you have now is not the correct one for six months ago or six months from now. The resources I’ve found and managers I’ve interviewed have given different and often contradictory solutions. I have looked at how to set up a data science team for long enough to be confident that there isn’t a right answer. The best you can do as a manager is continuously realign your org structure, data architecture, workflow, and culture, to make sure you are proactively scaling your team to fulfill the current and future needs of your company.

The biggest question in data science organization: is your team centralized or embedded? Centralized: a data science team that sits and works together, performing the role of an internal consultant. The “time” of the DS team is a resource allocated to business stakeholders. Embedded: individual data scientists sit within specific product teams, their day-to-day is highly iterative within that team, and their range of projects is limited. DraftKings has a centralized team, as does Zillow. Wayfair has both. Airbnb moved from centralized to a hybrid for this reason:

We began with the centralized model, tempted by its offering of opportunities to learn from each other and stay aligned on metrics, methodologies, and knowledge of past work. While this was all true, we’re ultimately in the business of decision-making, and found we couldn’t do this successfully when silo’d: partner teams didn’t fully understand how to interact with us, and the data scientists on our team didn’t have the full context of what they were meant to solve or how to make it actionable.

Chuong Do has an excellent piece on centralized/embedded, emphasizing the some of the major drawbacks of both approaches. Decentralization can lead to knowledge silos, fewer opportunities for collaboration, tougher to implement standards (from codebases to hiring), and difficulties sharing infrastructure/best practices. Centralization can lead to teams that lack the business context or buy-in of partners, and can also lead to situations where, “Data science is treated as a support function, answering questions from product managers rather than operating as true thought partners and proactively driving conversations from a data-informed perspective.”

While interviewing DS managers, I’ve generally found that everyone thinks they have a hybrid solution that takes the best parts of centralization and embedded. This is both impossible, and a goal to strive towards. Recognize that immensely talented people at wildly successful companies disagree on the best way to do this, so your role as a manager is to make the best decision for your team at this point in time, and to be completely attuned to when that structure might no longer make sense. Here are my thoughts on composing an org structure, with the caveat that they are likely to change in the future.

Start with a centralized team. When you start hiring data scientists, you will have many business teams that need support, and too few data scientists to embed. At small companies with small teams, knowledge sharing/business context/cross functional collaboration is easier. The office can be so small, you know what everyone’s working on. There is less of a “knowledge silo” risk.

Embed data scientists when algorithm-heavy projects become big enough that they will perpetually need a data scientist. Wayfair slowly embedded data scientists on their pricing, recommendations, and marketing teams; recognizing that these business units are so algorithm and analytics based they need to be led by business-oriented data scientists. This is true of Zillow’s Zestimate team as well. Both Wayfair and Zillow kept a centralized data science team to support the rest of their business. Be wary, embedding data scientists is harder from a hiring and resource management perspective — these ICs need to have high caliber business acumen, and they won’t work on anything else. Thus, timing is more crucial when hiring for an embedded structure.

Embedded data scientists should report to a central data science unit for as long as possible. Embedding is guaranteed to cause knowledge silos for business context. The goal here to is to align everything else for as long as possible. How data scientists work, collaborate, share code and results, evaluate success. This is optimizing for knowledge transfer, which we’ll touch on later.

When you are stretched thin, look outward and expand your team size. But be very picky, especially earlier you are. Never underestimate the the compounding costs of bad hires. Be deliberate, not only about evaluating talent, but if it’s the right talent. Do you need a research style hire to consistently refine a core part of the business, or do you need a quick-moving IC who can support many teams, complete many 2–3 day problems to get a 90% accurate answer, and move on to the next thing? How will those needs change in the next six months?

Finally, make staffing decisions based on the caliber and support of your non-technical business team. How capable are they? What does it mean to be an analyst at this company: light Excel skill or heavy SQL? How aligned with the data science team is the rest of your company? Are the business stakeholders clamoring for DS solutions, or are they resistant to change? Most of the companies I spoke with are tech-native (<10 years old, exposed brick, free La Croix, standing desks, foosball) and want more data science support, but some of the most interesting conversations I had were with a 40+ year old media company that prides itself on not using technology and automation. The advice I got from their COO:

We didn’t want these people wandering across the organization looking for things to fix. The worse think you can do is bring them in and set them loose. You have to have a high touch model with stakeholders that are willing to take on a data culture. Spend the time to sit down with the target team and hear their pain points, opportunity points, what they are evaluated on. Make them ask, “Can you show us what you’re doing, how it’s frustrating, what we do every day and how we win.”

The COOs experience was eye-opening to me; everywhere I’ve worked, the business has wanted more data science resources. That is not a universal truth. As a manager, you need to be aware of the data culture of your business, and the ability of your stakeholders to give your ICs the context and support they need to be successful. You must develop the point of trust between the DS team and rest of the company, which requires hiring ICs that can partner with your stakeholders. The more business oriented your data scientists are, and technically competent and open your company is, the easier this becomes.

Maximize Your Team’s Knowledge Transfer

I started my project to communicate this point, and I can’t emphasize it enough. IC data scientists can treat their job like an individual sport, and it is your job to make them work as a team. Kaggle and most data science educational resources are solo acts. You can’t assume a collaborative team with a defined workflow and coding standards will appear on its own, in fact, I’ve seen the opposite — where individuals work side by side, solving the same problems multiple times, on their local machines, in Python 2.7, 3.5, and R.

Your goal must be to increase knowledge transfer and mentorship between team members, which will result in better code and a stronger team. This requires a cultural shift away from “individual contributor” to “team member”. Without this shift, you cannot develop an industrial caliber data science team. You must emphasize that IC evaluation will be based not just on projects, but also developing others and improving the team workflow, standards, and tools. Start with code review.

Code review: Often, people assume this is a way to detect flawed code before it takes down a website. This is certainly good and necessary, (I would have liked code review when I forgot a “DESC” in an “ORDER BY” to sort product rankings, causing Wayfair to recommend items from worst to best for four hours, which at the time was a $200k mistake) but often code review as seen as NOT necessary when code can’t take down a website, or no one is technically as capable as the person who wrote it.

This misses the best part of code review: it’s a learning opportunity, a dedicated “two way street” knowledge transfer, forcing a team member to learn how someone else writes code. The least experienced member of a team can learn so much from reviewing someone else’s code that they don’t understand, and the most experienced member can learn how to coach. Code reviews should not just answer the question, “Will this break the website?”. Reviews should focus on:

• Is this code readable, resilient, well-commented, up to team standards? Is it the right level of abstraction/parameterization/modularization? (computer science principles)
• Are these data science and statistical methods sound? (data science principles)
• Are these joins efficient, are these tables structured optimally for daily inserts, are they designed correctly based on whether they should be read or read/write? This is essential to a team using Hive/HDFS/Presto or ETL in any way. (DBA principles)
• Is this work correct, actionable, convincing, well-presented, implementable? (business principles)

For knowledge transfer and code review to be effective, a team must have a shared set of beliefs about what “good” looks like. Accordingly, managers must define a set of team standards and workflow in a collaborative, flexible manner — with the caveat that individuals are encouraged to look for more efficient and powerful ways to improve those standards and workflow. Reverse engineer an answer to this question: “If a data scientist on my team figures out a better way to do something, how does that knowledge transfer to everyone else in a non-opt out way, daily?” Domino writes: the goal is to enable compounding collaboration in a modular way. “High-performing data science teams work best when they stand on the shoulders of those before them.” Here is an example of a codified team workflow/set of standards in my ideal world:

Our team uses the same versions and of libraries Python, Scikit Learn, Presto, Tensorflow with DBeaver as a SQL IDE, Jupyter as a Python IDE, and Airflow for job scheduling. People have different preferences and different knowledge bases in all of these areas. This is a strength — help us come up with new and improved pipelines and standards. To the same extent that individuals are encouraged and rewarded for having business positive impacts, they should CONSTANTLY be looking for more efficient or powerful ways to improve the team’s workflow and habits. There will be code review, both for business case, efficiency case, and whether the code is written properly. (PEP8 compliant, linters). Only properly formatted code will be committed. Code review is seen as an opportunity to teach and learn, and to catch bugs. Repos are organized at the project level with people working collaboratively, to get new ICs up to speed and give experienced ICs the ability to take on a teaching/leadership role.

Knowledge on business context and past team projects must be transferred as well. Companies have many solutions with many names: standups, storytelling events, learning academies, lunch and learns, brainstorming sessions, knowledge repos, wikis. Airbnb has a great blog post about democratizing data, and all their projects start with a one pager on “what is the point of the project, how am I going to do it, how am I going to measure success”.

Any of these solutions can work for a team at a given point in time, but many of the individuals I spoke with cautioned that they don’t scale well. On storytelling events: “We used to have two teams presenting once a month. Now there are nine teams — I have people that have been working for six months and don’t know what three teams do.” On brainstorming sessions: “It worked when we were smaller, now I have people I haven’t met giving me advice on how to solve a problem they haven’t seen before.” On lunch and learns: “People stopped going once it got too big”.

Code review must be executed in a non-opt-out way that scales when your organization triples in size, and knowledge transfer must as well. Your job is to align your team behind processes that work, and communicate that a critical part of the data science IC’s job is documenting the problem he or she tried to solve, how they solved it, and the success of the solution.

Be Intentional with Your Prioritization Process

A common refrain from both IC data scientists and business stakeholders was that they have no idea how projects are prioritized. Prioritization ranges from highly data driven (“We look at everything on a estimated revenue increase per data-scientist-week”) to deliberately not data driven (“There’s no prioritization necessary, we just work on what’s important, things don’t need to be justified or prioritized”) to somewhat political (“It’s a small enough company to know if you’re overusing or underusing data science resources”) to highly political (“Every team we support gets equal access to our time”). Similarly, the role IC data scientists play ranged from “I pick my projects” to “I don’t pick my projects, ten projects get assigned to ten people and they don’t know why.”

As a data science leader, you must force your company to align on both what “business impact” is and how to measure it, and you must make sure that conversations on prioritization are bottom up and top down between your ICs and business stakeholders. You must introduce the transparency not commonly associated with data science teams. One DS manager told me, “Each project has a different value it brings, we can’t compare them to each other.” If you can’t compare projects, how can you possibly allocate them other than politically? To the same extent, you must recognize that (again, relevant XKCD) in data science, it can be hard to explain the difference between the easy and the virtually impossible. Your data scientists certainly have imperfect information on the business problem to be done, your stakeholders certainly have imperfect information on how much work is necessary for a solution. You must source projects from business stakeholders and ICs, aggregate a complete set of information on the expected cost/benefit of those projects, assemble that data, and present it in a way that makes prioritization self evident to all parties.

Crucial to this process is establishing that your data science team exists to partner with, not as a service provider. Your team must develop trust and authority by justifying their value: sourcing good projects, getting huge wins, reporting out on it. This allows the team to start the culture: “If you have an idea, we’ll work with you on it, but we have a seat on the table.” You should avoid ticket systems and political, pro-rata allocation of team resources to the dependent stakeholders. The best way to do this is by proving that you and your team have enough business context and can partner with others — if you give us the space to solve these things, we will.

Emphasize the Minimum Sophistication of a Solution

There is a high level of overlap between the Know When You’re Done section of the data science IC playbook and this section. To retread: IC data scientists educated through academic or online means are not trained to evaluate the complexity of their algorithms. Many IC’s admitted they did not know when a problem was done, most stated that time management is something they’re working on improving. It is your job to establish the frameworks for accuracy tradeoffs with time and complexity, and to help your direct reports figure out when they they should move on. Recognize that your ICs will have a natural tendency to take too long to build solutions that are too complex. Imposing a guardrail is a huge part of the job, as is developing a team’s common sense and intuition on the right way to start solving a problem.

You must balance those standards, with personal development: recognize that people do want space to learn and grow, which requires solving new problems with new technologies. Data science organizations famously suffer from a fixation on tools that I’ve been on both sides of: I deliberately placed myself on lower priority clickstream projects to learn distributed computing principles/Hadoop/Hive earlier in my career, and I’ve been frustrated when teammates have done the same, building convolutional neural nets for the sake of building convolutional neural nets (we’ll figure out how image classification improves the business later). Currently, this fixation on tools is Tensorflow, Spark Streaming, GANs and RNNs, but it has been other things in the past and will be other things in the future. Domino calls this, “A culture of ‘silver bullet’ thinking where some data scientists believe just getting Spark Streaming and TensorFlow will solve all their problems…In many cases, data scientists wear their tool wrangling as a badge of honor and the badge is wrapped up in their identity.” One IC told me, “I know Tensorflow isn’t the right way to solve my problems, but there is always peer pressure to use the cool thing. The company blog posts and learning academies aren’t about reducing customer churn with random forest classifiers. This industry moves so fast and I need to keep up.” Another said she measures her personal growth what new technologies she’s implemented.

Neither data scientist is not being unreasonable here. Having Tensorflow and Spark Streaming experience can make a big difference on a resume, and you don’t learn as much solving slightly different problems with the same tools. DS managers must be aware of the tension between solutions with the correct tools/complexity, and solutions as learning opportunity. You must balance properly engineering solutions with team happiness and development. Like everything else, recognize that this is something you need to be intentional, transparent, and flexible on. A great solution is to give your team a clearly defined time period where they can work on something they want to with the tools they want to learn, but also provide them guidance and pushback: “If you’re want to do a more complex think, tell me why you’re going to pay that cost.” Talk to your team to make sure they feel like they’re learning, and allocate “stretch” projects accordingly when you have the opportunity to allow someone to solve a real problem with a solution or technology that is new to them.