Is Anybody Actually Good at This?

Fire GarPax. Joe Dumars on two phones. Isiah Thomas and the Knicks. Every fan knows a front office that blew it. It isn’t a debate; it’s something we were raised on.

I was raised on the other side of it. Growing up in Massachusetts, I believed the public schools and the universities made everything here run a little smarter, and that the Celtics hung all those banners because Red Auerbach knew things nobody else knew. Like every fan, I also assumed that in the hot seat I’d know better, which is why I’ve spent more nights arguing about trades than watching games.

Here’s the trouble with believing that. There’s an old con called the Baltimore stockbroker. He mails 10,240 strangers a stock tip: half are told it goes up, half that it goes down. Next week he writes only to the 5,120 who got the right call, and splits them again. After ten weeks, ten people have watched him call the market ten times in a row without a miss. They’d hand him their savings. He never knew anything.

Thirty front offices have been making calls for decades. Some of them were always going to look like geniuses. The question is whether any of them actually are.

I built a draft model and a trade model because I was bored. The draft model liked Jokić more than any team did: 4th in his class, when the league took him 41st. It liked Shai 3rd in 2018, when he went 11th. It still had no idea they should have gone first, Jokić by a mile. In fairness, neither did the GMs we call geniuses. So, two things to find out: whether our heroes are just our villains with better luck, and whether a model like mine could beat them. I’ll take the second one first, because if the model can’t beat anyone, you shouldn’t trust it to judge them.

Could the model win? Yes

Before you trust any of this, a confession: I know how these careers turned out, and so does anything I build today. The fix is to make the model take the test the GMs took. Every draft on the board was predicted by a version of the model trained only on the drafts before it: 2009 never saw Steph Curry’s career, and 2014 never saw Jokić’s.¹ The trade grades work the same way: each player’s rating, his Marginal Plus Minus (mPM), is the one known on the day of the trade. Two things I can’t scrub. One, the scouting reports were scored by an AI model trained after these careers played out; it sees only the report, never the name, but a report can be recognisable. Two, I chose how to build the model knowing who turned into who. So treat the gap below as the most a club could have had, not what it would have.

Here’s the test. Take one pick at a time. Everything before it happens exactly as it did; at that pick, the club takes the board’s favourite of the players still available instead. Then add up the gaps, club by club.² Each pick is scored on what the player was worth to the club that drafted him: everything he produced in his first four seasons, at what it would cost on the open market. After that, if his club re-signs him, what he produced minus what he was paid, which goes negative when he falls short of his deal.³

The board would have changed 86% of 1,133 picks since 2007, and every club but Miami would have come out ahead with it, by +$105M a pick on average. Miami did beat the board, just not by much: $6.2M a pick over 22.

Most of the gain is in the first round: +$156M a pick, where the board’s player turned out better in 65% of the picks it changed, against +$53M in the second. A lot of it is one man. Jokić’s career is worth about $2.1B to Denver in today’s money, counting what’s left of it, and the board takes him at 24 different picks in 2014, so those swaps alone are about two-fifths of the gain: per changed pick, +$122M with them and +$72M without. The typical first-round swap gains +$37M. Take away the 10 players the board benefited from most, Jokić included, and it still comes out ahead, if only by +$13M a pick; it takes 16 to wipe the gain out.⁴

The biggest swaps each way, in the players’ value to the club that drafted them over their careers, in today’s money.

The board isn’t infallible either. It would have passed on Stephen Curry, Donovan Mitchell and Paul George. If Jokić is its best idea, Nick Calathes is its worst: it takes him at 34 different picks in 2009, over Jrue Holiday, Ty Lawson and Jeff Teague, and those swaps cost about $2.4B. It misses stars too, just more cheaply than the people doing it for a living. It also can’t get tricked by Thon Maker’s mixtape (like I was).

Trades get a similar test. Let the model veto every trade it graded negative for one club, and let that club simply not make it.⁵ The model is a grump: it would have vetoed 56% of 1,948 trade sides. It was also right to be. The trades it hated lost $54M a side on average, the ones it liked made $43M, and every club comes out ahead with the veto. That isn’t just from trading less: vetoing the same number of a club’s trades at random does worse for all 30 clubs, and clearly worse for 28 of them.⁶

Then there is my team. The veto would have saved Boston $1.4B, and vetoing the same number of Boston’s trades at random would have cost it $389M, so the model found the Celtics’ bad trades and not just any trades. (Sigh.) No club comes out behind with the veto, but it does throw away a few legends along the way: Utah’s 2013 trade of Rudy Gobert and Erick Green (+$910M), Chicago’s 2004 trade of Luol Deng and Jackson Vroman (+$625M) and New Orleans’s 2013 trade of Jrue Holiday, Pierre Jackson and Nerlens Noel (+$403M).

Heroes, villains and a lot of luck

Now the first question: do front offices differ from each other? Not who made the best trade or drafted the best player. Somebody always did. Whether knowing who made a decision tells you anything about how it turned out, and whether that difference is clear enough that it isn’t just the dice.⁷

Two things to be good at. Process is making the decision my model liked on the day: for a trade, the Trade Lab’s grade; for a pick, the same flip as above, scored on draft night, which is how much of the draft board’s value a club left on the table by not taking the board’s favourite. Process has no luck in how it turned out, since it’s my model’s own grade of each decision. It can still be luck in which decisions came a club’s way, so I check it against a reshuffle of who made which decision. Results are what happened, and here luck is everything, so I keep four kinds apart: what a trade returned; what it returned beyond what trades graded like it usually return; out-picking the board (did the player you took produce more than the one the board would have taken?); and beating the forecast (did the player you took produce more than the board expected of him?). There are also two ways to sort the decisions: by franchise, and by the person running it on the day.⁸

Here’s how you tell. Take the real decisions and shuffle who made them: every club keeps the same number of decisions in the same seasons, and a draft pick is only swapped with picks from the same draft and the same range of slots, since a top-five pick has far more to win or lose. Do that 20,000 times and count how often the shuffled clubs end up as far apart as the real ones. That’s the luck test.⁹ Then there’s the stockbroker problem: with 30 teams, one of them will always look special. The fix is a false discovery rate, which asks how often you’d be wrong calling a club this far out real, given how many you looked at.

Trades

Process first. 2,471 trade sides since 1999-00, and the average one graded −$11M on the day.¹⁰ Utah’s trades graded best, +$8.3M a side, then Boston’s, +$7.3M (invest in schools, the children are our future), and Houston’s, +$5.6M. By executive, Magic Johnson is first. None of it is more than the luck of which deals came their way: reshuffles spread the clubs this far 49% of the time, and the executives 96%. The model can tell a good trade from a bad one. It can’t pick out a good trader.

Now what the trades actually returned. Do some clubs’ trades work out better than others’ by more than luck? Boston’s returned +$12M a side, 3rd of 30, behind Utah and San Antonio. To check whether that order means anything, I shuffle again, and 75% of the shuffles spread the clubs out at least as far as they really are. A single trade can land $125M better or worse than average, so a club’s average over a few dozen trades lands wherever the dice put it. The executives look the same. Only Utah clears even the usual bar on its own, and with 30 clubs to look at, calling it real you’d be wrong more often than not.¹¹

What each franchise’s trades returned, per trade side, trades through 2020-21, in today’s money, with the range that covers 95% of what the dice alone would do. Top five and bottom five.

What about the villains from the top? One of them is about where you’d put him. Isiah Thomas’s trades graded 30th of 111 executives on the day, and returned the second least of all 98: −$134M a side, over 8 trades, ahead of only Tony DiLeo’s five. That clears the usual bar on its own, but with 98 executives to look at, calling it real you’d be wrong more often than not.¹² The rest are ordinary. Joe Dumars is 25th on what his trades returned and Gar Forman 68th, on trades that graded a little above average on the day: −$10.0M a side, against −$11M. John Paxson, the other half of GarPax, is 8th, on trades the model graded 85th of 111: it didn’t like them, and they worked. As franchises, Chicago is 26th of 30 and New York 29th: below average, not far enough below to tell.

The draft

Process first again. On draft night the average pick left $99M on the table: the gap, by the board’s own forecast, between the player it liked best and the one the club took. New Orleans, Dallas and the Lakers left the least; Phoenix, Charlotte and Washington the most. Shuffle again, this time only among picks from the same draft and range of slots, and the clubs differ by more than chance: a spread this wide turns up 2% of the time.¹³ New Orleans, Dallas, Philadelphia and Milwaukee clear the usual bar on their own, but not once you allow for all 30. By executive, the spread is ordinary (32%), but one man clears the bar on his own: R.C. Buford, whose Spurs left $52M a pick on the table against $97M for the average executive.¹⁴ The caveat: “good process” here means agreeing with my model, which might be good drafting or might just mean liking the same kind of player it does.

Now what the picks actually returned, and the question I care about most: does anybody out-pick the board? It’s the flip from the top of this column: one pick at a time, the player’s career value to his club against the board’s pick’s at the same spot. When the board would have taken the same player, the gap is zero, so Denver gets nothing for Jokić: at 41, the board takes him too. The board beats every club but Miami. The question is whether it beats some by less than others, by more than luck would. It doesn’t. Shuffle again and the clubs spread at least this far 94% of the time; the executives, 68%. Nine executives’ picks did beat the board’s on average, led by Larry Riley and Jeff Peterson, and Miami beat it as a club, but none of them by more than luck would give.¹⁵

The last question is different: did their players beat the forecast? Did the player a club took return more than even the board expected of him, over his whole career?¹⁶ This is where Jokić lives, and it isn’t evidence of choosing better than the model, since the board would have made the same pick. It’s mostly luck. The one skill that could hide in it is development: what a club does with a player after it takes him. If anyone is better at that, it doesn’t show. Shuffle again and the executives spread this far 45% of the time, the clubs 21%, with Oklahoma City first.¹⁷

Here is what the top of it is made of:

The ten picks since 2007 that landed furthest from what the board expected them to return the club that drafted them, all ten above it: classes through 2025, over the seasons each has played, in today’s money.

Larry Riley, Ed Stefanski and Larry Bird lead the executives, and Lawrence Frank is next: Riley on Stephen Curry, Stefanski on Jrue Holiday and Frank on Shai Gilgeous-Alexander, whom his Clippers took 11th and traded a year later for Paul George.¹⁸ Ten picks, ten different executives: each of them did it once, and one draw can’t tell an eye for stars from standing in the right place.¹⁹

One more way to see it. If nobody were better than anybody, luck alone would push about 13 clubs and executives over the usual bar in trades, and about 10 in the draft. The real counts are 6 and 9: about as many as you’d expect if everyone were equally good and some just got lucky.

None of this says nobody is better. It says a club would have to be enormously better to show up. To be picked out reliably from 30, a club would need to return about $53M a trade side more than the rest, or $216M a pick. A real edge of a few million a decision, which is a lot of money over a decade, is invisible in 22 seasons of trades and 19 drafts.²⁰

Why it’s so hard to tell

Part of it is that the job pulls front offices together. They copy each other: the analytics department one club builds this year, the rest build next year, and a good idea doesn’t stay anyone’s edge for long. Part of it is that the worst of them don’t last. A truly terrible decision-maker gets fired before making enough decisions to show up in a test like this, so what I’m comparing is mostly the survivors.²¹ Neither means the job doesn’t matter. It means the people who keep it are closer to each other than to the people who lost it. It may also be why the model beats almost all of them at once. If everyone learns from everyone else, they share the same blind spots, and a model that never falls for the room’s favourite idea gets to step around all of them.

Every one of these trades is on the trades page, graded on the day and on how it turned out, and every one of these drafts is on the draft board. Go find your team’s favourite deal and see whether the model would have let it happen. There’s plenty of room for improvement in there. My grandfather would be thrilled.

Figures: every trade side on the trades page, 1999-00 to 2025-26, and every pick on the draft board, 2007 on; executives from Basketball-Reference’s team pages; in today’s money, as of this build. Wins are on this model’s own scale and dollars at the open-market price of a win; the method explains both.

1. Statisticians call these out-of-fold, or walk-forward, predictions. Not ordinary cross-validation, which would predict each class from all the others, later ones included, and let what the league learned afterwards leak backwards. The board’s calibration and its contract fits follow the same rule: earlier classes only. ↩
2. The board chooses only among players who were actually drafted, at that pick or later, because an undrafted favourite has no career to score. ↩
3. A career still running is projected from here: his wins from the Trade Lab’s projection, valued by the same rules, with later deals weighted by the chance he re-signs and paid the max or what players like him re-sign for. The recent classes are mostly projection. ↩
4. Without the twenty largest gains, +$80M per changed pick. Each club’s number is its own: the board takes Jokić at picks of 17 different clubs, so the clubs’ gains don’t add up. ↩
5. Nothing else changes: no knock-on trades, and the other club’s ledger is untouched. Trades through 2020-21, so the results are mostly in, and no sign-and-trades. ↩
6. Clearly worse after allowing for having looked at 30 clubs: a 5% false discovery rate. ↩
7. Every number from here on is raw: a club’s or an executive’s own decisions, added up, with nothing taken out. ↩
8. The executive is whoever Basketball-Reference names on each team’s season page, the president of basketball operations or the general manager. In a season that names two, trades before March go to the one who was there the season before, and the draft goes to the one who is there the season after. Executives with fewer than 5 trade sides, or 5 picks, are left out. ↩
9. Randomization inference on the F statistic of the units’ raw averages, with each unit also tested against its own shuffled average. ↩
10. Leaving out sign-and-trades: 125 of them, 292 sides. In a sign-and-trade the player picks his club and the trade is the paperwork, so the club he picks grades like it pulled off a steal it didn’t make. ↩
11. The Benjamini–Hochberg q-value: 0.79 at best. No franchise or executive, in any results test, trades or draft, is picked out at a 10% false discovery rate. ↩
12. The Benjamini–Hochberg q-value: 0.91. Thomas ran the Knicks from December 2003; the trades are dated against the day each executive took over, not the season. ↩
13. A franchise’s average in odd-year drafts also predicts its average in even-year ones, at a correlation of 0.55 (against reshuffled teams, under 1% of the time), though part of that is bad clubs picking high year after year. ↩
14. A false discovery rate of 3%: allowing for having looked at 68 executives. ↩
15. Testing every club and executive against its own reshuffle, the best false discovery rate is 93% for a club and 71% for an executive. ↩
16. For a player his club still holds as 2025-26 ends, the rest of his career is projected and added to what he has already returned: his wins from the Trade Lab’s projection (today’s mPM, the aging curve, availability), valued by the board’s rules, which count the rookie deal at market, a later deal at market minus the salary, and future deals by the chance he re-signs and what he’d be paid. The board’s pick at the same spot gets the same treatment. Players already gone count only what they returned. ↩
17. No single club or executive comes close to standing out either. Tested one by one and allowing for how many I looked at, the best case would still be a false alarm 26% of the time for a club and 63% for an executive, against a bar of 10%. The recent classes are mostly projection, so this could move as they play. ↩
18. A drafted player keeps counting for the club that drafted him even after it trades him, for as long as he stays where it sent him: a trade cashes him in. Walking away in free agency or being waived ends it. ↩
19. Telling skill from luck means estimating how much a person’s results vary around their own average, and you can’t estimate a spread from one observation. Nobody gets ten more drafts. ↩
20. Detectable with 80% power at a 5% false discovery rate: about four of a typical club’s standard errors. ↩
21. An economist would call the second one selection: the sample is cut on the outcome. Both shrink the spread I can measure, so the true spread among everyone who ever got the job is probably wider than the one here. ↩