With its vast knowledge of strategy and technique and a wealth of  financial resources at their disposal, NFL organizations not only have the potential to discover if that linebacker prospect fits their scheme, but they can also find out what he did with that blue pencil sharpener in Mrs. Beam’s second grade social studies class. So why do they still have a huge opportunity to improve as evaluators of talent? The answer is in the process.

During his 2011 NFL Draft Confidential special that aired on ESPN,  Bill Parcells describes football as a “talent-acquisition” industry.  And during the show’s next 90 minutes, Parcells provides great insights into the scouting process, how it generates a draft board, and its economic impact on the game.

But what got my attention as a former operations manager and director of a service sector business is the issues that NFL teams face to do consistent, accurate, and quality work.

One of the things I took away from the show is that the NFL has a lot in common with other businesses – especially those in the service and manufacturing sectors.

As with other industries, the word tenure isn’t a common way to describe jobs in the NFL. General managers, coaches, and personnel feel tremendous pressure to win now, which can lead to a results-driven mentality.

This is perfectly natural. However during those long hours of work in an urgent quest to attain these results, it’s difficult for an organization to feel the same urgency to scrutinize its processes that are in use to reach its goals.

The Inexact Science of Evaluation

During the ESPN special, Parcells repeatedly described talent evaluation as an “inexact science.” Once again, this is no different than the service and manufacturing industries where statistics are essential to measure productivity.

But what makes talent evaluation an inexact science is the fact that statistics cannot provide a full or accurate measure of an individual’s performance. Nor can statistics alone gauge talent or project future performance.

Because measuring and projecting individual performance deals with both objective and subjective criteria, it is vitally important that there is a strong methodology in place to ensure that evaluators are consistent with their approach to the work. Many service and manufacturing businesses have figured this out by embracing an approach that I call “quality-driven processes.”

These processes not only generate results that are more accurate and productive, but the structure of the process itself also helps these businesses get better at what they do with each passing year while saving money.

Don’t Blame The Evaluators, Focus on The Process

Based on the processes Parcells explained in this show, as well as conversations I have had with former scouts, I believe that as knowledgeable as teams are about the game, they lack of a high-quality, well-defined evaluation process. The current process isn’t designed to help them continuously improve and I think it is a reason why NFL teams frequently contribute to their own scouting mistakes.

Exhibit one is the NFL’s grading system. Most teams use a grading system that inherently create a high level of variability. And when individual evaluators have a different understanding of how a system is supposed to work because the system isn’t well-defined, differences of opinion among evaluators can be avoided.

Here is the basic system that most NFL teams use. There will be differences in the range of numbers, but this is essentially it.

Typical NFL Prospect Grading Scale

• 9.00 –  A player for the ages (Jim Brown).
• 8.00-8.99 – A perennial All-Pro.
• 7.50 – 7.99 – Future All-Pro.
• 7.00-7.49 – Pro-Bowl-caliber potential.
• 6.50-6.99 – First-round-caliber player with Pro Bowl potential.  
• 6.00-6.49 – Potential to become a quality NFL starter.
•  5.50-5.99 – Potential starter and likely first-day pick.
• 5.10-5.49 – Potential to make an NFL roster and contribute.
• 5.01-5.09 – Has a better than 50/50 chance to make a roster.
• 5.00 – Has an even chance to make a roster.
• 4.75–4.99 – Training camp player.
• 4.50-4.74 – Potential invitee to an NFL training camp.
• 4.00-4.49 – Needs developmental time in another league.

On the surface this might seem like a very clear scoring system, but it’s not clear at all.

After talking with former scouts with recent stints in the NFL within the past 10 years, all of them explained to me that the score is a “hard grade.” This means the scout watches the player, writes some notes, and then assigns an overall grade according to these type of general definitions.

The potential problem is that it appears that none of these grades explicitly define what NFL scouts should be considering when evaluating a player:

• Athletic skills (speed, flexibility, strength, agility, etc.)
• Position-specific techniques (pad level, routes, blocking, etc.)
• Conceptual knowledge of the game (vision, pocket presence, etc.)

You might argue, of course scouts understand what to look for when evaluating a player – that’s their job!

However, look at any industry that hasn’t really examined its processes and there are frequent errors that occur among employees with regards to how they define the criteria they use to evaluate performance. This is especially the case in the NFL.

According to ex-scouts there is an unwritten truism called the 25/25 Rule, which describes the tendency for NFL organizations to fire veteran scouts and replace them with new scouts in their mid-twenties (25) at an annual salary of 25K. This practice often occurs when teams change leadership. It also keeps scout salaries low – a nice side benefit.

However, the 25/25 Rule also creates an environment where scouts are more reticent to stand up for their takes on players. Working for the NFL is a dream job for many and job security is already a tenuous thing.

Training for scouts also appears to be lacking with some teams. One former scout for a team in the AFC North explained to me that his team never had a defined training for the position. New scouts were put to work and given the option to attend position meetings at the team complex when not engaged in 90-hour work weeks.

He explained that the skills portion of his job interview was to watch film and write notes about what he saw, but he never knew exactly what he specifically did that separated him from the other candidates for the job. Obviously, he was doing something right, but imagine what kind of things he was doing wrong that were tacitly reinforced because he didn’t get any formalized training.

The dynamic this creates within a team’s scouting department is similar to any business where there is a group of individuals made up of  different ages, different levels of job experience, and different levels of knowledge about the job.

Think I’m wrong? Next time you’re at work, gather a group of your peers and ask them all to define in writing how a simple task: how the receptionist should answer the telephone.

Then pretend you’re the receptionist answering the phone while they grade your performance using this 1-5 scale:

1. Poor
2. Fair
3. Meets Expectations
4. Good
5. Excellent

Unless your company has very clear guidelines for every process, you’ll not only find that each of your peers has a different answer how the receptionist should answer the phone, but they also will have a very different idea of how well/poorly the job was done. To compound the problem, ask them after the fact how they define each of these grades and you’ll likely get a different answer from each person.

The problem this example underscores is that a lack of clearly defined criteria increases variation among those assigned to judge performance.

Where one scout might define a player’s performance as Good, another may define it as Meets Expectations. The difference between these two scores in the NFL might be the difference between a player projected to become a starter and one projected only to make a roster.

Undefined Processes + Process Variation = Poor Results

Former 49ers, Rams, and Cardinals scout Dave Razzano’s account [as told to Yahoo! writer Michael Silver] of a much-publicized run-in with Rams GM Charlie Armey over his scouting report of Utah QB Alex Smith is a glaring example.

Razzano’s refusal to fall in line with the widespread belief that Smith was a big-time quarterback prospect led to a heated confrontation with Armey in a meeting at Rams headquarters a couple of weeks before the ’05 draft. Razzano’s report on the former Utah quarterback opined that Smith was “not as good as our backup, Jeff Smoker. Backup only for the Rams.”

Armey, who declined to discuss the incident after it was initially reported by Santa Rosa Press Democrat’s Matt Maiocco, solicited the input of other scouts and coaches who’d studied far less tape (if any) of Smith, who ended up being picked No. 1 overall by the 49ers.

“There were 12 guys around the table, and Charley had them rate him on every attribute – arm strength; accuracy short; accuracy long; judgment; game management; ad-lib ability under pressure. And he put a highlight tape on the projector. I mean, obviously, he’s gonna be 30 out of 30, and every throw’s a great pass … it’s a highlight tape!

“He said, ‘Are you gonna sit there and be stubborn? Why can’t you see what we see?’ I got heated. I said, ‘I’ve watched seven tapes, and I’m not changing my grade.’ He told one of our assistants, ‘Go get all seven tapes.’ I started screaming, ‘You’re gonna look at highlight tapes? That’s how Akili Smith got drafted!’ [Scout] Tom Marino had me in a bear hug. I just lost my mind.”

Not only did they differ on how to score the player’s overall performance, but also what type of criteria (highlights of preselected plays or actual game conditions) to use to arrive at that grade.

Another former scout of an AFC East team said he saw his peers paraphrase material from print and Internet publications to complete his scouting reports.

These examples aren’t meant to cast NFL teams in a bad light. These are common issues in any industry where its processes aren’t given the scrutiny they deserve.  These are symptoms of poor processes management and poor processes create variation that can have a negative impact on the team on the field as well as economically.

Overestimate a player’s skill level and a team could wind up overpaying a player who cannot start for them. Underestimate a player’s skill level and a team can miss on the opportunity to acquire him.

This is the kind of variation that can be corrected with a good process, which:

1. Defines specifically in writing what the team values in players.
2. Defines which settings scouts can use to grade players.
3. Clearly defines a grading system.
4. Uses a system that incorporates all skills and techniques that a team wants to see from its prospects into the grading system.
5. Prioritizes the importance of those skills and techniques with a weighted score the contributes to the overall evaluation.
6. Scores players as only meeting or not meeting expectations of those scoring criteria rather than using a highly subjective number system.

No evaluation process is perfect, but I believe NFL teams will see great improvement to its talent evaluation process – and bottom line – once it decides to explore best practices in process management.

Doing so will help them create an evaluation process that will help their scouts and management stay on the same page and prevent issues that they have control over. In addition, when they do have a vast difference in opinion the process should be structured so it will help address the larger problem and continuously refine what they are doing as the game evolves.

The Rookie Scouting Portfolio already does this because it adopts and customizes best-practice methodologies for its performance monitoring that Fortune 500 companies use in the service and manufacturing sectors. I believe if an NFL team, and its wealth of resources and vast knowledge about the game, applied similar best practices in process management they would produce an incredibly strong scouting department that could give them a huge edge over their competition and ultimately save their team money.

Although I came to the Rookie Scouting Portfolio lacking NFL-caliber knowledge or football experience, I possessed the knowledge to build a process that would help me use others’ knowledge of the game to successfully evaluate the skills of NFL prospects. Moreover, my process continues to help me refine my knowledge and end product.

I don’t think it will be long before teams explore this avenue to improve their scouting process. The Saints recently purchased a system to refine their tracking and management of data. This is a step in the right direction. However, the same could be said about the use of video tape to record games for scouting.

What NFL teams need to consider is that a system only works well if you have a strong process to incorporate that system to fulfill an overall objective.

Otherwise, it’s just expensive technology.