Risk Assessment Engine - Detail

The Risk Assessment Engine is a feature I designed for DAT One as a part of efforts to combat fraud. The Figma prototype below is a recreation and not a direct 1:1 to the final handoff.

Background and Market Position

Background

DAT is the Craigslist of the trucking industry with extra analytics tools. DAT’s flagship product is DAT One, a service intended to combine various disparate functionalities across the freight industry into a single platform. The product connects people with freight (Shippers) or their representatives (Brokers) to the people who own trucks (Carriers).

Normal contacts fell through across the board as demand spiked during the height of the pandemic. Everyone had to scramble for people to carry their freight so they turned to DAT en masse for help sourcing new capacity. This made the pandemic a ludicrously profitable period of time for DAT and its users.

Market Position

DAT and its suite of products are the primary player in both load posting and analytics of the trucking industry. Microsoft when the first iPod was released is a good analogy for DAT; there is an overwhelmingly dominant company in the market but smaller companies are causing significant disruption.

Currently DAT falls into the “last subscription I cancel” category, but there is a risk of that changing in the years to come. The industry is changing rapidly and DAT has invested in new functionality as much as their competitors.

User Problem

Fraud

When there is a sudden increase in the profitability of an industry an increase in fraud is sure to follow. DAT received numerous reports of significant fraud from every customer segment starting mid-to-late 2020 which peaked in Spring 2023.

When asked: “What are you doing about the fraud on your platform?” the only correct answer is everything. DAT established multiple working groups focused on making their platform as secure as possible.

The main type of fraud DAT deals with is Double Brokering which is when a nefarious fraudster acts as an additional middle man to the process in order to siphon off funds. The accordion below contains a detailed description of this specific type of fraud.

Vetting potential partners to weed out the double brokers takes up a significant amount of energy and man hours. Roughly half a given user’s day was spent vetting potential partners.

The financial repercussions on a load that was double-brokered averaged around ten times the load’s monetary value through a combination of lawsuits and increased insurance premiums.

Our Answer

How might we reduce fraud and save our customers’ time?

The answer: The Risk Assessment Engine. It generates an easily consumable rating based on known indicators of fraudulent behavior both proprietary and public.

The Timeline

The project kicked off in late-June and DAT publicly announced the Risk Assessment Engine alongside a working version of the feature on our test server in October.

Research

Goals

• Confirm the value of the featureset. Previous studies indicated that scores or ratings would be successful but never to the point that it was added to the road map.

• Document our customers’ vetting process. This would contribute to the data model, helping us understand what should result in a “Good” or a “Bad” rating.

• Validate a single score / assessment. Internal conversations focused on whether or not simplifying the vetting process to a single score / icon would provide value to our customers.

• Uncover carriers’ concerns with being rated. The health of the carrier user base is of the utmost importance to DAT. We were concerned that rating carriers could cause them to switch to a competitor.

• What are we missing? I include this question in all my conversations with users; it is especially important in this context. Vetting potential partners in the freight industry is hilariously complex and the people working on the project have never done it themselves.

Studies

I collaborated with our research team to provide mockups, assist in moderating, and analyze the results from these studies:

• Foundational / Initial Study - A different study was already in the works as we were kicking off the Risk Assessment Engine and was converted to address this topic. We already had participants scheduled so we dove right in.

  Broker Specific Study - Interviews with Broker customers, focused on the segments spending the most time vetting partners

  Carrier Specific Study - Interviews with Carrier customers focused on the demographic most often emulated by fraudulent actors.

  Surveys - Pop-over survey prompts shown on pages relevant to our target user segments.

  Card Sorting - As our first study wrapped up sessions with users we knew a single score wasn’t the correct path so we targeted our broker users again to define categories for sub-scores

Recommendations

• Act quickly on the Risk Assessment Engine. Everyone, even the carriers who would be rated, were excited about Risk Assessment Engine and we frequently heard things like “This will be a gamechanger for us.”

• Information must to be available at the time of action. Our users continuously reiterated that the at-a-glance assessment needs to be visible wherever they come across a potential partner.

• Transparency and customization are paramount. After the initial excitement faded, we were consistently told, “I don’t trust you to make this decision for me.” The assessment should be a sign that points to a problem rather than proof positive of a potential partner’s nature. All companies use different standards; everyone we spoke to was concerned about handing over control of vetting partners to a third party. Transparency in calculation methodology and customization options would allay these concerns.

• Giving carriers a “bad” rating is useless without context. Users need to know the reason for the rating and have a path to view the source information themselves.

• Comparisons must be made against appropriate peer groups. An independent owner-operator of a flatbed truck should not be compared to a thousand truck fleet.

• The score needs to be updated as often as possible. The public data in the trucking industry is frequently out of date. Any scoring system needs to be validated by the proprietary data DAT has access to.

In addition we were able to gather a significant amount of feedback to pass on to our data science team to inform the construction of the data model.

Ideation

Pulling on previous studies and incorporating new information from the feature-specific studies mentioned above, I dove into defining user flows and sketching out interfaces.

Persona

We initially focused on our Broker segment of users for the Risk Assessment Engine, prioritizing scoring / assessing carriers. This decision was based on a combination of technical limitations and data access restrictions as well as the fact that brokers are the most common target to initiate Double Broker scams.

Company Search

While defining this flow it became apparent a Company Search function would be required to access the assessment. The most common steps for finding new partners are:

1. A broker posts a load

2. A carrier finds the load and calls the broker

3. They have an initial conversation

4. The broker then goes through their vetting process

5. The load is booked with that carrier or not

Brokers need to be able to look up companies. When we started this project, there was no way to look up a company in DAT One. I advocated for expanding scope to include a company search and that change was approved.

Flow

This flow represents the state once both Company Search and the Risk Assessment Engine are fully launched.

Company Profile

This sketch shows the Company Profile page where a more detailed breakdown of the assessment would be shown.

In-Context Assessments

Here is an early sketch of how scores could be displayed in the general load listing pages, reduced to an icon to save real estate for other important information.

Assessment Customization

This is a sketch of what customization of scores might look like including the idea of presets.

Data Model

We spent significant time discussing and constructing the data model.

Example: calculate the ratio of safety violations to the number of trucks and compare that number to similarly sized legitimate operators. In this case, a ratio outside the normal range could be a red flag.

We intended the Risk Assessment Engine to be a living model that compares the current carrier base against an expected curve. When results deviated too far from the expected curve, DAT would investigate and adjust the various parameters. The plan was to build an AI to manage adjustments to the data model.

Scoring System

We settled on a three-state icon as a scoring system:

• Good - No problems here, this should be a good partner.

• Check - We’ve noticed some information that might indicate a problem and you should check their information.

• Review - There is something fundamentally off about this company and you should review your plans to work with them.

We danced around these labels for a long time. Both our research team and DAT’s legal department did not want us to make authoritative statements like “This carrier is safe to work with.” Instead the assessment should be a signpost pointing to potential issues.

We would rate carriers on three areas:

• General Information - A carrier’s safety record and general operating information

• Insurance Details - The carrier’s insurance company, history of payment, and specific policy details

• Carrier Insights - Proprietary DAT data

There would be an overall single score but only as a representation of the lowest common denominator across the subscores. If they rate “Review” on insurance, the overall score would be “Review”.

Design, Prototyping, Testing

Design Scope

Final design scope for this project ended up being:

• Designs for the new Company Search

  • New primary navigation item

  • Landing Page

  • Predictive search

  • Results Page

• Design of the assessment

  • 3-State icons

  • Display of icons in various places across DAT One

  • Hover states

  • A card with a higher level of information on the Company Profile

The Company Search Page - new with this feature

The assessment shown in-line with the new company search results.

The Company Profile - with expanded Assessment

The expanded assessment card is intended to be the hover state on interacting with the icon assessment throughout the product. If you click a subscore, the page scrolls to the relevant information and highlights the card.

Prototype

Recreation, not 1:1 to final design handoff.

Testing

We announced the Risk Assessment Engine at DAT’s annual convention and held a rigorous round of usability testing / concept review with our customers. I moderated the sessions and wrote the final report.

The primary takeaways:

• Very positive response

• Widespread interest in how we would calculate the assessment

• Concerns over transparency and customization in-line with pre-announcement studies

Next Steps

Moving forward these were the next steps in continuing to develop this feature:

• Add more data to the fledgling model

• Test the score against known good and bad actors

• Improve information architecture and availability

• Explore customization of the assessment

• Continue to adapt and iterate based on feedback from users