What Healthcare Organizations Must Know About AI Token Usage and Costs

How Can AI Token Usage Affect Healthcare Costs? 

Organizations have been especially interested in using AI to streamline clinical workflows and reduce administrative burden for care teams. Applying AI to healthcare-specific tasks requires a significant amount of reasoning. 

Prior authorizations, for example, can take up a majority of a physician’s time. Using AI to improve revenue operations (RevOps) provides significant financial ROI and will be a foundational AI use case across healthcare organizations, but it is a complex process that can’t be accomplished with a simple prompt. 

READ MORE: Cleveland Clinic grounds tech change in reliability and scalability.

One approach organizations are taking is to build AI agents that read through a doctor’s note, gather the supporting material for a prior authorization, find the right codes for the procedure from the specific insurance company and then submit it. But any anomaly — such as if a doctor’s note is unclear and the AI agent has to do more intensive or redundant reasoning — means AI token costs can rise from cents to dollars for the process, turning ROI significantly negative. 

From a physician’s standpoint, using an AI agent can save time, but from the finance department’s perspective, the organization doesn’t get the money back for that timesavings right away. The finance team has to reconcile the bill up front every month. So, if use cases are being created that have negative ROI potential, that requires proactive financial planning and real-time transparency linked fully to enterprise architecture, AI design and AI operations. 

Rethinking AI Token Usage and Its Potential Costs 

Standard AI usage can be relatively cheap and a significant competitive differentiator. When organizations start to rely on AI agents to retrieve information from an electronic health records system, cross-reference compliance standards or comb through payer guidelines, that involves more intensive token usage that’s not readily predictable. 

In contrast, cloud usage is fairly deterministic. There’s auto-scaling, and when organizations rent X server, they know it’ll cost Y per month. From a billing perspective, there aren’t many surprises. AI model usage does not currently map out like that. 

Organizations are also struggling to gain visibility into how their users are relying on generative AI services. Unless they have a gateway or some other method of observing usage, they just get one bill for all of it, whether or not users have had successful outcomes. 

DISCOVER: Healthcare IT leaders seek a new north star in the age of AI.

Think of an organization that has 10 AI agents and five departments using them, with multiple workflows, but it can’t track which AI agent is providing what value for that cost. Assigning an ROI to a single workflow that uses an agent is difficult. Enterprises can receive a surprise bill with no visibility into what the usage was applied to or where the negative ROI is occurring. 

AI is experiencing its own Jevons paradox — even though it’s getting cheaper to use, demand has increased, which means overall spending on it has skyrocketed as well. As costs decrease, people are finding more ways to use AI. Usage and visibility needs are rapidly exceeding organizations’ existing governance maturity and capabilities. 

Improving Token Consumption Visibility and Budgeting 

Healthcare organizations need not reinvent the wheel when it comes to managing token-based AI costs. There’s plenty from FinOps that can help to structure TokenOps, but like any enterprise architecture, it has to connect throughout the stack and become a key part of AI governance. It shouldn’t be siloed to one area of the organization.

Similar to the cloud, organizations must learn that finance teams should work with IT and stakeholders of AI use cases to get the right billing, visibility and guardrails. 

Establishing gateways and deploying frameworks such as OpenTelemetry that provide visibility into token consumption across all models and use cases can help. This will start providing you with the data for finance so you can at least see who's doing what, what the spending is and where the negative ROI is occurring with granular, transparent and real-time visibility. 

A gateway might work like this: A user sends a prompt through a gateway that your organization controls. That gateway can then determine which model to use for that prompt, track the user, how many tokens used and other important data, for every user or agent prompt that occurs. 

EXPLORE: Instead of models and GPUs, focus on the right workloads for AI.

Putting in an enterprise AI gateway between all the prompts, internal apps, chatbots and data feeds helps with smart routing and token budgeting. It can also provide data loss prevention, such as semantic-level scrubbing of protected health information, or Model Context Protocol (MCP) to FHIR integration for accessing sensitive health data. A gateway is something every organization should think about implementing and is a foundational component of healthcare AI architecture. 

Another change for organizations that gateways and governance maturity enable is measuring and optimizing “token yield,” which connects token usage to a useful business outcome. Out of a million tokens, what percentage of the tokens purchased and consumed contributed to improving direct patient care, for example? Where did the tokens that did not contribute occur and why? Token yield is an emerging metric to help organizations understand what they’re even spending money on tokens for. 

Best Practices for AI Token Management 

Although many AI models accessed through a service provider are essentially black boxes, organizations should do their best to understand each model’s capabilities. That way, they can determine the appropriate model for the appropriate workload. 

This may also inspire organizations to explore whether they want to host a model on-premises in a hybrid environment, which can be more cost-effective for certain cases. Incoming regulations may also impact AI agent architecture, such as ensuring there’s a human in the loop for certain steps. Some platforms and product solutions offer some simple-to-consume standard AI capabilities, such as Epic’s Penny, which is purpose-built for basic RevOps workflows to reduce implementation and operational complexity. 

DIVE DEEPER: How to use AI to power employee productivity.

Each AI model has a different cost per token. Let’s say an AI agent has five steps to accomplish for its designated task. For one of the steps, it may use one model, but for the next, simpler step, it may use the cheapest model. If these considerations are not taken, you may end up architecting an AI agent that isn’t using the models effectively and is generating significant costs. For the cloud, it’s similar to renting the biggest server to run the smallest workflow and then enabling unlimited scalability. AI agent design and model selection then become major parts of managing AI token costs and usage. 

As AI becomes more embedded in healthcare workflows, organizations that fail to manage token usage strategically risk runaway costs that can have serious impacts. Inefficient token consumption also introduces latency and reliability issues that can compromise time-sensitive decisions. Beyond cost and performance, thoughtful token management reflects a broader commitment to responsible AI governance. Providers that treat token efficiency as a core operational discipline today will be far better positioned to scale AI safely and sustainably as the technology continues to evolve.

This article is part of HealthTech’s MonITor blog series featuring analysis and recommendations from CDW experts.