How Vision AI Models Analyze Skin Images Clinically

A photo of a rash or a mole looks simple to read. A dermatologist sees shape, color, border, and location all at once, and weighs them against a patient's history before saying a word. That's the task vision AI has to replicate, and it's why the simplest version of this technology, a model trained to spot one thing in one type of image, keeps falling short of what clinics actually need.

Why analyzing a skin image is hard

Skin disease covers a huge range: common acne and rosacea, inflammatory conditions, pigmentary disorders, and cancers that can be fatal if caught late. A model trained only on dermoscopic images of melanoma has no way to help with a flare of rosacea or a patch of eczema. Reading a single image correctly means taking in the shape of a lesion, how it's colored, where it sits on the body, what type of image it even is, and the patient's own history, all at the same time.

Researchers behind PanDerm, a foundation model for dermatology, put the problem directly: current AI models for the field "remain limited to isolated tasks" and "struggle to integrate various data types and imaging modalities, reducing their utility in different real-world clinical settings. That's a fair description of where the field stood for years. A real dermatology workflow asks a system to screen for skin cancer across the whole body, work through a differential diagnosis spanning hundreds of conditions, read multiple image types, interpret pathology slides, track a lesion over months, and estimate how a condition might progress. No single image classifier was ever built to do all of that, and clinicians see that breadth of presentations every day, so platforms built for patient care, Nolla among them, pair AI image analysis with a licensed clinician who reviews the findings and signs off before anything gets prescribed.

The pipeline from raw pixels to a clinical signal

Any clinical AI claim rests on a sequence of smaller steps, and each one produces an output the next step depends on. The first is segmentation: the model has to isolate the lesion or rash from the surrounding skin before it can say anything useful about it. Get that step wrong, and every measurement built on top of it is unreliable.

Once the area of interest is isolated, the model measures morphology, the visual features a dermatologist would name out loud: how irregular the border is, how many colors appear within the lesion, the texture, the overall shape. A model extracts these descriptors instead of an eye, but they are the same ones used in clinical notes.

Location matters just as much as shape. A pigmented spot on the face carries a different set of likely explanations than the same-looking spot on the sole of the foot, so distribution and body-site analysis is its own step in the pipeline, not an afterthought. If a lesion is tracked over time, the model also has to compare a current image against past ones to flag meaningful change.

Earlier tools handled each of these tasks with separate convolutional neural networks, trained on carefully curated dermoscopic image sets. They did well on the narrow job they were built for. What they couldn't do was carry that performance across different types of images or different conditions, which meant a clinic needed a separate tool for every task in the pipeline rather than one system that handled the sequence end to end.

What foundation models changed about this pipeline

Foundation models changed the economics of that pipeline. Instead of training a separate model for segmentation, a separate one for morphology, and another for distribution, a single architecture pretrained on a large, varied set of images through self-supervised learning can be adapted to each stage. PanDerm was pretrained on more than 2 million skin images spanning clinical photography, dermoscopy, total-body photography, and dermatopathology, then evaluated across skin cancer screening, risk stratification, differential diagnosis of common and rare conditions, lesion segmentation, longitudinal monitoring, and metastasis prediction.

One of the more important findings from that work is about efficiency, not just breadth: PanDerm reached strong performance using a fraction of the labeled data that earlier approaches required. In medicine, where expert-annotated images are expensive and slow to produce, that's a real constraint being loosened, not a minor detail. A newer system called DermFM-Zero, also referred to as PanDerm-2, was described in a February 2026 preprint, and it was trained on more than 4 million images paired with clinical descriptions. Preprint results haven't gone through peer review, so you should read those figures as early and unconfirmed rather than place them alongside PanDerm's published findings.

The framing that fits these tools best is augmented intelligence: systems that raise the floor for a non-specialist clinician and extend the reach of pattern recognition a dermatologist has spent years building. That framing matters for what comes next, because the pipeline doesn't stop at the image.

How multimodal reasoning adds patient data to image data

An image alone often isn't enough to settle a diagnosis. Acne, rosacea, and seborrheic dermatitis can produce strikingly similar photographs, and what separates them is often found outside the frame: what triggers a flare, whether hormones are involved, how a patient responded to a prior treatment. Multimodal models close that gap by processing patient narratives, electronic health records, pathology reports, and clinical notes together with the image, making the photo one input among several.

Matching a visual finding to a pathology result takes text and image reasoning working together, and multimodal models trained on large, labeled clinical datasets, combining patient history, symptom narratives, and images, are where this kind of reasoning becomes workable inside a telehealth visit. When a model processes both the photo and the patient's own account of onset, triggers, and prior treatments, clinician review starts with a far richer signal than the image could provide by itself. Nolla's platform is built around this kind of combined input: it feeds structured patient history alongside the photo, so the physician reviewing the case has both pieces in front of them before any decision gets made.

The quality of a multimodal assessment depends partly on what the patient submits alongside the photo. A more complete clinical history, written clearly, gives the AI triage step a stronger signal to work from and gives the reviewing clinician more to evaluate.

Image quality, input modality, and model performance

A model that performs well on a standardized clinical photo doesn't automatically perform the same way on a photo taken on a phone in a bathroom with mixed lighting. That gap between controlled and uncontrolled image capture is one of the more practical constraints in deploying any of this to patients directly.

A 2026 study tested an AI skin assessment framework, built on a CoAtNet-4 backbone, across DSLR, tablet, and smartphone images from 1,099 participants, grading results against assessments from five board-certified dermatologists. Performance stayed high on mobile images overall. But pigmentation-related accuracy dropped noticeably when the input moved from DSLR to smartphone. Correlations with dermatologist grading were strong for wrinkles, pores, and sagging, while hydration and elasticity only showed moderate correlation regardless of which camera took the photo, since those are physical properties a flat image struggles to capture no matter how good the lens is.

Moving an AI model out of a controlled clinical setting and into a patient's own camera roll is a known challenge in the field, and image quality is one of the biggest variables determining how well that move goes. For a patient-facing platform, the practical answer is to build in image enhancement and quality checks rather than assume every uploaded photo meets a clinical standard, and to be honest with users that a blurry or poorly lit photo limits how much confidence any AI step can offer. A system that flags a weak image and asks for a retake, or routes it straight to a clinician's judgment, is letting a human make the call when the input isn't good enough for the algorithm to.

The diagnostic cost of the skin-tone gap in training data

Image quality isn't the only input problem. Who got photographed in the first place shapes what these models can see accurately, and that shapes who they help.

A 2026 audit called SkinMap reviewed more than 1.1 million images across 29 public dermatology datasets and found that darker skin tones, Fitzpatrick types V and VI, make up only a small share of the total, while common neoplasms on fair skin form dense, well-represented clusters. HAM10000, one of the most widely used dermoscopy training sets, was sourced from Australia and Austria and carries no explicit metadata on ethnicity or skin tone. Because of that absence, nobody can check, from inside the dataset itself, whether a model trained on it treats every skin type with equal accuracy. SkinMap also found that pediatric patients make up only a small share of the available images, so the gap isn't limited to skin tone alone.

The diagnostic cost of this imbalance is specific: models trained this way are more likely to misclassify a cancerous lesion as benign in a patient with darker skin, which is the exact error a screening tool exists to prevent. A 2026 atlas of digital dermatology datasets put a number on the scale of the problem, finding that Fitzpatrick V and VI skin tones make up just 5.8% of images across unified dermatology datasets. It's a data problem, built into the training sets most of these models learn from, so SkinMap's framework was built to measure where those blind spots sit and guide future data collection toward the regions that are missing, rather than adding more images to the categories that are already full.

How mandatory clinician sign-off addresses the black box problem

A correct output from a model isn't the same as a usable one. If a lesion gets flagged as high-risk and the model can't show why, a clinician has nothing to check that judgment against, and no way to challenge it if something looks off, an opacity that is a direct consequence of everything described above: a pipeline built on uneven training data, tested unevenly across image quality and skin tone, needs a human checkpoint before its output becomes a medical decision.

Errors in AI dermatology assessments cluster in a particular place: cases that require pulling together clinical, pathological, and microbiological information, with accuracy dropping specifically where a microbiological method is needed. Those are the atypical, complicated presentations where a clinician's judgment carries the most weight, and they're also the cases where a model's confidence is least trustworthy.

Regulators have started treating this as a requirement. The EU AI Act, passed in 2024, classifies AI-based medical diagnostic systems as high-risk. Algorithmic transparency, quality management, data governance standards, and human oversight aren't optional extras but conditions for deployment. Tools like Grad-CAM, used in the 2026 smartphone imaging study to visualize which parts of an image drove a given prediction, are one attempt at making these outputs checkable.

The argument that high aggregate accuracy justifies loosening oversight runs into trouble exactly where it matters most. If darker skin tones make up only 5.8% of the images in unified dermatology datasets, the patients most exposed to an unchecked error are often the same patients already underserved by the health system generally. Platforms that require a physician to review every AI finding before any prescription goes out, a model Nolla follows, build the oversight directly into the structure of the visit, so it isn't just a fallback for edge cases.

What these systems can reliably do with clinicians in the loop

None of this argues against using AI in dermatology. It argues for a specific arrangement: AI handles structured triage and pattern recognition, and a licensed clinician reviews the output before it becomes a decision. Under that arrangement, the evidence for improvement is concrete.

PanDerm's own evaluation showed it improved clinicians' diagnostic accuracy on dermoscopy images, and it improved non-dermatologist providers' differential diagnosis across a wide range of skin conditions on ordinary clinical photographs. The combination of model and clinician outperformed either one working alone.

A real-world pathway presented at the EADV Congress 2026 in Vienna, run by Thomas, Macedo, and Fearfield, tracked 8,391 patients representing the vast majority of urgent suspected skin cancer referrals across two hospitals. A CE-marked Class III AI medical device analyzed dermoscopic smartphone images: it discharged patients with benign-looking lesions on its own and routed every higher-risk case to a teledermatologist for review. The clinician stayed in the loop for every case where risk was elevated, with no exceptions.

The asynchronous, store-and-forward model, where a patient or referring doctor uploads images and history and a dermatologist reviews within a day or two, is the most common format for this kind of care and fits AI-assisted triage well. It's the structure Nolla's platform runs on: multimodal AI trained on a large library of labeled clinical cases, paired with review from a licensed clinician before any treatment decision is finalized. For a patient who would otherwise go without a specialist opinion at all, an asynchronous consultation built this way is a real intervention, not a stand-in for an in-person exam but a credible, supervised first step toward one.

The same technology that expands access can just as easily deepen the disparities already present in the data, if it isn't built with that risk in mind from the start. So keep a human reviewer in the loop, and keep diversifying the data these models learn from, together, as standard practice rather than a patch you apply after something goes wrong. The value in this pipeline was never going to come from the model alone or the clinician alone. It comes from running both, in sequence, every time.