返回资讯中心
外部精选
软件工程
#开发者实时流

Eval of 19 ways to crop images using LLMs, Jev and image classification models

Every crop of a photo should keep the part that matters. On new images an editor can click that spot. On older content nobody ever did, so I tested whether a model can do it instead.

Hacker News 最新marcus_johansso2026/10/18 · 2026/10/1823 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Every crop of a photo should keep the part that matters. On new images an editor can click that spot. On older content nobody ever did, so I tested whether a model can do it instead.

A site rarely shows a photo the way it was uploaded. The same image becomes a square on a card, a wide band at the top of an article and a tall picture on a phone. Each of those is a crop, and a crop has to decide what to cut away.

The focal point is the spot that every crop must keep: the face, the eyes, the thing the photo is about. In Drupal, the Focal Point module lets an editor click that spot when they add an image, and the image styles crop around it. Without a focal point, the crop keeps the centre and hopes for the best.

Setting a focal point takes an editor a second, on the day the image is uploaded. The trouble is everything that was uploaded before. A site that has been running for a few years has thousands of images, and none of them have a focal point. When you add a new layout with square cards or tall phone images, every one of those old images gets cropped around its centre. Heads get cut off, the person the story is about ends up outside the frame, and the card shows a wall.

Migrations make it worse. Content moved over from another system often has no focal point at all, or has one stored in a way that is hard to extract: in the old system's own crop data, in a custom field, or as a crop that was baked into the image file itself.

Fixing that by hand means opening every image, finding the spot and saving it again. Nobody budgets for that, so it doesn't happen. That makes it a job worth handing to a model: look at the photo, say where the focal point is, and do it for the whole media library in one go.

I picked 19 photos from Pexels: people, groups, a dog, a car, a camera dial, a stadium and a couple of photos where the subject is tiny. All the photos in this post come from Pexels. I marked the focal point on each one myself. That is the answer every method is measured against.

Each method got one of three prompts that explain what a focal point is: a short one (18 tokens), a medium one (228) and a long one (1,529). None of them mentions what is in these photos. Every combination ran three times, so you can see how much the answers move between runs.

Each method adds its own short instruction on how to answer (pixels, a grid cell, a number on a box). Tokens counted with OpenAI's o200k_base tokenizer.

The models: GPT-5.5 with reasoning and GPT-5.4 mini without, both from OpenAI; Claude Haiku 5.5 from Anthropic, only asked for x and y; Jev from TypeSafe; and two small models I ran on my own graphics card, Gemma 4 E2B and Qwen2.5-VL 3B, through Ollama. The object detectors ran on the same card, one model at a time.

The score is the distance between a method's answer and my point, in percent of the photo's diagonal. 5% is close enough that you would hardly notice; at 20% the crop is probably about something else. I also cut four crops around every answer (square, 16:9, 9:16 and a close square thumbnail) and counted how often my point was still inside.

One real run each, on photo 1 with GPT-5.5 and the short prompt, picked because the methods disagree. The images are exactly what the model was sent. Rulers and the grid went for the man walking away; the cursor loop caught its own mistake.

Here are all 19 photos with my point (the white ring) and the answers. Pick the methods, models and prompt on the left. Small dots are the three runs; the square is the middle of the three, which is what you would use if you ran a method three times and wanted one answer. Switch on crop frames to see what a site would cut, or open the Crops view to compare one method's crops with mine.

Answers from every run, as measured. All photos from Pexels. Distances are in % of the photo's diagonal; the middle of three runs is the median of their x and of their y.

In % of the photo's diagonal. The centre baseline is 14.4%.

GPT-5.5 came closest. Simply asked for x and y, it landed 6.2% from my point on average, with a median of 4.1%, and 96% of the crops kept my point, for about $16 per 1,000 photos and almost seven seconds per photo. Letting it check and move its own answer (the cursor loop) got it to 5.8%, but that costs more than three times as much and takes 24 seconds per photo. Rulers and grids didn't beat the plain question.

Claude Haiku 5.5 was the cheap surprise. Asked for x and y, it had the best median of all, 3.8%, and the most answers within 5% of my point, for 19 cents per 1,000 photos and just over a second per photo. When it missed, though, it missed by a lot, so its average is 8.9%. GPT-5.4 mini, at four times the price and just as fast, was less accurate on every measure.

Jev was the other surprise. A box around the focal point, take its centre: 8.7% on average, 89% of the crops kept, for six cents per 1,000 photos and half a second per photo. That is about 250 times cheaper than GPT-5.5 for an answer that is a little further off.

The small local models split. Qwen2.5-VL 3B, asked to point in its own format, got a median of 4.8% and was very steady between runs, but when it missed, it missed by a lot. Gemma 4 E2B, asked for x and y, did worse than always picking the centre.

The object detectors were the most interesting failure. Grounding DINO nearly always found a box around the right thing: if you could pick the best of its top ten boxes, you would be 3.5% from my point. But neither a simple rule nor Gemma could tell which box mattered, so they ended up at 13.5% to 17%, about as good as the centre. GPT-5.4 mini picked better: 10.6% with Grounding DINO's boxes. That is still behind asking the same model for x and y, partly because a person's box has its middle on the chest, not the eyes.

The long prompt helped the small models the most. GPT-5.5 didn't need it: asked for x and y, it did best with the short prompt (5.9% against 6.4% with the long one).

We came up with the rulers, the grid and the cursor loop because they gave much better results with older models like GPT-4.1, and we weren't the only ones: the Set-of-Mark paper showed in 2023 that numbered marks on an image made GPT-4V far better at pointing at things, and developers on OpenAI's forum reported that a numbered grid helped GPT-4o and GPT-4.1 give usable coordinates. Like a lot of things with AI, the magic workarounds aren't needed anymore: GPT-5.5 does best when you simply ask it for x and y. The same goes for the prompt: with GPT-4.1 we had to spell out what a focal point is in a long prompt, while GPT-5.5 did best with the 18-token one. The cursor loop is still marginally better (5.8% against 6.2%), but at more than three times the cost and 24 seconds per photo, it's probably not worth it.

Averages hide how different these two feel in use. Here both crop the same photos at the speed and price I measured, on their way to 1,000 photos. Speed it up to see where the bill ends.

Each at its measured average time and cost per photo, with the prompt that worked best for it: GPT-5.5 asked for x,y with the short prompt, Jev's box with the long one. The 19 photos repeat with the next run's answers. The blue square is the answer, the white ring my point, the clear window the crop.

For AI Image Crop, Jev support and image classification support go without saying. We can probably turn off the loop as well, if you use a frontier model. The module already has a bulk action for existing content, and that is where Jev shines: a whole media library for a few cents, at half a second per image. An object detection model like Grounding DINO would let a site do the same on its own server, without sending its images anywhere. And the loop didn't help the smaller models either (GPT-5.4 mini went from 9.9% to 9.8%), so one call with a good model is the way to go.

A focal point tells a crop what to keep, but sometimes an editor wants to choose the crop itself: exactly this square, exactly this band. We will do something similar for manual cropping later, and see whether a model can pick the crops an editor would.

AI takes my rambling notes, structures them and writes a first draft. I then prompt it to use interactive components, and at the end I read through it and rewrite it until it says what I want it to say.

Listen to our AI generated podcast · 5:15

View or present as slides · 4:20

Summary in 3 sentences

Podcast: Can AI set the focal point of a photo?

Podcast episode

4:20

1×

Present the slides yourself Close

The post in three sentences

Sites with years of content have thousands of images without a focal point, and nobody will go back and set them by hand. I asked models to set them on 19 Pexels photos and measured how far they landed from my own points. GPT-5.5 came closest, Jev was nearly as good at a fraction of the cost and time, and most small local models and object detectors barely beat a crop around the centre.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →