Rendered at 19:30:26 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mistercow 18 hours ago [-]
Aside from the fact that this was obviously never viable and the entire problem is clearly unsolvable if you sit down and really probe it for fifteen minutes, what I find most frustrating about this is that the false promise of preserving photos as reliable evidence is actively harmful.
You will not build a perfect system, or even something near perfect. The best you're going to do is make it so that it's hard to casually present AI photos as real, leaving only the cases where it really matters. In the "best" case, you've just made the public more trusting of photos in general, so that when there's actual money or power on the line that makes jumping through the hoops to fake authenticity worth it, the public is more susceptible.
The best outcome at this point is for everyone to get on the same page that photos have roughly the same probative value now as drawings. Poorly thought out snake oil efforts to prove authenticity are only going to delay that.
indutny 17 hours ago [-]
In my opinion, the benefits for end users are rather minimal since I doubt an average person would ever be checking C2PA provenance data, but there is a commercial incentive for Google and others to promote C2PA, since it makes preparing training material for Machine Learning significantly easier, and perhaps as a smaller benefit justifies hardware attestation that locks users down into proprietary OSes.
chrisjj 11 hours ago [-]
> it makes preparing training material for Machine Learning significantly easier
How?
lazide 11 hours ago [-]
A huge problem with ML training is the ‘ouroboros issue’ - training ML with input from other MLs breaks things in very deep (but difficult to stop/detect when it’s happening) ways due to the way the internal math works (model collapse).
Right now, the Internet training set is becoming more and more contaminated with better and better generative AI images and video.
It makes the models more screwed up, and makes it very difficult for humans to figure out what is original and not too.
If there was some signal that could at least make it easier to identify ‘original’/real images…
The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.
lazide 11 hours ago [-]
Not when you also including poisoning attacks.
I remember the original paper showing issues with even a couple percent of certain kinds of synthetic data too, not 100%.
ainch 10 hours ago [-]
What do you mean by poisoning attacks - stuff like Nightshade or Glaze? I was under the impression that those have largely failed to achieve their goals.
chrisjj 4 hours ago [-]
Those aren't poisoners. The are attempted protectors.
kawogi 9 hours ago [-]
I think the term "AI incest" would sum this up :)
chrisjj 8 hours ago [-]
OK, but any reliance on a fakable signal such as this seems guarateed to invite poisoning.
People didn't have the ability to distribute forged images to millions to form a mob in seconds though.
Gigachad 16 hours ago [-]
I don’t think there is a technical solution to this problem but I think there is a legal one.
Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.
No, this would be literally you (the user) checking "this is AI generated" before you post something publicly. Platforms can auto mark it if they detect a watermark like SynthID but they can also leave the choice up to you.
The point is making it easier to go after bot accounts that spam generated videos to influence politics. They need to disclose that its AI and lose their power or lie and get criminally prosecuted.
It's not the same thing as model providers adding a mandatory watermark to everything, and even worse one you can't verify yourself and have to trust that Anthropic is telling everyone the truth. People should have the option to use undetectable AI tools in private, even if it's illegal to post the outputs on social media.
lbriner 10 hours ago [-]
> Enforce penalties
Really, everyone should know by now it is not as simple as that. We already have lots of laws that are not enforced (fully or at all) because there is a cost to enforcing the law - time and money. Some random person uploads an AI image to Facebook and you really expect the police to expend a few thousand $ prosecuting them for what? To Server as a warning?
Some people will be untraceable. Some will claim innocence or mitigation or some disability that excuses them. Even those that might be prosecuted will be fined and won't pay which costs even more to follow up.
We should know that the legal system does not create an obedient society.
SkyBelow 7 hours ago [-]
That hasn't done anything to stop scamming, so why would it apply to AI? People located outside of areas with these laws won't have to follow them, and this reduces building a immune system to such actions, making people more likely to fall for it when done by those not bound by the laws.
In a real like political misinformation, this will have the effect of making people trust non-watermarked images more, which will then be used by foreign actors to pass off propaganda as legitimate.
Also, if you don't hold people responsible for spreading an image they know is fake, bad actors can take advantage of this even within the US (they purposefully spread a image they have reason to think is fake but lacking a watermark), but holding people responsible for a strict liability crime for spreading AI without knowing it is AI seems an even worse route.
I'm not sure a law even makes the issue better in a 'don't let perfect be the enemy of good' sort of way.
akoboldfrying 14 hours ago [-]
I think public key cryptography offers a technical solution that is nearly as ideal here as for its existing uses for securing communication between physically remote actors -- please see my comment here for details: https://news.ycombinator.com/item?id=49444227
I say "nearly", because as soon as you need to keep a private key secure from someone with direct physical access to the device, you're entering dangerous territory. TTBOMK there's no way to make a "perfect black box", so it becomes an arms race between defensive "obfuscation" and tamper detection mechanisms in the one hand and stealth scanning techniques on the other. But this is already the case for TPMs -- that is, the situation is no worse than for an already widely accepted technology.
fwipsy 16 hours ago [-]
I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing.
People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless.
"C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.
Retr0id 16 hours ago [-]
When I search for "C2PA" on the google play store, there are more AI-watermark-removal apps than there are signing apps. Certain types will jump through ridiculous hoops if they think it will affect their algorithmic reach on social media.
Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.
fwipsy 14 hours ago [-]
You're arguing that lots of people can spoof this, the other guy is arguing that nobody will know it can be spoofed so it will do more damage. But these are contradictory -- if fakes become common, then they will also become common knowledge. The impact of any given fake is reduced if there are more of them. The technology doesn't need to provide 100% assurance. If it adds even a little friction to the slop mills then that's increasing the signal to noise ratio.
treyd 16 hours ago [-]
It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic.
When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
fwipsy 14 hours ago [-]
Read the rest of my comment please. Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together? Probably not, since if there's really all that much riding on it, people will point out it can be bypassed.
Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.
The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
Gormo 7 hours ago [-]
> Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together?
Yes, absolutely. Probably moreso. The whole point of these proposals is to try to solve for the "motivated malicious user" who is engaging in actual high-stakes fraud. There is no point in applying techniques that suppress inconsequential pranks while making serious crimes easier to get away with.
This really seems like a rehash of the perennial DRM argument: DRM restrictions provably do not reduce large-scale motivated copyright infringement, they just annoy legitimate paying users. This is the same class of solution, in that it is effective only where the stakes are low and the impact is minimal.
anonreplier 6 hours ago [-]
Why is this being framed as 2 types of users, lovable pranksters and fraudsters? There's a whole spectrum between these 2.
Also I'd like to know if a "joke" is likely fake.
Gormo 5 hours ago [-]
I don't suspect there is a uniform spectrum between those two. I think this is something that's going to be clinal, which we see in a lot of other comparable social contexts. The number of people actually willing to cross a moral threshold into outright crime is relatively small, but those are precisely the people who cause the most damage when they get away with their behavior.
Bur I don't even really think that's really relevant anyway, because whatever the density of "malicious" motivations is, the point here is that the fact that it only is an effort/motivation threshold that allows this technique to "block" malicious uses, and the motivation to overcome that threshold correlates directly with the stakes involved in the malicious use.
In other words, the more malicious the abuse is, the less effective this solution will be: the boundary of its usefulness will be wherever the line between pranksters and actual criminals happens to lie.
treyd 6 hours ago [-]
You're conflating the effectiveness of the mechanism with its systemic impact.
* Pangram: Yes we should really be discouraging people from putting trust in tools like this because they can't be made totally reliable.
* Antivirus: We should be building application environments with robust security models so that malicious software has a limited blast radius (like we do on mobile, like the Linux ecosystem is trying to do with Flatpak, etc).
* HTTPS: HTTPS is a strict upgrade from HTTP so we should be using it everywhere possible. The UI symbols to indicate to users the security expectations they're getting are good practice.
* ssh: This is just an inappropriate comparison.
The HTTPS comparison would make more sense if actually 0.1% of the time when their browser said they were using HTTPS it was just lying.
hypfer 14 hours ago [-]
You're missing all of the points that there could be by focussing on random people.
While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.
This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
fwipsy 13 hours ago [-]
It's funny how people always say something is "a tragedy at the individual level" when they mean "it's not my problem." It's even crazier to dismiss the value of a security feature, just because it might make people feel more secure. That's true of every security feature! Very little of the technology that the web is built on is proof against state actors.
I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
Melatonic 11 hours ago [-]
Yeah I think any additional security is good. At worst this could help in a lot of court cases. Someone presents photo evidence - it could be manipulated - it could be not. This happens already. Then someone produces an original higher quality version (like a raw photo - which I take even on my phone at all times now) and experts can verify that as the original.
And I agree on state actors. If a major one is invested in something like this they might have well compromised the signing project itself, the verification process, or even the court system or media. That seems like a rare and extremely high bar to guard against.
hypfer 13 hours ago [-]
I am repeating myself, but this is about systems, and not about people.
It is however in the interest of the people to keep the systems running in an untainted way.
As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.
C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors".
Banality of evil. Again.
___
Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.
Those are different things.
Argh and I ran with your term aah
Gormo 7 hours ago [-]
> I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing.
Exactly: most people don't bother editing visible watermarks out of AI-generated media, because they have little or no incentive to bother doing so. This will "defeat" the 99.9% of users who are not actually trying to do anything malicious, but will be a minor annoyance to the 0.1% of users who are actively engaging in fraud, fabrication of evidence, etc.
The upshot is that not only us this not useful for its intended purpose, it will lure people into a false sense of security by creating expectations that AI-generated media will always be easily identifiable as such, and reduce the level of scrutiny that gets applied to the stuff that actually is malicious.
rcxdude 11 hours ago [-]
What percentage of users are malicious, do you think? To me the issue is precisely what the product is trying to solve: propaganda using faked footage passed off as real, which generally has no real difficulty with resources available to boost their message. Giving that any kind of stamp of authenticity is a bad idea, IMO, and the fact that it'll work on 99% of the cases that don't matter makes it even worse.
mistercow 7 hours ago [-]
> I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing.
Everyone realizing that photos don't prove anything would defeat 100% of malicious users. I don't understand what people incorrectly trusting photos is supposed to achieve at this point, in your view.
fightfake-ai 13 hours ago [-]
I agree with "I'm fairly certain this will defeat 99.9% of malicious users".
Also, note that C2PA should have something like Level 3 as well: "The image is mathematically proven to have come from the physical camera sensor."
It's somehow difficult to achieve this, but it's possible (although the attacks will always be possible of course).
dTal 10 hours ago [-]
Difficult and also solves nothing, since you can just point the camera at a screen.
Have you ever tried pointing a camera at a screen?
The screen is doing all kinds of crazy things that are not apparent to your eyes, but that show up clearly on camera.
blincoln 7 hours ago [-]
Some screens are like that, but it's not an inherent property of all visual displays.
For example, you'll see a sort of barber-pole effect when pointing a video camera at a raster-scan digital display whose refresh rate isn't synced to the camera's frame rate. To avoid that, sync them, or maybe use a colour e-ink display.
Alternatively, print a high-resolution version with a decent photograph printer and take a picture of the print with the C2PA camera.
qurren 18 hours ago [-]
Are we entering a world where if I took a picture with a film camera and scanned it, it would be rejected as not real?
This is ridiculous.
CapitalistCartr 18 hours ago [-]
It's not a matter of "rejected as not real", it's "There's no way to know if this is real or not".
account42 6 hours ago [-]
When 99% of people are posting pics from their big tech approved smartphones with the "200% real no way to fake this" badge it doesn't matter that absence of the badge technically doesn't mean fake, that's what it will be interpreted as.
spaqin 9 hours ago [-]
You can even do it wholly in an analogue process, do a darkroom print, and it could be seen as fake as well - darkroom techniques are quite flexible as well. And a negative itself? That could be a copy of a prepared printed slide.
lelandfe 17 hours ago [-]
The defendant submitted a remarkably high quality video of you saying you generated it with Nano Banana.
ted_dunning 16 hours ago [-]
And their video was C2PA signed.
account42 7 hours ago [-]
Worse than not working, this will likely be used as yet another excuse to attack computing freedom.
akoboldfrying 14 hours ago [-]
What makes you so certain that this valuable research showing weaknesses in today's systems will render the C2PA concept useless forever?
Yes, software LPEs are a risk -- as they are in every nontrivial computer system. New ones will appear, and old ones will be closed in time, as TFA acknowledges.
Re hardware attacks: The (neat!) glitch injection attack the author describes in the linked "lighter" page only raises the implementation cost of doing image certification properly. For example, if the camera module presented only an interface that dumped raw RGB or JPEG-encoded data plus a digital signature that used a private key known only to the manufacturer, then all that would be required to verify a "downstream" image would be to keep a copy of those original bytes inside the final (potentially cropped, filtered, AI-ed, etc.) image, in the worst case roughly doubling its size on disk (though certainly more efficient schemes could be designed). Any interested third party could then compare the original and final images by eye and decide for themselves whether or not the subsequent processing materially changed the image's "meaning".
Finally: Does the existence of lock picks or bolt cutters render padlocks pointless today? Does it corrode society by encouraging people to mistakenly believe that anything they put behind a $5 padlock will be safe forever? No, and no.
blincoln 6 hours ago [-]
> Does the existence of lock picks or bolt cutters render padlocks pointless today
A padlock shouldn't be the only component of the bigger physical security picture.
A random person carrying/using bolt cutters or trying to pick a lock looks suspicious, and should be noticed by on-site staff or whoever is monitoring the security cameras.
If the padlock is decent, there will also be an inherent delay involved in bypassing it using those tools, which should increase the likelihood of the person being noticed.
If the padlock is used in an unattended/unmonitored location (i.e. a remote vacation house with no neighbours and no security cameras), then the padlock is probably only good for keeping honest people honest.
A lot of physical security => information security analogies are misleading for the same reason. In information security, it's very possible for an adversary to have effectively unlimited time to perform their attack (or to develop the means to perform the attack quickly).
Imagine a scenario where any determined person could e.g. spend a month in their home workshop and develop a pair of gloves containing transducers that would vibrate any padlock open in seconds. They could mimic the action of using the key or entering the combination to avoid looking suspicious. Also, the gloves have a button that instantly creates another pair of the same gloves at no cost. How much value would a padlock have in that scenario? That's the kind of asymmetric playing field one has to consider in information security contexts.
rcxdude 11 hours ago [-]
The comparison to padlocks and bolt cutters is not relevant, they are quite different shapes of problems. A certificate that is weak enough that it should not be trusted in any case where it matters is actively harmful, not just mildly less helpful.
(This is true in general: if you add a trust signal that is mainly just a bit of effort to broadcast, you'll find that scammers and fraudsters will show that signal much more reliably than honest actors. For example, every email spammer has DMARC and DKIM set up absolutely perfectly)
akoboldfrying 7 hours ago [-]
I guess you're making the argument that, while padlocks sometimes protect things of low value, thus justifying the use of cheap, easily broken locks in those cases, the times where authenticity of an image is important are nearly always cases of high importance, meaning that there's no case where an assertion that's actually fairly cheap to forge (assuming a similar "lighter" glitch attack works on phones, less than $1000 for a modern smartphone that may get bricked, a soldering iron and an afternoon's work) makes sense? Perhaps so. It's certainly easy to think of images or video clips of very high importance, and outside of kids uploading AI clips to Reddit and pretending they're real, I can't think of any similarly low-stakes cases for image authentication.
What do you think of my digital signature idea?
hypfer 13 hours ago [-]
This is yet another of these absolutely caustic takes that come with a veneer of intellectualism, but actually just ignorantly deconstruct reality.
Each of them weaponizing the rules of the platform that (for sensible reasons) demand you engage with the strongest interpretation of the message/argument you see.
The asymmetry of effort there is unsustainable, and that's exactly the point.
No idea how that could be solved. Maybe a meta comment like this one helps.
__
I mean if you think about it, it shouldn't be possible for some anon account to drop this and sound like it's a worthy contribution to a debate against some real person with a real name, a track record and multiple thousand dollars of bricked hardware leading up to that assessment.
(Nor would it make sense for a non-anon but equally empty account)
It makes no sense, and the guarantees regarding protection of speech and all do not apply to these topics, because it's not an opinion that would get your real name in jail.
What can we do about these social exploits. Someone tell me please. It's driving me up the walls
dTal 10 hours ago [-]
That's a very long-ass way of saying "I disagree with this comment, I can't be bothered explaining why (too much work), so instead I think this user should be banned". I extracted nothing useful from your comment, which bears no relation to its parents that I can discern, am irritated to have spent time reading it and composing this reply, and also find your attitude pretty rich when you're also an anon account, and with less karma too. If you want to "call someone out" you've gotta be on damn solid ground and bring receipts. As it is you've just decreased the signal/noise ratio of the thread and contributed to a toxic atmosphere. I would just flag and move on but in this case I think feedback is important, and unlike you I can be bothered to explain what it is that's got me riled up.
hypfer 10 hours ago [-]
This is what I'd call the "enabler" archetype that gets unhappy about friction (rightfully!) but takes some.. uh.. mental shortcuts when it comes to tracing back the source of it.
I'm sorry man. The problem is that this exact described dynamic (see also the other reply by me which you seem to have conveniently missed), is that it affects _all_ internet spaces and makes them utterly miserable.
I agree though that this meta stuff doesn't exactly improve SNR, and I am open for better solutions. But those need to actually solve the problem, instead of just silencing the immune response.
___
Interesting side-thought
> with less karma too
I think this might point at (one of the) root cause(s) of the dysfunction I'm pointing at here.
Internet points ceased to be a metric documenting value quite a while ago. Something something Goodhart
dTal 9 hours ago [-]
Gonna make this real short and simple: it's totally unclear what you're on about. You've described nothing, merely complained, infuriatingly, in the vaguest possible terms.
hypfer 9 hours ago [-]
Don't take this as dismissive or a dunk or anything like that, but maybe an LLM can help with unpacking?
At least I know that they understand my writing and that they can help me understand other writing I don't.
I can assure you that there is a point. Whether it is worth bothering is of course your decision.
I'd wager probably not, but I guess that depends. No hard feelings either way.
senordevnyc 7 hours ago [-]
Maybe you should give your comments to an LLM, ask them to untangle wtf you're on about, and then have them rewrite into something coherent for the rest of us. Maybe ask for some pointers for future comments as well.
akoboldfrying 11 hours ago [-]
There is no content in your response. At all. I honestly don't think I've come across a post this devoid of content on this site before.
All I can perceive in it is a generalised hate towards some broader thing that you feel certain I'm consciously aligned with somehow, and that you think is inherently and obviously evil. Because... I don't use my real name on here? (Is your real name "hypfer"?) Because I don't have bricked hardware to back up my opinions?
If you want to convince me or others that something I wrote is wrong, stop hyperventilating and engage with at least one of the specific claims in my post. For your own sake, I'd also suggest editing or outright deleting your original post.
hypfer 10 hours ago [-]
It is correct that I have completely side-stepped your frame and not even attempted to engage with it. This is on purpose, because wanting people to engage with that specifically engineered frame is exactly the bad faith mechanism I've described.
What is interesting is what happens when these tactics are called out in a meta comment like I did, because it's actually a common thing that people start flailing like this once you side-step the script that was supposed to be followed.
Someone acting in good faith would not counter with an attack and a double-down like this, but with intent to resolve the situation (+ probably feeling a bit bad about an exchange having failed and being misunderstood).
I also like the "hyperventilating" claim. Easiest way to get out of a situation is to invalidate the source, and easiest way to do that is to claim emotions.
And the fake concern (for what, even?) of course.
> For your own sake, I'd also suggest editing or outright deleting your original post.
___
Anyway. I think the convincing people is actually working quite well here.
Though not in the sense you'd think.
I'm just pointing a spotlight at what I believe is violating the spirit of the rules (while staying within the letter of the rules).
What other people make out of that is their decision to make.
senordevnyc 7 hours ago [-]
What on earth is happening here? I seriously have no idea what you're even talking about at this point. You seem to be five levels of meta deep and talking in circles about yourself?
Very bizarre.
adabovehuman 12 hours ago [-]
> and the entire problem is clearly unsolvable
The problem is not identifying AI generated media
The problem is identifying real media. That it can actually do.
trentor 18 hours ago [-]
It's compliance for advertising. Your client doesn't want AI in their project you show them the audit trail and if it turns out to be faked you point to the supplier who faked it. Our agency signed a insurance not only because of clients who don't want to use AI in their artwork but also because of the EU AI act. They c2pa to get their money back from a cheating supplier if they are not compliant with the AI act.
uqers 21 hours ago [-]
I'm very surprised Google put in so much effort to implement an approach that is basically the equivalent of client-side verification of passwords. Did no one designing it mention that it could be defeated by any rooted device?
12_throw_away 20 hours ago [-]
Actually I think this approach is very forward looking! Attestation is on the cusp of becoming a very powerful technique. We just need to figure out how to build 100% bug-free and 100% secure hardware and software, and then it's gonna work great.
genxy 18 hours ago [-]
Wait a minute. I think you might have forgotten a /s, I can spot this kinda thing.
akersten 20 hours ago [-]
Well, all one has to do is look at the bigger picture of how rooted devices are being shuffled into 3rd rate/totally blocked experiences and the overall direction of things starts to take very clear shape.
Someone better figure out how to make computing devices at home from everyday parts because the only way I see this (shockingly rapid) arms race end is legally mandated, cryptographically locked down hardware and software (or even thin clients) everywhere.
RMS must be having daily nightmares at this point.
P.S.: Fantastic talk you linked there.
hypfer 14 hours ago [-]
I think it might tell us something about the culture there by now.
Doesn't sound like it's engineering-driven, even though they still do have a lot of capable engineers sitting there and atrophying.
I also wouldn't rule out that the less capable ones actually believed that the systems they've built are unrootable or something like that.
demibabs 21 hours ago [-]
Not any rooted device, it must be rooted via an exploit. Still pretty bad, though
LiamPowell 17 hours ago [-]
I always got the impression that C2PA is a way to say "this photo came from the BBC (for example) and they've only signed it because they've verified the supplied edit chain". It's always been obvious that one could point a camera at a screen, I don't think anyone involved with C2PA has claimed otherwise.
It seems like there's a big disconnect between what C2PA says it's for and what certain journalists think it's for.
Retr0id 16 hours ago [-]
C2PA is a bit nebulous as a concept, which is part of the problem. It is both of these things. The BBC type use case where a publisher signs their own content with their own keys seems reasonable to me, or at least, not obviously broken.
However I question the value-add when e.g. the BBC website is already authenticated by nature of being served over HTTPS, and anyone who redistributes BBC content can and should link back to the source.
> It's always been obvious that one could point a camera at a screen, I don't think anyone involved with C2PA has claimed otherwise.
They haven't claimed otherwise exactly, but some have implied it's a solvable problem. Here's where the "learn more" link goes, for when Youtube annotates a video as having C2PA metadata: https://support.google.com/youtube/answer/15446725 (Google is a C2PA Steering Committee member)
> The metadata that leads to a 'Captured with a camera' disclosure is made by a third party (for example, a camera manufacturer). This means that there is some risk that someone could take a photo of another screen showing synthetic content. Because the other screen shows an image that has been modified, it wouldn't be eligible for the 'Captured with a camera' disclosure. This issue is called 'air-gapping'. Camera manufacturers will continue to develop detection measures to prevent 'air-gapping', but the sophistication of those detection measures may vary in the near term.
Interestingly they do not mention any of the other known limitations. Their phrasing is highly weasel-wordy, but the implication is clearly that they imagine picture-of-screen detection to become robust (somehow) in the medium-to-long term.
LiamPowell 14 hours ago [-]
> However I question the value-add when e.g. the BBC website is already authenticated by nature of being served over HTTPS, and anyone who redistributes BBC content can and should link back to the source.
The value would be in images reposted to social media where the website an show a badge that says it came from a certain source.
fedpost 16 hours ago [-]
Why not just have BBC sign stuff with their own creds then? You can originate the chain of custody anywhere and relying on the camera is kind of silly when the reputation of the original publisher in a more meaningful backstop.
LiamPowell 14 hours ago [-]
It's useful to have all your tools automatically apply metadata in a standard way instead of having to keep track of it manually. Most cameras already add metadata that says what camera and lens were used, but you lose that as soon as you import it into Photoshop and export as a jpeg.
deathanatos 2 hours ago [-]
And never mind should someone want to edit¹ the image that comes off the camera. Now it cannot remain cryptographically verified, which means it's now in the pool of "dunno" images, where AI fakes can thrive alongside it.
¹in a non-AI, traditional way, that doesn't mislead the viewer and this entire footnote should reveal how subjective and intractable this problem is
randomblock1 21 hours ago [-]
Even at the hardware level, if it was a separate chip that the camera data passed through or something, that's not really good enough either, people have broken TPMs before. It'd have to be baked into the camera sensor. Even then, you could attack it from the next level up, with some fancy optics and a display, or something like that.
I don't think completely solving this sort of problem is even possible.
duskwuff 21 hours ago [-]
And I'm not sure it's even useful to solve. The presence/absence of a digital signature will never be the deciding factor in whether people accept/reject an image as authentic.
timcobb 20 hours ago [-]
Yeah this is what I don't get why are people even spending time on this.
HWR_14 20 hours ago [-]
Is this picture real or AI is a real problem it is worth money to solve.
duskwuff 16 hours ago [-]
What I'm getting at is that metadata that claims that a photo is "real" won't necessarily convince people that it is, and the lack of that metadata doesn't mean anything at all. About the only real use I've seen for C2PA is to confirm that an image bearing that metadata is AI-generated - and that marker is easily lost through editing or retransmission.
HWR_14 4 hours ago [-]
I'm not saying it's an easy problem or that this is the right solution. I'm saying it's a valuable problem and so they are trying this solution.
timcobb 6 hours ago [-]
Yeah we live in the era of alternative facts and this is just more, at best, "fact checking"
timcobb 6 hours ago [-]
It is (not sure I agree, but I can see how one would think this) but you're not going to solve it like this
silon42 12 hours ago [-]
And it's very real problem to workaround (aka, hack the device into faking the signature).
SoftTalker 19 hours ago [-]
Why? An image should never be proof of anything, by itself.
HWR_14 4 hours ago [-]
What should be sufficient proof for you that AI wont fake trivially?
EA-3167 20 hours ago [-]
A desperate attempt to preempt regulation.
akersten 20 hours ago [-]
A desperate attempt to establish their version of regulatory capture and not have to pay licensing fees to the other guy
rackp 19 hours ago [-]
[dead]
jasonjayr 21 hours ago [-]
And in 2026, I don't think it's too big of a stretch to imagine that there are going to be people in power that can add + remove the metadata to whatever image they want, at will, to tell whatever story they want to create. Sadly.
gruez 20 hours ago [-]
There were similar fears about the webtrust CA system, but AFAIK there's no known incidents where a government strongarmed a CA into misissuing a MITM certificate, and then it was used in MITM attacks. The closest is some misissued certificates seemingly due to incompetence but weren't used in attacks.
SoftTalker 19 hours ago [-]
And there are unicorns living at the end of the rainbow.
yjftsjthsd-h 20 hours ago [-]
> Even then, you could attack it from the next level up, with some fancy optics and a display, or something like that.
The analog hole is alive and well:)
taneq 19 hours ago [-]
And at that level, it’s a matter of definition anyway. What is “AI generated”? A photo of a screen showing an AI picture is still a photo. If that’s “AI” then what about a photo of an AI generated billboard? Or a photo of a bus with an AI graphic on the side?
TOMDM 13 hours ago [-]
I'd always thought the usefulness of C2PA was limited to verified devices in custody by trusted actors.
Like a security camera with a tamper evident enclosure, or an organisation being able to attest that they recorded the imagery.
The idea that it could be used to attest the authenticity of any random person or device surely wasn't a thing serious people expected was it?
hypfer 13 hours ago [-]
> was limited to verified devices in custody by trusted actors. Like a security camera with a tamper evident enclosure, or an organisation being able to attest that they recorded the imagery.
But that is also not the case, because whatever keys are in those devices may have been duplicated in the factory or somewhere along the supply chain.
Or the stuff is cloud connected and an exploit can be executed via that.
Or, as written in the blog post you're commenting on, software exploits.
The whole idea is that the concept works for no one.
wisty 21 hours ago [-]
I can break it with zero skills. Tripod, camera, clear monitor in a dark room ... just take a real photo of a fake photo.
Terr_ 21 hours ago [-]
That might be detectable if they signed content contains focal-length metadata... but even then, some foresight and a collection of lenses would hide it.
ipython 19 hours ago [-]
Wouldn’t the introduction of the lidar signals embedded in the photo (say used with apple’s faceid system) help here?
C2PA was never going to work to prevent abuse, but what it will do it give false confidence.
Most people don't read HN, or understand tech, and so if the device says "captured with camera" then they will believe it since Google says it's true.
Accusing someone of a crime, with fake but verified photographic or video evidence will be trivial, and claiming it's fake just makes someone tap the "Google says it's true" sign.
MelonArmiger 11 hours ago [-]
[dead]
ethagknight 22 hours ago [-]
I got a good laugh out of the "unblur to verify" first image. I dont know what I was expecting to see.
andrewflnr 20 hours ago [-]
As far as AI-generated images go, that was a good one.
yaro330 9 hours ago [-]
All these funny jabs at Android's security meanwhile Pixels are literally one of the most secure devices in the industry and Google invest tons into security.
Hardware attestation stands unbroken to my knowledge, only software attestation can be faked, and even then it's a constant cat and mouse game which Google continues playing until they are done with Pixel 3 generation.
C2PA is not immune to the analogue hole, sure, but dark room + photo of a photo approach falls apart the moment you bake in depth data into the image, which is already done.
fedpost 16 hours ago [-]
For the time being, I wish more workflows revolved around RAWs. You can pretty much prove the origins of an image sans cryptography by just possessing the image sensor data that you used to produce the final. It's not foolproof but AI image models can't really generate a convincing bayer raw and it's not something anyone is going to spend significant time training.
jauntywundrkind 16 hours ago [-]
Until it matters. Which you are proposing happening.
Melatonic 11 hours ago [-]
Exactly
e_l 13 hours ago [-]
I fear that the long-term result might be a small cartel of "trusted" companies that holds the private keys which can attest and lead to the death (or at least limiting widespread adopting) of open-source operating systems.
It won't stop fraud by governments and/or determined/well-resourced attackers, but it'll make it difficult enough for 99.99% of the public. And as usage drops over time, it becomes more socially acceptable to justify further limitations.
Governments and corporations (e.g. banks) will require that you use a "trusted" (meaning locked-down) devices to interact with their services. We're already seeing some companies block GrapheneOS/LineageOS.
16 hours ago [-]
Hard_Space 12 hours ago [-]
I sat in on a number of tedious C2PA meetings about three years ago. It was clear that no-one wanted to hear the hard truths, and one was certainly not rewarded for introducing them. I'm guessing there were a lot of consultancies to maintain, and a lot of snouts in the trough.
This was a technology that required international governmental prescription, not voluntary or viral adoption. Either way, it was horrifically flawed from the outset.
vanyle 9 hours ago [-]
This reads a bit like "Door locks do not survive contact with reality."
The point is not to prevent state-sponsored actors to produce fake media, but to add friction and avoid shady marketing agency and photographs from claiming their pictures are genuine.
ACCount37 9 hours ago [-]
Yeah, the target is not the cryptographic "safe forever", but a real world "safer than not having it".
If we're trying to decide whether a high profile politician has committed a crime, then yeah, C2PA on the footage isn't fully trustworthy. However, every bit of footage you get that corroborates the story raises the threshold of the attack.
If we're trying to decide whether Joe Everyman has crossed a double solid while driving his truck? C2PA is probably good enough. The chances of that footage being faked by a malicious party would be low even without C2PA, but C2PA makes them even lower.
spaqin 9 hours ago [-]
I do find it somewhat ironic that it's implemented in Pixels first, considering the amount of AI processing phone cameras do to get a pleasant image.
jazzyjackson 22 hours ago [-]
I would be interested in a note on whether Sony / Leica / Olympus “content credentials” do any better with their hardware to ensure a signature is assigned to data straight off the sensor.
Legend2440 22 hours ago [-]
My bet is they do considerably worse. Digital cameras are not designed with security in mind. Arbitrary code execution has been achieved on many DSLRs and there's even been open-source firmware projects for some.
Retr0id 22 hours ago [-]
Unfortunately they're a little outside of my tinkering budget, but if anyone wants to send me some I'll do my best to pwn them. Can't be any harder than a Google flagship, one would imagine.
I have ordered a faulty Sony A7 IV motherboard, but due to its faulty-ness and the lack of the rest of the camera, I'm not sure how far I'll be able to get with it.
kiddico 19 hours ago [-]
Could you make use of a Sony a6000?
EmbarrassedHelp 20 hours ago [-]
Why would someone paying for an expensive camera to damage the pixels of their images with "invisible" watermarks?
MadnessASAP 16 hours ago [-]
The signature doesn't touch the image data in any way. It's just a piece of metadata attached to the image, like any other metadata tag.
squidsoup 16 hours ago [-]
Isn't film photography verifiable with a physical negative? i.e. you could verify a digital reproduction of the analog photo by comparing the digital image with the negative. Tedious, but sound?
fedpost 16 hours ago [-]
You can expose a negative with whatever the fuck you want. It doesn't have to be a real scene.
16 hours ago [-]
roywiggins 15 hours ago [-]
Transferring electronic images to film is a long-solved problem.
Does that prove you didn't photograph a display showing the digital image?
16 hours ago [-]
jcarrano 11 hours ago [-]
If this could be made to work reliably, it would imply that cameras would have the ability to doxx users.
mikewarot 13 hours ago [-]
You could actually make this secure, by having firmware in the camera module itself do the signature before handing it off to the rest of the system. The key is to prevent ingress of control, as long as the module only emits signed photographs or video, and has tightly controlled input channels, for zoom, aperture, etc... it seems a quite reasonable project.
hypfer 13 hours ago [-]
No, you can't make this secure. You're just tweaking who can pull that off and how.
And, given that the main threat here is disinformation by nation state actors, they can also attack the camera module (or its supply chain) instead.
xyzsparetimexyz 20 hours ago [-]
Surely the easiest thing to target is photos taken by journalists and modifications, down sampling etc when shared to twitter?
14 hours ago [-]
9 hours ago [-]
tescreal 22 hours ago [-]
I expect the only plausible chance (and it is a stretch) will be at-the-censor marking. Quantum bla bla magic pixie dust or unicorn farts something. The chance of a trustworthy (including from nation-state tampering a la Stalin et al) means of verification of digital anything is as good as dead imho.
hydraterms 20 hours ago [-]
[flagged]
SecuriLayer 21 hours ago [-]
[flagged]
fenestella 18 hours ago [-]
[flagged]
Ozzie-D 18 hours ago [-]
[flagged]
tashian 22 hours ago [-]
I have a feeling Apple is going to knock it out of the park on this when they get around to it. They have a great foundation for doing image provenance well. The device attestation workflows are already there. And the same attacks that work against Android won't be as easy or effective because of Secure Enclave. Apple could run the whole signing process inside SEP.
And, Apple could choose to integrate a LiDAR depth map into the signed photo as a mitigation against the analog attacks (eg. pictures of screens).
gyomu 21 hours ago [-]
Apple isn’t going to touch this with a 10-foot pole.
The provenance “proof” these approaches provide is very tenuous and nowhere near the “this is a real photo of a real world event taken by a real camera and not an AI image” proof that marketing types like to push.
Apple doesn’t want a PR disaster where some crazy image is totally fake but becomes world news because it is “cryptographically signed as being from a real iPhone so it must be real!”
>Images captured with an opt-in Reference mode can be authenticated to confirm they were taken with an iPhone. Authenticating is done by tapping the Reference badge on the image, which sends the raw image, sensor signatures, capture time frame, and the unique hardware identifiers of the sensor to Apple's Private Cloud Compute (PCC) servers. PCC uses the information to determine whether the camera captured the photo, gives it a unique ID, and then returns an authenticated version to the user's device.
gyomu 14 hours ago [-]
Yeah, it might never ship, or it might not ship as described, or that might be exactly what they do - I love being wrong.
If it does ship like that, it’s hard to not imagine a situation as I described earlier - an “iPhone Reference Image” being used to propagate fake news, at which point the credibility of the feature goes to 0 (and Apple’s takes a severe hit).
You will not build a perfect system, or even something near perfect. The best you're going to do is make it so that it's hard to casually present AI photos as real, leaving only the cases where it really matters. In the "best" case, you've just made the public more trusting of photos in general, so that when there's actual money or power on the line that makes jumping through the hoops to fake authenticity worth it, the public is more susceptible.
The best outcome at this point is for everyone to get on the same page that photos have roughly the same probative value now as drawings. Poorly thought out snake oil efforts to prove authenticity are only going to delay that.
How?
Right now, the Internet training set is becoming more and more contaminated with better and better generative AI images and video.
It makes the models more screwed up, and makes it very difficult for humans to figure out what is original and not too.
If there was some signal that could at least make it easier to identify ‘original’/real images…
The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.
I remember the original paper showing issues with even a couple percent of certain kinds of synthetic data too, not 100%.
Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.
The point is making it easier to go after bot accounts that spam generated videos to influence politics. They need to disclose that its AI and lose their power or lie and get criminally prosecuted.
It's not the same thing as model providers adding a mandatory watermark to everything, and even worse one you can't verify yourself and have to trust that Anthropic is telling everyone the truth. People should have the option to use undetectable AI tools in private, even if it's illegal to post the outputs on social media.
Really, everyone should know by now it is not as simple as that. We already have lots of laws that are not enforced (fully or at all) because there is a cost to enforcing the law - time and money. Some random person uploads an AI image to Facebook and you really expect the police to expend a few thousand $ prosecuting them for what? To Server as a warning?
Some people will be untraceable. Some will claim innocence or mitigation or some disability that excuses them. Even those that might be prosecuted will be fined and won't pay which costs even more to follow up.
We should know that the legal system does not create an obedient society.
In a real like political misinformation, this will have the effect of making people trust non-watermarked images more, which will then be used by foreign actors to pass off propaganda as legitimate.
Also, if you don't hold people responsible for spreading an image they know is fake, bad actors can take advantage of this even within the US (they purposefully spread a image they have reason to think is fake but lacking a watermark), but holding people responsible for a strict liability crime for spreading AI without knowing it is AI seems an even worse route.
I'm not sure a law even makes the issue better in a 'don't let perfect be the enemy of good' sort of way.
I say "nearly", because as soon as you need to keep a private key secure from someone with direct physical access to the device, you're entering dangerous territory. TTBOMK there's no way to make a "perfect black box", so it becomes an arms race between defensive "obfuscation" and tamper detection mechanisms in the one hand and stealth scanning techniques on the other. But this is already the case for TPMs -- that is, the situation is no worse than for an already widely accepted technology.
People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless.
"C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.
Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.
When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.
The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
Yes, absolutely. Probably moreso. The whole point of these proposals is to try to solve for the "motivated malicious user" who is engaging in actual high-stakes fraud. There is no point in applying techniques that suppress inconsequential pranks while making serious crimes easier to get away with.
This really seems like a rehash of the perennial DRM argument: DRM restrictions provably do not reduce large-scale motivated copyright infringement, they just annoy legitimate paying users. This is the same class of solution, in that it is effective only where the stakes are low and the impact is minimal.
Also I'd like to know if a "joke" is likely fake.
Bur I don't even really think that's really relevant anyway, because whatever the density of "malicious" motivations is, the point here is that the fact that it only is an effort/motivation threshold that allows this technique to "block" malicious uses, and the motivation to overcome that threshold correlates directly with the stakes involved in the malicious use.
In other words, the more malicious the abuse is, the less effective this solution will be: the boundary of its usefulness will be wherever the line between pranksters and actual criminals happens to lie.
* Pangram: Yes we should really be discouraging people from putting trust in tools like this because they can't be made totally reliable.
* Antivirus: We should be building application environments with robust security models so that malicious software has a limited blast radius (like we do on mobile, like the Linux ecosystem is trying to do with Flatpak, etc).
* HTTPS: HTTPS is a strict upgrade from HTTP so we should be using it everywhere possible. The UI symbols to indicate to users the security expectations they're getting are good practice.
* ssh: This is just an inappropriate comparison.
The HTTPS comparison would make more sense if actually 0.1% of the time when their browser said they were using HTTPS it was just lying.
While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.
This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
And I agree on state actors. If a major one is invested in something like this they might have well compromised the signing project itself, the verification process, or even the court system or media. That seems like a rare and extremely high bar to guard against.
It is however in the interest of the people to keep the systems running in an untainted way.
As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.
C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors". Banality of evil. Again.
___
Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.
Those are different things. Argh and I ran with your term aah
Exactly: most people don't bother editing visible watermarks out of AI-generated media, because they have little or no incentive to bother doing so. This will "defeat" the 99.9% of users who are not actually trying to do anything malicious, but will be a minor annoyance to the 0.1% of users who are actively engaging in fraud, fabrication of evidence, etc.
The upshot is that not only us this not useful for its intended purpose, it will lure people into a false sense of security by creating expectations that AI-generated media will always be easily identifiable as such, and reduce the level of scrutiny that gets applied to the stuff that actually is malicious.
Everyone realizing that photos don't prove anything would defeat 100% of malicious users. I don't understand what people incorrectly trusting photos is supposed to achieve at this point, in your view.
Also, note that C2PA should have something like Level 3 as well: "The image is mathematically proven to have come from the physical camera sensor."
It's somehow difficult to achieve this, but it's possible (although the attacks will always be possible of course).
But yeah, difficult too :)
The screen is doing all kinds of crazy things that are not apparent to your eyes, but that show up clearly on camera.
For example, you'll see a sort of barber-pole effect when pointing a video camera at a raster-scan digital display whose refresh rate isn't synced to the camera's frame rate. To avoid that, sync them, or maybe use a colour e-ink display.
Alternatively, print a high-resolution version with a decent photograph printer and take a picture of the print with the C2PA camera.
This is ridiculous.
Yes, software LPEs are a risk -- as they are in every nontrivial computer system. New ones will appear, and old ones will be closed in time, as TFA acknowledges.
Re hardware attacks: The (neat!) glitch injection attack the author describes in the linked "lighter" page only raises the implementation cost of doing image certification properly. For example, if the camera module presented only an interface that dumped raw RGB or JPEG-encoded data plus a digital signature that used a private key known only to the manufacturer, then all that would be required to verify a "downstream" image would be to keep a copy of those original bytes inside the final (potentially cropped, filtered, AI-ed, etc.) image, in the worst case roughly doubling its size on disk (though certainly more efficient schemes could be designed). Any interested third party could then compare the original and final images by eye and decide for themselves whether or not the subsequent processing materially changed the image's "meaning".
Finally: Does the existence of lock picks or bolt cutters render padlocks pointless today? Does it corrode society by encouraging people to mistakenly believe that anything they put behind a $5 padlock will be safe forever? No, and no.
A padlock shouldn't be the only component of the bigger physical security picture.
A random person carrying/using bolt cutters or trying to pick a lock looks suspicious, and should be noticed by on-site staff or whoever is monitoring the security cameras.
If the padlock is decent, there will also be an inherent delay involved in bypassing it using those tools, which should increase the likelihood of the person being noticed.
If the padlock is used in an unattended/unmonitored location (i.e. a remote vacation house with no neighbours and no security cameras), then the padlock is probably only good for keeping honest people honest.
A lot of physical security => information security analogies are misleading for the same reason. In information security, it's very possible for an adversary to have effectively unlimited time to perform their attack (or to develop the means to perform the attack quickly).
Imagine a scenario where any determined person could e.g. spend a month in their home workshop and develop a pair of gloves containing transducers that would vibrate any padlock open in seconds. They could mimic the action of using the key or entering the combination to avoid looking suspicious. Also, the gloves have a button that instantly creates another pair of the same gloves at no cost. How much value would a padlock have in that scenario? That's the kind of asymmetric playing field one has to consider in information security contexts.
(This is true in general: if you add a trust signal that is mainly just a bit of effort to broadcast, you'll find that scammers and fraudsters will show that signal much more reliably than honest actors. For example, every email spammer has DMARC and DKIM set up absolutely perfectly)
What do you think of my digital signature idea?
Each of them weaponizing the rules of the platform that (for sensible reasons) demand you engage with the strongest interpretation of the message/argument you see.
The asymmetry of effort there is unsustainable, and that's exactly the point.
No idea how that could be solved. Maybe a meta comment like this one helps.
__
I mean if you think about it, it shouldn't be possible for some anon account to drop this and sound like it's a worthy contribution to a debate against some real person with a real name, a track record and multiple thousand dollars of bricked hardware leading up to that assessment. (Nor would it make sense for a non-anon but equally empty account)
It makes no sense, and the guarantees regarding protection of speech and all do not apply to these topics, because it's not an opinion that would get your real name in jail.
What can we do about these social exploits. Someone tell me please. It's driving me up the walls
I'm sorry man. The problem is that this exact described dynamic (see also the other reply by me which you seem to have conveniently missed), is that it affects _all_ internet spaces and makes them utterly miserable.
I agree though that this meta stuff doesn't exactly improve SNR, and I am open for better solutions. But those need to actually solve the problem, instead of just silencing the immune response.
___
Interesting side-thought
> with less karma too
I think this might point at (one of the) root cause(s) of the dysfunction I'm pointing at here.
Internet points ceased to be a metric documenting value quite a while ago. Something something Goodhart
At least I know that they understand my writing and that they can help me understand other writing I don't.
I can assure you that there is a point. Whether it is worth bothering is of course your decision. I'd wager probably not, but I guess that depends. No hard feelings either way.
All I can perceive in it is a generalised hate towards some broader thing that you feel certain I'm consciously aligned with somehow, and that you think is inherently and obviously evil. Because... I don't use my real name on here? (Is your real name "hypfer"?) Because I don't have bricked hardware to back up my opinions?
If you want to convince me or others that something I wrote is wrong, stop hyperventilating and engage with at least one of the specific claims in my post. For your own sake, I'd also suggest editing or outright deleting your original post.
What is interesting is what happens when these tactics are called out in a meta comment like I did, because it's actually a common thing that people start flailing like this once you side-step the script that was supposed to be followed.
Someone acting in good faith would not counter with an attack and a double-down like this, but with intent to resolve the situation (+ probably feeling a bit bad about an exchange having failed and being misunderstood).
I also like the "hyperventilating" claim. Easiest way to get out of a situation is to invalidate the source, and easiest way to do that is to claim emotions.
And the fake concern (for what, even?) of course.
> For your own sake, I'd also suggest editing or outright deleting your original post.
___
Anyway. I think the convincing people is actually working quite well here. Though not in the sense you'd think.
I'm just pointing a spotlight at what I believe is violating the spirit of the rules (while staying within the letter of the rules).
What other people make out of that is their decision to make.
Very bizarre.
The problem is not identifying AI generated media
The problem is identifying real media. That it can actually do.
At over a decade old, still prescient as ever: https://www.youtube.com/watch?v=HUEvRyemKSg
RMS must be having daily nightmares at this point.
P.S.: Fantastic talk you linked there.
Doesn't sound like it's engineering-driven, even though they still do have a lot of capable engineers sitting there and atrophying.
I also wouldn't rule out that the less capable ones actually believed that the systems they've built are unrootable or something like that.
It seems like there's a big disconnect between what C2PA says it's for and what certain journalists think it's for.
However I question the value-add when e.g. the BBC website is already authenticated by nature of being served over HTTPS, and anyone who redistributes BBC content can and should link back to the source.
> It's always been obvious that one could point a camera at a screen, I don't think anyone involved with C2PA has claimed otherwise.
They haven't claimed otherwise exactly, but some have implied it's a solvable problem. Here's where the "learn more" link goes, for when Youtube annotates a video as having C2PA metadata: https://support.google.com/youtube/answer/15446725 (Google is a C2PA Steering Committee member)
> The metadata that leads to a 'Captured with a camera' disclosure is made by a third party (for example, a camera manufacturer). This means that there is some risk that someone could take a photo of another screen showing synthetic content. Because the other screen shows an image that has been modified, it wouldn't be eligible for the 'Captured with a camera' disclosure. This issue is called 'air-gapping'. Camera manufacturers will continue to develop detection measures to prevent 'air-gapping', but the sophistication of those detection measures may vary in the near term.
Interestingly they do not mention any of the other known limitations. Their phrasing is highly weasel-wordy, but the implication is clearly that they imagine picture-of-screen detection to become robust (somehow) in the medium-to-long term.
The value would be in images reposted to social media where the website an show a badge that says it came from a certain source.
¹in a non-AI, traditional way, that doesn't mislead the viewer and this entire footnote should reveal how subjective and intractable this problem is
I don't think completely solving this sort of problem is even possible.
The analog hole is alive and well:)
The idea that it could be used to attest the authenticity of any random person or device surely wasn't a thing serious people expected was it?
But that is also not the case, because whatever keys are in those devices may have been duplicated in the factory or somewhere along the supply chain. Or the stuff is cloud connected and an exploit can be executed via that.
Or, as written in the blog post you're commenting on, software exploits.
The whole idea is that the concept works for no one.
Most people don't read HN, or understand tech, and so if the device says "captured with camera" then they will believe it since Google says it's true.
Accusing someone of a crime, with fake but verified photographic or video evidence will be trivial, and claiming it's fake just makes someone tap the "Google says it's true" sign.
Hardware attestation stands unbroken to my knowledge, only software attestation can be faked, and even then it's a constant cat and mouse game which Google continues playing until they are done with Pixel 3 generation.
C2PA is not immune to the analogue hole, sure, but dark room + photo of a photo approach falls apart the moment you bake in depth data into the image, which is already done.
It won't stop fraud by governments and/or determined/well-resourced attackers, but it'll make it difficult enough for 99.99% of the public. And as usage drops over time, it becomes more socially acceptable to justify further limitations.
Governments and corporations (e.g. banks) will require that you use a "trusted" (meaning locked-down) devices to interact with their services. We're already seeing some companies block GrapheneOS/LineageOS.
This was a technology that required international governmental prescription, not voluntary or viral adoption. Either way, it was horrifically flawed from the outset.
The point is not to prevent state-sponsored actors to produce fake media, but to add friction and avoid shady marketing agency and photographs from claiming their pictures are genuine.
If we're trying to decide whether a high profile politician has committed a crime, then yeah, C2PA on the footage isn't fully trustworthy. However, every bit of footage you get that corroborates the story raises the threshold of the attack.
If we're trying to decide whether Joe Everyman has crossed a double solid while driving his truck? C2PA is probably good enough. The chances of that footage being faked by a malicious party would be low even without C2PA, but C2PA makes them even lower.
I have ordered a faulty Sony A7 IV motherboard, but due to its faulty-ness and the lack of the rest of the camera, I'm not sure how far I'll be able to get with it.
https://en.wikipedia.org/wiki/Film_recorder
And, given that the main threat here is disinformation by nation state actors, they can also attack the camera module (or its supply chain) instead.
And, Apple could choose to integrate a LiDAR depth map into the signed photo as a mitigation against the analog attacks (eg. pictures of screens).
The provenance “proof” these approaches provide is very tenuous and nowhere near the “this is a real photo of a real world event taken by a real camera and not an AI image” proof that marketing types like to push.
Apple doesn’t want a PR disaster where some crazy image is totally fake but becomes world news because it is “cryptographically signed as being from a real iPhone so it must be real!”
>Images captured with an opt-in Reference mode can be authenticated to confirm they were taken with an iPhone. Authenticating is done by tapping the Reference badge on the image, which sends the raw image, sensor signatures, capture time frame, and the unique hardware identifiers of the sensor to Apple's Private Cloud Compute (PCC) servers. PCC uses the information to determine whether the camera captured the photo, gives it a unique ID, and then returns an authenticated version to the user's device.
If it does ship like that, it’s hard to not imagine a situation as I described earlier - an “iPhone Reference Image” being used to propagate fake news, at which point the credibility of the feature goes to 0 (and Apple’s takes a severe hit).
Wait & see.