Question about safety

Hi everyone, I wanted to create a text-to-image generator but after reading the comments and criticisms from a user on the lemmy forum who felt Perchance was too lax, I started asking myself a few technical and policy questions regarding how the text-to-image-plugin backend handles security, specifically concerning CSAM prevention and the ephemeral nature of the generations. I’m trying to understand the exact pipeline when a user attempts to bypass the safety filters using jailbroken or heavily obfuscated prompts. Filtering Mechanism: Does the backend primarily rely on post-generation safety checkers (like the CompVis Safety Checker) which block the output and return a black square/error? Or does it implement in-generation mitigation like Safe Latent Diffusion (SLD) that actively alters the trajectory to render a safe/clothed image instead? NCMEC Reporting on Altered/Blocked Content: If a malicious prompt is successfully mitigated by the system (meaning the backend outputs a safe/clothed image, or blocks it entirely with an error), is a report still generated and sent to the NCMEC based on the user's prompt and IP? Or is NCMEC reporting strictly triggered only if an actual CSAM image is generated and caught by the filter? Ephemeral Storage & Evidence: Since Perchance doesn't store generated images by default (to protect user privacy and save server costs), how does the system handle evidence collection? If the backend safety filter catches an illicit generation, does it temporarily quarantine the image, the prompt, and the IP address in a secure log specifically to fulfill the NCMEC CyberTipline reporting requirements? Thank you for your time and for clarifying how the platform balances strict user privacy with legal safety obligations.

-4 points · 8 comments · view on lemmy.world

8 Comments

edwardthefma99@lemmy.world · 3 pts · 21d (3 replies)

wow thats a mouthfull sadly i dont know that but i do know that they try to prevent from illeagle content being genarated but with ai its like a huge cat and mouse game thats why i developed https://perchance.org/admin-content-guard

newuser222222@lemmy.world · 0 pts · 21d (2 replies)

So you mean you don't know either if the site provides this? So each creator has to build their own NSFW security system to prevent CSAM? so you created this "admin content guard" by yourself to prevent that?

edwardthefma99@lemmy.world · 3 pts · 21d (1 reply)

I don't know anything technical but I do know that they do some things with messing with the prompts on the back end I developed content guard by myself to allow people to filter out illegal content because people will always figure out a way to get around the system basically you give it a word like girl and if you add a word that would be generally used with a NSFW team like let's say naked what it does is it adds the word naked to the negative prompt with a strong weight then it adds wearing business suit with coat and pants to the positive prompt basically covering up any nudity you're free to copy it if you want

newuser222222@lemmy.world · 1 pts · 21d

thank you

yamit@lemmy.world · 3 pts · 20d (2 replies)

Nothing is reported to NCMEC and I hope it never will. Some keywords are changed backend and the output is something generic and very well clothed. Trying to bypass it will generate some horror and distorted images in most cases. If you are worried about csam you can prevent it with simple regex using the function beforeGenerate(), constantly erasing the user's suspicious prompt keyword will make them grow tired and give up using your generator.

newuser222222@lemmy.world · 1 pts · 20d (1 reply)

how do you know? The dev said so?

edwardthefma99@lemmy.world · 1 pts · 20d

if they did start reporting it the whole sight would need to be shut down for forensic auditers wich costs time and money also all the images that makes up the models are all adults so with the correct promting you can get what looks like a child but it has adult gentalea also it sits in a leagle gray zone i just wish the perverts would just genrate thare own smut localy with thare own gpu power

newuser222222@lemmy.world · 2 pts · 21d