Whatever she racked her brain she couldn’t know him.

No. I'm not sure what you mean—maybe "However she racked her brain, she couldn't remember where she knew him from."

by "whatever" i mean, "she tried a lot..."
maybe this is the correct form:
how many times she racked her brain she couldn't remember him.

No.

"No matter how many times she racked her brain, she couldn't remember him." We wouldn't normally speak of racking our brain more than once. I would expect "No matter how she racked her brain, she couldn't remember him." "However she racked her brain, she couldn't remember him" is also correct but is perhaps a bit formal for the idea.

i really appreciate you for answering

you know! what is important for me is to show that "she tried a lot to remember"
does "however" show trying?

"However she racked" here means "in whatever manner she racked ..." or "as much as she might rack ...". "However she racked" might be better as "However much she racked...", now that I think about it, and it would show the trying more.