I agree with you on this point. I got a comment below on my YouTube video. "your Japanese accent is quite strong in your speaking". So, I think it's difficult to get rid of this Japanese accent especially when we grow old. But at the same time I got this kind of comment bellow. "Your English speaking is quite well!" So now I am convinced that even if we grow old, we can get started to speak our second language. At least, we can start to speak it! It's enough, I suppose.
My YouTube channel is ESPINMC