I'm similar. I went in with certain (incorrect) expectations as to how it would play, and was a bit disappointed as a result. Leaving it for a few weeks to go back to the head-scratching bits made things worse, resulting in the months-long gap since I last played it. I think I'd be better of resetting the game now and trying again - with a notebook this time. Actually, do I even have a choice, given the update bug I read about some time back.
The thing is, I am a gamer that requires a feedback loop, Fez very much doesn't provide an interactive feedback loop at the point I described. As an example, I assumed that it had something to do with the lightning so I spent ages trying to piece together some kind of timing related thing or something to do with illumination but as I had nothing to interact with to give me a clue I just gave up.
I'm not sure if I have always been this way regarding feedback loops but trial and error is very much part of how I play games, probably through conditioning. This idea being subverted is a good way for games to move as it requires more lateral thinking but I wasn't prepared for that and so I just gave up.
What would really raise this game into something more people could get into would be some sort of interactive notepad into which you could c/p symbols from the game, from which to organise and make sense of them. Could be brill on Wii U.