You’re both right and both wrong. As pointed out by @flavrans9, MP4, or rather the MP4 file format as defined in MPEG-4 Part 14, although mostly used for video data is a “digital multimedia format” or a container format and can contain audio, video, both, or something different altogether.
I am not sure if there is an mp4 audio visualizer. What I do know is that it should be possible to extract the audio data from the MP4 format and store it in a lossless file format (ensuring no data is lost by conversion), using the same tools or similar to those used to store it as MP3. I believe at least some of the several audio visualizers out there support reading such formats, too.
Although, honestly? I don’t think it matters. MP3, unless exported using horrible settings, should be more than fine for the details you’re looking for in the spectograms.