There are text files and there are image, video, and music files. Why are image, video, and music files considered binary files, and why are text files not?
Obviously every file is binary when it comes down to it, since every file is stored as bytes and bits. But colloquially, binary file just means any file which is not a text file. And text file means a file which consists of characters (in some character encoding or character set) and which you can open and edit in an ordinary text editor.
It's a matter of levels of indirection and degree of convention, as so many things in CS are. All files are stored as binary, but some (text files) are stored with a sufficiently simple binary format (a text encoding) that any of a very wide range of programs can display at least a basically correct and useful rendering of the contents that can be edited by straightforward keyboard usage. This includes not only actual unstructured plain text, but also more heavily structured formats such as Markdown, XML, JSON, and so on … as well as source code in nearly any programming language.
Contrast this with sophisticated formats like MPEG-4 or XLSB, which require a much more substantial decoder to make any sense out of the file, and can only be edited (in the general case) with a complex and specialized UI.
Text files are also binary files. They are just binary files with file formats which follow certain conventions, like certain bit-sequences representing certain characters. But even those conventions aren't uniform, because there are many different character encodings, and different standards for line endings.
Everything in computers are stored as 0 or 1 i.e. so called binary values. You can open any file and read it in binary format irrespective of the file extension.
But the special logic is needed to read the file if those 0's and 1's means different to for that file. e.g. 1000001 can refer to its binary equivalent or decimal equivalent of 65 or if you want to read as ASCII then it would mean A.
All the logic is how you want to read it. Similarly the JPEG files are also binary but you read its values differently using different encoders.
As Shakespeare said "Beauty lies in the eyes of beholder."