You mentioned Swift explicitly, so I'll answer about why Swift doesn't have protected.

Unlike many other languages, Swift lets you write "extensions" to other types (classes, structs, enums and protocols alike), even those which you don't own. Such extensions can even allow you to making library A's type conform to library B's protocol (an example of "retroactive modelling"). For example, you might have an Image object (from library A) that you would like to conform to your ORM's protocol DatabaseSerializable (from library B) so that it could be serialized to a database. In most languages, which requires wrapping up everything adapters all over the place. In Swift, you just extend the Image directly to conform to DatabaseSerializable

extension Image: DatabaseSerializable {
    func serailize(to db: Database) {
        // do whatever is necessary to save to the db or whatever
    }

They're a very core feature that heavily influecnes the style of programming done in Swift. For example, they're frequently used to visually separate conformances to multiple protocols, for example:

class Person {
    let firstName: String
    let lastName: String

    init(firstName: String, lastName: String) {
         self.firstName = firstName
         self.firstName = lastName
    }
}

// This impl can be auto-synthesized by the compiler, but I'm showing it here as an example anyway
extension Person: Equatable {
    static func == (lhs: Person, rhs: Person) -> Bool {
        return lhs.firstName == rhs.firstName && lhs.lastName == rhs.lastName
    }
}

// This impl can be auto-synthesized by the compiler, but I'm showing it here as an example anyway
extension Person: Hashable {
    func hash(into hasher: inout Hasher) {
        hasher.combine(self.firstName)
        hasher.combine(self.lastName)
    }
}

extension Person: CustomStringConvertible {
    var description: String { "\(firstName) \(lastName)" }
}

Now in this example, imagine there was a protected field, socialInsuranceNumber. If I'm in the context of some other class, it shouldn't be accessible. If I'm in the Person class or a subclass, it should be accessible. But what happens if I'm in the context of a Person extension? Should it depend on where the extension is done? (e.g. allow it in the same module as Person, but disallow access from extension in other modules). What happens if I do this?

extension Person {
    public var publicSocialInsuranceNumber: SIN {
        self.socialInsuranceNumber // this should be protected!
    }
}

I've just trivially circumvented the protections a protected access level would offer.

Instead, Swift has fileprivate, which acts like private, except the field is accessible from the defining file. So extension to Person within Person.swift can access socialInsuranceNumber, but Person extensions defined anywhere else can't.