Using Spotlight with macOS

2020-09-23 2087 words 10 minutes

Contents

Spotlight is a system wide metadata indexing system that Apple has been shipping since OS X 10.4 Tiger, and has been improved over the years with each OS. I like using Spotlight for some tasks and general searching for files in Terminal.app. I also use it for finding anything on my Mac. Typically, I do not use my mouse or trackpad to find files or navigate to folders. One just needs to hit cmd + spacebar on their Mac to pull up the Spotlight search menu and start typing. If you use a Mac, there is a good chance you also use this regularly like I do. This isn’t anything groundbreaking or new, but I have enjoyed using Spotlight over the years when it fits as a good tool.

Spotlight in the Terminal getting attributes

There are two binaries Apple supplies on macOS which you can leverage. They are mdls and mdfind, there is also another tool called xattr which also allows you to manipulate metadata in code. To start using these binaries we can just take a look at an application like say Firefox. Refer to their man pages for the documentation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


% mdls /Applications/Firefox.app       
_kMDItemDisplayNameWithExtensions      = "Firefox.app"
kMDItemAlternateNames                  = (
    "Firefox.app"
)
kMDItemAppStoreCategory                = "Productivity"
kMDItemAppStoreCategoryType            = "public.app-category.productivity"
kMDItemCFBundleIdentifier              = "org.mozilla.firefox"
kMDItemContentCreationDate             = 2020-08-31 18:05:40 +0000
kMDItemContentCreationDate_Ranking     = 2020-08-31 00:00:00 +0000
kMDItemContentModificationDate         = 2020-09-08 17:37:19 +0000
kMDItemContentModificationDate_Ranking = 2020-09-08 00:00:00 +0000
kMDItemContentType                     = "com.apple.application-bundle"
kMDItemContentTypeTree                 = (
    "com.apple.application-bundle",
    "com.apple.application",
    "public.executable",
    "com.apple.localizable-name-bundle",
    "com.apple.bundle",
    "public.directory",
    "public.item",
    "com.apple.package"
)
kMDItemDateAdded                       = 2020-09-08 17:35:57 +0000
kMDItemDateAdded_Ranking               = 2020-09-08 00:00:00 +0000
kMDItemDisplayName                     = "Firefox"
kMDItemDocumentIdentifier              = 0
kMDItemExecutableArchitectures         = (
    "x86_64"
)
kMDItemFSContentChangeDate             = 2020-09-08 17:37:19 +0000
kMDItemFSCreationDate                  = 2020-08-31 18:05:40 +0000
kMDItemFSCreatorCode                   = ""
kMDItemFSFinderFlags                   = 0
kMDItemFSHasCustomIcon                 = (null)
kMDItemFSInvisible                     = 0
kMDItemFSIsExtensionHidden             = 1
kMDItemFSIsStationery                  = (null)
kMDItemFSLabel                         = 0
kMDItemFSName                          = "Firefox.app"
kMDItemFSNodeCount                     = 1
kMDItemFSOwnerGroupID                  = 80
kMDItemFSOwnerUserID                   = 0
kMDItemFSSize                          = 210713091
kMDItemFSTypeCode                      = ""
kMDItemInterestingDate_Ranking         = 2020-09-08 00:00:00 +0000
kMDItemKind                            = "Application"
kMDItemLastUsedDate                    = 2020-09-08 17:37:09 +0000
kMDItemLastUsedDate_Ranking            = 2020-09-08 00:00:00 +0000
kMDItemLogicalSize                     = 210713091
kMDItemPhysicalSize                    = 211038208
kMDItemUseCount                        = 1
kMDItemUsedDates                       = (
    "2020-09-08 07:00:00 +0000"
)
kMDItemVersion                         = "80.0.1"

There are many metadata tags we can get right from the shell as well.

1
2


% mdls /Applications/Firefox.app -name kMDItemVersion
kMDItemVersion = "80.0.1"

You can see this is easy to get the metadata from an object on disk, it is also very fast as it searches the indexes and returns the indexed data.

1
2


% mdls /Applications/Firefox.app -name kMDItemVersion -raw
80.0.1                        

There is also no need to pipe to awk or grep as the tool gives you an argument of -raw to just return the data. Easily get multiple metadata attributes by passing multiple arguments.

1
2
3
4


% mdls /Applications/Firefox.app -name kMDItemVersion -name kMDItemFSName -name kMDItemLastUsedDate    
kMDItemFSName       = "Firefox.app"
kMDItemLastUsedDate = 2020-09-08 17:37:09 +0000
kMDItemVersion      = "80.0.1"

Spotlight in the Terminal searching

mdls is great for listing the metadata for objects on disk, and you can probably guess what mdfind is used for. After getting the metadata tags from mdls we can use them in mdfind you can also pass strings to mdfind and it performs searches similar to when you use it in the Finder.

1
2
3


% mdfind "using-spotlight"
/Users/tlarkin/blog/tlark/content/posts/using-spotlight-macos.md
/Users/tlarkin/Library/Application Support/JetBrains/IdeaIC2020.1/workspace/1hksXRhYWu8MI2RnUXoH3oitxRW.xml

Like finding the blog post I am currently working on, which my IDE is also storing data about on disk. This output makes sense considering I do all my blog work in my IDE with hugo along with built in tools. We can also use the metadata tags which we see in mdls output.

1
2


% mdfind "kMDItemFSName = Firefox.app"
/Applications/Firefox.app

You can specify file paths if you have an idea or what to limit search scope for metadata searching.

1
2


% mdfind -onlyin /Users/tlarkin "kMDItemContentType = public.python-script" | wc -l
   11582

I have a lot of Python scripts on my Mac, so I just did a line count of the output. I have a lot of repos cloned, open source tools cloned, and there are a lot of apps that use Python, so a big chunk of these results are from the software I have installed. However, every object in my home folder that has kMDItemContentType = public.python-script tag will return from this search.

Using mdfind with mdls

Of course with things like pipes in the shell you can easily use these tools together.

1

% mdfind -0 -onlyin /Applications "kMDItemContentType = com.apple.application-bundle" | xargs -0 mdls -name kMDItemCFBundleIdentifier -name kMDItemDisplayName -name kMDItemVersion

There is a lot of output I won’t paste into a code block here, but you can play around with this to see the different output you can get from these binaries. Last year there was a pretty nasty iTerm2 Vuln that we detected through our security tools, and the fact it was all over Twitter and the rest of the Internet. We wanted to detect how many vulnerable versions we had and then plan on patching after assessing how many vuln versions we had. A simple spotlight script was deployed, and anything that wasn’t the current version got flagged. Since iTerm2 is just a zipped App bundle, we had no idea where the user installed it. We also do not block users from installing their own software and tools, so there could be multiple versions present. Tools like Jamf Pro will not search for Apps outside of a default path unless you specify so in the inventory collection preferences. A quick Spotlight script made sense and it didn’t require much effort. There are tools out there like OSquery which is probably a better answer to get data about your fleet on a regular basis.

1
2
3
4
5
6
7
8


#!/bin/zsh

results=$(mdfind -name "kMDItemCFBundleIdentifier = com.googlecode.iterm2 && kMDItemVersion != 3.3.6")

if [[ "${results}" == "" ]]
    then echo "<result>false</result>"
    else echo "<result>true</result>"
fi

With a day we had the data back, and we had very few users with a vulnerable version on their system as most users were updating their versions on their own. So, we just had to send an update to a few systems. I do think there are better ways and better tools to manage application state out there, but if you don’t have those every Mac has Spotlight on it.

Metadata Tagging

None of this is new, and I am certainly not the first person to use these tools or blog about this stuff. The linked blog has some pretty good reading material you can reference. A quick reference how you can do this below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# create a file
touch file.txt
# add a custom tag
xattr -w com.apple.metadata:"_customTag" "customVal1" file.txt
# get the attribute
mdls -name _customTag file.txt -raw                           
customVal1
# search for it
mdfind "_customTag = customVal1"
/Users/tlarkin/file.txt

There are a lot of things you could do with metadata tagging in macOS. You can programmatically search for metadata tags, as well as set your own tags.

Caveats with Custom Tagging

When tagging things on disk that get overwritten, like contents of an installer package, the tags could be deleted. So, if you were to say tag an app you installed with your automation tools, then a user downloads another version and overwrites it, your tags, are now probably gone.

Some file paths are not indexed by Spotlight. Paths like /tmp are not indexed. So, make sure you test the paths you are using if you use these tools. I have definitely forgotten this before and tested custom tags with xattr in /tmp and completely spaced that this doesn’t work.

Some file names are error prone with certain characters. Apple, I think has fixed this bug as I can now tag a file located in a folder with a . in the folder name. This was broken at some point in time, Mac Mule discovered it. I was not able to locate his blog post on it, but will update with a link if I can find it.

Python Objc Bridge

You can also do all of these neat things in Python as well. I wrote this script not too long ago to search and find all the Parallels VMs on a Mac. The end goal was to collect the OS versions of all VMs and ship that data to Snowflake, so we could report on them for vulnerabilities on the OS.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85


#!/usr/bin/python

"""
this is a script to detect what VMs are on a system and escrow what OS they are
You can run this daily or in another workflow
It will use Spotlight to find files with the .pvm extension to locate where the files are on the file system
then parse the PvInfo file for Parallels VMs

credit:  https://github.com/munki/munki/blob/b6a4b015297c262124fb80086b6b55e329a4fec0/code/client/munkilib/info.py#L362-L396

"""

# import modules
import xml.etree.ElementTree as et
from Foundation import NSMetadataQuery, NSPredicate, NSRunLoop, NSDate


# start functions


def get_vms():
    """use spotlight to find parallels VM files"""
    file_list = []
    query = NSMetadataQuery.alloc().init()
    query.setPredicate_(
        NSPredicate.predicateWithFormat_(
            "(kMDItemContentType == 'com.parallels.vm.vmpackage')"
        )
    )
    query.setSearchScopes_(["/Applications", "/Users"])
    query.startQuery()
    start_time = 0
    max_time = 20
    while query.isGathering() and start_time <= max_time:
        start_time += 0.3
        NSRunLoop.currentRunLoop().runUntilDate_(
            NSDate.dateWithTimeIntervalSinceNow_(0.3)
        )
    query.stopQuery()
    # get the results of the file names, and find their file path via spotlight attribute
    for item in query.results():
        pathname = item.valueForAttribute_("kMDItemPath")
        if pathname:
            file_list.append(pathname)
    return file_list


def get_vm_os(vm_list):
    """feed this function a list of results from Spotlight to parse what VM is running what OS"""
    # blank list to populate data in
    os_list = []
    # loop through return from spotlight and grab needed info
    for vm in vm_list:
        path = str(vm + "/VmInfo.pvi")
        # load XML file into parser
        tree = et.ElementTree(file=path)
        root = tree.getroot()
        # find the text of the tags we want
        for element in root.iter():
            if element.tag == "RealOsType" and element.text is not None:
                # only want the data up until the first comma
                os_type = element.text.split(",")[0]
            if element.tag == "RealOsVersion" and element.text is not None:
                os_ver = element.text
                # combine the two pieces of data into single string for the list
                cstring = os_type + " " + os_ver
                os_list.append(cstring)
    return os_list


def main():
    """main to rule them all"""
    # get VM XML files
    files = get_vms()
    # parse XML files
    results = get_vm_os(files)
    # loop through and print out multi value EA for jamf inventory
    print("<result>")
    for item in results:
        print(item)
    print("</result>")


if __name__ == "__main__":
    main()

The above code was pretty much stolen from the Munki Project

Munki is probably the best online resource for Python ObjectiveC code. It is filled with tons of gems from the maintainer as well as the community.

I think that BYOD deployments are a very bad idea for many reasons. To sum up the biggest two reasons I will simply put:

Good luck putting that BYOD device on term hold
Good luck putting that BYOD device on legal hold

That being stated, over a few beers a year or two ago I was chatting with a buddy of mine about this topic. Spotlight tagging came into the conversation and I found myself bored about a week later one evening so decided to just add something like this to a DEP Notify workflow. So, I posted it here on my GitHub

That project is not tested (except for like once in a VM), nor maintained and was really just meant as a code example. Use at your own risk, and think about not doing BYOD if you can. The code uses similar tooling as the binaries above, but allows for the flexibility of Python. It also interacts with the macOS APIs, so some extended functionality is there.

So, there are many uses for Spotlight one can use in IT/Ops workflows.