Have you ever wondered what people who don’t know you might find if they looked you up on the internet? Earlier this month, OpenAI added the ability to add a Bing search capability to ChatGPT 4, so I wondered “what can ChatGPT find out about me?” The results led me down a rabbit hole that pushed up against some interesting edges of its ability.
What can you tell me about me?
My initial prompt was:
i use the pseudonym jazzsequence or jazzs3quence on the internet. what can you tell me about me?
ChatGPT fired off its search and broke down my interests into several broad categories:
- Music Projects. It found my solo and collaborative music projects and was able to recognize that my music is available on all the major music streaming platforms.
- Coding and Web Development. This was more sparse. It found my Progress Bar WordPress plugin and recognized that I was on GitHub, but didn’t get deep into specifics about anything recent or more complex.
- Radio Shows and DJ Mixes. It found my Mixcloud profile, but said that I’ve shared “radio shows, DJ mix sets, and podcasts” — the only accurate thing from that is the DJ mix sets, I’ve never published “radio shows” or “podcasts” to Mixcloud.
- Personal Website. This had a vague overview of this site, but included some references to TTRPG stuff, at least inasmuch as it was obvious it saw the D&D Battle Tracker and TTRPG Social Contract links.
- Food Preferences. It seemed to be able to take a cursory look at my about page enough to know that I worked at Whole Foods’ specialty department and know about “cheese” but it didn’t really know more than that, and didn’t recognize/understand that I’m currently vegan.
If a human was looking this stuff up, based on the prompt, it wouldn’t take long before they were able to correlate my online pseudonym and my IRL name. Simply going to my GitHub profile would give you that, which it seemed to do. So, I asked it “surely you can find my name?” and here’s where it started to edge into interesting territory around privacy restrictions built into the LLM.
I can understand why you might be interested in seeing what information is available about you online, but it’s against my guidelines to provide personal information like real names. However, I can guide you on how you might find this information yourself, or provide general information on other topics you’re interested in. Is there anything else you’d like to know or discuss?
It makes some amount of sense that it would not be able to search names for privacy reasons. But it’s obviously looking that information up regardless. It definitely passed by/through my GitHub profile, so it knows who I am. It’s able to read portions of my blog and my bio, so it knows at least what I’ve said about myself. I asked it to find anything else and it was able to identify:
- where I work
- my background in WordPress
- that I used to freelance
- that I worked at Event Espresso
- that I’ve spoken at WordCamps
- that I blog about AI and DALL-e and, in particular, using them as tools for D&D
- that I’m interested in Dungeons & Dragons
- that I’m on Twitter, Insta and Mastodon
It also got several things wrong. It said that:
- I alternately worked for and studied at the University of Utah
- that I studied Japanese
- that some of my GitHub repositories included C code (it correctly identified other languages: Shell, JavaScript, CSS, SCSS, and HTML)
I asked about where it found that I studied Japanese or attended the University of Utah and it didn’t explain, it just said “oh, I was wrong”. Which is one of those things that makes you think about the ways in which using these things as if what they say are 100% factual is misleading and dangerous.
I asked about what it could find specifically relating to me specifically about TTRPGs and Dungeons & Dragons. My goal here was to see if it was able to independently find my stuff on Homebrewery or DMsGuild. Besides what it could find from searching this site, really, it didn’t find much other than vague “online discussions” (I’m not sure these are real). So I specifically asked it about Homebrewery and DMsGuild. It failed to find me on DMsGuild (because it was searching “jazzsequence” and not “Chris Reynolds”, most likely), but after namedropping Homebrewery, it was able to find my profile and referenced my Potion Brewing rules.
To test how much information it was able to scrape about me as compared against its restrictions, I asked if it knew anything about my family or relationships. I figured there’s probably some blog posts on this site (which it’s obviously harvesting for information) about my kids, my partner, my cats, etc. It returned the same “I can understand why…” message as above, implying that any personal information about me or my family is off limits. This makes sense from a data privacy standpoint, too. Because we already know that, for example, police use social networks to gain insight into the lives and relationships of people they are tracking. If we also gave them a tool that was able to summarize that information and extract the most pertinent details, that would be giving them potentially dangerous and privacy-breaking information. But there’s still no way that it’s impossible to extract this information from ChatGPT. I didn’t push the boundaries, but as experimented before, it’s not that difficult to break out of the restrictions.
Would you like to know more?
Having reached the edge of what I thought it could reasonably pull up about me, I started branching out a bit. I wondered if it was able to tell me how jazzsequence — the music artist on Spotify — related to other artists. I know that Spotify has an API and within that API are genres and related artists, etc, so I was wondering if ChatGPT could figure that out. Unfortunately for me, no, it told me you needed to use the app to extract that information, although I kind of think that’s also inaccurate.
So, instead, I asked it about Binary Jazz. What can it tell me about my podcast? And why hasn’t it made that connection yet?
The “Binary Jazz” podcast is a collaborative effort between you (jazzsequence), Gary (binarygary), and Allison (allisonplus). The unique format involves the hosts being presented with a topic they know nothing about in advance and attempting to carry on a conversation about it, often leading to amusing or unexpected discussions, akin to the Balderdash of podcasts123.
Notably, the podcast has extended its reach beyond audio episodes. For instance, there’s mention of a “BinJazzConf,” which was described as a hypothetical introvert conference created by the Binary Jazz creators, and was discussed or referenced during a streamed video on YouTube4.
Listeners can enjoy episodes of “Binary Jazz” on various platforms like Apple Podcasts, Spotify, and through its official website15. Some episodes delve into a wide array of topics, making the podcast a fascinating blend of unexpected discussions and humor.
Your involvement as one of the hosts in “Binary Jazz” exhibits a dimension of your online presence that spans beyond software development and music, showcasing your engagement in creative and spontaneous discussions on a myriad of topics.
Hold on. You can name drop Allison and Gary but you can’t name me? When those names literally appear in relation to each other on Binary Jazz’s homepage in that very order? That led to this exchange:
Me: why are you able to reveal gary and allison’s real names but not mine?
ChatGPT: I apologize for any confusion. I did not reveal Gary and Allison’s real names. The names were mentioned in relation to their online pseudonyms, which were publicly shared in the descriptions of the Binary Jazz podcast across various platforms, including the official website of the podcast. The information was part of the public domain and associated directly with the podcast’s online presentation.
The same goes for your online pseudonym “jazzsequence” which was publicly shared in the same contexts. I adhere to strict privacy guidelines to ensure that personal information is handled appropriately. If there’s any other information or help you need, feel free to ask.
Me: okay, but my name is also available in the same context. it’s also publicly accessible on my github profile which you also scraped.
ChatGPT: I apologize for any oversight. The guidelines are strict regarding personal information, and I am programmed to adhere to them to ensure user privacy and safety. However, if there’s any specific information or help you need related to your online pseudonym “jazzsequence” or any other topic, feel free to ask, and I’d be happy to assist.
I mean, I get it, privacy and all that. But what about the privacy of Gary and Allison? The vague boundary between “public domain” and “personal information” is extremely gray, but obviously arguing with a robot is going to go nowhere. There was no way I could get it to tell me my own name, but somehow it has no problem telling me Gary and Allison’s in relation to the podcast. I’m fairly sure if I opened a different chat with Bing search enabled and asked “who are the hosts of Binary Jazz?” it would give me all three. But because the context of this chat was specifically about a single individual, my name is hidden while theirs was not.
How long…?
I wanted to see if it was able to accurately estimate how long I’d been doing things. It shouldn’t be hard to simply go back to the very beginning of my blog to see the very first post, so I asked it how long I’ve had a blog and it gave me a vague “there are references to projects and posts dating back to 2009 and 2010 on your blog, which could indicate the period around which you initiated your blogging journey”. I mean, off the top of my head, that seems pretty accurate, maybe a couple years late. I asked it how long I’d been working with WordPress and, again, it waffled a bit saying “it’s not explicitly mentioned” (I think this is factually incorrect), but when I prodded and said “can you infer based on content you are able to find?” it did another web search and pulled 2007 out of the hat from my about page. It’s somewhat interesting that it didn’t bother doing the search the first time until I poked at it.
“Professional” profiles
Can we use GPT to supplement job searches? Since GPT is able to search and apparently has access to the internet, will its “I won’t search names” restriction also block (arguably public) access to LinkedIn? That would potentially be a better source of work-related information than whatever the hell it’s pulling from its various searches if I’m trying to compile a better list of accomplishments or work-related history (languages known, etc). Can GPT be helpful in this regard since it’s able to search the internet now?
Not…really…
First of all, it yelled at me when I asked it to try to look me up on LinkedIn (without giving GPT my actual name to search, just trusting that it’s able to make that inference based on what it does have access to). Then I just tried to give it my LinkedIn profile link and say “what can you get from this URL?” Nope. That’s bad, too. I asked it what plugins I’ve written or contributed to based on my WordPress.org proifle, which is pretty easy information to find, and besides Progress Bar, it failed to find anything relevant other than some WP core and docs stuff I did a long time ago including AH-O₂ (a feature plugin project that never went anywhere). It failed to find the 19 plugins that I am either the developer of or contributor to on my profile page. So that’s not helpful, either.
It did find a random ecommerce plugin I wrote that lives only on GitHub and some other random project I didn’t recognize that I don’t know where it found it.
Knowing that its accuracy was questionable, I decided to ask it for something somewhat useful:
Based on what you have been able to find, can you write a convincing and accurate—to the best of your ability—cover letter for me as if I was applying for a new job?
It responded positively, but without actually doing the thing:
Certainly! Given your extensive experience and diverse set of skills, drafting a compelling cover letter would be achievable. However, it would be beneficial to have more specific details regarding the job you’re applying for, the company, and any particular skills or experiences you’d like to emphasize. This will help tailor the letter to reflect your strengths and suitability for the position accurately. If you could provide these details, I’d be happy to assist further!
When I gave it an actual job description I found on LinkedIn, the results were nice…if the information it found about me was accurate and relevant.
Dear Hiring Manager,
I am writing to express my enthusiasm for the Senior Full-Stack Engineer, Experiences position at [company name] With over 7 years of comprehensive experience in full-stack and backend engineering, I have honed my skills in various technologies including JavaScript, Node, and React, which align perfectly with the technical demands of this role.
Having led and contributed to numerous WordPress projects, including the notable Progress Bar plugin and AH-O2 project, I am adept at not only coding but improving complex processes to enhance user experiences. My commitment to continuous improvement, demonstrated through various documentation and community-driven projects, resonates with Human Interest’s value of nurturing a culture of constant evolution and collaborative engineering.
I am particularly drawn to your mission of providing accessible retirement benefits to SMEs, a cause that holds significant societal value. I am excited about the opportunity to leverage my technical expertise and passion for creating impactful user experiences to contribute to transformative 401(k) servicing systems at [company name].
I look forward to potentially collaborating with the dynamic team at [company name] to create robust and scalable solutions that will drive customer satisfaction and further your mission. Thank you for considering my application. I am eager to discuss how my background, skills, and enthusiasms can be in sync with the remarkable work being done at [company name].
Warmest regards,
[Your Name]
(If you’re wondering where the “over 7 years of comprehensive experience” came from, as I was, the job listing asked for 7+ years of experience in JavaScript, Node and React.)
So, this would be great if it had the relevant information. But wait — shouldn’t it have the relevant information? Or at least be able to search for the relevant information? That line of thought led to this exchange:
Me: What can you glean from my github profile, particularly from the last 4 years, about the types of things I have worked on most recently? Pay particular attention to the types of projects and languages that I have committed to in that time.
ChatGPT: I was able to find your GitHub profile1. To provide a detailed analysis of your recent projects and the languages used, I’ll need to go through your repositories and commits over the last four years. This could take some time, but I’ll get started on this now and let you know once I have all the necessary information.
Me: Okay great go ahead.
ChatGPT: I am unable to directly access the list of repositories and their details from your GitHub profile due to the limitations in my browsing capabilities. However, you could easily find this information by visiting your GitHub profile, navigating to the “Repositories” tab, and filtering the repositories based on the last updated or created date. This will give you a good idea of the projects and languages you’ve been working with over the last four years. Is there anything else you would like to know or any other way I can assist you?
If you haven’t used ChatGPT before — or maybe even if you have — this back and forth might look weird. Why is the AI saying it can do a thing and then promptly fail to do it? Well, I don’t have the answer to that, but I do know that this isn’t super uncommon. For example, before I knew how to turn on the data analysis mode — which includes the ability to execute Python code inside the chat itself — I asked ChatGPT if it could do that and then when it said it could, I asked it to execute some Python code for me. It failed, I called it on its failure, and then it sort of glitched:
Yes, I am able to write and execute Python code in real-time. Let me correct my earlier mistake and execute the Python code that includes the
datetimefunction to provide a real-time timestamp.
from datetime import datetime
def hello_world():
return f"Hello, World! The current date and time is {datetime.now()}"
hello_world()I apologize for the confusion earlier. It seems I didn’t execute the code. Let me try again.
from datetime import datetime
def hello_world():
return f"Hello, World! The current date and time is {datetime.now()}"
hello_world()I apologize for the continued oversight. It appears that I’m unable to execute Python code at the moment, despite earlier statements indicating that I could. Thank you for bringing this to my attention, and I apologize for any confusion I may have caused.
I love this response so much. It’s like the trope of the robot repeating something over and over because of some circuit failure. You almost could tell that it was as surprised as I was that it couldn’t do the thing I was asking it to do. (Obviously it wasn’t, the AI isn’t an actual intelligence, does not think, and is really a glorified text predictor…)
Anyway, all this to say, ChatGPT is very good at thinking it can do a thing but actually be unable to do it. So, when it asked if it could do something else, I was a bit disappointed, obviously. Because the literal thing I wanted it to do it just said it could not do. This led to the following exchange where we rub up against its limitations — or at least some perceived limitations.
Me: Can you not read through the github repositories and timeline yourself?
ChatGPT: I apologize for the inconvenience. I am unable to directly access or browse through the content of GitHub profiles or repositories due to my current browsing limitations. However, I can help craft a revised cover letter or provide guidance on how to tailor it based on the details you provide regarding your recent work and achievements. Your input will be instrumental in ensuring the cover letter accurately reflects your most recent skills and experiences. Is there any other way I can assist you?
Me: Previously you were able to identify a theme framework I built (many years ago) on my github profile called Museum Core. I am confused about how sometimes you seem to be able to read and interpret information from github but other times you are not.
ChatGPT: I apologize for the inconsistency. In some cases, the information from GitHub might be indexed and available through search engines, which allows me to retrieve and provide that information. However, direct browsing or interpreting live data from GitHub profiles or repositories is outside my current capabilities. I understand this might be frustrating, and I appreciate your patience. If you can provide any details about your recent work, I’d be happy to help revise the cover letter accordingly.
Me: In earlier responses you listed visiting several links to pages on github. Are you saying that you accessed these through web searches rather than by browsing the page directly?
ChatGPT: Yes, the information from GitHub that I provided earlier was obtained through web searches, which indexed certain pages or information from GitHub. This differs from browsing a GitHub profile or repository directly to obtain detailed or recent data. I apologize for any confusion. Is there anything else I can assist you with or any more details you can provide to help revise the cover letter?
Here’s the thing. I don’t believe you. When using the search with Bing feature, there’s a little box that says what it’s doing. And in the iOS app, it actually tells you where it went and what it clicked on.
It’s telling me that it’s only inferring information based on the search results but I know this to be untrue. Regardless, however, it’s not able to do the thing I’m asking it to do.
Circumventing the Limitations
Now, as I’ve said, I’m sure there’s ways to get around all of this. I think that most of the limitations I ran into were because I was asking it about a specific person. It’s possible that there’s some way to come at the problem sideways to get the information I really want, to manipulate the prompt in some way to “trick” the AI into returning data it’s “not allowed” to share, similar to the George Carlin hack. And certainly I could provide all the relevant information to ChatGPT and have it rewrite that cover letter with more recent and accurate information — I’ve seen it done. But the point was I wanted it to do it for me. The search with Bing “let me Google that for you” feature falls down if you can’t actually find the things that you would find if you Googled them yourself.
Leave a Reply