Nosey Parker is an Apache-licensed secrets detector that we built from the ground up for offensive security. It is a secrets detector: essentially, a special-purpose `grep`-like tool with many built-in patterns that pick out things like passwords, credentials, and API tokens that an attacker can use to gain unauthorized access to systems.
There are many pattern-based secret detectors. However, Nosey Parker finds things that other tools miss. It’s played a part in reaching objectives on multiple Red Team engagements against Fortune 100 companies. Security engineers have called it “by far one of the most useful tools on red team engagements these days” and “[behind] quite a few high-risk cloud vulns and attack chains.”
Two years ago, we debuted Nosey Parker at BlackHat Arsenal EU 2022.
In the meantime, a lot has changed, both in terms of Nosey Parker and the environment. This article will outline some of those developments.
Are misplaced secrets actually a problem?
You may wonder if misplaced secrets are actually a problem. How would secrets end up in source code in the first place? Surely, no professional software developer does that!
But the reality is more nuanced and complicated. For many reasons, secrets get written down, waiting to be found by an attacker. For example:
Expediency:
Imagine you are a software engineer, and the project you are working on needs an established approach to managing the secrets that it needs for its operation. Your performance is not measured for doing that kind of work, so you intentionally hard-code the credentials to make your feature work. The repository is private; who is going to see it anyway? (The issue here may be one of misaligned incentives.)
Accident:
Suppose you are a software engineer working on a new feature requiring authenticated access to another system. While developing that feature, you hard-code some credentials just to see if the rest of your code works, to properly handle the credentials later. You commit your in-progress changes to get feedback from your CI system. Alas! The credentials are now preserved in the commit history.
Ignorance:
If you are working with a new technology or framework, you might be following a tutorial that uses a hardcoded token with the comment `// Fill in your token here. `
If this is your first time working in security, you might not even realize when you are embedding credentials in your project. (A few rules in Nosey Parker are derived from example code and tutorials like this that we’ve seen replicated in the wild.)
Shifting assumptions:
The scenario: Your company has an internal project that you are now releasing as open-source. You add the entire source tree to your new Git repository, unaware that it contains hardcoded secrets that were previously deemed an accepted risk.
But now they’re public.
Nosey Parker: How things started vs. how they’re going
The initial release of Nosey Parker in 2022 had only 53 rules and could only scan the filesystem and local Git clones. Findings were reported without any Git metadata, like the commit or pathname, making it challenging at times to review findings.
Now, 11 releases later, Nosey Parker has:
- 161 rules, developed with feedback from security engineers
- Native support for scanning content from GitHub
- A novel high-performance algorithm to determine where in Git history a secret was introduced
- Improved performance and scalability
- A couple dozen contributors
We have also continued development on Nosey Parker++, a Praetorian internal version that adds ML-powered denoising (aka false positive filtering) and ML-powered secrets detection. Praetorian’s security engineers use this during engagements, but it is also available indirectly through Chariot, Praetorian’s Continuous Threat Exposure Management platform.
Additionally, we’ve developed Nosey Parker Explorer, an interactive terminal-based review tool that significantly simplifies the review process, making it feasible to go through thousands of findings within hours.
This tool was instrumental in one “assumed breach” engagement, where we reviewed thousands of findings from scanning 20TB of content that a client had inadvertently exposed through a misconfigured Artifactory repository. We will be releasing this tool under the Apache license soon.
What’s changed in the environment with respect to secrets? What’s the same?
One thing that has changed
Several services, including GitHub and GitLab, have enabled automated secrets detection, revocation, and push protection by default for public repositories in the past two years.
Today, it is less likely that you will find valid hardcoded secrets in public Git repositories on the internet.
We have had more success finding public secrets by looking at sources without automated secrets detection and revocation, such as cloud buckets and AWS Lambda functions.
One thing that has remained the same
On offensive security engagements, we commonly can move laterally with secrets obtained with Nosey Parker’s help.
Once initial access to internal systems is obtained, hardcoded secrets are still readily available. Perimeter security is still a thing in 2024.
What’s next?
We are continuing to develop Nosey Parker based on feedback from offensive security engineers. Future developments will include additional detection capabilities, new rules, and a continued focus on improving signal-to-noise. We may add native support for additional input types, such as GitLab or Docker.
Nosey Parker is an open-source project; you can contribute to it!
One kind of contribution that is especially welcome is for new high-precision rules. Developers don’t stop creating new APIs and services, so there is a perpetual need for additional detection.
We will soon release Nosey Parker Explorer, the interactive review tool, as an open-source project.
