Getting Shodan’s python API to work with an Internal CA Certificate

TL;DR

Find certifi\cacert.pem and add your internal root ca’s certificate information to the bottom.

I’m learning Python and I’m learning Shodan. Why not put the two together and use Shodan’s python API? Great idea.

Problem: We have a proxy at work. That means that my computer thinks the certificate authority that is vouching for the website I’m visiting is actually my internal CA – which signs the certificate used by the proxy.

This has been a speed bump every time I try to do anything in Python involving the internet. I would get the now vanquished:

[SSL: CERTIFICATE_VERIFY_FAILED]

This came up when testing requests or installing via pip. I discovered work arounds for those.

For PIP – as the article states you can use the flag 

--trustedhosts

Figuring out which hosts needed to be trusted could take a few minutes of head scratching but I’d figure it out.

OR for PIP you can also add your certificate path to the global.cert flag

For requests – I found several workarounds.

First, I could just turn off SSL verification like so:

requests.get('https://github.org', verify=False)

That wasn’t very secure and not a long term solution. So in the same article I found:

requests.get('https://github.com', verify='/path/to/certfile')

This was the precursor to my Shodan fix. It allowed the requests to use my internal CA certificate and verify the SSL connection.

Those workarounds didn’t work with Shodan. It didn’t have verify=false, or verify=[pathtorootcer] or any of those. I looked over the docs and saw no obvious way to either feed shodan a certificate of my choosing OR having it ignore the SSL verification.

At some point while you are troubleshooting an error you realize you aren’t going to find the answer without Google by digging through code. So I started up the traceback. My plan was to find out where in ssl.py or urllib3.py or any other library that I could feed it my certificate.

While navigating the file path looking for some of the libraries, I saw a folder:

In my troubleshooting I saw a post about certifi. I opened cacert.pem and I see a list of certificates. Huh. I wonder if I add my certificate to this list if this will work.

Voila!

Added my certificate which looks like this:

—–BEGIN CERTIFICATE—–
Certificate material
—–END CERTIFICATE—–

And I am now in the business of combing Shodan’s API.

Phew that took a while to crack.

I love cracking difficult issues.

Tips

  • While your searching for how to fix an internal CA issue – I find a lot of info on self-signed certificates. Helps to search for self-signed certificates.
  • When you export the certificate from the web browser – make sure you export the ROOT and not one further down the chain.

References: