How To Run a (Mostly) Static Website in Google App Engine


At the beginning of 2015, I moved all of the deciphertools.com website to the Google App Engine. Most of the content on our website is static and our traffic is moderate but bursty, so running our own virtual server on Rackspace to host the website seemed wasteful (of administration time and money.) Our virtual server was also performing poorly: during those bursty times, we would have poor latency, and our website was very open to (extremely lame) DoS attacks.

After the move, our website is cheaper to run, and performs beautifully. I can rest easy at night knowing that if we have spikes in traffic (which should be cause for celebration), our infrastructure will scale to handle the load. To this day our static site costs have been free. We do pay for some other dynamic site projects as well as Google Cloud Storage to host our larger software downloads, at rates comparable to Rackspace Cloud Files. The entirety is still MUCH cheaper than spinning up a Rackspace cloud server ourselves and running lighttpd or apache.

Finding simple directions to host static pages on the Google App Engine was difficult, so this is my contribution of instructions. Please feel free to post comments with questions — questions usually make me learn something new!

Step 0: Download the App Engine SDK

Download the SDK (and administration program) from the App Engine download page.

Edit diciembre 18, 2017: Google replaced the original App Engine SDK with a more complicated toolset, which is some pain to get up to speed with for simple sites. However, there was enough complaint about it that they made original SDK available again. Go to this Google App Engine download page, scroll down, and click "Download and install the original App Engine SDK for PHP." to download the older SDK. This SDK is much easier to use for new App Engine users (including a great big Deploy button). My link is to the PHP download page, but if you're hosting a truly static site, then it doesn't matter which programming language you pick.

More on picking a development language for app engine: most sites end up needing some redirections and email scripts, so pick something you're comfortable scripting in. If you're used to web scripting in PHP and the site is really only going to do basic stuff like redirection and email, then PHP is great. If you're going to want to use the App Engine datastore and other APIs eventually, then I recommend Python. (You can also change your mind later.)

Step 1: Setup your project folder structure

Every app engine project has an app.yaml file, and then all of the supporting scripts and files. For (mostly) static sites, I like to have the following folder structure:

- my_project
  |- app_engine
     |- app.yaml
     |- public
        |- (all of my static site files, examples here below...)
        |- index.html
        |- favicon.ico
        |- images
           |- image1.png
           |- image2.png
        |- js
           |- bootstrap.min.js
        |- css
           |- ... you get the idea

I make that app_engine folder to house the Google App Engine project in case I have external resources to generate pieces of the website. For example, I use blogofile to generate our website and blog, so in addition to the app_engine folder, I also have a blogofile project that houses the blogofile templates. (I generate the site files from my blogofile templates, then copy the results into app_engine/public when I am happy with them.)

Step 1.5: Setup your static file folder

The public folder is going to house the static files that your site serves. Structure this just like it were your normal site folder on a web-server setup. In my cool ascii-art diagram above, you can see a typical index page, images subfolder, and Javascript/CSS folders.

Step 2: Make a simple app.yaml for all static content

You might be appalled by the cases, I'm ok with that!

application: your-application-name-here
version: 1
runtime: php
api_version: 1
threadsafe: yes

handlers:

# Handle the main page by serving the index page.
# Note the $ to specify the end of the path, since app.yaml does prefix matching.
- url: /$
  static_files: public/index.html
  upload: public/index.html


# Handle folder urls by serving the index.html page inside.
- url: /(.*)/$
  static_files: public/\1/index.html
  upload: public/.*/index.html

# Handle nearly every other file by just serving it.
- url: /(.+)
  static_files: public/\1
  upload: public/(.*)


# Recommended file skipping declaration from the GAE tutorials
skip_files:
  - ^(.*/)?app\.yaml
  - ^(.*/)?app\.yml
  - ^(.*/)?#.*#
  - ^(.*/)?.*~
  - ^(.*/)?.*\.py[co]
  - ^(.*/)?.*/RCS/.*
  - ^(.*/)?\..*
  - ^(.*/)?tests$
  - ^(.*/)?test$
  - ^test/(.*/)?
  - ^COPYING.LESSER
  - ^README\..*
  - \.gitignore
  - ^\.git/.*
  - \.*\.lint$
  - ^fabfile\.py
  - ^testrunner\.py
  - ^grunt\.js
  - ^node_modules/(.*/)?

Step 3: Handling your 404 baggage with redirects

Our website has evolved over many years, so we have a hefty pile of 301 redirects that need serving. To serve the 301 redirect header, you need to use some scripting; for this example I'm using PHP. (If anyone knows how to serve the 301 statically, I would LOVE to know how.) You may be grossed out by the following script, but it works, and it reminds me of lighttpd's mod_redirect, so I'm happy enough.

  • I setup a script named redirector.php. I was lazy and it's just sitting in my app_engine folder; it would probably be better in a scripts folder or something clean like that.
<?
// REMINDER: ALL OF THESE NEED TO BE IN app.yaml too
$direct_redirects = array(
  "/blog" => "https://yoururl.com/blog/",
  "/products.html" => "https://yoururl.com/index.html",
  ... many many MANY... MANY other mappings...
);

$path = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);
$redirect_url = $direct_redirects[$path];
if(!is_null($redirect_url)) {
  header("HTTP/1.1 301 Moved Permanently"); 
  header("Location: $redirect_url"); 
}
?>

(Yep, you have to map every url you want to redirect. Get fancier if you need, but I just use this to map crawl errors and moved pages, so it works for me.)

  • As the script so nicely reminds me, the urls that need redirection need to be in your app.yaml file. Add your paths to the handlers section ABOVE the other rules, since those rules match many things.
# Note the $ to specify the end of the path, since app.yaml does prefix matching.
- url: /blog$|/products.html$
  script: redirector.php

Step 4: Installing SSL and Setting Up Your Domain with Google Apps

If you want to serve your site using HTTPS, then you'll need to install SSL certificates in GAE.

Update agosto 24, 2018: If you don't need special SSL certificates, like EV (Extended Validation, for the green bar in the web browser), then using Google's free managed SSL for Google App Engine may be just what you need.

To enable managed SSL in App Engine:

  1. Go to your Google App Engine dashboard.
  2. Click the menu icon in the upper left corner, and under Compute, click App Engine > Settings.
  3. Select the Custom domains section.
  4. Check the domains you want to secure with managed SSL and then click the Enable managed security button.
  5. Take this time you'd usually spend banging on SSL setup to get a coffee or go for a walk.

Or, if you prefer to set up your own SSL certificates, keep reading.

If you need to install SSL on your App Engine app, you will need to setup your domain with Google Apps. If you want to support us, you can use our Google Apps referral link to sign up for Google Apps. Update 2016: You no longer need a Google Apps account to install SSL certificates on your App Engine site. Refer to the “Adding SSL to your custom domain” section of the instructions from Google in the next paragraph.

I don't know if there is something wrong with me, but I never remember how to do HTTPS/SSL setup. There are copious outdated documentation pages lurking around, along with poor instructions from third parties. I HIGHLY recommend these instructions from Google augmented with these instructions about the actual SSL installation from the Neutron Drive Blog. Update diciembre 18, 2017: My favorite SSL blog post is no longer maintained, so here is my version of those same instructions.

A little bit about CloudFlare

If you need CDN caching, threat mitigation, SSL, site redirections, and a pile of other awesome services, I highly recommend CloudFlare. We used to use the Pro Plan for the Decipher Tools website, and I use the free plan for every static site I setup (at a bare minimum to use their lovely DNS interface that doesn't make me want to shove a spoon in my eye like most others.) Now that we have a static site, and our own SSL setup from step 4, we don't need the paid CloudFlare features. However I still really dig their DNS interface, and their caching is excellent if you have need.


13 Comments



MOHIT GUPTA

MOHIT GUPTA julio 7, 2016

Hi Kelly HW ,

Need to ask something I have made my project using build.xml and my all urls are on http and now i want to move them on HTTPS how can i do that and my web site is based on google app engine

Dileep P G

Dileep P G diciembre 12, 2015

Hi, Neat tutorial! Helped quick start my website on GAE.

As for redirects, I do not have any specific page redirects. But I did need the one domain redirect from naked to www. And I opted for the Domain Forwarding feature available on GoDaddy (My Domain Registrar) instead of using the 301 re-director script. Works good so far. Let me know if any pitfalls going that route.

Thanks!

Kelly HW

Kelly HW diciembre 12, 2015

I think it's great if you can get the forwarding "for free" using another technique besides paying for app engine processing time :)

One thing we should check though: if you have a page in the url (not just the domain itself), does the domain forward keep the page in the URL intact? That can be important for SEO/page rank, if someone posts a link to a page using your naked domain, you want it to redirect to the same page with the www subdomain.

Ferdinand

Ferdinand junio 6, 2015

Do you really need a CDN if it is hosted on the app engine? Is it not suppose scale "auto-magically"? Do you find any speed improvement by using the CDN?

Kelly HW

Kelly HW junio 6, 2015

That is a great question Ferdinand. We used CloudFlare prior to the switch, and I've left it for a number of reasons that may be moot now (SSL and cache headers are the first two that come to mind, and those should be much better/easier to address directly in GAE now.)

When I get a chance (hopefully next week? since I'm at WWDC right now) I'm going to do a little bit of poking, measuring, and Google Page Speed Insights experiments moving a few things I do in our CDN configuration directly into GAE now.

GAE does scale automagically :) I'm curious to see if there is a difference in latency worldwide for static resources with and without the CDN.

Ferdinand

Ferdinand junio 6, 2015

Thanks. I have had varying degrees of issues with CDN and WordPress while using CloudFlare. Cache headers was one of them. Also, the caching while using the admin interface was pretty bad at that time (couple of years back). I moved to the now defunct Google PSS which was much better, but still had intermittent issues with caching.

Finally, when PSS was downgraded I moved to Compute Engine to host my blog. After having to scrape clean my blog from Pharma hacks a couple of times, I now have a convoluted process where the WordPress now runs locally behind a firewall, and I just publish the static rendition of the blog to GAE. It is much better with the speed (as everything is static), security (because there is no backend database or php processing) and cost (as there is no front end instances to run, just outbound bandwidth to pay for).

It is definitely faster than hosting with a provider. I was curious if i needed a CDN to speed it up further. Let me know if and when you get around to measure the effects of CDN with GAE at some point.

ChrisBertrand

ChrisBertrand agosto 8, 2017

Hi Kelly
Did you checked the speed without the CDN ?
Is the CDN useful/necessary to deliver file archives/executable (application for instance) ?

Kelly HW

Kelly HW agosto 8, 2017

Hi Chris,

We stopped using the CDN for delivery of our static pages, since GAE itself delivers them just fine. (I don't have any stats off the top of my head unfortunately!)

I would definitely definitely encourage CDN for large files (over the static file size cap). Our installer downloads are larger than the static file size cap for GAE, so we serve them from Google Cloud Storage, which incurs a cost for downloads. Using Cloudflare CDN for those downloads saves us around $35/month (it's not free since we do have to serve some of the files before the CDN caches it). But, if your larger files are small enough to fit under the static file size cap, I encourage you to try that first :)

ChrisBertrand

ChrisBertrand agosto 8, 2017

Thank you very much for these details.
A real experience is very useful.

Decipher Tools Software

Decipher Tools Software agosto 8, 2017

Our pleasure, Chris! ;-)

Rita

Rita mayo 5, 2015

How did you handle 404? What would be entry in app.yaml for 'url' handler since other url handlers are already catch all. Please suggest as I see you do have custom 404. Thanks!!

Kelly HW

Kelly HW mayo 5, 2015

Hi Rita,

Good question!

Our custom 404 is handled via CloudFlare SmartErrors (which is available in the free plan, if you wanted that particular 404 page.)

Or, you can specify the custom page (as long as it is under 10KB) with the error_handler rule in your app.yaml.

https://cloud.google.com/ap...

Kelly HW

Kelly HW abril 4, 2015

If you need to install SSL on your App Engine app (which requires you to setup your domain with Google Apps), I HIGHLY recommend the instructions from Google (https://cloud.google.com/ap..., along with the instructions about the actual SSL installation here: http://blog.neutrondrive.co...